The Federal Motor Carrier Safety Administration (FMCSA) has proposed some significant, highly anticipated changes to the system it uses to identify motor carriers that pose the greatest safety risks.
In proposed changes to its Safety Measurement System (SMS), the agency declined to adopt a more convoluted item response theory (IRT) model as an alternative to the existing SMS methodology. Proposed revisions to SMS, which uses data from roadside inspections, crash reports, and investigations to identify and prioritize carriers for intervention, include:
- Reorganizing and updating safety categories (formerly known as the seven Behavior Analysis and Safety Improvement Categories, or BASICs)
- Consolidating violations
- Simplifying violation severity weights
- Using proportionate percentiles instead of safety event groups
- Improving intervention thresholds
- Placing greater emphasis on more recent violations
- Updating the Utilization Factor
“FMCSA’s announcement falls far short of the comprehensive systemic overhauls needed to accurately identify at-risk carriers and reduce truck crashes. The overwhelming majority of data the agency collects and analyzes has no connection to crashes," said Todd Spencer, president of the Owner-Operator Independent Drivers Association (OOIDA). "The cause of highway safety would be best served by the agency moving ahead swiftly with the Large Truck Safety Causal factors study authorized by Congress. You never fix any problem without a clear understanding of what caused it.”
A new website, the Compliance Safety Accountability (CSA) Prioritization Preview, is the first phase of planned updates to the agency’s SMS. Carriers can visit the website to preview how their data would appear under the proposed changes.
Changes to the FMCSA's SMS are detailed in the Federal Register. Public comments on the proposal must be submitted by May 16.
Dan Horvath, American Trucking Associations' VP of safety policy, said, “We are reviewing the proposed changes, but we support efforts to refine the CSA program so it can be a more effective tool for prioritizing carriers for intervention."
See also: FMCSA revokes second ELD in a week
Why the agency nixed IRT
The FMCSA first implemented its SMS in 2010 as part of the broader CSA program to identify high-risk carriers and prioritize them for enforcement. Through SMS, carriers are scored in seven BASICs based on roadside inspection violations and compliance reviews.
Amid heavy criticism of the system, in 2017, Congress tasked FMCSA with funding an independent study of SMS. FMCSA, in turn, engaged the National Academy of Sciences (NAS) to conduct that study. NAS ultimately made six recommendations to improve the system via a statistical model called IRT. That IRT model, however, created more concerns and questions than answers.
The latest notice from FMCSA calls out various limitations and practical challenges with using an IRT model. FMCSA pinpoints five main concerns with IRT:
- IRT is “heavily biased” towards identifying smaller carriers that have few inspections with violations and limited on-road exposure to crash risk. When the safety event groups and data sufficiency standards used in SMS were applied to the IRT model, IRT produced similar results to SMS.
- IRT does not use vehicle miles traveled (VMT) or power units (PUs) to adjust for differences in on-road exposure in the Unsafe Driving BASIC. As a result, IRT identified carriers with much lower crash rates in that BASIC compared to SMS.
- IRT modeling is not readily understandable by most stakeholders or the public, according to FMCSA. “IRT’s inherent complexity makes it challenging for the industry and public to replicate and interpret results,” the agency noted. “While SMS results can be reproduced and explained using simple math, IRT requires an advanced understanding of statistical modeling and analysis.”
- A motor carrier cannot independently compute its IRT results. IRT results can be computed only for the entire carrier population.
- IRT’s runtime is incompatible with FMCSA’s operational needs, as it takes the IRT model four weeks to run compared to two days for SMS.
“As a result, FMCSA has concluded that IRT modeling does not perform well for the agency’s use in identifying motor carriers for safety interventions, and therefore, does not improve overall safety,” according to FMCSA.
A breakdown of the proposed changes
Aside from eliminating IRT, the biggest change is FMCSA's proposal to reorganize safety categories. Rather than the BASICs, the new safety categories would be unsafe driving, crash indicator, hours of service compliance, vehicle maintenance, vehicle maintenance: driver observed, hazardous materials compliance, and driver fitness.
In addition, the safety category reorganization would combine the existing drug and alcohol BASICs into the Unsafe Driving category. FMCSA explained that because roadside drug and alcohol violations are rare, merging those into the Unsafe Driving category would streamline SMS.
The reorganization would also split the Vehicle Maintenance BASIC into two categories: Vehicle Maintenance and Vehicle Maintenance: Driver Observed. FMCSA explained that because vehicle maintenance is currently the largest BASIC, “breaking this category into two separate categories would provide greater specificity to help carriers improve and enforcement officials to conduct targeted investigations.”
Vehicle maintenance problems that could be identified by drivers in pre- and post-trip inspections fall under the Driver Observed category, and any other violations would fall under the general Vehicle Maintenance category.
When it comes to severity weights, SMS currently measures the crash risk of safety issues by assigning each violation a severity weight of one to 10. FMCSA states that this has been criticized as overly subjective. Instead, FMCSA proposed that each violation group is assigned a weight of either one or two: a violation that does not result in an out-of-service order and a violation that does.
The agency is also reviewing how carriers are organized into safety event groups based on their number of inspections and crashes. In the current SMS, each carrier is assigned a percentile rank that compares them to other carriers in the same group—a higher percentile being higher risk. However, sometimes an additional inspection, even if there was no violation, bumps a carrier up to the next safety event group, which sometimes results in a large percentile jump.
Under the new system, SMS would use safety event groups only to calculate the benchmark median values of each group. Carriers' proportionate percentiles would be calculated from a weighted average of percentiles based on those benchmarks. Once a benchmark is calculated, a carrier's percentile would be based solely on the carrier's own performance rather than the performance of other carriers.
In addition, FMCSA currently prioritizes carriers for safety interventions when their SMS percentiles reach or exceed pre-established levels called intervention thresholds. These thresholds vary for each BASIC because certain categories have been shown to be correlated with higher risk. The categories that correlate with higher risk have lower intervention thresholds, representing more stringent criteria. FMCSA proposed changes in intervention thresholds to allow the agency to focus on the most high-risk carriers.
To focus more on carriers with recent violations, FMSCA proposed assigning percentiles only to carriers that had at least one violation in that category in the past 12 months. This means that if all a carrier’s violations in a particular safety category are 12 months or older, the carrier will not be assigned a percentile in that category.
"On the surface level, it appears the FMCSA is moving SMS in the right direction, addressing many of the concerns the industry has expressed over the last decade, like the subjectivity of violation severity weights,” said Brandon Wiseman, a transportation attorney and the president and founder of Trucksafe, a regulatory compliance consulting service. “There are still a lot of unanswered questions, which I hope the agency will flesh out as it begins to receive comments and holds its listening sessions over the next few months."
This article was originally published on FleetOwner.com.
