“Thanks for sharing this with us, Irwin. We really appreciate your efforts.” I’ve heard that line more times than I can count over the past six years.
Sometimes it was a genuine acknowledgement of gratitude from co-workers, supervisors and other members of the trucking industry for work I had done to show definitive flaws in the Federal Motor Carrier Safety Administration's Safety Measurement Scoring (SMS) system. That is part of the agency’s Compliance, Safety, Accountability (CSA) method of reviewing a carrier’s safety performance against other carriers.
CSA is a data-driven safety compliance and enforcement program that holds motor carriers and drivers accountable for their role in safety.
More often than not, however, I heard “thanks” from various leaders of the FMCSA at the end of listening sessions that had convened to allow for public comment on the CSA/SMS system. While they managed to sound sincere when they said it, the implication became very clear as I attended more and more of these sessions on behalf of my company. The agency had no intention of using the data I had provided them to make meaningful changes to the scoring system.
The FMCSA, having put a lot of time and money into research to revise and overhaul the old SafeStat scoring system, was going to use CSA – then known as CSA 2010 – no matter what. As long as it forced carriers to improving their safety policies and procedures, then the new system would be labeled a success.
Unsafe Driving
Much of my research focused on the Unsafe Driving BASIC, one of seven safety Behavior Analysis and Safety Improvement Categories the CSA/SMS uses to target deficiencies in a carrier’s overall profile. For Panther, it was the only score over the acceptable “threshold” level, which left a large golden triangle with an exclamation point in it next to our score (see accompanying graphic).
The other six BASICs are:
- Crash Indicator
- Hours of Service Compliance
- Vehicle Maintenance
- Controlled Substances/Alcohol
- Hazardous Materials Compliance
- Driver Fitness
I was tasked by my supervisors in early 2011 to figure out why Panther’s score was so high and what we needed to do to get the score down to an acceptable level. Over the next two years, more stringent enforcement and training methods allowed us to reduce our number of Unsafe Driving violations by more than 50 percent, yet our CSA/SMS Unsafe Driving score actually went up by a small fraction over the same timeframe.
Frustrated with a score that didn’t accurately reflect how much our company had improved, I began digging into the mathematics of the scoring system and discovered why the groupings the FMCSA used, as well as the concept of a percentile based
scoring system, created unintended bias toward carriers that ran primarily over-the-road, yet used straight trucks as the majority of its fleet, instead of tractor trailers.
There was also a state-by-state bias, as different states enforced different BASIC categories at varying levels.
Position Papers
Once I had prepared position papers on these flaws, I tried to share them with members of the FMCSA whenever they had a public listening session, or a meeting of one of its committees where the public was invited and given a chance to make a comment. Each of these sessions became more frustrating as the agency continued to downplay concerns raised, ignore the recommendations made in my papers, and make only minor tweaks in the scoring system instead of necessary, large-scale revisions.
Finally, with the support of several groups that had seen my papers – including the Expedite Association of North America (TEANA) and the Alliance for Safe, Efficient and Competitive Truck Transportation (ASECTT), I took my information to Congress. After numerous sessions with members of the House Transportation and Infrastructure Committee over the course of two years, the members added a key provision to the Fixing America’s Surface Transportation (FAST) Act of 2015.
Review Board
The CSA/SMS scores were removed from public view and the National Academies of Sciences, Engineering and Medicine – an organization that provides nonpartisan, objective guidance for decision makers on pressing issues – convened a review board to:
- Go over the CSA/SMS scoring system thoroughly
- Determine what flaws need to be corrected
- Offer recommendations that the FMCSA must use to revise the entire system
I was able to present my documentation of the data flaws to the Academies’ review board at the end of August 2016. This group repeated the same quote I had heard so many times before: “Thanks for sharing this with us, Irwin. We really appreciate your efforts.”
I can only hope that these experts in the fields of math and statistics were being sincere when they said it. The review board is scheduled to issue its findings and recommendations paper this month.
At that point, I’ll know if my journey taken through the CSA/SMS minefield was worth it.
Irwin Shires is the government qualification coordinator for Panther Premium Logistics (www.pantherpremium.com). It is one of the largest premium logistics companies in the world, providing services that include time-sensitive, mission-critical and white glove delivery.
