Learning from the Clipper 2 Launch Problems
More than two months after the initial launch date in December, customer upgrades remained on hold due to ongoing issues, according to staff report for the 2/23 Clipper Executive Board meeting.
These delays are undermining the rollout of one of the region’s signature achievements to make Bay Area transit more rider-friendly, with the introduction of free and reduced priced transfers, which have the potential to increase transit ridership and save riders thousands of dollars per year. The plan was one of the top recommendations of the Blue Ribbon Transit Recovery Task Force to regain ridership in the wake of the pandemic.
In a region with many agencies, the goal of common fare media and programs is essential to achieve a customer-friendly, high ridership system that supports housing, social equity and climate goals, and supports travel and tourism and reduces peak congestion.
Once Cubic cleans up the rollout problems, we strongly encourage an after-action review of the design, testing and launch process in order to identify what led to the problematic launch of Clipper 2.0, to move forward with actions to improve the region’s future capacity for fare and payment product development and procurement.
Responsibility lies with Cubic and with MTC
The goal is to learn from this experience to ensure we do not repeat it.
A big part of responsibility for the problematic launch is that of Cubic, whose system designers, programmers, and quality assurance staff are directly responsible for many of the problems. Cubic should be held financially accountable for delays and costs according to their contractual agreement with MTC.
But significant responsibility also lies on the shoulders of MTC, which manages the Clipper project. A review should analyze multiple responsibilities for the flawed launch. Given that MTC holds the ultimate responsibility for the outcomes, the review should be clear about what actions MTC can take in the future to do better and minimize the potential for failure.
Preventable problems
The Bay Area has hundreds of thousands of people with information technology skills, including grassroots transit advocates who observed the problems and Cubic’s explanation at the January Clipper Executive Board meeting and saw that they were eminently preventable, revealing mis-steps in system design, development and testing. Such preventable problems include
Account data migration failures due to expired credit cards or expired passes. It is an elementary programming practice to develop tests for predictable cases, such as expired credit cards and passes, and to enable the system to fail gracefully when these predictable conditions occur.
Riders unable to add money to Clipper cards at ticket vending machines due to incomplete transactions. Designs for transaction safety have been known for decades.
Failure to scale the system. There were multiple examples of the system timing out under real-world and amounts of data. Testing under simulated load and network conditions, could have helped uncover issues at the predicted usage levels prior to rollout. For example the system was initially configured with a standalone database query server (SQL server) rather than a cluster.
Fare inspection devices take an unacceptably long time to validate fares. Solutions for system designs to provide acceptable performance with networked devices have been known for many years.
These are issues that Cubic could have prevented. And with additional skills on the project team MTC might have been able to spot warning signs for the failure to meet reasonable design and schedule goals.
The costs and risks of a flawed rollout
The delays and poor customer experience caused by the launch of this project, including people unable to access their funds, delayed rollout of the free and reduced price transfer policy, spending long times with tech support to attempt to access their funds and data and fix problems, are an especially serious problem in a year when public perception of public transit is existentially important. Not to mention the problems with the reconciliation process that cause challenges for transit agencies with currently vulnerable finances.
The Bay Area has recovered from other flawed technology projects such as Caltrain’s signalling system that needed to be rebuilt after a false start on what turned out to be an unsuitable platform choice.
An easy-to-use fare and payment system is important and will need further phases of development. It is critically important to learn from this experience to be able to continue to improve a well-coordinated, rider-friendly system for the future.
In summary, we reiterate our call for a detailed after-action review of the design, testing and launch process in order to identify what led to the problematic launch of Clipper 2.0, and the development of an action plan to improve MTC’s future capacity for to manage complex product launches and procurements in the future.
Systemic challenges with public sector technology projects
MTC and agencies have the opportunity to learn and improve the public sector’s capacity to be able to deliver a unified system in a competent and customer-friendly way.
The US public sector - including in the Bay Area, the global hotbed of technology development - has well-known challenges in procuring and developing technology systems, described in books such as Recoding America by Jennifer Pahlka.
There are multiple root causes of public sector challenges in developing technology projects. These challenges range from lack of sufficient in-house expertise to manage technology projects, including in product management and system architecture as disciplines; complicated business rules, waterfall-style procurement and development practices drawn from concrete-and-steel construction projects.
The problems with the Clipper 2.0 launch pose an opportunity to learn from these problems and develop greater competency to procure and manage such projects in the future including future features of the region’s transit fare and payment system.
