Recent theories about the cause of the widespread State unemployment system failures of 2020 continue to suspect legacy systems. It is almost to the point of claiming victimhood instead of dealing with the root cause. The issues that caused these critical government systems to fail last year manifested over many years out of States’ neglect to maintain them. The accusations began with COBOL; then, when that became an indefensible position, the blame was placed with legacy systems which is code for, “we failed to manage the life-cycle of our legacy application”. COBOL and the mainframe are not legacy systems; a system’s legacy status is not classified by its age. “Legacy” or outdated end-of-life technology specifically has the inability to perform critical business operations.
An application is often inaccurately defined as “legacy” due to simple neglect. Maintaining legacy systems is behind the scenes work, it is not glamorous, and no one gets an award or recognition for a job well done. It is something the team just knows is part of the daily work “in the trenches” that needs to be done. It is something the team just knows is part of the daily work ‘in the trenches” that needs to be done. The pandemic exposed the penalties of decades of neglect. The unfortunate reality is the dependence on these systems will be with us for years, or decades to come. Many are suggesting, to simply move the to the cloud. You don’t need to work in technology to know that a system so critical to daily operations cannot simply be moved to the cloud. Even small enhancements to these undocumented systems often produce major disruptions. These issues have been around for years and the can has been kicked down the road year after year. Proclaiming it can be resolved quickly is a make-believe solution for the next CIO to hopefully address. The issue arose over time and it will take time to resolve.
How about doing the obvious and owning the problem? Instead of search for the elusive Easy Button so often cited inappropriately as “the cloud”, how about initiate a smooth conversion to modern technology? And do this only after committing the resources and money on the operational oversight needed for stabilization and maintenance of the current systems. The Tech Industry has been trying to replace mainframes for as long as I have been around sleeping in a MOBIDIC as my father worked late into the night. Many State modernization efforts have been absolute failures often taking the rip and replace approach and not expending the time and effort to document and define what problem they were trying to solve. We should find ways to discover a smooth transition methodology that involves moving incrementally so that we fail fast, fail small but always fail forward.
26 States have attributed unemployment processing issues to problems with legacy unemployment IT systems. Last year, New Jersey Governor Phil Murphy pleaded for volunteer COBOL programmers to assist with the State’s broken system. The Nebraska Department of Labor took a methodical approach over several years to modernize their services and integrate their suite solution into a single cohesive whole. This is our practice across the State when it applies to maintaining legacy systems. Best practice for any programming language includes source code organization, maintaining available documentation and code comments with easy-to-follow coding conventions. The premise that new code is better than old code is only true if you did not maintain the old code.
Bugs don’t mysteriously appear in the code base. I was never called in the middle of the night during my time in operations because a legacy system had extemporaneously developed a bug. It would have been because we introduced a poorly tested new code release. All deployments should follow the NIST SA-3 SDLC as a basic guideline of best practices to return you to a stable state. Ensure that your test environments are as close to production as possible. Utilize automated deployment and ensure a complete production back-up prior to deployment. This allows you to return to a stable state quickly and confidently without customer impact. The goal in the deployment process should be repetitive and uneventful. Be ready with your rollback steps in case something goes wrong.
There is always a big appeal to having something shiny and new, but if you maintain your current system it will have the functionality you need. If it doesn’t have the functionality it can be modified to meet your needs with little to no investment justification. CIO’s must hold internal stakeholders and vendors to task when they are petitioning for a new system. Ask Why? What functionality is missing? And, how will this help our customers (in our case, the citizens) and internal business users? Most IT modernization projects do not perform the due diligence necessary to follow these simple guidelines.
Everyone wishes for the elusive easy button. Keep in mind that vendors benefit financially when you move to their latest platform and even more so if they can convince you to migrate from your current vendor platform to theirs. Unquestionably, this will come with a large migration fee, accompanied by increased annual maintenance fees. When I look at a modernization project, I always ask if I have done a thorough investigation of the current platform. I look at code base and business processes that are being incorporated in the current environment. If you don’t fix the issues you have prior to embarking on a modernization project or eliminating the current issues via a managed service agreement, the core issues never get addressed and will reappear in your quest to eliminate the problem.
When I hear the words “legacy code”, I interpret that to mean there is a lack of process which results in untested and undocumented code in production. If you don’t maintain and deploy your code properly, whatever the language, you will soon be calling it and the entire system it runs on a “legacy” system.
As always, I appreciate what you do every day for the State of Nebraska.
Ed Toner