Ruminations

Relics and Remains

The complacency that industries embrace can often result in falling behind the times. This became painfully clear when the auto industry slowed to a crawl when they had no plans in place to deal with supply chain shortages of decades old semiconductor technology used in many parts and subsystems of every vehicle.

Even with the chip shortages impacting new cars, existing cars were still able to drive. That isn't always the case when it comes to other forms of transportation. When railroads have system failures, both riders and cargo are negatively affected. Though, to be fair, railroad issues don't get anywhere near the news coverage that airline problems do.

The recent Southwest Airlines system failures resulted in more than 20,000 flights being canceled. Southwest's problems were due to an old computer system from the 1990's that couldn't handle the number of rescheduled flights and crews caused by weather related cancelations.

But we're not here today to pile on Southwest airlines, even though they promoted five executives after their holiday fiasco. And we're not here to complain about the air traffic system failure that delayed hundreds of flights in Florida just last week.

We're here today to talk about the FAA's NOTUM system, and other systems that are in the same boat.

On January 11th, 2023, the FAA's NOTAM system went down at 3:28am. This stopped all domestic flights from receiving notifications of real-time hazards while in flight. This resulted in delays of thousands of flights, with at least 1,300 flights being canceled.

The reported cause of this system failure was a corrupt database file. The system outage started to clear up around 11am, but as of 3pm all services had yet to be fully restored. The FAA's outage only affected about half as many flights, crews and passengers as the Southwest Airlines debacle did. But every hour that passed brought it closer Southwest's numbers.

Both the FAA's NOTAM system, and Southwest Airlines' Skysolver system, are aging and outdated. Neither has a backup system in case of emergencies, though there are redundancies for critical hardware failures (but not for software failures or data corruption).

It shouldn't surprise anyone that the GPS and location technology in your car and your phone are decades ahead of the technology used by the FAA to track and manage planes in flight. The old FAA technology even results in flight paths being miles longer than necessary, increasing the time and costs of each flight.

Airline systems simply aren't using 21st century technology. They don't have adequate failover redundancy mainly because they use mainframe technology and aren't capable of taking advantage of distributed data centers or cloud technology.

Though the FAA has made progress improving the amount of time and fuel planes waste sitting on the tarmac, they have simply struggled to modernize their computer systems.

The airline industry is not alone in its lack of agility. Other industries, such as banking, insurance, and utilities are still well rooted in old technology. And the FAA isn't the lone government agency stuck in the past. Both the IRS and Medicare are also tied to old tech. But the reasons are very different.

On the surface, it seems that the IRS and Medicare would be prime candidates for newer technology. Both track and process hundreds of millions of transactions annually, mostly in batch mode. So why are these systems stuck in the stone age? To put it simply, math.

Computers do a lot of math, and they are very, very good at it. But calculations use resources, and calculations need to be very accurate. And when it comes to things like IRS refunds or Medicare payments, they need to be very accurate and perform very fast calculations. So instead of using floating-point math, Medicare and the IRS use fixed-point math.

The use of fixed-point calculations makes perfect sense for applications like the IRS or Medicare. They are keeping track of decimal data (e.g., dollars and cents). But modern chips and modern programming languages are geared towards floating-point calculations.

floating-point math is used for displaying graphics, which is an important part of that vast majority of all computer systems in use today.

Modern CPUs have circuitry specifically designed to perform floating-point math called an FPU, and many systems now have separate graphics cards which utilize powerful Graphics Processing Units (GPUs). Even though these calculations get hardware assistance, they are still not as accurate as fixed-point calculations.

I could write a long, boring overview of the difference between using fixed-point and floating-point calculations, and why the difference is important. But you would be better served by reading this excellent article on the subject.

What this math problem comes down to is computer languages. The computer language used by the IRS, Medicare and the airline industry is COBOL. COBOL is also still used in a number of large industries. It is also used for other federal and state government agencies, including most unemployment agencies and even disability systems. As of 2017, it was estimated that 220 billion lines of COBOL code were still in use.

COBOL is an old computer language, dating back to 1960. Early computers did not have floating-point hardware support (or much need for it), so fixed-point calculations were the norm. COBOL's native fixed-point support was just another part of the language.

COBOL is one of the first computer languages that I learned. Just another trait contributing to my dinosaur status.

There are many newer computer languages than COBOL, but native fixed-point support is extremely hard to find. These newer languages are easier to program in, have much larger pools of available programmers, and have support communities that COBOL cannot rival. But COBOL programmers are retiring and are hard to find. I'm not suggesting that the IRS and Medicare switch languages and start outsourcing their programming to Fiverr. But finding a new, modern language would greatly benefit government organizations and companies that are tied to these legacy systems moving forward.

The real solution for this fixed-point dependency is to develop compiled fixed-point arithmetic libraries for a few of the newer languages. They need to be compiled so the libraries can function as native extensions of the language. This will give them the speed necessary to perform the millions of accurate fixed-point calculations per second that are required by these types of applications and many others.

And who is going to develop these high precision specialty libraries? My suggestion is a federally funded collaboration with the top computer science universities in the US. Pick the top five or six, let them pick the language they want to develop for (resolving any mutual picks), and let their graduate students solve this issue once and for all.

These universities would each receive generous compensation, and bragging rights. This effort would result in the IRS and Medicare saving tens of millions of dollars each year once they migrated to more modern systems and languages. These libraries would then become part of the public domain and any company or user developing in a compatible language would be able to use the libraries. This would help many industries, saving potentially hundreds of millions of dollars per year.

COBOL programmers are generally more expensive than programmers using modern languages because COBOL programmers are much harder to find. And the hardware that supports COBOL is generally much more expensive than what is available via the cloud. Also, the hardware requirements make having full hardware backup systems financially impractical. So, migrating the systems that handle more than 10% of the US GDP to something more modern, and capable of having redundancies and failover, sounds like a no brainer.

The idea of replacing COBOL is not new. The US Office or Personnel Management outlined the staffing concerns a decade ago, as did others. And the costs of rewriting legacy COBOL systems will not be cheap, though an industry has emerged to reduce those costs.

For some companies, the cost to migrate away from COBOL may come close to, or even exceed, running their current systems - at least in the short term. And why spend the money when in many industries these systems still work? But it also gets expensive to maintain COBOL systems.

The industries that rely on COBOL don't seem to be in a hurry to find a solution, though I'd be willing to bet that Southwest Airlines will have some sort of plan when it is called before Congress to explain what happened last December. But we'll just have to wait and see.

---