The old system hid the bill

In July 2023, an independent investigation of Project Cornerstone, the billing replacement at WSSC Water in Maryland, went to the utility's commission. The report says the first contract for the new billing system was approved for $15 million. It puts the final cost at about $121 million.

I believe an old billing system is a loan that a utility keeps paying interest on, and the interest stays out of sight until the day the system has to change. WSSC and the Los Angeles Department of Water and Power are two cases where a public report wrote that bill down.

1. What technical debt means

Ward Cunningham, a software developer, gave us the idea in a 1992 conference paper: "Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite." He added: "Every minute spent on not-quite-right code counts as interest on that debt."

In a utility, the same thing happens with a system that stays in service for decades. Each year it saves the cost of a replacement. Each year it also gets harder and more expensive to change.

2. WSSC: a 1990 mainframe and a new rate structure

The investigators describe the old system plainly. It was an IBM mainframe "built in the late-1980s and put into service in 1990." The rate software used COBOL, and "fewer and fewer programmers were proficient in programming and patching the rate structure software."

WSSC replaced it because the system was "severely outdated" and "could not accommodate a new rate structure for customers." That is the interest coming due. The system ran for almost 30 years, and when the rates had to change, it could not follow.

WSSC's own records show the interest years before the report. Its FY2013 proposed budget counted 624 "mainframe support services" in FY2010 and planned for 800 in FY2013. In September 2017, chief financial officer Joseph Beach told the commission why the old system could not bill the tiered rates it was studying. The field that held each customer's usage was "only six bytes long," measured in thousands of gallons, "with no ability to use decimals." The workarounds meant losing "approximately $23 million per year" or rounding customer bills. "We only have two people who actually work on this existing system," he said, and then: "So why didn't we know about these existing billing system limitations before testing? Quite honestly, we should have." Management recommended moving the new rate structure to FY20.

The new billing system went live on July 2, 2019. After go-live, WSSC spent about $51.4 million more "on stabilization, knowledge transfer, software licensing, staff augmentation, and managed services." The report also says the project "ultimately succeeded in implementing a working billing system." Both statements are in the same document, and I think both are fair.

The plan and the outcome are both on the record. WSSC's FY2020 budget scheduled the new billing system for May 2019 and a second release, for work and asset management, in FY2020. The report says about $15.8 million was spent on that second release before it was dropped. The FY2027 proposed budget, published in 2026, still funds "modernization of some of our legacy systems, including work and asset management," and its IT section still supports databases "including mainframe."

3. LADWP: a 40-year-old system, highly customized

The California State Auditor reviewed the LADWP billing project in March 2015 (report 2014-105). The new customer information system replaced "a 40-year-old, highly customized system that was technologically outdated." The department "originally budgeted $87 million" and "more than doubled that budget to nearly $181 million over time." The report's title warns that total costs may go beyond $200 million.

The system launched in early September 2013 for roughly 3.8 million customers. Customers then complained "of late utility bills, unwarranted shut-off notices, and excessive wait times." In the three months after launch, call waits averaged about 24 minutes against a goal of three. The auditor found that the department "minimized or ignored the severity of the issues" when it decided to launch.

LADWP's own board records show how long the tail runs. In March 2012, a board resolution said the new system was "estimated to cost approximately $132 million." After launch, the board hired outside project managers for "remediation of the defectively implemented CC&B Billing System" under a class action settlement: $1.3 million in October 2015, then up to $30 million in June 2017. For the year ending June 30, 2017, it recorded $41.0 million of class action settlement costs as a regulatory asset. In September 2021, it approved a sole-source support agreement with Oracle for up to $23.7 million. Since January 2014, customer bills have also carried a charge for training staff on the system, now spread over 15 years. On June 30, 2024, the water system still had $7.2 million of it to collect.

Part of that tail ended in federal court. In February 2015, answering the State Auditor, LADWP's board said an investigation had "preliminarily found that a vendor hired to assist in implementing CIS intentionally misled the department." The auditor reported this as the board's assertion, and noted that neither the department nor the board had told it about an investigation or possible litigation until then. In March 2015 the city sued PricewaterhouseCoopers, alleging, in the California Supreme Court's words, that it "had fraudulently misrepresented its qualifications." That claim was never tried. Federal prosecutors later described a "collusive lawsuit" brought by a ratepayer and settled for $67 million, which the city's outside lawyers used to settle the billing claims on the city's terms. The $1.3 million project management contract above went to the law firm of the city's special counsel against PwC. The $30 million contract went to Aventador, a company he set up. In November 2023 the special counsel was sentenced to 33 months in federal prison over a kickback of nearly $2.2 million, and he admitted bribing LADWP officials to win the Aventador contract. LADWP's general manager at the time was sentenced to six years. The city dismissed its suit against PwC in September 2019, and a trial court ordered the city to pay $2.5 million in discovery sanctions.

4. Where the interest came from

Read the two reports side by side and the same two words show up: customization and people. LADWP's old system was "highly customized." At WSSC, fewer people each year could patch the code. Then the people who ran the replacement left too. Of the 20 WSSC staff who held key roles on Cornerstone, "only four remain currently employed," and the report links that turnover to "loss of institutional and project specific knowledge."

None of this blames the people who built those systems. A custom system was a reasonable choice in the 1970s and in 1990. The cost of that choice grows a little every year, and nobody sends a monthly statement.

5. What it means for utilities

The question I would ask first is this: what does your oldest system cost you every month? Count the staff hours of workarounds, the changes you postponed, and the rate options you never studied because the software could not bill them.

Three questions help put a number on it:

1. Which systems hold your billing, asset and GIS data, and how old is each one?
2. Who knows how the oldest system works, and what happens when they retire?
3. What did your last big IT project cost compared to plan?

6. What it means for vendors

Do not take over scope you cannot deliver. The WSSC report found that the utility "permitted Oracle to dominate key elements" of the project, and that Oracle "made a late change to the scope of work to effectively remove Oracle's responsibility to provide well-defined deliverables." The report also says WSSC "did not have the personnel qualified to act as the SI" (systems integrator). In an internal review that the report quotes, a WSSC staff member said: "[w]e became the SI."

In my view, a vendor that wins a project with a loose scope wins a dispute later. Write down who integrates, who cleans the data and who supports the system after go-live, before the contract is signed.

7. What to watch next

1. Rate structure changes. WSSC's trigger was a rate structure the old system could not bill. Watch rate studies and public service commission rulings.
2. Contract amendments on billing projects. Change orders and "post-go-live support" items appear in board agendas and minutes long before an audit does.
3. Audit and inspector general reports. State auditors, city auditors and utility inspectors general publish these. They are where the full bill gets written down.

The budget lines, minutes and audited statements above come from the public records we collect at AquaIntel. Read year by year, they show the interest long before an audit adds it up.

The bill for an old system arrives a little every month. The replacement project is where somebody finally adds it up.