AUSTIN, Texas -- Four years after the Department of Veterans Affairs awarded a half-billion-dollar information technology contract -- so big that one executive predicted it would jump-start an entire industry -- Austin-based VA officials warned that the fledgling effort to digitally track medical equipment was in danger of "catastrophic failure."
Internal documents obtained by the Austin American-Statesman show that last year, even as government overseers were taking the VA to task for failures in other high-profile technology projects, VA officials worried that the department's $543 million contract with Hewlett-Packard Enterprise Services to implement a real-time locating system was careening off the rails.
The system, which consists of tagging and wirelessly tracking everything from catheters to hospital beds, has been hailed as a way to potentially save millions of dollars in lost or misplaced equipment.
The VA has also vowed the project would prevent death and disease from unsterilized equipment, a persistent problem at the VA. The contract was awarded in 2012, soon after colonoscopies with dirty equipment led to a rash of hepatitis and HIV infection at veterans hospitals in Florida and Georgia.
But the contract has been beset by a host of problems, including failed operational tests, questions over the reliability of equipment tags and fundamental concerns over whether the department's Wi-Fi can support the system, according to thousands of pages of emails, reports and documents obtained by the American-Statesman using the Freedom of Information Act.
The previously undisclosed setbacks with the tracking system underscore the department's larger information technology challenges as it embarks on its most ambitious information technology project to date: replacing its decades-old VistA medical record system. At an estimated cost of up to $16 billion, it represents the first major technology test of President Donald Trump's administration.
The tracking system should have gone live in VA facilities across the nation last month. Central Texas VA officials have said work stalled about 18 months ago due to contract issues. Officials say they have since fixed many of the problems and plan to launch the project nationwide by June 2018.
The failure to implement the project in a timely manner has already contributed to at least one troubling situation. Following up on a complaint, VA internal investigators visited the Washington, D.C., Medical Center in March, where they found the hospital had no functioning inventory system and was unable to ensure its equipment was properly sterilized.
That led to a series of supply-related crises: In recent months the hospital has run out of bloodlines for dialysis patients, clip appliers to close blood vessels during surgery, and supplies as basic as alcohol pads and wound dressings. The hospital has been forced to cancel prostate biopsies and, in at least one case, a surgeon used expired surgical equipment.
In all, the medical center had $154 million in supplies that were not being inventoried. The audit also found that employees were shunning another troubled information technology program, called Catamaran, that manages supplies through predictive analytics. The program, the result of a $275 million contract with Houston-based Shipcom Wireless, was terminated last year, according to documents obtained by the Pittsburgh Tribune-Review, which reported that employees complained of frequent crashes that made it difficult to order supplies.
"The Medical Center placed patients at unnecessary risk by failing to ensure that appropriate medical supplies and equipment were available to providers when needed," investigators wrote.
Archaic systems
When the locating system's contract was awarded in 2012, the system was seen as an opportunity for the VA to shed its reputation as an old-school tech dinosaur and chart a path toward what leaders were calling a "transformation initiative."
The same year the contract was awarded, the VA's inspector general warned that despite calls for digitalization, thousands of pages of paper benefit files were stacked so high in VA offices that they were compromising the structural integrity of buildings.
A year earlier, the VA's nine-year, $127 million effort to modernize its outpatient scheduling software was declared a bust. In 2013, the VA and Department of Defense abandoned their effort to merge medical records, after spending $564 million. Today, the VA spends more than half of its $4.3 billion information technology budget on maintaining archaic legacy systems.
The real-time locating system, on the other hand, was a foray into the "Internet of Things," a cutting-edge technology with a track record of making factories, warehouses and hospitals more efficient. The Hospital Corporation of America credits its locating system with reducing time spent looking for equipment by 75 percent.
Driven in part by growth in medical facilities, the real-time locating system industry is expected to expand five times over between 2015 and 2022, to more than $8 billion.
Industry experts predicted the massive VA contract would be a boon to the nascent field.
"It's pretty rare that something this big comes along to jump-start a whole industry," said Marcus Ruark, former vice president of locating system subcontractor Intelligent InSites Inc., in 2013.
The system doesn't just track equipment. It also measures temperature, air pressure and humidity, so hospitals can remotely monitor temperature-sensitive medications and tissue and get quick alerts on water leaks.
"[Real-time locating system] technology is as transformative to hospital operations as Uber is to personal transportation," wrote Ari Naim, chief executive officer of system developer CenTrak, last year.
But documents suggest the development of a nationwide tracking system at the VA has been besieged by problems since work began.
Six months after the contract was awarded in December 2012, the VA complained that tags were too wide for smaller pieces of equipment and "could be knocked off upon contact."
In September 2013, the VA declared Hewlett-Packard's underlying data architecture "unacceptable."
Hewlett-Packard in turn blamed the VA's wireless Internet infrastructure for the problems, accusing the VA of cutting corners.
The root problem for the failures lay in "performance errors that it appears VA has failed to take responsibility to correct," Hewlett Packard wrote, adding that the company "should not be held accountable for the VA's Wi-Fi limitations."
Stop work order
In mid-2016, when a VA supervisor refused to let Hewlett-Packard access its cloud-based backup systems, another VA employee wondered why in an email chain.
"I was also surprised," a third VA employee wrote in an email. "But I read it as, I am 100 percent fed up with these nitwits and I'm not going to give on anything anymore."
The VA later issued a stop work order and renegotiated new or modified contracts with Hewlett-Packard. The VA downplayed the issues mentioned in internal documents, telling the Statesman that the stop work order "allowed the VA to refocus efforts toward delivering capabilities in an incremental approach beginning with those that were more mature" and "provided both parties the opportunity to finalize the path forward and realign the schedule to achieve those goals."
Hewlett Packard Enterprise Services, which has since become DXC Technology, gave a similar explanation.
"With any large-scale project, priorities and available capabilities evolve over time and often the prudent action is to take a step back to give both parties an opportunity to re-align before moving forward," a company spokesman said in a statement. "We are aggressively working toward the full RTLS implementation."
After a new round of testing, the plan is to release "the fully integrated solution" by June 2018. The VA says a host of issues have been resolved, including equipment tags and the Wi-Fi limitations and the system has been installed in a dozen facilities.
U.S. Rep. Phil Roe, R-Tenn., the new chairman of the House Veterans Affairs Committee, has made information technology contracts a key part of the committee's oversight of the department. Tech challenges were the subject of the committee's first hearing after the new Congress was seated in January.
In June, VA Secretary David Shulkin announced the VA would purchase an off-the-shelf system to replace its decades-old, homegrown VistA health records system. The VA, he said, would use the same vendor that is building the Department of Defense's new records system. The decision would allow the two departments to more easily sync patient records.
Trump praised the decision on Twitter, calling it "one of the biggest wins for our veterans in decades."
But some lawmakers grumbled that in announcing it would purchase from the same vendor used by the Department of Defense -- Missouri-based Cerner Corp. -- the department was losing the pricing leverage of a bidding process. While the Defense Department is paying $4.3 billion for its system, the price tag for the much larger VA could come out to four times that amount.
Critics say the experience on the real-time locating system project shows that hiring a commercial vendor to build a new system does not ensure smooth sailing.
Roe has called this year and next "pivotal" for the VA's information technology department, noting that it is trying to fix problems after years of failure.
"Now," he said during a February committee meeting, "is the last chance to get them right."
A Section on 07/03/2017