“Once you turn on the spigot to go-live, that doesn’t mean the project is done; it just means you’re going on to the next phase,” said Michael Uretz, executive director of EHR Group in Issaquah, Wash. “I don’t care if it’s a manufacturing or inventory system or an EMR and billing system, it has to be tested and the results audited before and after implementation.”

Testing before you go live is not sufficient—you have to test in a live environment as well, Uretz said. The reason is simple: reality has a way of being different from theory. “There’s no way you can test for all the crazy scenarios that take place in real life,” he said. “In almost 30 years of implementing systems, I’ve learned that there’s always going to be an oddball situation that comes up that you can’t test for pre go-live.”

Weeding out errors is not the only reason to audit your new EMR system. You’ll also want to verify that the vendor has met its contractual requirements and that the system is performing up to expectations, said Laura Jantos, principal in charge of healthcare information and technology for ECG Management Consultants in Seattle.

Early on in an institution’s system-selection process, most likely, a selection committee drew up a list of goals and objectives that the ideal EMR would meet. Once the institution chose its vendor, a contract identified another set of goals to be guaranteed by the vendor. Audits conducted shortly after go-live determine whether both sets of objectives are being achieved.

Complex systems leave lots of opportunity for error or omissions. With EMRs, the potential is nearly limitless. Billing codes may not be captured correctly. Medical data may not be stored correctly. Employees may be taking system-compromising shortcuts.

“It’s a matter of stepping back and saying, ‘Okay, where might there be pitfalls in this? If something is going to go wrong, where is it likely to be?’ And then it’s a matter of trying to figure out how to asses that,” said Mary Jo Flynn, vice president of audit services at Allina Hospitals & Clinics, based in Minneapolis.

Three test areas
After go-live, Jantos recommends conducting check-ups in three general areas: technical functionality, clinical functionality, and operational impact. “On the technical side, we want to make sure the system is meeting any guarantees we’ve established with the vendor, for example, the response-time guarantee: how quickly is the system supposed to move from screen to screen?” Communication with the clinical staff of the technical expectations is critical. “Sometimes people won’t tell you about problems; they’ll assume the system is supposed to be slow,” Jantos said.

Security and data integrity are also critical technical processes to test. “With EMR, there’s the potential for interfaces that didn’t exist before, such as into imaging systems and billing systems,” Flynn said. “You need to create those interfaces and audit them to see are they working correctly.”

In terms of clinical function, Jantos said most organizations conduct robust testing before go-live. “Still, you’ll want to test that functions such as dosage calculators and coding mechanisms are operating correctly, making sure the data that’s going in is being stored accurately, that any calculations are correct, and that the data coming out is being reported accurately,” she said.

Early audits are also a good time to make sure that clinical objectives are being met, Jantos said. “For example, can we pull up all patients on a certain medication if that medication is recalled?”

On the operations side, Jantos noted that it won’t matter how good the new system is if people don’t use it correctly. “This means going back to the flow charts you designed and seeing if people are using the processes you designed, and if not, why not,” she said. “Are you reducing your amount of transcription? If not, it may be that the right templates aren’t in place to allow effective documentation.”

At each implementation site in the Allina system, Flynn’s department conducted a charging accuracy review a few months after go-live. The audit not only checked to make sure charges were being captured correctly by the EMR, but also whether people were using the system correctly.

“Sometimes you’ll find you’re not capturing all the charge codes you should be capturing,” Jantos verified. “That might be an area that requires you go back and do some retraining, or you can force different types of required fields.”

Management controls may also be part of operational audits, Flynn noted. “In the days of paper charts, after the patient went home, the chart would end up on a coder’s desk to review. If you walked by someone’s desk and saw a pile of charts, you knew they were behind.” But in an electronic world, how can you tell? Checking for work queues is one solution.

Engage the vendor
Uretz, whose consulting company often helps hospitals and physicians groups select EMR vendors and negotiate contracts, offered three tips for post-go-live audits:

• Plan for them in the early stages of contract negotiation
• Involve the vendor in the audit process
• Tie the final payment milestone to a 30- to 60-day successful post go-live audit

“You need to engage your vendor in the process with you. They’re the ones that will have the best procedures to check the accuracy of their software,” Uretz said. “People who haven’t been electronic might not have the knowledge that your vendor would.”

Auditors who have worked in a manual world do need a little time to get up to speed, said Flynn, whose Allina group has tested the Epic EMR installation at about 65 clinics and eight hospitals, with three more hospitals to go. “It’s been a learning experience for all of us; every time we do one, we get a little better at it,” she said. “Practitioners just need to know their piece of it, but when the auditors are coming in, they’re looking at everything from A to Z. And there isn’t any training designed for auditors.”

Even after the initial post-go-live testing is done, there should be regular upgrades, and those upgrades need to be tested, too. “Especially around major upgrades, you’ll want to go back and make sure things are working,” Jantos said. “We don’t always know how every upgrade is going to work in a live environment. Quite honestly, we see people putting in a system and then not staying on top of upgrades.” An auditor’s work is never done.

Deborah Geering, This e-mail address is being protected from spam bots, you need JavaScript enabled to view it , is a freelance writer based in Atlanta.