Latest / VoraPrep Audio — Professional Licensure & Certification Exam Prep / [CISA] CISA3: World-Class Guide to Post-Implementation Review (PIR) and Project Closure
Transcript
- Hey, welcome to VoraPrep Audio. Glad you could tune in. Let's talk about that weird feeling when a project is declared a huge success... but then you find out it's actually a huge problem. There's this story about a global logistics firm that spent over a billion dollars on a new automated sorting system. A billion. The project finished on time, on budget. Everybody high-fived. The project team popped the champagne and moved on. Six months later, the company’s earnings report comes out. Shipping errors were through the roof. Customer satisfaction had dropped 15 percent. The system *worked*… but it wasn't delivering the value it promised. And it turns out, no one ever went back to formally check. That's why we're talking about the Post-Implementation Review, or PIR. Because for an auditor, project completion does *not* equal project success. So, what is a PIR? It’s a formal, structured look-back after a new system is up and running. Your job as the IS auditor is to provide an independent, objective opinion on what really happened. Did the project deliver? Are the controls working? Was the whole process managed well? This independent viewpoint is everything. The project team, the business sponsors… they’re often biased. They *want* it to be a success. Think of yourself, the IS auditor, like a home inspector checking out a brand-new house. The builder—that's the project team—is proud it was built on time and on budget. They declare victory. But you're the one who comes in to check if the plumbing actually works, if the foundation is sound, and if the house really meets the homeowner's needs. The PIR is that independent inspection for an IT system. We’re focused on business value, not just construction milestones. To keep the core objectives straight in your head on exam day, I use the mnemonic VALUE. V-A-L-U-E. It’s a perfect way to remember what an auditor is looking for. Let's start with V, for Verify Benefits. This is the absolute heart of the PIR. Did the project deliver the business value and the return on investment that was promised in the original business case? You're going to be comparing the performance *after* the system went live against the baseline from *before*. Some of these benefits are tangible, easy to measure—like reduced operating costs. You'll want to see the numbers. But some are intangible, like better customer satisfaction. For those, you might look at things like user surveys or market feedback. Next up, A is for Assess Performance. Okay, so the system is live. But is it working well? Is it efficient? Here, you’re looking at the technical side of things. You'll check performance against the service level agreements—the SLAs. Things like system uptime, response time, how many transactions it can handle. You’re also looking at how well it supports the actual business process. Are there bottlenecks? Is it a nightmare for users to navigate? Then we have L, for Learn Lessons. What went right? And maybe more importantly, what went wrong? The goal here isn't to point fingers or assign blame. It's about capturing knowledge so the organization doesn't make the same mistakes on the next big project. These findings need to be documented and stored somewhere central, like in a Project Management Office knowledge base, so they actually get used. Now, this next one is pure auditor territory. U is for Understand Controls. Are the internal controls designed correctly and are they actually working in the live environment? You can have a system that meets all its functional goals but is leaking sensitive data because the controls are weak. So you'll be testing things like input validation, access controls, segregation of duties… all the good stuff. Finally, E is for Evaluate Process. This is where you look at how the project itself was managed. Did the team follow the company's policies and its System Development Life Cycle, the SDLC? You’ll review the project plan, the risk register, the change logs. You want to see if the project governance was effective and if any major deviations from the plan were handled properly. So when you see a CISA question about a PIR, here's how to think it through. First, identify the auditor's real goal. Is the question about project metrics, like scope and budget? Or is it about business value and risk? The CISA answer will always, always prioritize value, controls, and benefits over just finishing on time. Next, look at the project's timing. Is it a brand-new implementation, say one to three months old? Or has it been running for six months or more? An early review is going to focus more on technical stability. A later review is where you really dig into whether those strategic benefits are showing up. Always look for the business case. It's your north star. The best audit action is almost always comparing the actual results against what was promised in that document. And be careful to distinguish between what an *auditor* does versus what *management* does. Your role is to identify an issue and recommend that management fix it. The correct answer choice won't be "retrain the staff." It will be "recommend that management implement a retraining program." It's a subtle but critical difference. And finally, remember that objective evidence—like system logs and control tests—is always stronger than subjective sources, like just talking to the project team. They have a vested interest. Of course, you'll need to talk to a lot of different people to get the full picture. The key stakeholders in a PIR include the business users on the front lines—they'll tell you what it's really like to use the system. You'll talk to senior management and the project sponsors, who care about the ROI and strategic goals. The IT operations folks can tell you about stability and support. The project team gives you context. And ultimately, your independent report goes to the Audit Committee, who relies on it to oversee these big IT investments. Let's make this real with an example. Picture a hospital, "HealthFirst Medical," that just rolled out a new Electronic Health Record system. The business case promised two big things: a 20% reduction in patient check-in time and better compliance with privacy rules. You're the IS auditor, coming in six months later to do the PIR. First, you'd get that business case and project charter. You now have your key metrics: check-in time and HIPAA compliance. Next, you gather evidence. For check-in time, you'd pull system logs for the three months before the new system and the three months after. You'd also look at application logs for data entry error rates. For the compliance piece, you’d do some control testing—maybe sample the user access logs to make sure only authorized people were looking at patient records. You'd also send out surveys to the nurses and admin staff to see how things are going on the ground. Then you analyze what you found. Let's say the data shows that check-in time went from 15 minutes down to about 13.8 minutes. That’s an 8% improvement, not the 20% they were promised. So, a miss. But, you also see that unauthorized access incidents dropped from 12 per quarter to just one. That's a huge win. The data entry errors also improved, but still missed the target. Your job is to figure out why. The user interviews reveal the problem with check-in times: the staff wasn't properly trained on the new insurance verification workflow. It's confusing and slowing them down. So, your final report is balanced. You'd conclude that while the project was a major success for security and compliance, it's under-delivering on efficiency. Your recommendations would be for management, not for you. Things like: recommend management funds a targeted training program for front-desk staff. Recommend IT looks into system latency during peak hours. And recommend the PMO updates its methodology to require better user-interface testing on future projects. See how that works? If you had picked an answer on the exam like "Verify the project manager signed off on all deliverables," you'd be wrong. That's a project management task. It doesn't tell you if the hospital is actually running better. This all ties into a bigger concept called Benefits Realization Management. It's the whole process of making sure a project actually delivers the value it promised. The auditor's job is to check that this process exists and is being followed. You're looking for a formal Benefits Realization Plan that was created early on, with specific, measurable, SMART goals. You want to see that a business owner—not the project manager—is assigned to each benefit, accountable for making it happen long after the project team is gone. And you absolutely have to verify that a baseline was measured *before* the project started. Without a baseline, any claim of improvement is just an opinion. The Lessons Learned process is part of this too. It needs to be a formal, structured activity, not just a casual chat. It’s about finding root causes, without blame, and putting that knowledge into a central place where it can help the next project. And finally, there's the Project Closure Process. This is the formal end. It's about administrative closure, like archiving documents. Contractual closure, like paying vendors. Financial closure, closing out the budget. And releasing the team. It’s the wrap-up. A common pitfall is confusing this formal sign-off with success. A project can be closed out perfectly, on time and on budget, and still be a total failure from a business value perspective. As an auditor, you have to see that difference. So, on exam day, if you see a scenario that says a project was completed 'on time and on budget,' a little alarm bell should go off in your head. That's very often a distractor. The CISA mindset forces you to look past that and ask the real questions: Did it deliver the business value? Are the controls effective? Does it actually help the organization? The best answer will always be tied to the core functions of a post-implementation review. If you're looking for a way to get these concepts down cold, with practice questions that test you on exactly these kinds of distinctions, check out the VoraPrep app. It’s a full exam-prep system with lessons, thousands of questions, and even an AI tutor to help you work through the logic. You can get started and try it for free at Vora Prep dot com. Alright, so let's quickly recap. A Post-Implementation Review is about business objectives, not just project management metrics. Your role as the IS auditor is to bring that independent, objective view. Remember the VALUE mnemonic: Verify Benefits, Assess Performance, Learn Lessons, Understand Controls, and Evaluate Process. And never, ever confuse a clean project sign-off with real business success. You’ve got this. Keep up the great work, and I'll talk to you on the next one.