Shaping the Future of eCOA User Acceptance Testing

User acceptance testing (UAT) plays a pivotal role in ensuring the success and reliability of eCOA data; however, it is often overlooked until the final stages of delivery. UAT ensures the eCOA solution has been designed to meet the protocol requirements and performs appropriately. In many cases, sponsors do not realize they are responsible for UAT, not software providers, to ensure the assessment is independent of the validation activities.

Large pharma is more likely to have a process in place, but small and midsize organizations may only run one or two studies that include eCOA per year, so study teams don’t have the knowledge, experience, time, or resources to conduct UAT. As a consequence, UAT becomes an afterthought, only picked up at the end of the study build when pressure is on to go live.

Smaller organizations may not have conducted UAT before, and they might lack the understanding of how much UAT is to be performed. This means issues arise once in production leading to urgent change requests and can also potentially put the study data at risk.

Instead, sponsors need to consider UAT at the beginning of the build process. This aids both the sponsor and the eCOA vendor as early review and feedback ensure the build is progressing in line with requirements and reduces the likelihood of major issues being raised in the final UAT testing.

Limiting UAT to a small number of people among the various stakeholder groups (e.g., clinical operations, biostatistics) is also important, as it keeps timelines to a minimum and ensures testing remains focused on the objective. Problems often arise when new people get involved in the testing even though they weren’t part of the design stages. This leads to questions about the design, which are too late, and must either be ignored or the build has to be repeated, which delays the study start.

Providing visibility for key stakeholders throughout the build process can prevent these concerns. Such early reviews reduce the temptation to get involved at the UAT stage and ensure that any discrepancies between assumptions and the requirements specification are addressed early in the build process so there aren’t any surprises down the line.

When it comes to the UAT itself, there are several ways to increase efficiency. Creating test cases is key. Vendors should not provide the test scripts as they may be too similar to those used during their own testing and validation, which will clearly lead to no findings. Sponsors should take a risk-based approach to agree on the scenarios that need to be included, screens, questionnaires, etc., resulting in detailed test cases that walk the user through every step. Each test case should be limited to manageable completion times, e.g., two hours, to ensure all testers, especially those from sponsors, can fit it into their busy schedules.

New test cases do not need to be created for every study. Much like instrument libraries and reusable code for software, sponsors can develop a library of test cases with links to each questionnaire, for example, “If you include SF36, this is the standard UAT test case.” Taking a centralized approach across studies and vendors means libraries of test cases can be created and reused across the organization. This not only reduces the burden on study teams, it also accelerates timelines and increases quality and consistency across the sponsor portfolio, regardless of vendor.

One of the most challenging aspects for UAT is ‘time travel’, the ability to simulate moving through visits or completing questionnaires over time. Understanding how the system is set up to support this capability is very important and needs to be considered during the test case development.

Just as for patient-device provisioning, shipping devices presents a logistical headache for UAT with devices getting lost in transit or stuck in customs, causing potential delays to study go-live. This issue would be significantly reduced if testing could be completed on the same device for multiple studies, e.g., ship devices to each country once and then update the app/apk for each study. Even better, an alternative such as emulators or online methods would eliminate the need for physical device shipments to support UAT altogether.

Is it time to be more agile?

With proactive planning, sponsors can increase UAT quality, efficiency, and consistency across studies and vendors through the development of reusable test cases. Software development uses agile methodology, but the rest of the operational delivery is waterfall, when in fact everything needs to start at the same time. Involving UAT during the initial eCOA design will highlight potential issues early and reduce the likelihood of significant issues arising during UAT.

This principle applies across the spectrum of operational delivery for patient outcomes. While software is at the heart of eCOA, the surrounding processes and deliverables are the source of the greatest delivery challenges; UAT, licensing, translations, provisioning, helpdesk support, and more.

It is relatively easy to put questionnaires onto a screen. It is the supporting activities on the operational side, things that people don’t necessarily think of initially, that make the difference to overall delivery and site/patient compliance.

Read the accompanying blog post from Willie Muehlhausen and Tim Davis: Addressing the Challenges of eCOA Licensing.

About Gauri Nagrani

Gauri is co-founder of Safira Clinical Research. She began her career as a formulation scientist, however later moved to the clinical research arena, where she brings more than 19 years of experience. She has held leadership roles at large pharma and CROs within technology and innovation, data management, and clinical operations areas (i.e., eCOA, IRT, eConsent, EDC, etc.). She is a subject matter expert on operationalizing technology and has rich experience in technical project management, audits, supporting inspections, vendor governance, UAT, streamlining processes, and developing standards.

She pursued her graduate degrees in biochemistry, pharmacology, and pharmaceutical research and development in Canada at McGill University.

In 2023, she was recognized as a Tech Wizard by PharmaVoice 100.