In March 2010, IEC 62366:2007, “Medical Devices–Application of Usability Engineering to Medical Devices,” went into effect, and compliance to this standard is now required by the European regulatory bodies.1 Compliance to the standard’s predecessor, ANSI/AAMI HE74:2001, “Human Factors Design Process for Medical Devices,” has been required by the FDA for more than ten years.2 Both documents state that medical device manufacturers must demonstrate that all potential use-related hazards in their devices have been identified, tested, and mitigated. The device must be shown to be usable by the target population in the intended environment. These requirements reflect an overall movement toward reducing risk to patients by ensuring that medical devices incorporate good human factors engineering practices. This in turn has prompted device manufactures to learn the differences between typical usability testing and usability testing for regulatory approval.
Usability testing should be conducted iteratively throughout the development of any device, but especially medical devices. In so-called formative usability testing, target users are presented with a prototype design and asked to perform tasks—unaided—while observers collect data. Users typically think aloud, and the goal is to uncover potential usability issues and areas for improvement. This process is repeated as the design becomes more refined and the prototypes gain higher fidelity, helping to verify the design. These tests are typically conducted in a controlled environment, such as a testing laboratory, market research facility, or even a conference room. The resulting test reports become components of the usability engineering file or, for FDA, the human factors engineering file.
Validation testing is the last usability testing a medical device manufacturer completes, and in many ways it is the cornerstone of the usability engineering file. The goal is to validate the final design to prove that all use-related hazards have been mitigated. Because of this, there are more precise requirements for conducting validation tests, such as the following:
- Validation tests must be performed with representative end users. In fact, all usability testing should be conducted with representative end users, but with validation testing, it is even more crucial. Medical device manufacturers should be sure to include users representing each user type (e.g., physicians, nurses, and biomedical engineers).
- Validation tests must be conducted in an actual or simulated use environment. This is because with medical devices, the use environment can significantly affect users’ ability to interact with the device safely and effectively. In fact, the U.S. Quality System Regulation 21 CFR 820 states that, “Design validation shall ensure that devices conform to defined user needs and intended uses, and shall include testing of production units under actual or simulated use conditions.”3 In the hospital setting, factors such as competing alarms, stress, fatigue, and distractions need to be taken into account. With devices used in the home, factors such as temperature extremes, poor lighting conditions, interruptions, and family members come into play.
- Validation testing requires a larger sample size than formative testing. While there are no hard and fast rules for the minimum acceptable sample size, FDA has stated that upwards of twenty-five participants of each user type are usually satisfactory. (Formative usability tests typically have smaller sample sizes, such as eight to ten participants, after which there are diminishing returns in uncovering usability issues.) Factors to consider when deciding on the appropriate sample size include the following: the variability of user demographics, the variability in how the device is used, the device’s complexity, and the potential risks associated with the device.
- Validation testing must be conducted with production equivalent units. This means the actual hardware, software, labeling, instructions-for-use, and training materials. This is in contrast to formative usability testing, which is conducted with models, prototypes, and/or simulations.
- The tasks performed by users during a validation test must include the primary operating functions (frequent and critical tasks) and any tasks with associated safety risks. When planning a final validation test, medical device manufacturers must tie the tasks back to the hazards and risks analysis. According to FDA, the validation test report must present a clear link between the test results and the device’s safety.
The push to increase patient safety through better design is not new. FDA has been promoting the application of human factors engineering since the mid-1990s.4,5 The requirements of IEC 62366:2007 have simply brought this issue to the attention of more medical device manufacturers. Conducting iterative usability tests during the development process demonstrates a manufacturer’s commitment to use safety. Correctly planning, executing, and reporting the final validation test is the capstone to the process, allowing manufacturers to feel confident in the safety of their products.
*This article was originally featured in Consultant's Corner, a Qmed publication, on November 10, 2010.
References
1. ISO/IEC 62366:2007- Medical Devices -- Application of Usability Engineering to Medical Devices, (Geneva, International Electrotechnical Commission, 2007).
2. ANSI/AAMI HE74:2001, Human Factors Design Process for Medical Devices (Washington, DC: AAMI, 2001).
3. Code of Federal Regulations, 21 CFR 820.
4. Sawyer, Dick. “Do It By Design: An Introduction to Human Factors in Medical Devices,” (US FDA, 1996). Retrieved from: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm094957.htm.
5. Kaye, Ron and Crowley, Jay. “Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk Management,” (US FDA, 2000). Retrieved from: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm094460.htm.
