It is well documented that different evaluators conducting usability evaluations of the same product often come up with disparate findings. Does the suspect reliability of usability evaluations mean we should stop conducting them all together? The answer is a clear and resounding no.
The reality is that usability evaluations can yield different results, depending on the way in which they are conducted. More importantly, however, usability evaluations identify important use-related challenges and hazards, and provide insight into how a design can be improved.
In this blog, we’ll explore challenges associated with conducting reliable usability evaluations and offer insights as to how to overcome these challenges. We’ll also discuss how to improve usability testing practices to ensure we are identifying the most important issues.
Usability Evaluation Methods
There are two primary methods of evaluating the usability of a product: (1) usability tests and (2) expert reviews. The key difference between the two is that usability tests are conducted with representative users, while expert reviews (e.g., heuristic analyses, cognitive walkthroughs) are typically performed by usability professionals and/or domain experts. Both methods use a set of tasks that help evaluators identify usability issues, and in both methods usability professionals analyze the data in order categorize problems, rate them according to a defined severity, and offer recommendations for design improvement. While there are numerous methods for evaluating usability, one thing is clear: different evaluators can produce different results.
Overlap of Usability Issues
One might reasonably assume that expert usability professionals conducting different evaluations of the same product would uncover the same usability problems. Unfortunately, the reality is quite different. Numerous studies have explored this issue, the most prominent being the Comparative Usability Evaluation (CUE) series. A striking portrait of the lack of overlap is painted when you look at the number of unique issues reported by single teams across the first four studies of the series:
- CUE-1 – 91%
- CUE-2 – 75%
- CUE-3 – 61%
- CUE-4 – 60%
The CUE-4 study represents the most comprehensive comparison of usability studies to date, involving 17 usability teams in total, nine of which performed expert reviews and eight of which conducted usability tests (Molich & Dumas, 2008). As seen above, 60% of all problems reported were identified by only one team. Many others have found strikingly similar overlap in their own research. See, for example, Jeff Sauro’s blog “How Effective are Heuristic Evaluations?”
Factors Affecting the Reliability of Usability Evaluations
A large part of the problem can be attributed to the variables affecting both types of usability evaluations. We’ll discuss several of the most important variables affecting usability evaluations, and offer some practical insights as to how to reduce their impact on reliability. This list is far from comprehensive, and we invite readers to add additional variables to the commentary at the end.
Task selection. Task selection is an important aspect of both usability tests and expert reviews because the tasks performed greatly affect the interaction that test participants and/or evaluators experience. As Molich and Dumas point out, “A usability problem—even a critical one—normally will only be discovered if there are tasks that focus on the interface where the problem occurs” (p.275). Development teams should define the primary operating functions and frequently used functions, and then create tasks that will allow participants to interact with these areas. In medical device development, tasks should also be created to address potential use-related hazards defined during risk analysis.
Test protocol. Unfortunately, evaluation teams may use different instructions and ways of interacting with participants during a usability test, and these subtle differences can bias the results. At Farm each moderator closely follows the same protocol. We also conduct pilot sessions to ensure participants fully understand the questions and are not biased by the way questions are framed. During formative testing, the think-aloud protocol may also uncover instances where the task instructions are misleading the user and/or causing confusion. For a more comprehensive discussion of creating a successful protocol, see Beth Loring and Joe Dumas’ “Effectively Moderating Usability Tests.”
Categorization of problems. The categorization of usability problems is also important. In the CUE-4 study, participants were asked to use predefined categories. The authors found that identical issues were sometimes classified as positive findings and sometimes classified as usability issues. In other studies, including previous CUE studies, evaluators were asked to define their own categories and scales. The language used to define problems will undoubtedly affect the way people, including clients, understand test results. It is important that categories be easily defined and understood by the development team.
Domain knowledge. In a study of heuristic evaluations conducted by Nielsen (1992), “double experts,” or usability experts with extensive knowledge of the specific domain being studied, performed better than usability professionals without domain expertise. It is sometimes suggested that usability professionals who become too knowledgeable about a device can lead test participants during the evaluation, but we have found that evaluators who take the time to understand a product will produce a better and more relevant list of issues than those who do not.
Providing recommendations. There is no hiding the subjective nature of providing recommendations. Nevertheless, this is a critical juncture in the process, one that represents a shift from research to solving problems. Some common problems include: overly vague recommendations, recommendations that are in direct conflict with business goals, recommendations that reflect personal opinion alone, and implicit recommendations. To avoid some of these pitfalls, it is critical that evaluators provide solid evidence for how a recommendation supports the issue that is uncovered. Similarly, we have found that recommendations are most useful when the usability team has been closely involved in the development process.
It is important to know that in a medical device summative report, third-party evaluators such as Farm are not supposed to suggest how an issue will be mitigated. We simply report the issue and provide the root cause from the user’s perspective. It is up to the device manufacturer to report to the FDA how they fixed the problem and re-tested the issue.
The Value of Usability Testing
According to available research, the results of usability testing and expert reviews can be inconsistent across evaluators. Fortunately, they can be made more reliable by applying rigor to various aspects of usability evaluations, including the test protocol and task selection. A usability evaluation, while based on the fundamental principles of behavioral science, is a tool used to provide better and safer products, and it should be judged on its ability to inform design change, to improve the user experience, and to improve the safety of medical devices. The science of evaluating products is not perfect, but if we keep the end goal in mind, we will have a better appreciation for the positive impact that usability evaluations have on product development.
Molich, R., & Dumas, J.S. (2008). Comparative usability evaluation (CUE-4). Behavior & Information Technology, 27(3), 263-281.
Nielsen, J., and Molich, R. (1990, April). Heuristic evaluation of user interfaces. CHI 1990 Proceedings, 249-256. Seatle, Washington.
Massive Market and Growth
The growth of the mHealth industry has product developers and software engineers increasingly focused on the regulatory acceptance criteria they may face when developing these apps. Research2Guidance, a global mobile research group, states that 500 million smartphone users are expected to be using mHealth medical apps by 2015. Today there are approximately 97,000 mHealth medical apps available, and sales of medical apps are expected to reach $26 million by 2017.
In the Farm blog The Rise of Mobile Health and the Importance of Human Factors, many of the drivers contributing to the explosive growth of mobile software development are highlighted, including economic and technology trends and the ubiquity and convenience of mobile platforms. It’s clear that healthcare information, made available to both consumers and healthcare providers via mobile devices, has the potential to reduce healthcare costs and improve care across multiple point-of-care environments.
To provide clarity and direction for mHealth medical app developers, the U.S. Food and Drug Administration (FDA) has developed guidance outlining the suggested path to approval and including a set of definitions on what is and is not regulated. The FDA differentiates between a wellness app and a potential medical device (such as a device that uses an mHealth medical app). This is an important distinction for developers, since the FDA has said it will not regulate wellness apps, and thus the documentation burden for the developer falls within more traditional development practices. Per these guidelines, an mHealth medical app will be regulated if it is:
- An app that displays, stores, analyzes, or transmits patient-specific medical device data
- An app that transforms or makes a mobile platform into a regulated medical device
- An app that performs actual medical device functions
- An app that allows users to input patient-specific information and provides patient-specific results, diagnosis, or treatment recommendations used in clinical practice or to assist in making clinical decisions
Present and Evolving Guidance
The current guidance for mobile medical applications was issued in 2011. This guidance provides some support to manufacturers by categorizing mHealth medical apps by risk to patient and addressing the level of development (including testing) documentation expected for each risk class. This guidance helps the developer determine accurate costing and time estimates. These guidelines also help determine which of the existing mHealth medical apps are to be classified as Class I (general controls), Class II (special controls in addition to general controls), or Class III devices awaiting premarket approval (PMA). Class I devices or MDDS (medical device data systems) are considered to be the least risky, and the FDA has exempted them from any regulatory requirements. mHealth medical apps that provide electronic transfer, storage, or display of medical device data qualify as Class I devices. Class II devices require filing for 510(k) clearance and include products such as mHealth medical apps that display radiological images for diagnosis. Class III devices requiring PMA may require clinical trials if the app is novel (with no predicates). An mHealth medical app also must be FDA approved if it is an extension of an FDA-approved and regulated medical device.
The FDA is making an effort to address concerns and provide oversight on guidance issues. It is important to be aware that congress is also asking for clarification on this effort. According to MobileHealthNews, as of March 15, 2013, a letter has been sent to the FDA seeking clarification on the agency’s intentions:
In an attempt to fill the gap, the guidance goes on to call out additional standards and regulations for subjects such as:
- Software verification and validation
- Off-the-shelf software used in medical devices
- Cybersecurity for networked devices containing off-the-shelf software
- Radio frequency wireless technology in medical devices
These regulations and guidelines, as well as the guidance for the content of premarket submissions for software contained in medical devices, provide insight into what is expected in a premarket submission for performance and process documentation, the potential high-level risks an app may encounter, and areas where field issues have occurred. By reviewing these publications, an mHealth developer who has little or no experience working in the regulated medical device market will have the key information required to develop a plan and a process for creating, testing, and documenting an app that can be submitted with the best chance for clearance.
Software Development Regulation
Critical to mHealth software development is guidance for the development process itself. The FDA does not author its own standards, but has chosen to recognize the European standard for software development ANSI/AAMI/IEC 62304:2006. A detailed discussion of this standard is included in the online article Developing Medical Device Software to IEC 62304. The European standard uses a patient risk method for identifying risk associated with device software, and relies heavily on ISO 14101:2007 for a risk management approach to be followed throughout software development.
For mHealth developers who are unfamiliar with regulated software development, Happtique, a mobile health software development company, has created a voluntary program that provides a set of interoperability guidelines. Designers who follow these guidelines and apply to have their mHealth medical apps tested will receive certification for their mHealth medical apps, informing the consumer that they have achieved this level of interoperability. The program has been reviewed by the FDA and is seen as a complement to its regulatory requirements. This is covered in the article Happtique Publishes Final Standards for Mobile Health App Certification.
While these numerous standards, regulations, and guidances are complex and may prove confusing, there are some basic steps that developers can take in order to provide a clear and predictable development path:
- Thoroughly understand the app’s intended use. It’s important to be able to define the intended use for an mHealth application and communicate its use precisely, including benefits the app provides to the end user and patient and, if possible, including what the app is not intended to do. Not only will this have a positive impact on the marketing of an mHealth app, it helps define the depth of process and documentation needed throughout the development process
- Follow existing FDA and international guidance relating to communication, electrical, and platform hardware, so that key risks can be avoided. Create a development process that will identify risk factors contained within hardware components, wireless protocols, operating systems, and platform-specific interferences, and from both a technical and a user perspective. Identify critical issues and create an approach that will minimize or eliminate them. Consider using FDA’s MAUDE database in order to mitigate risk
- Create a software development process according to the ANSI/AAMI/IEC 62304:2006 standard in order to support safe software that meets performance requirements
- As defined in the blog article The Rise of Mobile Health and the Importance of Human Factors, follow ANSI/AAMI HE75:2009 to ensure that the mHealth app will be safe and easy to use
- Consider an iterative implementation approach, rank risks and design mitigations around the highest risks. Implement and test those first, both for performance and for usability (by testing with real users). Build small increments and test those, continuing until all functionality is implemented and the risks have been minimized or eliminated
- For mHealth apps that will be used on multiple platforms, target implementation by most-to-least market impact, then go through the same iterative development path
- According to regulations, developers must monitor released applications for safety and effectiveness issues. mHealth developers should follow FDA CAPA guidance to improve the development process
- Finally, use well-established development and design, and incorporate testing tools, to reduce the probability of defect introduction
Home healthcare and the use of medical devices outside of the professional healthcare environment are on the rise. Modern medicine allows us to live longer and provides those with chronic diseases the ability to receive medical care at home. Examples of home-use devices are oxygen concentrators, hospital beds, sleep apnea monitors, body-worn nerve and muscle stimulators, and dialysis machines, just to name a few.
According to the NAHC (National Association for Home Care & Hospice), approximately 7.6 million individuals are receiving home healthcare in the United States from roughly 17,000 paid providers. Not only does home healthcare improve recipients’ quality of life, but it also provides cost savings. Looking at the chart below, you can see the cost advantages to receiving care at home.
Despite the advantages, employment of devices outside of professional healthcare facilities increases the risk of harm through unintended or potential misuse, driving an even greater need for devices to be designed using human factors principles to mitigate these risks.
In the past, manufacturers of home healthcare equipment were required to comply with IEC 60601-1, demonstrating that their designs mitigate the risks associated with use in the home by patients or caregivers. In 2010, a new provision, IEC 60601-1-11 was published, turning attention specifically towards Requirements for Medical Electrical Equipment and Medical Electrical Systems Used in Home Care Applications. Also in 2010, the FDA started the Medical Device Home Use Initiative to "ensure the safety, quality, and usability of devices labeled for home use." Accordingly, the agency states that it will take the following actions to support the safety and safe use of medical devices in the home:
- Establish guidelines for manufacturers of home-use devices;
- Develop a home-use device labeling repository;
- Partner with home health accrediting bodies to support safe use;
- Enhance post-market oversight; and
- Increase public awareness and education.
These steps will help address the challenges associated with the use of medical devices in the home and provide greater protections for patients receiving home healthcare.
As part of the initiative, a new draft guidance has been written to help manufacturers "design risk out of the device." Draft Guidance for Industry and Food and Drug Administration Staff, Design Considerations for Devices Intended for Home Use is meant to provide advice and summarize other guidance documents available to manufacturers, citing over 24 documents and international standards. It is important to remember that the guidance was created to help manufacturers understand all variables that should be taken into account when designing a home-use device, but that it is not possible to follow all of the guidelines simultaneously. Designers should follow the guidelines to the extent possible, which will help the FDA to evaluate the device’s requirements, functionality, and safety.
How are “home-use devices” and “environments” classified?
A home-use device is a medical device intended for users in any environment outside of a professional healthcare facility or clinical laboratory. The term includes devices intended for use in both professional healthcare facilities and homes.
A user is a lay person such as a patient (care recipient), caregiver, or family member who directly uses a device or provides assistance to the patient in using the device.
A home is any environment other than a professional healthcare facility or clinical laboratory where a device may be used.
Note that the word "home" is being used loosely to mean ANY location in which you use the device outside of a professional healthcare facility or clinical laboratory. Thus, it could be your primary home, your vacation home, your car, public transportation, outdoors, or any other non-clinical location.
It is vital to take into consideration the potential environment(s) where the device may be used. Just a few of the challenges are:
- There may be few electrical fixtures and they may not be up to code;
- The lighting may be poor;
- The home could be quieter or louder than a hospital environment. For example, children or a loud TV can add to noise level;
- Homes, vehicles, and public spaces are not always designed to mitigate maneuverability barriers; and
- Sterility and cleanliness in a home will not be the same as in a clinical setting.
So what should manufacturers do when designing their next home-use device? Consider the following high-level summary based on my review of ANSI/AAMI HE75:2009, Section 25, on home healthcare and the draft guidance on home use:
- The User - Front and center to the design of any product is the user. Who is the ultimate user of the device: the patient, the caregiver, a visiting nurse, or other person? Can the device be used by the intended population? Might the user have sensory, cognitive, or physical limitations? For example, is the device designed in such a way that someone with dexterity issues can use it properly without having to ask for help? Can they hear audible feedback from the device and see the screen well enough?
- The Use Environment(s) - Just as important as the user, the use environment must be strongly considered. Is the device stationary or mobile? Whether in the home or on the go, it is not possible to rely on proper power outlets and thus would require an alternative power source (e.g., batteries). If alarms are part of the device, manufacturers must ensure that they can “be heard in uncontrolled noise environments typically found in the home." Also consider whether the device will be permanently attached to the user in both private and public settings. How will the user bathe? Will it be comfortable when he or she is sleeping? Should it be made as inconspicuous as possible? How will he or she get through security scanners? The use environment(s) also will play heavily into design considerations as it relates to durability, potential exposure to the elements, and whether or not it can safely function in the expected conditions.
- Device Considerations - Multiple guidance documents exist to aid in the design process, offering a mix of regulations and advice. Manufacturers will find considerations related to such things as:
Training - Device training can take many forms: instructions for use, device labels, information sheets, and formal training, among others. Keep in mind that some users may not be able to understand multiple steps or a long list of warnings and precautions. Additionally, manufacturers should weigh the importance of training users in addition to providing written instructions. Tutorials built into device software can be helpful in cases where the device is used infrequently or the user may be slightly impaired (e.g., low sugar clouding mental judgment).
Post Market - What kind of support will be provided to the user once he or she is home using the device? Will 24-hour customer support be available? What happens if the device malfunctions or breaks? Users need to understand their options and what to do if they run into problems.
- Design controls and software;
- Learnability and intuitiveness;
- Sound;Labeling; and
The standards and guidance documents can assist manufacturers in developing safe, usable products. However, home-use devices still require extensive testing with the intended users—in simulated environments and later in actual home-use environments—to ensure their usability, understandability, and safety. If manufacturers consider the human factors that are specific to home-use devices, then they, their users, and the healthcare system as a whole will realize the benefits.
In my medium-term professional life of nearly 20 years, I have experienced the rolling hills of good and bad workplace culture. In many companies the term “culture” is often thrown about by HR for hokey team-building exercises, but is rarely well defined. It is one of those emotional terms people instinctively feel is either good or bad. Culture is a complicated equation with multiple input variables like diversity, financial footing, and ideology that hopefully outputs success. The most positive and successful company cultures have the following high-level attributes.
An open atmosphere that encourages creative thought.
In the consulting world, we live and die by our creativity. The freedom to express an idea in a group brainstorming session is critical. Even a bad idea (I’ve had more than a couple) has the potential to inspire someone else into a winning concept. When employees are encouraged to express themselves, free of a creative dictator who bullies meetings, the best ideas quickly rise to the top because nothing is held back.
A diverse group of people who can look at a challenge from multiple angles.
Diversity obviously plays a key role in creativity and culture. The best ideas come from people with the farthest reaching experiences. Staffing people from different corporate, social, and interest backgrounds produces the most diverse concepts because the pool of knowledge is that much bigger. A great company culture needs a broad spectrum of viewpoints to both develop great products and provide an interesting workplace.
Cross-functional teams uninhibited by psychological or physical walls.
Once the key people are in place, the internal teams need to work together. This is probably the single largest hurdle to positive corporate culture in large organizations. I can swing a hammer with the best of them, but milling a precision slip fit is a totally different issue. I need to know where to go and who to see for the optimal solutions to my problems. Physical barriers like separate buildings or even high cubical walls often compartmentalize people and unintentionally nudge employees into “defending their space” instead of valuing the people they work with. An openness and mutual cooperation can naturally develop when groups of talents (like engineers and designers) are mixed in one location and interact face to face on a daily basis. Innovation does not stop at boundaries, it overcomes them.
A means for a group and individuals to control productivity.
An often underestimated catalyst to a positive company culture is a department budget. Speaking as an engineer, we need tools to most efficiently do our work. A budget we can spend as we see fit (free of multiple approval signatures) is wonderfully liberating. It improves productivity by allowing employees to purchase tools they are excited to learn while making their jobs easier. Shiny new toys also have the added benefit of keeping people engaged while at the same time improving the company’s overall capability.
An internal quality/regulatory system that puts the idea first.
A positive culture is also fostered by an internal development system free of creative impediments. There is nothing more inspiration killing than the thought of forms, signatures, and cross-functional team approvals that need to be navigated before pencil can even be put to paper. Without question, standards and internal quality requirements are critical to manufacturing safe and effective products, but at the knife’s edge of concept creation there should be total freedom. That is by far one of the best parts of working for a consultancy. Our internal procedures were created around the framework of unimpeded development. Creativity comes first.
Contrary to the musings of academics, there is no set path to a utopian society. Human beings are wonderfully unique in thought, passions, and personality, but there are a few simple building blocks companies can put into place to foster a positive corporate culture. Leveraging the unique qualities of individuals to work together toward a common goal can be achieved when they are motivated, feel they have some control of their destiny, and are free to express themselves.
The words “Design of Experiments (DOE)” and “Taguchi” usually conjure up images of cell phones coming off the manufacturing line while someone inspects them for defects. The intended application for DOE is in the manufacturing world, so it’s easy to see why the product design community often neglects these methodologies. However, my experience implementing DOE into product design shows otherwise. Using screening designs, where applicable, can save design firms (and the harried engineers doing endless testing) time, money, and resources.
A DOE is loosely defined as purposeful change to the inputs of a process to observe change in the outputs. Whether we choose a fractional factorial or a full factorial does not change the fact that we are using DOE, unless we are picking a method without being mindful of the advantages and disadvantages of our selection. The image below illustrates the relationship among DOE (also referred to as Experimental Design), screening designs, and the Taguchi method. The Taguchi method is a type of screening design, which is a type of fractional factorial, which falls under the umbrella of DOE along with full factorial.
Becoming an expert in Design of Experiments often requires entire courses, poring over textbooks devoted to the subject, and spending hundreds of hours on real-world experimental design and analyses. A little bit of linear algebra helps as well. The advantage of Taguchi screening designs is that they provide a taste of the subject without requiring a degree in statistics. Engineers can access standard tables and available software, and become “experts” in screening designs over the course of a few hours.
Types of projects in the product design world that might be good candidates for a screening design include:
- A product that is not meeting expected performance results
- A large number of concepts that need to be narrowed down to a few
Popular Experimental Design Methods
How many of us have gone into the lab only to return with more questions than answers? Mechanical engineers like me, who are not usually trained in statistical analysis, tend to employ one of two experimental methods: full factorial or one-variable-at-a-time. Full factorial involves testing every combination of variables in a separate test configuration. The output is a model that theoretically describes the behavior of the system. How often do clients ask for a complete mathematical model of their system? More often than not, the client just wants to make it work. One-variable-at-a-time involves performing one test, looking at the results, and then designing the next test based off those results. The idea is that each test will point you in the direction of the optimized solution. However, this type of testing can often lead to numerous tests with conflicting information and no quantitative numbers to back up observations. A screening design provides the best of both of these methods; it points to the optimal solution with minimal test setups. If a more precise output is preferred, a full factorial can be run with more thoughtful, and ideally less, main factors.
Overview of Screening Designs
Screening designs introduced by Genichi Taguchi replace traditional full factorial configurations with a fraction of the tests by sacrificing information about interactions among main factors. Usually consisting of eight, 16, or 27 separate orthogonal configurations, depending on the number of design variables, screening designs point to the optimal discrete setting of each main factor tested. This means that screening designs largely neglect interactions among main factors. For example, if you were to run an experiment testing seven variables at max./min. values, for a full factorial you would run 128 different configurations. A Taguchi screening design calls for eight. If the screening design eliminates two factors from being important, your full factorial count suddenly drops to 32, with just the addition of eight preliminary tests.
In one case we implemented a Taguchi screening design to simultaneously reduce testing time and improve performance in an environment with a high degree of noise. Noise is a factor that affects performance but that we have little control over, like swings in ambient temperature, machining tolerances, or in this case, differences in technique from surgeon to surgeon. Our client had a bench-top surgical device that was operating well in the lab but had sporadic performance in the surgical suite.
We identified five main factors on the device we were able to change (design variables) and two noise variables that accounted for most of the variation we observed. We were able to pare down four of the factor options to two or three levels: either a max./min. for continuous factors or a discrete setting for qualitative variables. Our qualitative variables were three swappable blade geometries, each coming in two different sizes. Because none of these can be interpolated between, they are defined as discrete.
With a full understanding of the design variables and noise variables, we debated doing a full factorial versus a Taguchi screening design. A full factorial would have required 2 x 2 x 2 x 2 x 3 = 48 separate test setups neglecting noise and 48 x 2 (levels for noise variable one) x 2 (levels for noise variable two) = 192 separate test setups with noise included (total testing time would be about 28 days). A Taguchi L18 required 18 test setups x 4 runs for each setup to account for noise = 72 tests (total time would be about 5 days). This means that we only perform the test setup, the most time-consuming portion of the test, 18 times, and run four samples through each setup. The Taguchi L18 was the clear winner because of the balance of information about noise and time savings. The Taguchi method would not have been a good choice if our output, system performance, was a nominal target rather than a minimizing or maximizing target, or if the factors had not been independent from one another.
After running 18 separate configurations, the output is an indication of what factor settings lead to an optimized system performance. Below is a commonly used chart called a Main Effects Plot that compares each design variable with the overall mean system performance.
This plot indicates that in order to maximize system performance, shown on the vertical axis, the optimized design variable levels are high speed, high mass, elliptical blade geometry, and small blade size. Since the slope of temperature is shallow, I can eliminate it as a major contributor to system performance. Using my optimized design variable settings, I plug the system performance for each into the expected performance equation. This equation predicts the output, which in this case is system performance, when the design variables are set to the desired levels. The expected performance equation indicates that I can expect an increase in system performance from 62% to 73%.
The last step is to run confirmatory tests. Now that I have a number for expected performance, it is necessary (and just common sense!) to prove that the optimized variables will get close to this number. The number of tests is up to your own discretion; I usually run enough so that I’m confident in the result. If the confirmatory tests do not match the expected performance number, the issue is probably either that main factors or noise variables are not accounted for, or that there is a high degree of interaction among main factors.
If more data is required to further optimize results, a full factorial or other type of design could be run with more efficiency due to the knowledge gained from the screening design. In this case, most of the variable settings were either linear or discrete, which means I can rely on the screening design data to choose the optimized variable setting. If I suspected one of the variables was nonlinear, like if the speed variable actually looked like a quadratic function, I would want to describe a more complete model to be sure that my results reflect the actual performance. I might add levels to speed and test against the two successful blade geometries to get a more finely tuned model. This means that my original full factorial experiment is reduced from 192 tests to 3 (levels of speed) x 2 (blade geometries) x 2 (levels of noise variable 1) x 2 (levels of noise variable 2) = 24 tests.
Taguchi or not, the key to getting meaningful data out of testing is the old standby: measure twice, cut once. It’s tempting to sprint into the lab, throw something together, and expect things to go a certain way. What usually happens is that you end up wasting half a day figuring out that you need better controls when you could have been spending that time carefully planning out a test plan. What variables do I really care about? What do I want to measure? What is my contingency plan if I don’t get meaningful data? Better yet, once you get a meaningful plan together, get the client to sign off on it so that everyone is on board in terms of expectations, timelines, and program risks. Although screening designs may not be suitable for all situations, when they work they save a huge amount of time, money, and resources.
Engineering Statistics Handbook. Ed. Carroll Croarkin and Paul Tobias. 1 June 2003. National Institute of Standards and Technology Information Technology Laboratory SEMATECH. 10 December 2012 .
Stephen R. Schmidt and Robert G. Launsby. 2005. Understanding Industrial Designed Experiments (4th Ed.). Air Academy Press, Colorado Springs, CO, USA.
Human factors engineering as applied to the design of medical devices has never been as important as it is today, especially since the release of the U.S. FDA’s draft guidance document Applying Human Factors and Usability Engineering to Optimize Medical Device Design in June 2011. With the imminent rise of mobile health applications (apps), human factors engineering principles will become even more vital to the success of this emerging industry and to the safety of the patients for whom they are designed.
Numerous factors have contributed to the recent explosion of mobile health applications and remote patient monitoring, creating a perfect storm of opportunity for this sector of the medical industry. Economic trends, such as the push for cost reduction and new regulations mandating disincentives or penalties for the readmission of Medicare patients into hospitals within a certain period of time, are pushing healthcare providers toward a more vested interest in keeping patients at home. Technology trends, such as the migration to electronic health records, the advancement of digital health applications, cloud computing, the widespread use of social media and the ubiquity of mobile devices, play a huge role. The ability to self-monitor and keep a diary of health issues through the use of mobile apps is strengthening relationships between healthcare providers and their patients.
Consider the following points:
- By 2015, 500 million smartphone users are expected to be using medical apps1, according to Research2Guidance, a global mobile research group;
- The market for mobile health apps is expected to quadruple to $400 million by 20162, according to ABI Research;
- Over three million patients will be monitored over cellular networks by 20163; and
- Three of every four dollars spent on U.S. healthcare is for chronic diseases, and family caregivers are estimated to provide 80% of all long-term care for chronic diseases.3
FDA to start regulating mobile health/medical apps
In July 2011, the FDA released a draft guidance document on mobile medical applications “to inform manufacturers, distributors, and other entities about how the FDA intends to apply its regulatory authorities to select software applications intended for use on mobile platforms.” 4
Since the release of this document, there has been movement within the app development industry to understand and anticipate exactly which mobile applications will require FDA approval. The guidance indicates that the following types of mobile applications would be subject to regulatory processes:
- Software applications that can be executed on a mobile platform, or Web-based software applications that are tailored to a mobile platform but are executed on a server; and
- Software applications that have an intended use within the scope of the concept of medical device as regulated by the FDA, and:
- Are used as an accessory to a regulated medical device (for example, an app that connects to a medical device for the purposes of controlling the device in some way); or
- Transform a mobile platform into a regulated medical device (for example, an app that remotely monitors patient vital signs).
(Note that according to ANSI/AAMI HE75:2009, a mobile medical device is not limited to only mobile phones and tablets, but any device that can be mobile, whether by carrying or rolling.)
You may notice that the guidance does not apply to mobile apps intended to analyze, process or interpret medical data; the FDA has indicated that it will address these types of mobile applications in a separate guidance document. However, the important takeaway is that the FDA will soon be releasing legally enforceable guidelines that will apply to a plethora of medical and health apps already on the market and many more under development. In fact, a bill set to be introduced in the U.S. House of Representatives called the Healthcare Innovation and Marketplace Technologies Act (HIMTA) proposes to establish an Office of Mobile Health at the FDA specifically to provide recommendations on mobile health issues and create a support program to help developers navigate HIPAA privacy regulations.
Importance of applying human factors to mobile apps
The implementation of human factors engineering throughout the design process will be critical for emerging mobile health applications, not only because the FDA is exerting its responsibility to protect and promote public health by regulating these new mobile medical devices, but because it’s good practice and is an essential tool for decreasing patient safety risks while increasing usability and effectiveness.
Take it from someone who has already been through the process. An article written by Brian Dolan for mobihealthnews in May 2011 describes a panel that he moderated with several mobile health app companies who have already navigated the FDA’s 510(k) process successfully. In the article, WellDoc Founder/CEO Ryan Sysko is quoted as saying that if he could change one part of the process, he “would have the FDA provide greater clarity around what successful human factors testing looked like.”5
Our research and usability team at Farm knows the detrimental consequences of failing to apply human factors engineering to product development efforts. We have helped many clients whose medical devices have been rejected by the FDA for lack of necessary or appropriate human factors evaluations. As we always remind our clients, human factors is not a one-time testing event that occurs at the end of the development cycle, but rather an ongoing iterative approach that starts at the very beginning.
The importance of implementing an iterative approach in the design of mobile medical apps is as relevant as applying the process to physical devices. As the more savvy companies have learned, a robust process starts with the gathering of user requirements and includes preference testing of multiple design concepts, design verification, which could include several rounds of formative usability testing of the product itself and related documentation, and a final summative validation test that proves the successful mitigation of use-related safety risks. As is the case with physical medical devices, mobile medical app developers will be expected to follow the user-centered design guidelines of the international standard IEC 62366: 2007.
During the development process, mobile app designers should turn to established human factors guidelines, particularly those set forth in the ANSI/AAMI HE75:2009. Below are some examples from HE75 that could apply to mobile medical devices and/or apps.
- Carefully analyze the conditions under which the mobile device is going to be used (for example, when a user is moving or being moved, in moving vehicles, while wearing the device or during stationary use, on a rack, above the head, etc.).
- The display on the mobile device should not be obstructed by additional accessories, wires or devices.
- Auditory indicators can be used to supplement visual indicators and should provide the ability to adjust volume, on/off and native language (when feasible).
- When possible, aim to work with existing technologies that already have protocols in place that work with medical industry standards, such as the IEEE 802.11 series of standards for LANs, Bluetooth and cell phone protocols.
- Carefully analyze the conditions under which the mobile device is going to be used and how detrimental it is when the battery runs low.
- Keep important tasks immediately identifiable.
- Ensure that the design takes into account the small size of the screen, limiting the amount of images and text.
- Remain consistent; place information in the same place over a series of screens.
- Offer more than one way to navigate through the system.
- Provide guidance such as prompts or pop-ups when applicable.6
Below are some mobile app design best practices published by the mHIMSS Design Tenet Workgroup in January 2012.
- Eighty percent of screen real estate should be dedicated to data; twenty percent to interface.
- For readability, a single sans-serif typeface and up to six type treatments for the typeface are used.
- Color is used sparingly and helps the information, the interaction and the user experience accomplish an apps’s intended purpose.
- The app displays controls in a progressive manner, only the ones needed at specific points along the intended workflow.
- The app works within mobile device limitations such as: no hover text feature, larger target size and smaller display.
- The app leverages new capabilities, such as touch-based interactions, location awareness, proximity sensitivity, integrated communications and push notifications.7
The rise of the mobile health industry is underway and offers an outstanding opportunity to revolutionize healthcare. In September it was announced that the FCC would act on key recommendations from its mHealth Task Force to adopt wireless health technology8.
In order for mobile health application developers to be successful, they must create safe, easy- to-use products that can pass the rigorous FDA approval process. The critical path to this success begins by stringently applying the principles of human factors engineering. Drawing a reference from the Hippocratic Oath, the ultimate goal for designers is to first do no harm, and then do everything possible to provide the best possible product experience for the patient. The only way to do this is to involve end users in the design process from start to finish.
- Lawmaker Pitches New FDA Office of Mobile Health, Jenny Gold, Kaiser Health News, September 26, 2012.
- 5 Ways Mobile Apps Will Transform Healthcare, Derek Newell, Forbes.com, June 4, 2012.
- Webinar: The Inevitable Imminent Rise of Remote Patient Monitoring, MobiHealthNews, September 2012.
- Draft Guidance for Industry and Food and Drug Administration Staff: Mobile Medical Applications, U.S. Department of Health and Human Services Food and Drug Administration, Center for Devices and Radiological Health, and Center for Biologics Evaluation and Research, July 21, 2011.
- Lessons Learned from FDA Cleared Mobile Health Companies, Brian Dolan, MobiHealthNews, May 5, 2011.
- ANSI/AAMI HE75:2009 Human Factors Engineering – Design of Medical Devices, American National Standard, October 21, 2009.
- Selecting a Mobile App: Evaluating the Usability of Medical Applications, mHIMSS App Usability Work Group, July 2012.
- Fact Sheet – mHealth Task Force Recommendations, Federal Communications Commission, September 24, 2012.
I recently started my ninth year as a mentor for the Lawrence MA “Gearheadz” High School FIRST robotics team. FIRST is a non-profit program that serves to inspire young people's interest and participation in science and technology through robotic competition. Over the years it has occurred to me how this experience very much parallels the “real-world” of product development. Both have to deal with challenges that present themselves at the start of any new project, like cost, schedule, and resources. In both cases, there is never enough time or budget. Engineers, admit it: When have you felt that, at the end of a project, you had plenty of both? And from a people side, we all have to learn to work efficiently with team members of varying skill sets and levels, including those who may not yet have developed the necessary skills to deal with a project’s challenges. In this post, I present some of the commonalities I have observed between students in an inner-city high school robotics club and professional designers and engineers who constantly learn new methods, get inspired to solve the hard problems, and ultimately develop a useful device.
Product development firms, like Farm, must constantly generate creative solutions by relying on two fundamental elements: 1) a sturdy yet flexible foundation of a development process, and 2) creative individuals. Process, to a large degree, is a map of those common steps required to complete the journey from problem to solution, or in Farm’s case, from the seed of an idea to the implemented end product. Farm’s process is based on these elements:
- Strategy (e.g., definition of user, marketing, and business needs)
- Specifications and planning (e.g., definition of design inputs and general planning)
- Development (e.g., generation of product concepts through prototyping)
- Verification and validation (e.g., testing to ensure the device does what it’s supposed to do)
- Transfer into production
Farm’s process has played a critical role in all sorts of successful medical device projects, from drug delivery devices to surgical robots to wearable casts.
As for the second critical element: The creative individual. At Farm, we look for a balance of many attributes in our technical staff; he/she is open-minded, an out-of-box thinker, internally motivated, intellectually curious, knowledgeable in multiple manufacturing processes, can conceptualize appropriate-to-the-market concepts, versatile in typical design tools (e.g., CAD), and a team thinker (ability to work on abstract concepts in a group setting). There is no ideal proportion of these skillsets, as each individual brings different backgrounds and personal strengths.
This is a great place to start, but like a good spaghetti sauce base, these elements are only the core ingredients. We still need to add more flavors, a bit of personality, and enough time to bring it all together. To stay sharp and grow, we all need to stretch ourselves. This is made easier when the company culture encourages the synergizing of personal and company interests by sponsoring training events that range from internally-developed courses on Design for Experiments to Web-based courses on Geometric Dimensioning and Tolerancing to intellectual property licensing exams.
And to foster a positive work atmosphere, organizations should not forget to include the fun stuff like group cook-outs, biking treks through the woods, or ultimate Frisbee games, as these social events contribute tremendously to a cooperative environment, which is key for the long-term health of an organization. This is the building a team part... it takes time for people to gel and get into a groove.
So how does this relate to a FIRST high school robotics club?
Let’s first consider the process. The odds are that a process-starved professional organization will eventually fester into true anarchy (and ultimately bankruptcy!); the only question is when. Similarly in a high school club, any teacher will agree that should you dare to combine the youthful energy, immaturity, and hormonal “issues” of teenagers with no process, then an even more elevated level of anarchy will manifest itself and at a much more accelerated rate.
Drawing from my experiences, I’d like to leave you with a few takeaways, which are observations that apply to both the FIRST high school robotics program and the “real world” of product development. Recognizing these similarities should help bring home the pertinence of the robotics program for today’s youth as well as illustrate how professionals can gain valuable usable experience by contributing to such programs.
Takeaway #1: Mentoring kids offers a vehicle for hyper-fast-paced training for a professional by: 1) evaluating how robust your process is (describing a process to a teenager is usually harder than to a professional peer or direct report), and 2) sharpening your saw on how to recognize potential weaknesses for additional training. This could be considered a kind of “HALT (Highly Accelerated Life Test) for managers” training program. To be sure you’ll see the defects right away in your management techniques and abilities. (And you’ll have to fix them without delay to keep things rolling.)
Takeaway #2: Tossing these high school kids into such new spaces that they are not yet fully prepared for can give them wonderful exposure to the dynamics of working in a real-world group, including having to learn technical and team-related skills “on the job.” Though on a different scale, young professionals also experience this perspective, so make a point to nudge your less experienced staff into higher-level tasks from time to time. It’s okay to try something, be challenged, and struggle a little as it is all a part of the learning process!
Now let’s consider the “staff”. In a professional setting, you can usually count on the trainee to already have a sound background through some combination of formal education and/or experience. Not so in the case of the high school robotics club. In most cases for the Gearheadz, we need to teach fundamental things like how to drill a hole (safely and relatively precisely) long before we address the more complicated aspects of conceptualizing or the fundamentals of mechanical and electronics elements and programming. The process is important and it takes time. Eventually, meeting on multiple evenings each week and on Saturdays during the six-week winter build season, the kids all experience how to turn a problem statement (the rules of the game that change from year to year) into concepts, and ultimately into a 100 pound, four-foot-tall remote controlled robot that competes with and against five other robots developed by similar teams from all over the world.
Takeaway #3: Training needs to be delivered in a timely fashion and at an appropriate level. Especially true when the student is taking his/her first steps, but this also applies at any experience level. None of us like it when the subject matter is presented at too high or too low a level. Knowing your audience (and being flexible) is the mantra for anyone who teaches or makes presentations.
In the end, if we all don't keep focused on the need to constantly train the less experienced as well as ourselves, then we take the first step towards their, as well as our, own obsolescence.
And what about the fun stuff? “Fun” and "success" can have many different definitions, but they’re not necessarily mutually exclusive. For us Gearheadz mentors, we watch for the blood, sweat, and tears that every student puts into each robot every year and how it can positively impact lives. One particularly memorable moment for me was that shriek of excitement from a freshman after drilling her first hole using the drill press. The student was later voted president of the robotics club as a senior and is now a senior robotics major at Worcester Polytechnic Institute. Another success is the quiet kid with struggling grades who eventually raised his grades to the team’s standards and morphed into a leader and a skilled robot driver. Still another is last year's co-captain, who joined the team as a junior with no intention of majoring in a science, but is now an engineering major in college.
Takeaway #4: Though it is still called work, keep it fun. You don’t need to be a teenager to admit that life is too short to not have fun!
In conclusion, there is no longer any question that America needs to bring its A-game to escalating the understanding of the sciences by today's students from kindergarten through college. Unfortunately, too many finger this as a problem for the schools to address. However, there are many opportunities for people to pitch in to help solve the problem, to expose kids to this fascinating world of technology, to permit a kid to practice leadership as part of a team, to further motivate the kid who already has that sparkle of interest in his/her eyes, or to draw the aimless kid away from the seemingly inevitable dead-end choice. FIRST is one of these programs.
And we as professionals get to sharpen our “work” skills in the mean time!
A critical indicator of progress during the early stages of product development is the pace at which new information is created; such information is simultaneously the foundation upon which new concepts are built and the lens through which current concepts are viewed.
So how best to create new information?
Experience helps; all things being equal, having done something before encourages success the next time around. Beyond that it’s prototype, evaluate, repeat. Quickly.
Prototyping isn’t the bottleneck; between rapid prototyping and traditional shop work, concepts can be realized almost as quickly as designers and engineers can think them up. The limiting factor is evaluation, particularly trying to quantify the fuzzy front end.
Evaluation in the early stages typically focuses on low-fidelity prototypes, demonstrations by expert users, videos of actual procedures, and benchmarking of competitor products. Much of the information gathered at this point is necessarily qualitative. Still, some quantifiable questions remain: how much deviation is there in the angle at which different docs hold a laparascopic device? What is the path of personnel during a procedure? How does the movement of someone wearing a brace compare to healthy person?
Questions such as these can be answered (some with surprising ease) using computer vision (CV). All that is required is a camera to capture the images/videos and a tool such as MATLAB or Python to analyze it.
I’m going to share a simple example of CV in just a moment, but first a word of caution: some circumstances lend themselves to CV more than others. Specifically, straightforward CV projects will share the following characteristics:
- Measurements of interest are 2D (3D requires stereoscopic vision)
- In a controlled environment (i.e. consistent lighting, plain background)
- Low precision (high precision requires more fine-tuning of the code, camera resolution is also a factor)
In the example below, a Post-It is placed near the ankle, knee, and thigh. From this modest setup, angle data is extracted during a portion of gait.
Setup time: 20 min.Coding time: 2 hours (without reusing code)
The great thing about computer vision as a development tool is that it’s expandable. Once you can perform a simple task such as the one I’ve shown here, it’s a short step to more advanced topics like real-time vision and skin detection. Once you start adding these tools like these to your tool belt, you’ll be surprised at the uses you find for them.
What is the most valuable engineering resource Farm has at its disposal? From a cost/benefit perspective, it’s hard to argue against the co-op student. The benefits are straightforward: a company identifies talent while utilizing low-cost labor and a student gains first-person knowledge of an industry and its practices. Farm currently utilizes two simultaneous co-ops in six-month employment cycles, and being a former co-op student myself (now an engineer at Farm), I feel compelled to share my thoughts on how both companies and co-ops can benefit most from the cooperative educational experience. While my previous co-op experiences were generally positive, I have also observed a number of negative experiences in the past. Here are some recommendations based upon traits that commonly create positive co-ops, from both an employer and student’s perspective.
For Co-op Employers
Have Work on Day One
Co-ops generally arrive to work on their first day with their motivation and ability at inversely proportional levels. They want to help, but they don’t know how. Having a task from the get-go is extremely important as it creates an immediate sense of value and worth. This is a challenge for many product development firms because co-op-appropriate work doesn’t always exist. In this situation, create work. If a co-op lacks CAD skills, create a design problem that requires grasping complex surfacing or master models. Try not to rely on stock tutorials and lessons as most students have already completed these. A unique and well-defined task can go a long way toward creating lasting co-op motivation.
Understand a Co-op’s Goals Before and After Hire
While interviewing a co-op, most companies will inquire about what a student wants to get out of the experience. Some will answer honestly, some will tell the interviewer what they want to hear, and some simply don’t know. Regardless, it is important to continue this dialogue after the student is hired and throughout the experience. Interests and aspirations change rapidly for co-ops, and showing that you value their goals is essential.
Throw Them into the Fire
With a good co-op student, a product development firm can take the burden off of its full-time employees and rely on the co-op to complete billable tasks. This inherently requires a certain level of trust; trusting the co-op to complete deliverable tasks at a high level. Because of this, many firms wait to give co-ops responsibility until they have definitively proven themselves over time. Instead, try throwing them into the fire. Treat co-ops like full-time employees while simultaneously checking their work exhaustively. This method might eat into company overhead in the short term, but it’s worth knowing a co-op’s true capabilities as soon as possible.
Create Positive Press
The primary goal for a co-op employer is to get quality work from an otherwise inexperienced employee. While this is a completely valid goal, co-op employers often overlook the goal of creating positive press for their companies. When co-op students return to class, they reflect on their experience. They make presentations, write reports, take surveys, and above all share their reflections. Because of this, a positive co-op experience for one student can completely influence the opinion of hundreds of others. Why not make a positive impression on the engineers and designers of tomorrow?
Understand Your Company
You need to do your homework! A co-op student who researches the ins and outs of a company will get in the game much faster once employed. Prior to employment, co-ops should strive to gain a full understanding of a company’s core competency, clientele, competitors, business practices, past projects, and future goals. It might not be easy to find, but the information is out there, especially if the company is classy enough to have a blog. Possessing this knowledge allows co-ops to understand where they can fit in and allows them to suggest how they might be most valuable to a company. It isn’t enough to follow instructions. Good co-ops should assume they are being underestimated, and use company knowledge to ensure that they are utilized as effectively as possible.
Know How to Learn
Learning at a product development firm is stressful. For co-ops, balancing personal educational goals with the hectic demands of contract product development can seem borderline unachievable. Fortunately, product development firms are among the most efficient educational outlets out there, provided you know what you’re getting into. Here’s what to expect:
When billable work doesn’t exist, co-ops are often used for company initiatives or are left to their own devices to complete lessons and tutorials. This is the best time to find work that benefits both co-op and company. For engineers, learning a relevant programming language, becoming familiar with GD&T, developing machine shop skills, and becoming acquainted with new manufacturing techniques are all examples of valuable ways to spend unbillable time. Try to focus on subjects that will be applicable to upcoming projects.
Every time I’m forced to learn something new for a project, Bill O’Reilly’s voice pops into my head yelling “We’ll do it live!” This is where the most efficient learning happens. You’re forced to master and execute a skill in a very short timeframe, and getting it wrong isn’t an option. You pull information from coworkers, past projects, and outside contractors simultaneously. It isn’t ideal from a low-stress point of view, but from an efficiency perspective, there’s nothing better. As a co-op, being ready and eager for this type of education is indispensable.
Show Off Your Skills
A major source of co-op frustration arises when students aren’t given challenging work. The cause is understandable from both perspectives; co-ops want their work to teach them new skills, but employers don’t want to give important tasks to inexperienced workers. Co-op skill also varies widely from student to student, so employers are often conservative with the work they delegate. To solve this problem, co-ops should try to advertise their capabilities as much as possible. A company-wide “About Me” presentation or public portfolio that outlines previous projects and experience can be extremely beneficial for both co-ops and employers. Additionally, “About Me” presentations should be used to express relevant educational goals and interests. It seems obvious, but many co-ops will complete entire internships without ever expressing their skills and goals.
A good co-op experience has the potential to produce both a valuable education for a student and a budget surplus for an employer, a win-win scenario. Unfortunately, co-ops are often underutilized in the contract product development world. Deadline-driven projects necessitate instant results, and project managers just don’t have the resources to bring co-ops up to speed. Instead, co-ops sit idle and the reciprocal relationship is broken. It’s a shame, because when the co-op experience is successful, the long term advantages can be immense. Budgets are balanced, talent is identified, and knowledge is gained. A quality co-op experience is something to strive for by both students and employers, reflected by the fact that almost all of Farms entry-level engineering hires are previous co-ops. It’s evidence that a well-executed co-op program benefits everyone.
Lee PaneckiMechanical Engineer at FarmNortheastern University Alum
Interested in being a Farm Co-op? Visit Farm's Career Page!
In this enlightened age of “intelligent engineering”, data and the interpretation of data have become critical to the success of a project. The development tools in our high-tech toolbox are always expanding and bringing new insight to increasingly complex systems. But without a detailed product specification roadmap, we fall into the trap of generating oceans of useless data. The design team of today needs to be keenly adept at gathering, filtering, and utilizing data for the best possible patient outcome as well as commercial success.
Engineers love data. We always want more high-speed cameras from more angles to record more prototype scenarios to verify more simulations. This brings us to one of a program manager’s greatest frustrations: usable data. Whether you are designing an autonomous surgical robot, a patient-specific knee implant, or a paperweight for all the new regulations, volumes of data are useless without the proper specification framework to filter and process it. This is the role of the Product Requirement Document (PRD).
The PRD is designed to keep the development team on task and focused on specific product functionality. If the PRD is written at the end or evolves during the development program, truth-seeking engineers will generate ever increasing volumes of data. We do this to provide program decision makers a broad spectrum of options with varying degrees of feasibility. Working without a PRD is absolutely the slowest and most expensive way to develop a product. In a perfect world, every program would start with a detailed PRD backed by historical benchmarking data, but this is not an option for truly innovative technology looking for a production platform. Even without a PRD, the development team will still need critical, effective processes to reach definitive milestones and eventually a market-conquering product. Without hard stops during the process to critically analyze data, you will get stuck in never-ending loops of data gathering and end up creating more variables than answers.
Because development resources are finite, brainstormed concepts need to be vetted quickly and either fail fast or prove promising and carried further. If hanging weights off of a rapid prototype strapped to a car bumper gets you pertinent data in hours as opposed to machining prototypes and a detailed test fixture over four weeks, go with the car bumper. The most challenging aspect is to define the critical variables and harmonize them into a succinct goal. A great development team finds the facts in the figures and keeps the thought process moving forward quickly and efficiently.
One of the most powerful tools to reduce and accelerate prototyping loops is finite element analysis (FEA). FEA has made giant leaps forward in capability over the past 15 years. Large assemblies are now translated directly from any CAD environment, contact pairs are automatically assigned, and a mesh is created after the click of one button. It is the perfect tool to fail fast in both positive and negative ways. The results are shown plain as a rainbow on the computer screen, but are they fact? The quickest way to further a concept with FEA is through comparative analysis. If option A predicts X stress and option B is 1/2X stress with the exact same setup, you are moving in the right direction. The question then becomes are the physical stress levels at X or 10X.
There are many ways to validate FEA simulations in the real world. With the ever decreasing cost of 3D printers, you can be up and testing concepts in hours. The 3D printing materials will not be exactly representative of production materials (build layers, for example, are ideal crack slip planes), but remember that stress is force/area. Whether a part is printed or machined, both will have the same high-stress location when tested in identical setups just different ultimate loads and displacements. The issue arises when your prototype fails in ways you did not predict or expect. The failure load and location are in essence only one data point in a complex system. Would thinning out a section and increasing flexibility actually be more beneficial than thickening and stiffening a part? Does adding a gusset to support a corner relocate the high stresses to an even more critical location? To fully comprehend the behavior of a part or assembly, you need multiple data points within the system to trend or predict how your concept is performing as a whole.
Strain gages provide very enlightening data for validating FEA simulations. I have found over the years that many people either don’t understand or are afraid of using strain gages. The key is to confirm your gage application and data gathering process on something simple, like a bending beam, to start with. Perform the hand calculations, confirm the calculations with an FEA run, and match your physical gage readings to the predicted results. Once you validate your process, a whole new array of data gathering is available to physically map the performance of your concept. Strain gages have been immensely useful for sorting out FEA constraint issues and uncovering inappropriate setup assumptions. It is rare in the real world that anything is truly fully fixed. The development team will still need to iterate between the FEA setup/results and gage readings, but at least they will have a physical data path to correlation. It takes an experienced technical lead to understand when the analysis and physical testing are aligned well enough to predict the trend for your specific functional requirement. Strain gages are cheap, apply them liberally.
The desire to compress timeframes through the “intelligent engineering process” has become a reality across multiple industries. Development teams of today have the most incredible physical and virtual tools at their disposal, but they need to know how to most efficiently utilize them. Even without a PRD, data still needs to be gathered, sorted, and acted upon to develop and validate a concept. The right data in the right hands at the right time does make all the difference. Budgets, personnel, and time to market have all been significantly reduced, while at the same time product expectations have never been higher.