Learnings from the existing health apps assessment frameworks: selection of innovative insights

Europe faces a lot of illnesses based on society behaviour. The demand for mHealth based solutions and applications is rising. These are developed either by large international companies or smaller predominantly local companies with new ideas for health services.

Focus of both is usually to address an international market with an emphasis on scalability. By stating clear rules of how patient data is handled (GDPR), the trust in mHealth apps will be increased for European citizens and the use of mobile apps might increase.

Creating a trustworthy environment is the key to wider adoption of mHealth solutions and apps by patients and medical professionals. This adoption supports decentralisation of healthcare and further promotes emerging mHealth and telemedicine solutions. Implementation of such solutions reduces costs and inflow of patients to hospitals potentially leading to a higher standard of care. One way of achieving this is for the assessment frameworks (AFs) to focus on how the personal data are managed in terms of access, retention policy and transmission methods.

Safety is regarded as a way to safeguard the user of the mHealth solution. For this end almost ¾ of the analysed AFs had this into consideration in one way or another. Most commonly, content quality that provides health benefit for users, is the issue more relevant for the larger majority of frameworks. Claiming a health benefit entails having carried out a benefit risk analysis. Health risks are to be as low as reasonably possible and health benefits are to outweigh health risks to provide users with a degree of safety.

An important aspect of safety for users of the mHealth solution is the capacity for a citizen/patient to communicate with a health professional, either by design or by default of the solution. Meaning that if something is wrong with the usage or with the user of the mHealth solution a procedure can be in place to automatically communicate such misuse, or problem, or on demand by its user. As an example, this communication is mostly used by monitoring apps for chronic or prolonged diseases.

As a consequence of the mentioned above, user input information safety is one criterion that is seldom present, and can be considered an innovative example for the AFs. As an example, Medappcare Quality Approach framework addresses this issue. The objective can be to assess if an mHealth solution has systems in place to verify or evaluate user input information and report if problems are detected.

Medical backing and valid information are of great concern when implementing mHealth solutions. This domain is assessed by more than half of the frameworks. It is important to have in mind two main paths to evaluate validity that are expressed in, for example, AppSaludable framework: the validity in terms of where the information is gathered and supports the content of the app, and the validity in terms of accountability to the information that supports the app. Both aspects are of great importance and should be considered. Some assessment frameworks go even further to assess the level of liabilities for the information provided.

Most of the frameworks only consider one of the paths (the source and proof of information) and thus there is an opportunity to improve frameworks in the validity domain. Also, what stands out from all the frameworks that address this domain is the fact that only a few are concerned with very clearly indicate the user to refer to their physician.

As it is previously stated, it is important to ensure that the app can maintain its level of performance in technically demanding events like a sudden increase in the number of users, the simultaneous connection of all users, a sudden increase in the amount of data, and everyday events like an interruption of the internet connection. To ensure apps’ level of performance in those events, app providers need to do performance testing. Some AFs do ask about testing, but with a focus on end-user testing.

Although they do not ask questions about testing while assessing applications, Tic Salut Social in their Developer’s handbook describes necessary tests that should be carried out to ensure that the software complies with the identified needs and with the design.

• Unit tests are those which evaluate the functionality of a method or a function for example, in isolation from the rest of the system.

• Integration tests consist of investigating how two or more elements which have previously been subject to unit tests work together.

• Stress tests focus on the software’s performance when it is tested to the limit, to see how long it manages to work normally and what happens when this threshold is exceeded.

• Penetration tests are tests involving a simulated malicious attack by someone using the latest techniques to violate the app’s security to extract its data, corrupt its operation, etc.

This description can be used for improving questions about testing in the technical stability domain.

Use of web content accessibility guidelines (WCAG 2.0) that apply to mobile web content, mobile web apps, native apps, and hybrid apps using web components inside native apps. For instance, mobile accessibility considerations must be related to the four accessibility principles: (1) perceivable, (2) operable, (3) understandable and (4) robust.

• Perceivable incorporate information about the screen size, zoom/magnification and contrast.
• Operable refers to control of touchscreen devices, gestures, device manipulation and placement of buttons.
• Understandable refers to screen orientation, consistent layout, scroll, grouping elements, actionable elements and customization of screen and gestures.
• Robust addresses data entry methods such as virtual keyboard and platform characteristics.

There are also examples of techniques that apply to mobile applications, such as text alternatives, navigation, predictability and compatibility. See French Haute Autorité de Santé “Good Practice Guidelines” (p. 40) and UK NHS Digital Assessment Questionnaire v2.1 (p. 34-35) for a short guidance.

Use of international standard ISO 9241-210:2019 (Ergonomics of human-system interaction — Human-centred design for interactive systems) to assess the usability of mobile health applications. In particular, to measure the effectiveness, efficiency and satisfaction of each mobile health application.

For instance:

• how usability relates to the purpose and use of the product, system or service (e.g. size, number of users, relationship with other systems, safety or health issues, accessibility, specialist application, extreme environments).
• The levels of the various types of risk that can result from poor usability (e.g. financial, poor product differentiation, safety, required level of usability, acceptance, user experience).
• The nature of the development environment (e.g. size of project, time to market, range of technologies, internal or external project, type of contract).
• Aspects related to timing and resources, where extra communication and discussion to identify and resolve usability issues early in the project will afford significant savings at later stages when changes are, inevitably, more costly.

In addition, see French Haute Autorité de Santé “Good Practice Guidelines” (p. 39-41) and UK NHS Digital Assessment Questionnaire v2.1 (p. 34-35) as a guidance.

Transparency for mobile Health applications requires information on several aspects like benefits and effects of such tools, as well as the actual use and possible harms. Based on the short development cycles, a transparency evaluation may partially only rely on data provided by the manufacturer.

One of the main aspects of transparency is the accurate information about the way an application handles, transmits, stores and secures user related data. This aspect also includes sharing of data with third parties. As transparency is directly linked to the aspects of privacy, safety and security, examples for the secondary use of data, or the connection to open data platforms are of special interest.

Creating a transparent approach should include full information about the way data is handled, transferred and stored. One way to do so maybe the usability of transparency enhancing tools, which are specifically designed to help users to improve their privacy. Transparency tools may also aim to check health-related data, like prescribed medicine or given diagnoses.

As one of the least captured assessment domains, second to Scalability, Reliability assessment examples can provide an innovative aspect to AFs. Reliability focuses primarily on consistency and stability of results. Other aspects, collected from the analysis, are the assessment of errors and how everything gets logged or documented. 

To this end there is one example (France Good Practices Guidelines) that assesses failure rates, measurement error rates, and hardware risks of all types. The data should be also evaluated and documented, which is a big focus in one instance that can be a good example to follow in other AFs. This is a critical domain because the accuracy of data collected may vary between the products available on the market and their intended uses. Users should be aware of the precision and reproducibility of data measured for the intended use.

Testing is also an important part of reliability, but even fewer frameworks address the issue. To this issue there is space to expand reliability assessment by means of introducing assessment of testing methods. Such a case is present in one of the AF (CEN ISO/TS 82304-2).

Europe comprises of different countries each having its own health IT infrastructure and often employing different healthcare strategies. By implementing standardized interfaces (based on international harmonized communications standards) the likelihood of a wider-scale adaptation of an app in multiple EU countries increases.

mHealth apps can be included into existing systems-of-systems by respecting common interfaces to share data. Hence, such mHealth apps address a bigger market (EU/international) compared to applications that establish proprietary data solutions and services without interoperable link to the overall IT infrastructure/system.

The AFs could include references to specific harmonized international IT standards. Furthermore, requirements could focus on disclosing the used data models and service specifications to facilitate interfaces with the mHealth app using inter process communication capabilities provided by the operating system. This step would facilitate direct communication between apps on the users’ devices.

As a result of assessing AFs, it is pointed out that more than 80% of reviewed frameworks can capture if the assessed app is claiming to have health benefits. Most of those frameworks capture what health benefits the assessed app is claiming to have and if is there evidence about claimed benefits. Only a few AFs are not focused only on health benefits but also ask about other types of benefits – economic, behavioral, psychological, social, etc.

An innovative example that has to be mentioned is a benefits question that includes the concept of ethics, asked by CEN/ISO: “Is evidence available of a positive effect of the health app on health inequalities, access to care for hard-to-reach populations or eliminating discrimination?”. When building or upgrading AF, it is recommended to think about including other types of benefits in addition to health benefits. Questions asked about the other types of benefits can be formed the same as questions asked about health benefits.

Capturing health risks and side effects of mobile applications is very important for patient safety. This criterion can be assessed under effectiveness, patient safety, clinical safety, device safety, quality, risks, but it is important that it is assessed. A lot of AFs ask a question about if is there a health risk or a side effect of mobile application, next question is usually if this risk or side effect can be captured. These questions can be expanded with a list of all possible risks and side effects mobile application can cause, and the question is if information on potential risks and side effects is available to the patient using the application. Improvement can be made on capturing a methodology used to identify possible risks or side effects. Also, AFs could ask about measures that have been put in place to prevent a recurrence of any reported events.

Scalability, as the least captured assessment domain, can be the one to look for to expansion the AF development. This domain is synonymous to growth and interconnection. Common platforms often limit the way applications can communicate with each other to ensure stability of the overall platform. Where the domain is captured, extensive guidelines exist to apply interconnection between services, such an example can be found in PCHA’s Continua Design Guidelines. These design guidelines are focused on enabling the interoperable exchange of information across a Services Interface that can be applied for several different use cases, including uploading of measurement data, completing questionnaires, and executing commands.

Another benefit of including Scalability, from the example of the PAS 277, is to include assessment of compatibility of apps with different platform configurations. This in turn will have a trickledown effect to assess ways that information collected or used by the app may be reused, under appropriate privacy controls. Also, a possible design feature would be functionality that is dependent upon the underlying platform, and so might need to be changed when the app is supported on a different platform, should be designed as a separate component and that can be replaced without affecting the rest of the app code.

Security, with more than 80% observance, is one of the most predominant domains in the analysed AFs. Security can range from security of data in the mHealth solution, cryptographic communications, protocols, network issues, and more. These are common traits of all the frameworks that address security.

Security of user’s data, either when logging or sharing it, is the most common assessment.

Nevertheless, some important details are not so common but equally important. One of such details, from CEN ISO/TS 82304-2, is the assessment of specifically assessing if data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, is processed by the mHealth solution, or app.

Also, a good approach identified in the assessment is the usage of very specific criteria from some assessment frameworks. In the example of the France Good Practices Guidelines, threat analysis is mentioned and an assessment of security by design and by default is made. The specificity goes further with the evaluation of the protection provided through a robust encryption protocol using state-of-the-art cipher suites, such as TLS. Specificity is hard to achieve and may lead to extensive frameworks, but it ensures quality criteria needed for health solutions.