- Measuring desired or intended result (e.g., improved health outcome) is covered only in a few AFs.
AFs should capture if the application can measure a desired or intended result in everyday use and particular environments. Some AFs do ask questions like if the app sets goals for users or allows them to set goals for themselves; if goals achievement is tracked; if there is a visibility of progress, so some results are measured.
There is a gap in measuring results by the app provider, controlling if data generated and recorded are accurate, if it is relevant to the range of values expected in the target population, and if it is possible to detect clinically relevant changes or responses.
Also, there is a gap in demonstrating relevant outcomes of application (e.g., behavioural or condition-related user outcomes such as reduction in smoking or improvement in condition management, evidence of positive behavioural change, user satisfaction), and presenting comparative data (e.g., relevant outcomes in a control group, use of historical controls, routinely collected data) by the app provider.
Note: example for relevant outcomes and present comparative data taken from Evidence Standards Framework for Digital Health Technologies: https://www.nice.org.uk/about/what-we-do/our-programmes/evidence-standards-framework-for-digital-health-technologies
- Few questions about ethical concepts are explicitly included in the AFs.
While assessing AFs for effectiveness criteria, specifically if the framework can capture the app’s applicability by distinguishing different subgroups of users (e.g., demographics, age, gender, health literacy, medical condition, health status), it has been observed that most AFs do not include questions concerning to ethical concepts.
When they exist, questions regarding ethical concepts are spread through different assessment domains.