The assessment of cognitive functioning is core to most mental health and chronic conditions. Cognitive problems are one of the most critical comorbidities of many diseases. Hence, it is essential to be able to assess cognitive functioning, such as working memory, reaction time, spatial-visual skills, executive functioning, etc.
Cognitive assessment has traditionally been done on pen-and-paper and in a clinic, where a clinician is responsible for instructing the test subject and performing the assessment. Over the years, a wide range of such cognitive tests has been designed and validated, including Multiple Object Tracking, Corsi Block Tapping, Flanker, Reaction Time, Trail Making, and many more. Even though some of these tests now are available on tablet PCs (e.g. the CANTAB system), most of these test batteries still resemble the pen-and-paper version and still require to be performed in the clinic by a clinician.
In order to move cognitive assessment “out of the clinic” and to be able to make large-scale cognitive assessments, potentially on a population level, we have been designing a set of new cognitive assessment tools. Each of these tools is targeted at a specific hardware platform, and each has its own benefits and drawbacks in different settings and studies.
The tools are:
- The Internet Cognition Assessment Tool (ICAT). This is a browser-based web application for running cognitive assessments following the Screen for Cognitive Impairment in Psychiatry (SCIP) protocol.
- Cognition Package. This is a set of 14 standard cognitive tests implemented to run on an interactive smartphone. Cognition Package is a Flutter package and can be used in the design and development of mHealth applications, which want to incorporate assessment of cognitive functioning.
- Ubiquitous Cognitive Assessment Tool (UbiCAT). This is a smartwatch-based platform supporting three simple cognitive tests for lightweight and simple cognitive assessment.
The Internet-based Cognitive Assessment Tool (ICAT)
ICAT is designed based on the cognitive tasks of the Screen for Cognitive Impairment in Psychiatry (SCIP) protocol. The cognitive measures of ICAT are;
- immediate recall
- delayed recall
- working memory
- psychomotor speed
In contrast to the SCIP test – which is to be administered by a clinician in a clinic – ICAT is designed to be performed by the test subject on his or her own, using his or her own computer at home. Hence, a core design goal of ICAT is to enable “cognitive assessment at home”.
Now since the test subject has to perform the tests on his or her own without any help from a clinician, ease-of-use in a broad sense is extremely important, and ICAT was thus designed and developed in a user-centered design process. Moreover, as a novel feature, ICAT makes extensive use of automatic speech recognition (ASR) for the assessment of verbal recall. Using ASR allows the test subject to “talk” to the computer, just like s/he would talk to the clinician in the clinic. This is used in the recall tasks, where the test subject can recall and speak the words, which then would be recognized by the ICAT system. Currently, ICAT – including support for ASR – is available in English and Danish, and more languages are in the planning.
A feasibility test showed good overall usability of ICAT and good accuracy of the ASR for verbal memory assessment. The word error rate (WER) for English and Danish responses was 17.8% and 6.3%, respectively. We also found a moderate correlation between the ICAT and SCIP scores [1].
A clinical study with Bipolar Disorder (BD) patients has investigated the validity of ICAT, i.e., whether ICAT is a valid tool for cognitive assessment as compared to the “golden standard” (the SCIP test). This was done by investigating (i) the sensitivity of ICAT to assess cognitive impairments, and (ii) the concurrent validity of the ICAT as compared to SCIP [2]. Read more on this blog post.
We are now in the process of updating ICAT to version 2, where an additional Facial Recognition Test will be added. We plan to make ICAT available as part of the CARP Platform.
Cognition Package
Cognition Package is a Flutter package for incorporating cognitive testing into mobile health (mHealth) apps on Android and iOS. Cognition Package is built using the Research Package and is a Flutter implementation of a cognitive test battery including 14 validated gold-standard cognitive tests spanning all 8 Neurocognitive domains. Moreover, Cognition Package is also a programming framework that allows mHealth application developers to implement new cognitive tests – if the set of 14 pre-made tests is not sufficient.
The cognitive tests in the Cognition Package can be added to any mobile app and hence allow designers and developers of mHealth applications to add cognitive assessment across multiple health domains. The CARP homepage contains a tutorial on how to use the Cognition Package.
We have used the Cognition Package in mHealth apps for mental health, diabetes, and neurology. Furthermore, by combining the Cognition Package with the CARP Mobile Sensing framework, it is possible to build mHealth applications that sample objective sensor data while doing cognitive assessment. For example, the collection of accelerometer data while doing finger tapping or reaction tests for Parkinson’s Disease. Or the collection of heart rate data while doing a Flanker test in mental health.
As illustrated in Figure 2, the CARP Studies app has been designed to support a study protocol that can incorporate cognitive assessments in combination with mobile sensing in general. The cognitive tests are added to the user’s “Task List” as s/he has to perform them. In this example, the task is called “Parkinson’s Assessment”.
We are continuously maintaining the Cognition Package and improving on the tests. Cognition Package is released as open-source under an MIT License and we welcome others to help maintain and add new cognitive tests to the package (as Pull Requests to the GitHub repository).
UbiCAT
The Ubiquitous Cognitive Assessment Tool (UbiCAT) [3, 4, 5] is a smartwatch-based set of cognitive assessment tests. UbiCAT implements three cognitive tests:
- Arrow test
- Letter test
- Color test
Each of these tests is adapted from the two-choice reaction-time, N-back, and Stroop tests, respectively. The tests were designed in cooperation with domain experts and in a user-centered design process ensuring high usability. UbiCAT is designed to work together with a smartphone for data collection and passive data sampling, as shown below.
We have evaluated the UbiCAT test measures against standard computer-based tests and Pearson correlation analysis between the total correct responses obtained from the UbiCAT and the computer-based tests revealed a significant strong correlation (r=.78, P<.001) [3].
Furthermore, UbiCAT has been subject to a clinical study where we analyzed daily attention, working memory, and executive functions of patients with Bipolar Disorder and healthy controls [4]. The study showed that UbiCAT was a valid measure of cognitive functioning as compared to state-of-the-art cognitive assessment tools and that it was possible to classify patients and healthy controls on the basis of their daily cognitive and mobile data. The study also investigated the relationship between cognitive performance and different “digital phenotypes” like activity, mobility, and sleep. For example, the study showed a significant correlation between sleep duration and the next-day working memory performance.
These findings demonstrated that UbiCAT is feasible for for in-the-wild cognitive assessments, i.e., outside the clinic and in the everyday life of patients.
In contrast to ICAT and the Cognition Package, UbiCAT is no longer maintained or available. Partly because the Fitbit Ionic watches used for UbiCAT have been discontinued. We might, however, at some point revisit this line of research and create a “drilled down” version of the Cognition Package that also runs on smartwatches.
References
[Bibtex]
@article{hafiz2019internet,
title={The internet-based cognitive assessment tool: System design and feasibility study},
author={Hafiz, Pegah and Miskowiak, Kamilla Woznica and Kessing, Lars Vedel and Jespersen, Andreas Elleby and Obenhausen, Kia and Gulyas, Lorant and {\.Z}ukowska, Katarzyna and Bardram, Jakob Eyvind},
journal={JMIR formative research},
volume={3},
number={3},
pages={e13898},
year={2019},
publisher={JMIR Publications Inc., Toronto, Canada}
}[Bibtex]
@article{miskowiak2021internet,
title={Internet-based cognitive assessment tool: Sensitivity and validity of a new online cognition screening tool for patients with bipolar disorder},
author={Miskowiak, KW and Jespersen, Andreas Elleby and Obenhausen, Kia and Hafiz, P and Hestb{\ae}k, E and Gulyas, L and Kessing, LV and Bardram, JE},
journal={Journal of Affective Disorders},
volume={289},
pages={125--134},
year={2021},
publisher={Elsevier}
}[Bibtex]
@article{hafiz2020ubiquitous,
title={The ubiquitous cognitive assessment tool for smartwatches: design, implementation, and evaluation study},
author={Hafiz, Pegah and Bardram, Jakob Eyvind},
journal={JMIR mHealth and uHealth},
volume={8},
number={6},
pages={e17506},
year={2020},
publisher={JMIR Publications Toronto, Canada}
}[Bibtex]
@article{hafiz2020wearable,
title={Wearable computing technology for assessment of cognitive functioning of bipolar patients and healthy controls},
author={Hafiz, Pegah and Miskowiak, Kamilla Woznica and Maxhuni, Alban and Kessing, Lars Vedel and Bardram, Jakob Eyvind},
journal={Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
volume={4},
number={4},
pages={1--22},
year={2020},
publisher={ACM New York, NY, USA}
}[Bibtex]
@inproceedings{hafiz2019ubicat,
title={UbiCAT: wearable technology for ubiquitous cognitive assessment},
author={Hafiz, Pegah and Bardram, Jakob E},
booktitle={Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers},
pages={278--279},
year={2019}
}