Conclusions
This doctoral thesis focused first on the usage of consumer devices – Android smartphones and Wear OS smartwatches – for HAR systems, to finally explore the possibility of avoiding the use of any wearable device, relying on infrastructure devices and Wi-Fi technology for sensing purposes.
First, data collection libraries for Android smartphones and smartwatches were developed. Although other libraries for such tasks exist, the developed libraries were strictly designed focusing on data collection reliability (i.e., minimizing missing data, collection stopped by energy restrictions, etc.)
Then, after performing actual experimentation, public datasets containing inertial data from smartphones and smartwatches were reviewed, putting the focus on the placement of the devices and the heterogeneity and diversity (e.g., age and gender) of the subjects that participated in the data collection process. From the five identified datasets, only two of them properly reported demographic information about their participants (Vaizman, Ellis, and Lanckriet 2017; Arrotta et al. 2023), while others failed in reporting information such as age or gender distribution (Stisen et al. 2015; Sztyler and Stuckenschmidt 2016; Weiss, Yoneda, and Hayajneh 2019). Therefore, a data collection process was carried out to build a dataset, carefully selecting the participants. As a result, the collected dataset contains the most diverse subjects in terms of age and the most equally distributed in terms of gender, compared to those found in literature, while having a moderate number of participants.
Next, some doubts before building a HAR system arose: how much data is required to train a HAR classifier? Which model will perform best? Which data source – smartphone, smartwatch, or both – will provide the best results? Few studies analyzed how the amount of data impacts classification performance, but none focused on smartphone and smartwatch data. On the other hand, while comparisons of models and data sources were common in the literature, conclusions differed from one article to another. Therefore, an extensive analysis combining these dimensions was carried out to provide an answer to these questions.
Following these analyses, the lessons learned were employed to put HAR into practice by implementing a real-life mHealth system to compute the results of a mobility test automatically. The system was designed to work on consumer Android smartphones and Wear OS smartwatches, worn in their natural position, and evaluated on a cohort of \(30\) participants.The results were similar to or even better than other state-of-the-art proposals which used specialised devices or positioned (multiple) data collection devices in unnatural positions (e.g., a smartphone on the head). This proved the feasibility of using smartphones and smartwatches as sensing devices for HAR, real-life mHealth systems.
Finally, the dissertation migrates from inertial-based to Wi-Fi CSI-based HAR, a recent trend that employs the channel status from a Wi-Fi infrastructure and promises recognition of human activities – among other applications – without other sensing devices. In this case, we used an ESP32 microcontroller to obtain the Wi-Fi CSI data and observed through several experiments that the collected data changes over time. Although very good results for HAR can be obtained when limiting the experiments to a determined time frame, this instability over time makes it difficult to transfer CSI-based systems to real-life scenarios.
In conclusion, the path followed during this dissertation constitutes a comprehensive research that ranges from the development of tools, data collection, analysis, development and evaluation of HAR systems, while focusing on their applicability in real life.
Accomplishment of objectives
In Introduction, the objectives of this research were defined. Next, a summary is provided for each goal indicating how it was fulfilled.
RO1: Develop software tools for reliable data collection in Android smartphones and Wear OS smartwatches
Data collection libraries described the Background Sensors, WearOS Sensors and NativeScript WearOS Sensors libraries, which grant access to some specific sensors in Android smartphones and smartwatches. The libraries were implemented following the guidelines presented by González-Pérez et al. (2022), which proved to achieve a reliable task execution (i.e., data collection) with less than \(1\%\) of missing data.
These libraries were developed and tested to fully support smartphones and smartwatches running up to Android 13 and Wear OS 4, respectively. Although these OS versions are not the latest, significant technical changes have not been introduced in most recent versions – Android 14 and 15 and Wear OS 51 – that would affect their operation.
1 At the time of writing this dissertation, Android 15 is in Beta phase and Wear OS 5 is in Developer Previews (i.e., features and technical requirements might still change in the release version).
The libraries were made available as open-source software. In addition, the libraries were integrated into the AwarNS Framework, a modular framework to ease the development of context-aware applications (González-Pérez et al. 2023).
RO2: Collect HAR dataset of smartphone and smartwatch data from heterogeneous subjects
Smartphone and smartwatch HAR dataset described a dataset containing accelerometer and gyroscope samples from a smartphone and a smartwatch associated with five human activities. The dataset was collected focusing on the heterogeneity and diversity of the participants, resulting in a dataset with these characteristics: the widest age range of its subjects and a balance of \(56\%\)/\(44\%\) male/female participants. However, when compared with related datasets, the collected dataset was limited in terms of the number of activities and participants.
RO3: Analyse and compare the effect of the amount of training data, the data source and the model architecture on the performance of ML- and DL-based HAR
Multidimensional analysis of ML and DL on HAR presented an analysis exploring how the classification performance on four ML and DL models and three data sources were affected by the amount of data used for training them. Therefore, a three-way statistical analysis considering the amount of data, type of model and data source was executed.
Results unveiled different patterns in how the performance of classification models evolves depending on the amount of training data and the data source employed. Regarding data sources, the smartwatch data provided the best results when the amount of available data was low, while the fusion of smartphone and smartwatch data generally obtained the best results as the amount of data increased. When focusing on the models, the CNN showed the best results in almost any circumstance, making it the clear choice for HAR systems.
RO4: Develop and evaluate a mHealth system using HAR with smartphones and smartwatches
HAR in mHealth: TUG test using smartphones and smartwatches a mHealth system to automate the execution of the TUG test was implemented and analytically validated. The system, composed of two applications, employed the inertial data from a consumer smartphone or smartwatch to automatically compute the test’s results.
Both systems were validated on \(30\) test subjects and compared, showing an average error of \(44\) ms and \(51\) ms when computing the total duration of the test with the smartwatch and smartphone inertial data, respectively. In general, results indicated that the system performed better with the smartwatch data than with the smartphone data.
Compared with other proposals in the literature, the developed system improved the Bland-Altman results of other state-of-the-art solutions, although it presented lower Intraclass Correlation Coefficient results than other approaches. Notwithstanding, the evaluation outcomes showed that HAR with consumer smartphones and smartwatches can successfully be used for real-life mHealth applications, which typically require a high degree of reliability.
RO5: Analyse the feasibility of Wi-Fi CSI data for real-life HAR systems
Looking into the future: Wi-Fi CSI based HAR explored the usage of the Wi-Fi CSI applied to HAR, while also taking advantage of its reported versatility adding a location dimension to the target activities.
A first experiment showed good results when evaluating the classification accuracy of a CNN in a limited timeframe, which however downgraded when evaluating it using data from a different time span than the one used for training it. Subsequent experiments indicated a clear similarity between the CSI data of adjacent days while being different from other days. These results suggested the instability of the CSI data, thus hampering its usage for real-life applications.
However, there is still work to be done since the observed instability might have been produced by hardware or software limitations in the employed devices or by uncontrollable interferences in the environment.
Contributions
Several research outcomes were produced during the development of this thesis. These outcomes are listed below and classified as journal or conference contributions, datasets, reproducibility packages, software.
Journals
- “Dataset of inertial measurements of smartphones and smartwatches for human activity recognition”. In: Data in Brief (Matey-Sanz, Casteleyn, and Granell 2023a).
- “Implementing and evaluating the Timed Up and Go test automation using smartphones and smartwatches”. In: IEEE Journal of Biomedical and Health Informatics [Submitted. Under second review round] (Matey-Sanz, González-Pérez, et al. 2024).
Conferences
- “Instrumented Timed Up and Go Test Using Inertial Sensors from Consumer Wearable Devices”. In: International Conference on Artificial Intelligence in Medicine (Matey-Sanz et al. 2022).
- “Analysis and Impact of Training Set Size in Cross-Subject Human Activity Recognition”. In: 26th Iberoamerican Congress on Pattern Recognition (Matey-Sanz, Torres-Sospedra, et al. 2024).
- “Temporal Stability on Human Activity Recognition based on Wi-Fi CSI”. In: 13th International Conference on Indoor Positioning and Indoor Navigation (Matey-Sanz, Torres-Sospedra, and Moreira 2023).
Datasets
- “Smartphone and smartwatch inertial measurements from heterogeneous subjects for human activity recognition”. In: Zenodo (Matey-Sanz, Casteleyn, and Granell 2023b).
Reproducibility packages
- Code and data resources for “Instrumented Timed Up and Go Test Using Inertial Sensors from Consumer Wearable Devices”. In Zenodo (Matey-Sanz 2022).
- Reproducible package for “Analysis and Impact of Training Set Size in Cross-Subject Human Activity Recognition”. In Zenodo (Matey-Sanz 2023b).
- Reproducible package for “Temporal Stability on Human Activity Recognition based on Wi-Fi CSI”. In Zenodo (Matey-Sanz 2023a).
Softwate
- Background Sensors [v1.3.0] (Matey-Sanz, Casteleyn, and Granell 2024a)
- WearOS Sensors [v1.2.1] (Matey-Sanz, Casteleyn, and Granell 2024c)
- NativeScript WearOS Sensors [v1.3.0] (Matey-Sanz, Casteleyn, and Granell 2024b)
- Tug Test Smartphone Application [v2.0.0] (Matey-Sanz and González-Pérez 2022a)
- Tug Test Smartwatch Application [v2.0.0] (Matey-Sanz and González-Pérez 2022b)