Introduction

The research in this dissertation is centred on the Human Activity Recognition topic, whose main objective is to characterize the activities performed by human subjects automatically through the analysis of digital footprints from the use of sensing devices. This research field has had a large trajectory and has a promising future due to its breadth of possibilities to create novel applications and room for improvement due to the advent of new sensing technology and devices.

Throughout this thesis, we have explored two sensing technologies, different devices and types of activities, in the realm of the two fundamental pillars captured in the dissertation’s title: the usage of consumer devices and the applicability of HAR to real-life scenarios. Instead of using complex networks of sensing devices scattered around the body or having expensive, purpose-built devices, common consumer devices such as smartphones and smartwatches – owned by more than \(7\) billion and \(200\) million users worldwide, respectively – or cheap devices such as ESP32 microcontrollers – around \(15\)€– are used as data sources. In addition to this, the feasibility of applying the research to real-life settings is a paramount aspect in this dissertation: the aim is that the research done can translate into benefits for vested stakeholders in particular and for society in general. Therefore, the analyses and experiments contained in this dissertation have been designed to evaluate the feasibility of consumer-based HAR systems in real-life conditions.

Motivation

HAR is a research area aiming to determine human behaviour, actions, and activities from physical signals extracted from sensing devices (Gupta et al. 2022). Historically, HAR has had two main approaches regarding sensing devices: vision- and inertial-based sensors (Minh Dang et al. 2020). More recently, a breakthrough approach was proposed: monitoring the state of the channels of Wi-Fi networks (i.e., Wi-Fi CSI), which can be affected by the presence and movement of individuals, among other factors (Ma, Zhou, and Wang 2019). In this thesis, we focus on inertial- and Wi-Fi CSI-based HAR.

Inertial-based HAR

The mainstream availability of smartphones and smartwatches equipped with IMUs offers an inexpensive, ubiquitous, and reliable way to develop HAR systems that can be employed in real-life scenarios. Due to the potential of these systems, some challenges and opportunities were identified:

  • Modern Android smartphones and Wear OS smartwatches suffer from energy restrictions that can affect the data collection. The guidelines proposed by González-Pérez et al. (2022) overcome these restrictions, showing only a \(1\%\) of missing data in their experiments. Therefore, providing data collection libraries implementing these guidelines might be useful for the HAR and other communities that need reliable data collection tools.
  • Only five public datasets with smartphone and smartwatch data for HAR exist, where four of them used the devices in their common placement (carried in a pocket and the wrist, respectively). From those, RealWorld (Sztyler and Stuckenschmidt 2016), ExtraSensory (Vaizman, Ellis, and Lanckriet 2017) and DOMINO (Arrotta et al. 2023) include data from diverse people in terms of age and gender, traits that might help build generalizable HAR systems. Therefore, the HAR research community would benefit from more public and comprehensive datasets that adequately satisfy these aspects.
  • There are no works exploring how the accuracy of ML and DL evolves regarding the amount of training data in smartphone- and smartwatch-based HAR. In addition, existing research presents contradictory results in terms of data sources and model comparisons. Therefore, analysing how ML and DL models evolve regarding the quantity of training data while also comparing which data source and model architecture perform best could provide valuable insights for the research community.
  • Most of the existing real-life smartphone- and smartwatch-HAR-based systems focus on mHealth applications. However, the evaluation of those systems is still lagging behind in validating them for real-life (with real users) usage. Therefore, developing a mHealth system to solve a real-life problem with its corresponding validation following Goldsack et al. (2020) V3 framework – as far as possible – would be a great research output.

Wi-FI CSI-based HAR

Regarding the new trend in Wi-Fi CSI-based HAR, we observed that few works employed ESP32 microcontrollers and that one of them pointed out a possible instability over time in the collected CSI data that could undermine the performance of HAR systems. Therefore, more research on this topic would be useful to shed light on the mentioned issue and determine its viability in real-life scenarios.

Research objectives

From the identified challenges and opportunities, the following research objectives were defined and will be addressed in successive chapters:

  • RO1: Provide software tools for reliable data collection in Android smartphones and Wear OS smartwatches. Create software libraries enabling the development of smartphone and smartwatch applications for data collection. Make these libraries available as open-source software solutions to the research community, serving as enablers for future HAR research.
  • RO2: Collect HAR dataset of smartphone and smartwatch data from heterogeneous subjects. Generate a public dataset composed of IMU data from a smartphone and smartwatch carefully selecting participants to preserve age diversity and gender balance.
  • RO3: Analyse and compare the effect of the amount of training data, the data source and the model architecture on the performance of ML- and DL-based HAR. Determine how the accuracy of several model architectures evolves regarding the amount of data used for training them and the nature of the data employed. Provide insights that might guide researchers when choosing the amount of data they need to collect, the most suitable data source or the most appropriate model for their HAR systems.
  • RO4: Develop and evaluate an mHealth system using HAR with smartphones and smartwatches. Create an application for off-the-shelve smartphones and smartwatches capable of automating the computation of results of a well-known mobility test and validating it for real-life usage.
  • RO5: Analyse the feasibility of Wi-Fi CSI data for real-life HAR systems. Design experiments to evaluate the use of CSI data collected with ESP32 microcontrollers taking into account the stability of the data over time.

As depicted in Figure 2.1, these research objectives constitute the path of the thesis, which focuses on inertial-based HAR with the development of data collection tools, the collection of a dataset, a study of the factors affecting the performance of ML and DL models, and the development and evaluation of an mHealth application focusing on inertial-based HAR, to finish with a feasibility study of the usage of the Wi-Fi CSI data for HAR.

Thesis structure

The rest of the dissertation is structured as follows: