The Onnela Lab at the Harvard T.H. Chan School of Public Health has developed the Forest library to analyze smartphone-based high-throughput digital phenotyping data. The main intellectual challenge in smartphone-based digital phenotyping has moved from data collection to data analysis. Our research focuses on the development of mathematical and statistical methods for analyzing intensive high-dimensional data. We are actively developing the Forest library for analyzing smartphone-based high-throughput digital phenotyping data collected with the Beiwe platform. Forest will implement our methods for analyzing Beiwe data as a Python package and is released under the BSD-3 open-source license. The Forest library will continue to grow over the coming years as we develop new analytical methods.

Forest can be run locally but is also integrated into the Beiwe back-end on AWS, consistent with the preferred big-data computing paradigm of moving computation to the data. Integrated with Beiwe, Forest can be used to generate on-demand analytics, most importantly daily or hourly summary statistics of collected data, which are stored in a relational database on AWS. The system also implements an API for Tableau, which supports the creation of customizable workbooks and dashboards to view data summaries and troubleshoot any issues with data collection. Tableau is commercial software but is available under free viewer licenses and may be free to academic users for the first year (see Tableau for more information).

For more detailed info on specific subpackages, see our Documentation.

Description

Description of how beiwe data looks (folder structure + on/off cycles)

Input: typically raw data from smartphones Output: typically summary files

• Creating synthetic data
  • Want to try out our methods, but don't have smartphone data at hand? Use bonsai
• Data preparation
  • Identifying time zones and unit conversion: use poplar
  • Collate Beiwe survey data into .csvs per participant or per study: use sycamore
• Data imputation
  • State-of-the-art GPS imputation: use jasmine
• Data summarizing (see tables below for summary metrics)
  • Mobility metrics from GPS data: use jasmine
  • Daily summaries of call & text metadata: use willow
  • Survey completion time from survey metadata: use sycamore

Usage

Please note that Forest is tested only against Python 3.11 and 3.12 (preferred). To install: console pip install beiwe-forest

Alternatively, install directly from GitHub with pip. As the repo is public, it won't prompt you to login. If you've used forest in the past, it might be prudent to do a pip uninstall forest first.

console pip install git+https://github.com/onnela-lab/forest

To immediately test out forest, adapt the filepaths in the code below and run:

```python

Currently, all imports from forest must be explicit. For the below example you need to import the following

In the future, it would be great to have all functions import automatically

import datetime

from forest.bonsai.simulatelogdata import simlogdata from forest.bonsai.simulategpsdata import simgpsdata, gpstocsv from forest.jasmine.traj2stats import Frequency, gpsstatsmain from forest.willow.logstats import logstats_main

1. If you don't have any smartphone data (yet) you can generate fake data

pathtosyntheticgpsdata = "ENTER/PATH1/HERE" pathtosyntheticlogdata = "ENTER/PATH2/HERE" pathtogpssummary = "ENTER/PATH/TO/DESIRED/OUTPUT/FOLDER1/HERE" pathtologsummary = "ENTER/PATH/TO/DESIRED/OUTPUT/FOLDER2/HERE"

Generate fake call and text logs

Because of the explicit imports, you don't have to precede the functions with forest.subpackage.

simlogdata(pathtosyntheticlogdata)

Generate synthetic gps data and communication logs data as csv files

Define parameters for generating the data

To save smartphone battery power, we typically collect location data intermittently: e.g. during an on-cycle of 3 minutes, followed by an off-cycle of 12 minutes. We'll generate data in this way

number of persons to generate

n_persons = 1

location of person to generate format: Country2letterISOcode/City_Name

location = "GB/Bristol"

start date of generated trajectories

start_date = datetime.date(2021, 10, 1)

end date of trajectories

end_date = datetime.date(2021, 10, 5)

api key for openroute service, generated from https://openrouteservice.org/

apikey = "mockapi_key"

Length of off-cycle + length of on-cycle in minutes

cycle = 15

Length off-cycle / (length off-cycle + length on-cycle)

percentage = 0.8

dictionary of personal attributes for each user, set to None if random, check Attributes class for usage in simulategpsdata module.

personalattributes = { "User 1": { "mainemployment": "none", "vehicle" : "car", "travellingstatus": 10, "activestatus": 7 },

"Users 2-4":
{
    "main_employment": "university",
    "vehicle" : "bicycle",
    "travelling_status": 8,
    "active_status": 8,
    "active_status-16": 2 
},

"User 5":
{
    "main_employment": "office",
    "vehicle" : "foot",
    "travelling_status": 9,
    "travelling_status-20": 1,
    "preferred_exits": ["cafe", "bar", "cinema"] 
}

} samplegpsdata = simgpsdata(npersons, location, startdate, enddate, cycle, percentage, apikey, personal_attributes)

save data in format of csv files

gpstocsv(samplegpsdata, pathtosyntheticgpsdata, startdate, enddate)

2. Specify parameters for imputation

See https://forest.beiwe.org/en/latest/jasmine.html for details

time zone where the study took place (assumes that all participants were always in this time zone)

tz_str = "Etc/GMT-1"

Generate summary metrics e.g. Frequency.HOURLY, Frequency.DAILY or Frequency.HOURLYANDDAILY (see Frequency class in constants.py)

frequency = Frequency.DAILY

Save imputed trajectories?

save_traj = False

Hyperparameters class for imputation (default leave None), from forest.jasmine.traj2stats import Hyperparameters

parameters = None

list of locations to track if visited, leave None if don't want these summary statistics

placesofinterest = ['cafe', 'bar', 'hospital']

list of OpenStreetMap tags to use for identifying locations, leave None to default to amenity and leisure tagged locations or if you don't want to use OSM (see OSMTags class in constants.py)

osm_tags = None

3. Impute location data and generate mobility summary metrics using the simulated data above

gpsstatsmain( studyfolder = pathtosyntheticgpsdata, outputfolder = pathtogpssummary, tzstr = tzstr, frequency = frequency, savetraj = savetraj, parameters = parameters, placesofinterest = placesofinterest, osmtags = osm_tags, )

4. Generate daily summary metrics for call/text logs

option = Frequency.DAILY timestart = None timeend = None participant_ids = None

logstatsmain(pathtosyntheticlogdata, pathtologsummary, tzstr, option, timestart, timeend, participant_ids) ```

More info

• Beiwe platform for smartphone data collection
• Onnela lab

Publications

• Straczkiewicz, M., Huang, E.J., and Onnela, JP. A “one-size-fits-most” walking recognition method for smartphones, smartwatches, and wearable accelerometers. npj Digit. Med. 6, 29 (2023) Open Access
• Huang E, Yan K, and Onnela JP. Smartphone-Based Activity Recognition Using Multistream Movelets Combining Accelerometer and Gyroscope Data. Sensors 22 (7), 2618 (2022)
• Onnela JP, Dixon C, Griffin K, Jaenicke T, Minowada L, Esterkin S, Siu A, Zagorsky J, and Jones E. Beiwe: A data collection platform for high-throughput digital phenotyping. Journal of Open Source Software, 6(68), 3417 (2021)
• Liu G and Onnela JP. Bidirectional imputation of spatial GPS trajectories with missingness using sparse online Gaussian Process. Journal of the American Medical Informatics Association 28(8), 1777 (2021)
• Barnett I and Onnela JP. Inferring mobility measures from GPS with missing data. Biostatistics 21:2, e98, 2020 Open Access
• Huang E and Onnela JP. Augmented movelet method for activity classification using smartphone gyroscope and accelerometer data. Sensors 20(13), 3706, 2020