Official Github repository for the paper "Accessible, At-Home Detection of Parkinson’s Disease via Multi-Task Video Analysis". The paper has been accepted at AAAI 2025 and the published version will appear soon. Until then, please refer to the arxiv version: https://arxiv.org/abs/2406.14856
Note: We are unable to share the raw patient videos to protect their identity and privacy. In this repository, we share the extracted features for the three tasks we experimented in our paper. If you need the frame-by-frame hand key-points, face mesh (facial key-points extracted with MediaPipe), facial action units (extracted using OpenFace), or raw audio files for the ``quick brown fox'' utterance, please reach out to us at [email protected]. Requests will be evaluated case-by-case and we will be able to share those detailed data if the purpose of data use aligns with our IRB protocol.
To access the extracted features, please go to /data folder in this repository.
- Metadata contains demographic information about the user, unique Participant_ID, and whether the subject has Parkinson's disease (pd column: yes indicates the participant has PD, no indicates Non-PD)
- List of all participants IDs contains the list of unique ids of all participants. They are later divided into Validation and Test sets. All other participants are considered part of the training set (IDs are not explicitly listed).
- For conformal prediction variants, a part of the training set participants is reserved as Calibration set. None of the Validation, Test, and Calibration should be used for model training where conformal prediction or Platt scaling is involved.
- We also include extracted features from YoutubePD [1] dataset videos. As in the original paper, test participants are separated, and not used for training the models.
- Task-specific features can be accessed using the following links: Finger-tapping, Smile, and Speech.
- Zhou, Andy, et al. "YouTubePD: A Multimodal Benchmark for Parkinson’s Disease Analysis." Advances in Neural Information Processing Systems 36 (2024).
Note: The feature extraction codes will be added to the repository soon (please email us if you need it earlier). Currently, the repository contains code for training and running the task-specific models and UFNet, the baselines we experimented, and the ablation studies. We also provide code for demographic information analysis.
-
To run the UFNet model, you first need to run and save the task-specific models. Please make sure the hyper-parameters are the same as we report in our paper (the arxiv paper has the detailed hyper-parameters for both task-specific models and the UFNet: https://arxiv.org/abs/2406.14856).
-
Next, run UFNet without prediction withholding or UFNet with withholding uncertain predictions. Also, you can train/run UFNet with different multi-task combinations by adjusting the hyper-parameters accordingly.
Re-training the Models on a New Dataset
If you have collected video data of similar tasks from PD patients and healthy individuals, you can use our codebase to extend our dataset and re-train the models.
- From the new videos, extract task-specific features (feature extraction codes will be added soon). Merge these new feature datasets with our previous datasets. Update the validation and test ids if you want some of your new data to be included for model selection and evaluation.
- Re-train the task-specific models (you can perform a hyper-param search as mentioned in the detailed version of our paper).
- Re-train and evaluate the UFNet model with the new datasets.
If this repository is relevant and helpful for your research, please cite our paper:
@article{islam2024accessible,
title={Accessible, At-Home Detection of Parkinson's Disease via Multi-task Video Analysis},
author={Islam, Md Saiful and Adnan, Tariq and Freyberg, Jan and Lee, Sangwu and Abdelkader, Abdelrahman and Pawlik, Meghan and Schwartz, Cathe and Jaffe, Karen and Schneider, Ruth B and Dorsey, E and others},
journal={arXiv preprint arXiv:2406.14856},
year={2024}
}