Skip to content

Commit ddc63b5

Browse files
pritamqupritamqu
committed
dataset release
1 parent 2283690 commit ddc63b5

18 files changed

+1694
-43
lines changed

‎.gitignore

+1-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
__pycache__/
22
.spyproject/
33
temp*
4-
zip_unzip_files.py
5-
codes/
4+
zip_unzip_files.py

‎LICENSE

-1
This file was deleted.

‎LICENSE.html

+136
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<body>
4+
5+
<h3>Database License Agreement</h3>
6+
<br>
7+
By accepting the terms of this document, the User (defined as the person named in the request for access and who has accepted the terms of this Database License Agreement), who will make use of the database or the database interface created by Pritam Sarkar and Ali Etemad (“Licensors”), agrees to the following:
8+
<br>
9+
10+
<ol>
11+
<li>
12+
<strong>Licensed Database</strong>
13+
<br>
14+
The Licensed Database means the database (including both the actual data as well as the interface to the database), or other material to which the User applied this Database License Agreement (DLA) entitled “A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work” and made available through the Queen’s University Dataverse portal.
15+
</li>
16+
<br>
17+
18+
<li>
19+
<strong>Scope of the License</strong>
20+
<br>
21+
<ol type=a>
22+
<li> <strong>Academic research </strong>
23+
<br>
24+
The User is granted a worldwide, royalty-free, non-sublicensable, non-exclusive, non-transferable, and non-sublicensable license to use the Licensed Database only for academic research.
25+
</li>
26+
<li> <strong>Moral Rights</strong>
27+
<br>
28+
Moral rights, such as the right of integrity, are not licensed under this Database License Agreement, nor are publicity, privacy, and/or other similar personality rights. Licensors to the extent possible, waive and/or agree not to assert any such rights held by the Licensors to the limited extent necessary to allow User, solely to exercise the rights licensed hereunder.
29+
</li>
30+
<li> <strong>Prohibited Use</strong>
31+
<br>
32+
The User acknowledges that the Licensed Database contains personal information of individuals that could be used to identify those individuals, either alone or in conjunction with other data. The User shall not use the contents of the Licenses Database in any research where the objective is to directly or indirectly identify the persons whose personal information may be contained therein. Any such use will be considered a breach of this Database License Agreement and will result in its termination with immediate effect in accordance with section 8(b) below.
33+
</li>
34+
<li><strong>Commercial use</strong>
35+
<br>
36+
The User may not use the Licensed Database for any non-academic purpose. The Licensed Database may not be used for any illegal or unlawful purposes whether commercial or not. Non-academic purposes include, but are not limited to:
37+
<br>
38+
<ul>
39+
<li> proving the efficiency of commercial systems
40+
<li> training or testing of commercial systems
41+
<li> selling data from the dataset creating military applications
42+
<li> developing governmental systems used in public spaces
43+
</ul>
44+
Where the User wishes to use the Licensed Database for non-academic purposes, such use must be under a separate agreement negotiated in good faith between the parties.
45+
</li>
46+
</ol>
47+
48+
<br>
49+
<li> <strong>Responsibility</strong>
50+
<br>
51+
This Database License Agreement must be accepted by a person with a permanent position at an academic institute.
52+
</li>
53+
54+
<br>
55+
<li> <strong>Distribution</strong>
56+
<br>
57+
The User may not grant anyone access to the Licensed Database by giving out their username and password. The User may not distribute the Licensed Database or portions thereof in any way, with the exception of using small portions of data for the exclusive purpose of clarifying academic publications or presentations. Note that publications will have to comply with the terms stated in Section 6.
58+
</li>
59+
60+
61+
<br>
62+
<li> <strong>Access</strong>
63+
<br>
64+
The User may only use the Licensed Database after the User has accepted the terms of this Database License Agreement (DLA).
65+
</li>
66+
67+
<br>
68+
<li> <strong>Publications</strong>
69+
<br>
70+
Publications include not only journal articles, but also presentations for conferences or educational purposes or other written disseminations, including pre-print repositories (e.g. arXiv), arising from research using the Licensed Database.
71+
<br>
72+
All publications that report on research that use any of the Licensed Database must cite the following paper as the source of the Licensed Database must cite the following paper as the source of the Licensed Database.
73+
<br><br>
74+
Sarkar, P., Posen, A. and Etemad, A., 2022. <i>AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work</i>. <i>arXiv preprint arXiv:2205.06887</i>.
75+
<br>
76+
<p>
77+
BibTeX entry:<br>
78+
<code>
79+
@misc{sarkar2022avcaffe,
80+
title={AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work},
81+
author={Pritam Sarkar and Aaron Posen and Ali Etemad},
82+
year={2022},
83+
eprint={2205.06887},
84+
archivePrefix={arXiv},
85+
primaryClass={cs.HC}}
86+
</code></p>
87+
</li>
88+
89+
90+
<br>
91+
<li> <strong>No Warranty</strong>
92+
<br>
93+
<strong>
94+
The Licensed Database is provided as-is and as-available, and without any representations or warranties of any kind, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable.
95+
<br>
96+
In no event will the Licensors be liable to the User on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Database License Agreement or use of the Licensed Database, even if the Licensors have been advised of the possibility of such losses, costs, expenses, or damages. User assumes all risk and any liability arising from its use of the Licensed Database.
97+
</strong>
98+
</li>
99+
100+
101+
102+
103+
<br>
104+
<li> <strong>Term and Termination</strong>
105+
<br>
106+
<br>
107+
<ol type=a>
108+
<li> Unless otherwise terminated by operation of law or by acts of the parties in accordance with the terms of this DLA, this DLA is in force from the date the terms are accepted by the User and remains in effect until the Licensed Database is no longer subject to a prohibition on its use without the approval the Licensors. </li>
109+
<li> If the User fails to comply with this DLA, then the rights granted hereunder will terminate automatically. </li>
110+
<li> Where the right to use the Licensed Database terminated under section 8(a), it reinstates:
111+
<ol type=i>
112+
<li> automatically as of the date the violation is cured, provided it is cured within 30 days of the User’s discovery of the violation; or </li>
113+
<li> upon express reinstatement by the Licensors. </li>
114+
</ol>
115+
<li>Section 8(c) does not affect any rights the Licensors may have to seek remedies for the User’s violation of this DLA.</li>
116+
<li>Sections 1, 6, 7, 8, and 9 survive termination of this DLA. </li>
117+
118+
</ol>
119+
</li>
120+
121+
122+
<br>
123+
<li> <strong>Interpretation</strong>
124+
<br>
125+
This DLA does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Database that could lawfully be made without permission under this DLA.
126+
<br>
127+
If any provision of this DLA is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this DLA without affecting the enforceability of the remaining terms and conditions.
128+
</li>
129+
130+
</ol>
131+
132+
133+
134+
135+
</body>
136+
</html>

‎README.md

100644100755
+172-32
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,172 @@
1-
<h1 align="center">
2-
AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work
3-
</h1>
4-
5-
<h3 align="center">
6-
Under review.
7-
</h3>
8-
9-
<h3 align="center">
10-
<a href="https://www.pritamsarkar.com">Pritam Sarkar</a>
11-
&nbsp; Aaron Posen
12-
&nbsp; Ali Etemad
13-
</h3>
14-
15-
<h3 align="center">
16-
<a href="https://arxiv.org/pdf/2205.06887.pdf">[Paper]</a> <a href="https://github.com/pritamqu/AVCAffe"> [Repository]</a> <a href="https://pritamqu.github.io/AVCAffe/"> [Project Page]</a>
17-
</h3>
18-
19-
20-
This is the official code repository of AVCAffe. Please check the [project website](https://pritamqu.github.io/AVCAffe/) for additional details.
21-
22-
23-
<!-- ### Items available -->
24-
### TODO List
25-
- [x] [Paper](https://arxiv.org/pdf/2205.06887.pdf)
26-
- [ ] Database release
27-
- [ ] Database license agreement
28-
- [ ] Instruction to download
29-
- [ ] Dataloader code
30-
31-
32-
Coming soon...
1+
<h1 align="center">
2+
AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work
3+
</h1>
4+
5+
<h3 align="center">
6+
AAAI 2023.
7+
</h3>
8+
9+
<h3 align="center">
10+
<a href="https://www.pritamsarkar.com">Pritam Sarkar</a>
11+
&nbsp; <a href="">Aaron Posen</a>
12+
&nbsp; <a href="">Ali Etemad</a>
13+
</h3>
14+
15+
<h3 align="center">
16+
<a href="https://arxiv.org/pdf/2205.06887.pdf">[Paper]</a> <!-- change with aaai link -->
17+
<a href="./docs/assets/files/avcaffe_supp.pdf"> [Appendix]</a>
18+
<a href="https://arxiv.org/pdf/2205.06887.pdf"> [ArXiv]</a>
19+
<a href="https://github.com/pritamqu/AVCAffe"> [Code]</a> <a href="https://pritamqu.github.io/AVCAffe/"> [Website]</a>
20+
</h3>
21+
22+
23+
This is the official code repository of AVCAffe. Please check the project website: https://pritamqu.github.io/AVCAffe/ for additional details. Before you download and/or use the AVCAffe Dataset, please make sure you have read the Database License Agreement available here: [DLA](./LICENSE.html).
24+
25+
<!-- ### Items available -->
26+
### Updates
27+
- [x] Paper
28+
- [x] Database license agreement
29+
- [x] Instruction to download
30+
- [x] Dataloader code
31+
- [ ] Database release
32+
- [x] Audio-visual recordings, ground truths, and additional meta data
33+
- [ ] Face-crops (coming soon, pending due to some technical issues)
34+
35+
36+
37+
### Overview
38+
39+
AVCAffe is hosted in [borealisdata.ca](https://borealisdata.ca/) under [Queen's University Dataverse](https://borealisdata.ca/dataverse/queens). Please follow the instructions below to download the dataset.
40+
41+
The directory structure of the dataset is as follows.
42+
43+
```
44+
├── .. # total size = 34.7 GB
45+
├── ground_truths # size = 192 KB
46+
├── info
47+
├── videos # size = 552 KB
48+
| │ ├── per_participant_per_task # size = 13.9 GB
49+
│ │ ├── aiim001
50+
│ │ ├── aiim002
51+
│ │ ├── ...
52+
│ │ ├── ...
53+
│ │ └── aiim108
54+
│ └── shorter_segments # size = 20.3 GB
55+
│ ├── aiim001
56+
│ ├── aiim002
57+
│ ├── ...
58+
│ ├── ...
59+
│ └── aiim108
60+
└── images
61+
└── shorter_segments_face # size = 41.7 GB
62+
├── aiim001
63+
├── aiim002
64+
├── ...
65+
├── ...
66+
└── aiim108
67+
68+
```
69+
70+
- `ground_truths` contains the self-reported ground-truths for affect and cognitive load.
71+
- `info` contains additional meta data, e.g., train-val split, pre-study responses, etc. Please find details below.
72+
- `videos/per_participant_per_task` contains full length videos of each participant per each task. Video length of 2.5-10 minutes, resolution of 640x360 pixel, format `.mp4`.
73+
- `videos/shorter_segments` contains segmented clips of the same videos in `per_participant_per_task`. Video length approximately 6 seconds, resized the shorter side at 256 pixel, format `.avi`. Note, the shorter clips are prepared for easy and efficient to use to train deep learning models.
74+
- `images/shorter_segments_face` contains the face crops of the participants. To train the baseline models we use just the face-crops instead of the full frames, which works better (at-least for simple models).
75+
76+
### How to Request Access?
77+
78+
79+
**Step 1:**
80+
To access this dataset you must have an account in the Dataverse using an institutional email address.
81+
Please go to this link to create an account: https://borealisdata.ca/loginpage.xhtml.
82+
83+
**Step 2:**
84+
Once you have an account, and signed in on the same browser simply click on this link https://borealisdata.ca/dataverseuser.xhtml?selectTab=apiTokenTab, it will show the your `API Token`. Alternatively, you can click on the drop down list shown under your name (top right corner), and click `API Token`. Please note the `API Token` which will be required at future step.
85+
86+
**Step 3:**
87+
Once the account is created please go to this link https://borealisdata.ca/dataset.xhtml?persistentId=doi:10.5683/SP3/PSWY62. **Select all 1,918 files in this dataset** and **Request Access**. Please read the **Terms of Use** and **Terms of Access** and **Accecpt** to submit a request. Please see the screenshot below.
88+
89+
![step 3](./docs/assets/images/request_access.png)
90+
91+
**Step 4:**
92+
Your request to access this dataset is successfully submitted. It may take 5-7 days to approve the request, if your access is not granted by that time, please write us an email at [email protected] and/or [email protected]. Once your request is approved please proceed to the next step.
93+
94+
### How to Download?
95+
96+
**Step 1:**
97+
We provide a script to download the dataset. Create a directory where you want to download the dataset and go to that directory. Please run the following commands from a terminal `mkdir avcaffe` and `cd avcaffe`.
98+
99+
**Step 2:**
100+
Please run the following command to clone the github repository in your current location.
101+
```
102+
git clone https://github.com/pritamqu/AVCAffe.git
103+
cd AVCAffe
104+
```
105+
106+
**Step 3:**
107+
Please open the `codes/downloader/downloader.py` using a text editor. Please update the API_TOKEN variable with your `API Token` noted earlier.
108+
109+
**Step 4:**
110+
Next, please ensure you have the required packages installed or you can install them by running
111+
```
112+
pip install -r codes/downloader/requirements.txt
113+
```
114+
115+
**Step 5:**
116+
You can download the entire dataset simply running the following command:
117+
```
118+
python codes/downloader/downloader.py
119+
```
120+
121+
**Step 6:**
122+
Congratualations! The dataset is downloaded.
123+
124+
125+
### Dataloader
126+
127+
We provide supporting codes for easy access of the dataset. A dataloader written in PyTorch is available in `avcaffe/codes/dataloader/`. A minimum example usage is shown below and details usage is presented [here](./codes/dataloader/README.MD):
128+
129+
```
130+
db = AVCAffe(ROOT,
131+
subset='train',
132+
return_video=True,
133+
video_clip_duration=2,
134+
video_fps=16.,
135+
return_audio=True,
136+
audio_clip_duration=2,
137+
audio_fps=16000,
138+
return_labels=True,
139+
class_name='mental_demand',
140+
mode='clip',
141+
clips_per_video=1,
142+
)
143+
144+
sample = db.__getitem__(1)
145+
146+
```
147+
148+
### Additional Details
149+
150+
We list some of the additional details here:
151+
- participants used for training and validation splits are mentioned in `info/train.txt` and `info/val.txt` respectively.
152+
- the outcomes of prestudy questionnaire are available in `info/prestudy_response.csv`.
153+
- the list of participants who agree to use their faces/images/videos for article or accopanied media contents, are available in `info/public_face_ids.txt`.
154+
- some of the clips available in `shorter_segments` have no speech when the participant were listening to the other participant or thinking or trying to solve the tasks, the file ids of such clips are available in `info/no_audio_files.txt`.
155+
156+
157+
### Citation
158+
If you find this repository useful, please consider giving a star :star: and citation using the given BibTeX entry:
159+
```
160+
@misc{sarkar2022avcaffe,
161+
title={AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work},
162+
author={Pritam Sarkar and Aaron Posen and Ali Etemad},
163+
year={2022},
164+
eprint={2205.06887},
165+
archivePrefix={arXiv},
166+
primaryClass={cs.HC}}
167+
```
168+
169+
### Question
170+
You may directly contact me at <[email protected]> or connect with me on [LinkedIn](https://www.linkedin.com/in/sarkarpritam/).
171+
172+

‎codes/dataloader/README.MD

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# AVCAffe Dataloader
2+
3+
We provide 2 dataloaders available in `avcaffe_vid.py` and in `avcaffe_img.py`.
4+
- Use the dataloader in `avcaffe_vid.py` if you want to load the raw video and audio.
5+
- Use the dataloader in `avcaffe_img.py` if you want to load the facial crops and audio.
6+
7+
## TODO
8+
- [ ] Detail example usages of the dataloaders.

0 commit comments

Comments
 (0)