Bad performance when finetuning on speech dataset

[bradshook]

I'm trying to reproduce the results for the BCIC2020-3 (speech) dataset. I downloaded the data from https://osf.io/pq7vb/files/osfstorage.

The data was downloaded using these steps:

wget -O dataset.zip "https://files.osf.io/v1/resources/pq7vb/providers/osfstorage/5e947fd8f1353503a7d55758/?zip="
7z x dataset.zip -ofiles/
Move files/Test set/Track3_Answer Sheet_Test.xlsx to speech/

Then I used your preprocessing script to preprocess the data.

I use this code snippet to finetune the model:

params = SimpleNamespace(
    seed=8888,
    cuda='1',
    epochs=50,
    batch_size=64,
    lr=1e-4,
    weight_decay=5e-2,
    optimizer='AdamW',
    clip_value=1,
    dropout=0.1,
    classifier='all_patch_reps',
    downstream_dataset='BCIC2020-3',
    datasets_dir='****/speech/processed',
    num_of_classes=5,
    model_dir='***/cbramod',
    num_workers=16,
    label_smoothing=0.1,
    multi_lr=True,
    frozen=False,
    use_pretrained_weights=True,
    foundation_dir='****/pretrained_weights.pth'
)

setup_seed(params.seed)

load_dataset = speech_dataset.LoadDataset(params)
data_loader = load_dataset.get_data_loader()
model = model_for_speech.Model(params)
t = Trainer(params, data_loader, model)
t.train_for_multiclass()

and the test evaluation is:

Test Evaluation: acc: 0.30267, kappa: 0.12833, f1: 0.30052

which is much lower than the reported metrics. Any idea what could be going on? I've used the same type of setup for other datasets like physio and shu and the performance was similar to what was reported.

[wsl2000]

I encountered the same problem, with acc: 0.30133, kappa: 0.12667, f1: 0.29598

[bradshook]

Yup. I've reproduced results for multiple datasets (using 5 seeds) and all of the performance metrics are lower than what was reported.

[wsl2000]

I got the same results on most of the datasets, except for this one.

[bradshook]

Yeh this one is the only one where the metrics are way lower than what is reported. I believe it may be a preprocessing bug because I have a separate model that performs just as poorly using the same preprocessed data.

[wsl2000]

I think this might help: the default value for '--multi_lr' is True, which should be set to False.

[samin9796]

@wsl2000 @brshook2023 Hi! Did you try to reproduce the results using baseline models (from the CBraMod paper) on the BCI 2020 imagined speech dataset? For example, using the BIOT model or SparcNet model? I get way too different baseline results from what was reported in the CBraMod paper. Using BIOT, I get approx. 30% accuracy and using SparcNet, I get around 70%.