[QEff Finetune]: Adding steps about how to fine tune on any custom dataset. #381

quic-swatia · 2025-04-28T11:14:42Z

Added steps on how to create the custom_dataset.py to run fine-tuning through QEfficient pipeline on any custom dataset. Also, added a detailed template for the user which covers how to create custom_dataset.py
Added the argument 'context_length' in the existing APIs which helps run fine tuning with padding for custom dataset.
Made alpaca_dataset as the default dataset.

Signed-off-by: Swati Allabadi <[email protected]>

quic-meetkuma

Good work in listing down the detailed steps for custom dataset, Swati! Please check on the comments. :)

quic-meetkuma · 2025-05-12T09:44:36Z

docs/source/finetune.md

+To run fine tuning for any user specific dataset, prepare the dataset using the following steps:
+
+    1) Create a  directory named 'dataset' inside efficient-transformers. 


double space between "a" and "directory"

quic-meetkuma · 2025-05-12T09:46:46Z

docs/source/finetune.md

+
+    def tokenize():
+


Add some comment as "Implement tokenization and prepare inputs for the training."

quic-meetkuma · 2025-05-12T09:49:04Z

docs/source/finetune.md

+    # load dataset
+    # based on split, retrieve only the specific portion of the dataset (train or eval) either here or at the last
+


Add one more comment as "Define a prompt template"

It's already there. ( # define prompt)

quic-meetkuma · 2025-05-12T09:49:32Z

docs/source/finetune.md

+
+    def apply_prompt_template():
+


Add some comment as "Convert the raw input into format as per the template defined earlier."

quic-meetkuma · 2025-05-12T09:53:16Z

docs/source/finetune.md

+    5) Inside get_custom_dataset(), dataset needs to prepared for fine tuning. So, the user needs to apply prompt and tokenize the dataset accordingly. Please refer the below template on how to define get_custom_dataset().
+    6) For examples, please refer python files present in efficient-transformers/QEfficient/finetune/dataset. In case of Samsum dataset, get_preprocessed_samsum() of efficient-transformers/QEfficient/finetune/dataset/samsum_dataset.py is called. 
+    7) In efficient-transformers/QEfficient/finetune/configs/dataset_config.py, for custom_dataset class, pass the appropriate value for train_split and test_split according to the dataset keys corresponding to train and test data points.


I think this is no longer needed after PR#289. We can directly pass --train_split and --test_split from the CLI.

Yes, correct! Thanks for pointing this. After PR 289, user can do this through both ways. Updated step #7 accordingly.

Signed-off-by: Swati Allabadi <[email protected]>

quic-meetkuma · 2025-05-19T10:50:58Z

QEfficient/finetune/configs/training.py

@@ -28,7 +28,7 @@ class train_config:
    use_fp16: bool = True
    use_autocast: bool = True
    val_batch_size: int = 1
-    dataset = "samsum_dataset"
+    dataset = "alpaca_dataset"


Good that you have added this change in this gerrit.

quic-meetkuma

Looks good to me.

quic-mamta · 2025-05-19T15:59:40Z

docs/source/finetune.md

+To run fine tuning for any user specific dataset, prepare the dataset using the following steps:
+
+    1) Create a directory named 'dataset' inside efficient-transformers. 


Add the location "at root of the repo."

quic-mamta · 2025-05-19T16:00:42Z

docs/source/finetune.md

+    2) Inside this directory, create a file named 'custom_dataset.py'. This is different than the custom_dataset.py present at efficient-transformers/QEfficient/finetune/dataset.
+    3) Inside the newly created efficient-transformers/dataset/custom_dataset.py, define a function named 'get_custom_dataset'. 
+    4) get_custom_dataset() should have following 4 parameters:  dataset_config, tokenizer, split, context_length. This function gets called twice through Qefficient/cloud/finetune.py with the name get_preprocessed_dataset. 


QEfficient not Qefficient

quic-mamta · 2025-05-19T16:01:32Z

docs/source/finetune.md

+    4) get_custom_dataset() should have following 4 parameters:  dataset_config, tokenizer, split, context_length. This function gets called twice through Qefficient/cloud/finetune.py with the name get_preprocessed_dataset. 
+    5) Inside get_custom_dataset(), dataset needs to prepared for fine tuning. So, the user needs to apply prompt and tokenize the dataset accordingly. Please refer the below template on how to define get_custom_dataset().
+    6) For examples, please refer python files present in efficient-transformers/QEfficient/finetune/dataset. In case of Samsum dataset, get_preprocessed_samsum() of efficient-transformers/QEfficient/finetune/dataset/samsum_dataset.py is called. 


since default dataset is changed, we should mention alpaca here.

The steps I have mentioned matches with the format of samsum_dataset.py. It doesn't match with alpaca_dataset.py. Hence, I didn't change it.

too verbose. Make it simple pointed steps

Signed-off-by: Swati Allabadi <[email protected]>

anujgupt-github · 2025-05-21T03:26:32Z

docs/source/finetune.md

+    4) get_custom_dataset() should have following 4 parameters:  dataset_config, tokenizer, split, context_length. This function gets called twice through Qefficient/cloud/finetune.py with the name get_preprocessed_dataset. 
+    5) Inside get_custom_dataset(), dataset needs to prepared for fine tuning. So, the user needs to apply prompt and tokenize the dataset accordingly. Please refer the below template on how to define get_custom_dataset().
+    6) For examples, please refer python files present in efficient-transformers/QEfficient/finetune/dataset. In case of Samsum dataset, get_preprocessed_samsum() of efficient-transformers/QEfficient/finetune/dataset/samsum_dataset.py is called. 


too verbose. Make it simple pointed steps

anujgupt-github · 2025-05-21T03:27:27Z

docs/source/finetune.md

+    5) Inside get_custom_dataset(), dataset needs to prepared for fine tuning. So, the user needs to apply prompt and tokenize the dataset accordingly. Please refer the below template on how to define get_custom_dataset().
+    6) For examples, please refer python files present in efficient-transformers/QEfficient/finetune/dataset. In case of Samsum dataset, get_preprocessed_samsum() of efficient-transformers/QEfficient/finetune/dataset/samsum_dataset.py is called. 
+    7) In efficient-transformers/QEfficient/finetune/configs/dataset_config.py, for custom_dataset class, pass the appropriate value for train_split and test_split according to the dataset keys corresponding to train and test data points. As an alternative, these values can be passed as command line arguemnets as well with the finetune command. For example "--train_split train". 


Add hyperlinks to the relative paths annotated in the steps below

anujgupt-github · 2025-05-21T03:28:55Z

docs/source/finetune.md

+```
+
+## Fine-Tuning on custom dataset


You should include details on how we use gradient accumulation, how the dataset is shuffled, how activation checkpointing is enabled in separate sections.
In custom dataset, add a point that if any user wants to use these, refer xyz section

quic-swatia requested review from quic-rishinr and ochougul as code owners April 28, 2025 11:14

quic-swatia self-assigned this Apr 28, 2025

quic-swatia requested a review from vbaddi April 28, 2025 11:16

quic-rishinr marked this pull request as draft April 28, 2025 16:51

Adding steps about how to fine tune on any custom dataset.

e757ab8

Signed-off-by: Swati Allabadi <[email protected]>

quic-swatia force-pushed the custom_dataset branch from 8485344 to e757ab8 Compare April 29, 2025 09:10

quic-mamta changed the title ~~Adding steps about how to fine tune on any custom dataset.~~ [QEff Finetune]: Adding steps about how to fine tune on any custom dataset. May 8, 2025

quic-swatia marked this pull request as ready for review May 9, 2025 10:39

quic-swatia requested a review from quic-amitraj as a code owner May 9, 2025 10:39

Update finetune.md

bae75d2

Signed-off-by: Swati Allabadi <[email protected]>

quic-rishinr added the fine-tuning label May 12, 2025

quic-swatia requested a review from quic-mamta May 12, 2025 05:35

quic-meetkuma suggested changes May 12, 2025

View reviewed changes

Adding alpaca_dataset as teh default dataset

05a385a

Signed-off-by: Swati Allabadi <[email protected]>

quic-swatia force-pushed the custom_dataset branch from f46f537 to 05a385a Compare May 19, 2025 10:45

quic-meetkuma reviewed May 19, 2025

View reviewed changes

quic-meetkuma approved these changes May 19, 2025

View reviewed changes

quic-mamta reviewed May 19, 2025

View reviewed changes

Update finetune.md

155bb77

Signed-off-by: Swati Allabadi <[email protected]>

quic-mamta approved these changes May 19, 2025

View reviewed changes

Merge branch 'main' into custom_dataset

43aab17

anujgupt-github reviewed May 21, 2025

View reviewed changes

		To run fine tuning for any user specific dataset, prepare the dataset using the following steps:

		1) Create a directory named 'dataset' inside efficient-transformers.

		# load dataset
		# based on split, retrieve only the specific portion of the dataset (train or eval) either here or at the last

[QEff Finetune]: Adding steps about how to fine tune on any custom dataset. #381

Are you sure you want to change the base?

[QEff Finetune]: Adding steps about how to fine tune on any custom dataset. #381

Conversation

quic-swatia commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

quic-meetkuma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quic-swatia May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quic-meetkuma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

quic-swatia commented Apr 28, 2025 •

edited

Loading

quic-swatia May 14, 2025 •

edited

Loading