Skip to content

Commit 779f899

Browse files
mengfei25mengfei25
authored
[CI] Reset models list for CI benchmark tests (#2166)
1. all huggingface models list file for nightly 2. all timm models list file for nightly 3. all torchbench models file for nightly 4. remove huggingface CamemBert due to removed in [pytorch/pytorch/pull/164815](https://github.com/pytorch/pytorch/pull/164815/files#diff-004303ad6116d64ab2a8356469ccb11b32d8caca702e1ad65cc0538600a76d2dL170) 5. add '--disable-cudagraphs' to reduce the impact of CUDA for XPU tests 6. align models list with pytorch for CI tests disable_build disable_ut disable_distributed
1 parent 0d2d924 commit 779f899

16 files changed

+572
-85
lines changed

‎.ci/benchmarks/huggingface.yaml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
skip:
2+
all:
3+
# Difficult to setup accuracy test because .eval() not supported
4+
- Reformer
5+
# Fails deepcopy
6+
- BlenderbotForConditionalGeneration
7+
- GPTNeoForCausalLM
8+
- GPTNeoForSequenceClassification
9+
# Fails with even batch size = 1
10+
- GPTJForCausalLM
11+
- GPTJForQuestionAnswering
12+
# Model too big
13+
- google/gemma-3-4b-it
14+
- openai/gpt-oss-20b
15+
- mistralai/Mistral-7B-Instruct-v0.3
16+
# Fails due to pytorch #164815 removed it
17+
- CamemBert
18+
19+
device:
20+
cpu:
21+
- meta-llama/Llama-3.2-1B
22+
- google/gemma-2-2b
23+
- google/gemma-3-4b-it
24+
- openai/whisper-tiny
25+
- Qwen/Qwen3-0.6B
26+
- mistralai/Mistral-7B-Instruct-v0.3
27+
- openai/gpt-oss-20b
28+
29+
control_flow:
30+
- AllenaiLongformerBase
31+
32+
batch_size:
33+
# TODO - Fails even after fake tensors
34+
divisors:
35+
AlbertForMaskedLM: 2
36+
AllenaiLongformerBase: 2
37+
BartForCausalLM: 2
38+
BertForMaskedLM: 2
39+
BlenderbotForCausalLM: 8
40+
# BlenderbotForConditionalGeneration : 16
41+
DebertaV2ForMaskedLM: 4
42+
DistilBertForMaskedLM: 2
43+
DistillGPT2: 2
44+
ElectraForCausalLM: 2
45+
GPT2ForSequenceClassification: 2
46+
# GPTJForCausalLM : 2
47+
# GPTJForQuestionAnswering : 2
48+
# GPTNeoForCausalLM : 32
49+
# GPTNeoForSequenceClassification : 2
50+
GoogleFnet: 2
51+
LayoutLMForMaskedLM: 2
52+
M2M100ForConditionalGeneration: 4
53+
MBartForCausalLM: 2
54+
MT5ForConditionalGeneration: 2
55+
MegatronBertForCausalLM: 4
56+
MobileBertForMaskedLM: 2
57+
OPTForCausalLM: 2
58+
PLBartForCausalLM: 2
59+
PegasusForCausalLM: 4
60+
RobertaForCausalLM: 2
61+
T5ForConditionalGeneration: 2
62+
T5Small: 2
63+
TrOCRForCausalLM: 2
64+
XGLMForCausalLM: 4
65+
XLNetLMHeadModel: 2
66+
YituTechConvBert: 2
67+
meta-llama/Llama-3.2-1B: 8
68+
google/gemma-2-2b: 8
69+
google/gemma-3-4b-it: 8
70+
openai/whisper-tiny: 8
71+
Qwen/Qwen3-0.6B: 8
72+
mistralai/Mistral-7B-Instruct-v0.3: 8
73+
openai/gpt-oss-20b: 8
74+
75+
76+
tolerance:
77+
higher_training:
78+
- MT5ForConditionalGeneration
79+
80+
higher_max_autotune_training: []
81+
82+
higher_inference:
83+
- GPT2ForSequenceClassification
84+
85+
higher_inference_cpu:
86+
- GPT2ForSequenceClassification
87+
88+
cosine: []
89+
90+
91+
accuracy:
92+
skip:
93+
large_models:
94+
# Models too large to have eager, dynamo and fp64_numbers simultaneously
95+
# even for 40 GB machine.
96+
- DebertaV2ForMaskedLM
97+
- BlenderbotForCausalLM
98+
99+
only_inference:
100+
# Fails with dynamo for train mode
101+
- M2M100ForConditionalGeneration
102+
103+
only_fp32:
104+
- GoogleFnet

‎.ci/benchmarks/huggingface_models_list.txt‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
AlbertForMaskedLM,8
2+
AlbertForQuestionAnswering,8
3+
AllenaiLongformerBase,8
4+
BartForCausalLM,8
5+
BartForConditionalGeneration,4
6+
BertForMaskedLM,32
7+
BertForQuestionAnswering,32
8+
BlenderbotForCausalLM,32
9+
BlenderbotForConditionalGeneration,16
10+
BlenderbotSmallForCausalLM,256
11+
BlenderbotSmallForConditionalGeneration,128
12+
CamemBert,32
13+
DebertaV2ForMaskedLM,8
14+
DebertaV2ForQuestionAnswering,8
15+
DistilBertForMaskedLM,256
16+
DistilBertForQuestionAnswering,512
17+
DistillGPT2,32
18+
ElectraForCausalLM,64
19+
ElectraForQuestionAnswering,128
20+
GPT2ForSequenceClassification,8
21+
GPTJForCausalLM,1
22+
GPTJForQuestionAnswering,1
23+
GPTNeoForCausalLM,32
24+
GPTNeoForSequenceClassification,32
25+
GoogleFnet,32
26+
LayoutLMForMaskedLM,32
27+
LayoutLMForSequenceClassification,32
28+
M2M100ForConditionalGeneration,64
29+
MBartForCausalLM,8
30+
MBartForConditionalGeneration,4
31+
MT5ForConditionalGeneration,32
32+
MegatronBertForCausalLM,16
33+
MegatronBertForQuestionAnswering,16
34+
MobileBertForMaskedLM,256
35+
MobileBertForQuestionAnswering,256
36+
OPTForCausalLM,4
37+
PLBartForCausalLM,16
38+
PLBartForConditionalGeneration,8
39+
PegasusForCausalLM,128
40+
PegasusForConditionalGeneration,64
41+
RobertaForCausalLM,32
42+
RobertaForQuestionAnswering,32
43+
T5ForConditionalGeneration,8
44+
T5Small,8
45+
TrOCRForCausalLM,64
46+
XGLMForCausalLM,32
47+
XLNetLMHeadModel,16
48+
YituTechConvBert,32
49+
meta-llama/Llama-3.2-1B,8
50+
google/gemma-2-2b,8
51+
google/gemma-3-4b-it,8
52+
openai/whisper-tiny,8
53+
Qwen/Qwen3-0.6B,8
54+
mistralai/Mistral-7B-Instruct-v0.3,8
55+
openai/gpt-oss-20b,8

‎.ci/benchmarks/timm_models.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# removing this file prevents the TimmRunner from dynamically skipping models
2+
skip:
3+
all:
4+
- ~
5+
device:
6+
cpu:
7+
- ~
8+
# Skip these additional models when running on aarch64
9+
cpu_aarch64:
10+
# timeout on aarch64
11+
- dm_nfnet_f0
12+
- nfnet_l0
13+
- visformer_small

‎.ci/benchmarks/timm_models_list.txt‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
adv_inception_v3 128
2+
beit_base_patch16_224 128
3+
botnet26t_256 128
4+
cait_m36_384 4
5+
coat_lite_mini 128
6+
convit_base 128
7+
convmixer_768_32 64
8+
convnext_base 128
9+
crossvit_9_240 256
10+
cspdarknet53 128
11+
deit_base_distilled_patch16_224 128
12+
dla102 128
13+
dm_nfnet_f0 128
14+
dpn107 64
15+
eca_botnext26ts_256 128
16+
eca_halonext26ts 128
17+
ese_vovnet19b_dw 256
18+
fbnetc_100 512
19+
fbnetv3_b 256
20+
gernet_l 128
21+
ghostnet_100 512
22+
gluon_inception_v3 256
23+
gmixer_24_224 128
24+
gmlp_s16_224 128
25+
hrnet_w18 128
26+
inception_v3 128
27+
jx_nest_base 128
28+
lcnet_050 256
29+
levit_128 1024
30+
mixer_b16_224 128
31+
mixnet_l 128
32+
mnasnet_100 512
33+
mobilenetv2_100 128
34+
mobilenetv3_large_100 512
35+
mobilevit_s 128
36+
nfnet_l0 128
37+
pit_b_224 64
38+
pnasnet5large 32
39+
poolformer_m36 128
40+
regnety_002 1024
41+
repvgg_a2 128
42+
res2net101_26w_4s 128
43+
res2net50_14w_8s 128
44+
res2next50 128
45+
resmlp_12_224 128
46+
resnest101e 128
47+
rexnet_100 256
48+
sebotnet33ts_256 64
49+
selecsls42b 128
50+
spnasnet_100 128
51+
swin_base_patch4_window7_224 128
52+
swsl_resnext101_32x16d 64
53+
tf_efficientnet_b0 128
54+
tf_mixnet_l 128
55+
tinynet_a 128
56+
tnt_s_patch16_224 128
57+
twins_pcpvt_base 128
58+
visformer_small 128
59+
vit_base_patch16_224 128
60+
volo_d1_224 128
61+
xcit_large_24_p8_224 16

0 commit comments

Comments
 (0)