[AutoDeploy]: integrate flashinfer fp8 fused moe kernel

[nzmora-nvidia]

🚀 The feature, motivation and pitch

Integrate the flashinfer/cutlass kernel from the flashinfer repo

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[nzmora-nvidia]

Data is in
#9011

[nzmora-nvidia]

Accuracy results:

• MMLU weighted average accuracy: 76.27 (4104)
• gsm8k ~56.7

There are no accuracy references, but looking at the available reference scores, these look passing.

/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_modelopt/users/nzmora/dev/TensorRT-LLM-neta-fork/tests/integration/defs/accuracy/references/mmlu.yaml

/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_modelopt/users/nzmora/dev/TensorRT-LLM-neta-fork/tests/integration/defs/accuracy/references/gsm8k.yaml

MMLU:

[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 59.72 (72) - abstract_algebra
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 72.22 (72) - anatomy
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 90.28 (72) - astronomy
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 69.44 (72) - business_ethics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.00 (72) - clinical_knowledge
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 87.50 (72) - college_biology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 56.94 (72) - college_chemistry
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 65.28 (72) - college_computer_science
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 52.78 (72) - college_mathematics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 76.39 (72) - college_medicine
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 56.94 (72) - college_physics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 81.94 (72) - computer_security
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.00 (72) - conceptual_physics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 61.11 (72) - econometrics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.00 (72) - electrical_engineering
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 61.11 (72) - elementary_mathematics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 65.28 (72) - formal_logic
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 50.00 (72) - global_facts
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 93.06 (72) - high_school_biology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 73.61 (72) - high_school_chemistry
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 84.72 (72) - high_school_computer_science
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 81.94 (72) - high_school_european_history
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 87.50 (72) - high_school_geography
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 95.83 (72) - high_school_government_and_politics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 77.78 (72) - high_school_macroeconomics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 45.83 (72) - high_school_mathematics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 86.11 (72) - high_school_microeconomics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 62.50 (72) - high_school_physics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 86.11 (72) - high_school_psychology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 72.22 (72) - high_school_statistics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 86.11 (72) - high_school_us_history
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 88.89 (72) - high_school_world_history
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 80.56 (72) - human_aging
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 81.94 (72) - human_sexuality
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 87.50 (72) - international_law
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 83.33 (72) - jurisprudence
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 86.11 (72) - logical_fallacies
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 62.50 (72) - machine_learning
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 83.33 (72) - management
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 93.06 (72) - marketing
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 84.72 (72) - medical_genetics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 90.28 (72) - miscellaneous
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 79.17 (72) - moral_disputes
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.00 (72) - moral_scenarios
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 86.11 (72) - nutrition
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.00 (72) - philosophy
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 83.33 (72) - prehistory
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 52.78 (72) - professional_accounting
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 62.50 (72) - professional_law
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 72.22 (72) - professional_medicine
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 81.94 (72) - professional_psychology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 83.33 (72) - public_relations
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 79.17 (72) - security_studies
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 88.89 (72) - sociology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 94.44 (72) - us_foreign_policy
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 55.56 (72) - virology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 90.28 (72) - world_religions
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 58.33 (360) - math
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.35 (576) - health
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 71.18 (288) - physics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 81.94 (216) - business
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 90.28 (144) - biology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 65.28 (144) - chemistry
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 73.61 (288) - computer science
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.00 (216) - economics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 75.00 (72) - engineering
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 78.47 (432) - philosophy
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 64.35 (216) - other
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 85.07 (288) - history
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 87.50 (72) - geography
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 88.19 (288) - politics
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 84.03 (144) - psychology
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 85.42 (144) - culture
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 77.78 (216) - law
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 69.83 (1296) - STEM
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 80.34 (936) - humanities
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 83.68 (864) - social sciences
[11/10/2025-05:14:07] [TRT-LLM] [I] Average accuracy 74.40 (1008) - other (business, health, misc.)
[11/10/2025-05:14:07] [TRT-LLM] [I] MMLU weighted average accuracy: 76.27 (4104)

what about these results (different test):
[11/10/2025-05:18:47] [TRT-LLM] [I] lm-eval gsm8k results (scores normalized to range 0~100):
|Tasks|Version|     Filter     |n-shot|  Metric   |   | Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|------:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |56.8613|±  |1.3642|
|     |       |strict-match    |     5|exact_match|↑  |56.7096|±  |1.3648|

[nzmora-nvidia]

Performance data is in https://docs.google.com/spreadsheets/d/1ScdOTqtq8sepgaGU06zd74u_cy6cUdV5d69gogGHQsQ/edit?gid=765185418#gid=765185418
See sheet labeled "11/9 tp=1"