Skip to content

Commit 7294dcd

Browse files
Darryl233Darryl233
authored
Merge pull request flagos-ai#4 from Darryl233/cicd
[CICD] flagos user tests management
2 parents d6fb612 + 96244cf commit 7294dcd

39 files changed

+2954
-0
lines changed

‎.github/CODEOWNERS‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# FlagOS DevOps - Code Owners
2+
3+
# Default owners for everything
4+
* @flagos-ai/devops-team
5+
6+
# CI/CD workflows
7+
.github/ @flagos-ai/devops-team
8+
9+
# Shared actions
10+
actions/ @flagos-ai/devops-team
11+
12+
# === User Tests ===
13+
14+
# FlagScale test cases
15+
flagos-user-tests/tests/flagscale/ @flagos-ai/flagscale-team
16+
17+
# FlagGems test cases
18+
flagos-user-tests/tests/flaggems/ @flagos-ai/flaggems-team
19+
20+
# FlagCX test cases
21+
flagos-user-tests/tests/flagcx/ @flagos-ai/flagcx-team
22+
23+
# FlagTree test cases
24+
flagos-user-tests/tests/flagtree/ @flagos-ai/flagtree-team
25+
26+
# vLLM-FL test cases
27+
flagos-user-tests/tests/vllm-fl/ @flagos-ai/vllm-team
28+
29+
# vLLM-plugin-FL test cases
30+
flagos-user-tests/tests/vllm-plugin-fl/ @flagos-ai/vllm-team
31+
32+
# TE-FL test cases
33+
flagos-user-tests/tests/te-fl/ @flagos-ai/te-team
34+
35+
# Megatron-LM-FL test cases
36+
flagos-user-tests/tests/megatron-lm-fl/ @flagos-ai/megatron-team
37+
38+
# Experimental test cases
39+
flagos-user-tests/tests/experimental/ @flagos-ai/devops-team
40+
41+
# Validation tools
42+
flagos-user-tests/tools/ @flagos-ai/devops-team

‎.github/ISSUE_TEMPLATE/new_test_case.yml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
name: New Test Case Submission
2+
description: Submit a new test case for FlagOS repositories
3+
title: "[Test Case] "
4+
labels: ["new-test-case"]
5+
body:
6+
- type: dropdown
7+
id: target-repo
8+
attributes:
9+
label: Target Repository
10+
description: Which FlagOS repository is this test case for?
11+
options:
12+
- FlagScale
13+
- FlagGems
14+
- FlagCX
15+
- FlagTree
16+
- vLLM-FL
17+
- vLLM-plugin-FL
18+
- TE-FL
19+
- Megatron-LM-FL
20+
validations:
21+
required: true
22+
23+
- type: dropdown
24+
id: test-type
25+
attributes:
26+
label: Test Type
27+
description: What type of test is this?
28+
options:
29+
- train
30+
- inference
31+
- hetero_train
32+
- unit
33+
- integration
34+
- benchmark
35+
validations:
36+
required: true
37+
38+
- type: input
39+
id: model-name
40+
attributes:
41+
label: Model Name
42+
description: Name of the model being tested (if applicable)
43+
placeholder: e.g., llama2, mixtral, deepseek
44+
45+
- type: textarea
46+
id: description
47+
attributes:
48+
label: Test Case Description
49+
description: Describe what this test case validates
50+
placeholder: |
51+
This test case validates ...
52+
validations:
53+
required: true
54+
55+
- type: textarea
56+
id: config
57+
attributes:
58+
label: Configuration
59+
description: Paste the YAML configuration for the test case
60+
render: yaml
61+
validations:
62+
required: true
63+
64+
- type: textarea
65+
id: gold-values
66+
attributes:
67+
label: Gold Values
68+
description: Paste the expected gold values (JSON format)
69+
render: json
70+
71+
- type: textarea
72+
id: environment
73+
attributes:
74+
label: Environment Requirements
75+
description: Describe the hardware/software requirements
76+
placeholder: |
77+
- GPU: 8x A100 80GB
78+
- CUDA: 12.1
79+
- Python: 3.10
80+
validations:
81+
required: true
82+
83+
- type: checkboxes
84+
id: checklist
85+
attributes:
86+
label: Submission Checklist
87+
options:
88+
- label: I have tested this test case locally
89+
required: true
90+
- label: I have included gold values (if applicable)
91+
- label: I have added a README.md with test description
92+
required: true
93+
- label: My YAML configuration follows the schema specification
94+
required: true

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
## Test Case PR
2+
3+
### Target Repository
4+
<!-- Which FlagOS repo does this test case target? -->
5+
- [ ] FlagScale
6+
- [ ] FlagGems
7+
- [ ] FlagCX
8+
- [ ] FlagTree
9+
- [ ] vLLM-FL
10+
- [ ] vLLM-plugin-FL
11+
- [ ] TE-FL
12+
- [ ] Megatron-LM-FL
13+
14+
### Test Type
15+
<!-- What kind of test is this? -->
16+
- [ ] train
17+
- [ ] inference
18+
- [ ] hetero_train
19+
- [ ] unit
20+
- [ ] integration
21+
22+
### Description
23+
<!-- Briefly describe what this test case validates -->
24+
25+
26+
### Environment Requirements
27+
<!-- What hardware/software is needed to run this test? -->
28+
- GPU:
29+
- CUDA:
30+
- Python:
31+
32+
### Checklist
33+
- [ ] YAML configuration passes schema validation
34+
- [ ] Gold values are included (if applicable)
35+
- [ ] README.md is present for each test case
36+
- [ ] Test case has been verified locally
37+
- [ ] No sensitive data (tokens, passwords, private paths) in configs

‎.github/scripts/detect_changed_repos.js‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
// Detect which repos have changed test cases.
2+
//
3+
// Outputs (via core.setOutput):
4+
// changed_cases — JSON array of case paths (manual single-case dispatch)
5+
// changed_repos — JSON object {repo, task, model} (manual repo dispatch or _none_)
6+
// changed_repos_list — JSON array of repo names (auto-detected from PR/push)
7+
//
8+
// Called from workflow via:
9+
// uses: actions/github-script@v7
10+
// with:
11+
// script: |
12+
// const run = require('./.github/scripts/detect_changed_repos.js');
13+
// await run({ github, context, core });
14+
15+
module.exports = async ({ github, context, core }) => {
16+
const inputCase = process.env.INPUT_CASE || '';
17+
const inputRepo = process.env.INPUT_REPO || '';
18+
const inputTask = process.env.INPUT_TASK || '';
19+
const inputModel = process.env.INPUT_MODEL || '';
20+
21+
// Manual dispatch — single case
22+
if (inputCase) {
23+
core.setOutput('changed_cases', JSON.stringify([inputCase]));
24+
return;
25+
}
26+
27+
// Manual dispatch — by repo
28+
if (inputRepo) {
29+
core.setOutput('changed_repos', JSON.stringify({
30+
repo: inputRepo,
31+
task: inputTask,
32+
model: inputModel,
33+
}));
34+
return;
35+
}
36+
37+
// Auto-detect from changed files
38+
let files = [];
39+
if (context.eventName === 'pull_request') {
40+
const resp = await github.paginate(
41+
github.rest.pulls.listFiles,
42+
{ owner: context.repo.owner, repo: context.repo.repo, pull_number: context.issue.number }
43+
);
44+
files = resp.map(f => f.filename);
45+
} else {
46+
const resp = await github.rest.repos.compareCommits({
47+
owner: context.repo.owner, repo: context.repo.repo,
48+
base: context.payload.before, head: context.payload.after,
49+
});
50+
files = resp.data.files.map(f => f.filename);
51+
}
52+
53+
// Extract unique repos from changed paths
54+
const repos = new Set();
55+
for (const f of files) {
56+
const m = f.match(/^flagos-user-tests\/tests\/([^/]+)\//);
57+
if (m && m[1] !== 'experimental') repos.add(m[1]);
58+
}
59+
60+
if (repos.size === 0) {
61+
core.setOutput('changed_repos', JSON.stringify({ repo: '_none_' }));
62+
} else {
63+
core.setOutput('changed_repos_list', JSON.stringify([...repos]));
64+
}
65+
};

‎.github/workflows/nightly_integration.yml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
name: Nightly Integration Test - User Tests
2+
3+
on:
4+
schedule:
5+
- cron: "0 2 * * *"
6+
workflow_dispatch:
7+
8+
defaults:
9+
run:
10+
working-directory: flagos-user-tests
11+
12+
jobs:
13+
discover-cases:
14+
runs-on: ubuntu-latest
15+
outputs:
16+
matrix: ${{ steps.resolve.outputs.matrix }}
17+
steps:
18+
- name: Checkout
19+
uses: actions/checkout@v4
20+
21+
- name: Setup Python
22+
uses: actions/setup-python@v5
23+
with:
24+
python-version: "3.10"
25+
26+
- name: Install dependencies
27+
run: pip install pyyaml
28+
29+
- name: Discover all test cases and resolve runner labels
30+
id: resolve
31+
working-directory: flagos-user-tests
32+
run: |
33+
python3 -c "
34+
import json, os, sys
35+
sys.path.insert(0, 'tools')
36+
from run_user_tests import list_test_resources
37+
from pathlib import Path
38+
39+
root = Path('.')
40+
resources_list = list_test_resources(root)
41+
42+
matrix_entries = []
43+
for entry in resources_list:
44+
matrix_entries.append({
45+
'case_path': entry['case_path'],
46+
'runner_labels': json.dumps(entry['runner_labels']),
47+
})
48+
49+
if not matrix_entries:
50+
matrix_entries.append({
51+
'case_path': '_none_',
52+
'runner_labels': json.dumps(['ubuntu-latest']),
53+
})
54+
55+
matrix = {'include': matrix_entries}
56+
output = json.dumps(matrix)
57+
print(f'Matrix: {output}')
58+
with open(os.environ['GITHUB_OUTPUT'], 'a') as f:
59+
f.write(f'matrix={output}\n')
60+
"
61+
62+
run-tests:
63+
needs: discover-cases
64+
if: ${{ !contains(needs.discover-cases.outputs.matrix, '_none_') }}
65+
runs-on: ${{ fromJson(matrix.runner_labels) }}
66+
strategy:
67+
fail-fast: false
68+
matrix: ${{ fromJson(needs.discover-cases.outputs.matrix) }}
69+
steps:
70+
- name: Checkout
71+
uses: actions/checkout@v4
72+
73+
- name: Setup Python
74+
uses: actions/setup-python@v5
75+
with:
76+
python-version: "3.12"
77+
78+
- name: Install runner dependencies
79+
run: pip install pyyaml
80+
81+
- name: Run test case
82+
run: python tools/run_user_tests.py --case ${{ matrix.case_path }}
83+
84+
notify:
85+
needs: run-tests
86+
if: always()
87+
runs-on: ubuntu-latest
88+
steps:
89+
- name: Generate summary
90+
run: |
91+
echo "## Nightly Integration Test Summary" >> $GITHUB_STEP_SUMMARY
92+
echo "" >> $GITHUB_STEP_SUMMARY
93+
echo "Run: ${{ github.run_number }}" >> $GITHUB_STEP_SUMMARY
94+
echo "Date: $(date -u '+%Y-%m-%d %H:%M UTC')" >> $GITHUB_STEP_SUMMARY

‎.github/workflows/post_test_cases.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: Post Test Cases Report
2+
3+
on:
4+
pull_request:
5+
branches: [main]
6+
types: [closed]
7+
workflow_dispatch:
8+
9+
defaults:
10+
run:
11+
working-directory: flagos-user-tests
12+
13+
jobs:
14+
post-report:
15+
if: ${{ github.event_name == 'workflow_dispatch' || github.event.pull_request.merged == true }}
16+
runs-on: ubuntu-latest
17+
steps:
18+
- name: Checkout
19+
uses: actions/checkout@v4
20+
with:
21+
fetch-depth: 0
22+
23+
- name: Setup Python
24+
uses: actions/setup-python@v5
25+
with:
26+
python-version: "3.10"
27+
28+
- name: Install dependencies
29+
run: pip install pyyaml
30+
31+
- name: Collect test cases
32+
run: python tools/collect_test_cases.py --output test_cases_report.json
33+
34+
- name: Post report
35+
uses: ./actions/post-benchmark-report
36+
with:
37+
backend_url: ${{ secrets.FLAGOPS_BACKEND_URL }}
38+
api_token: ${{ secrets.FLAGOPS_API_TOKEN }}
39+
report_path: flagos-user-tests/test_cases_report.json
40+
list_code: flagops-user-test-cases
41+
list_name: FlagOps User Test Cases
42+
header_config: >-
43+
[
44+
{"field": "case_id", "name": "用例ID", "required": true, "sortable": true, "type": "string"},
45+
{"field": "case_name", "name": "用例名称", "required": true, "sortable": false, "type": "string"},
46+
{"field": "repo", "name": "所属子仓库", "required": true, "sortable": true, "type": "string"},
47+
{"field": "updated_at", "name": "更新时间", "required": true, "sortable": true, "type": "string"}
48+
]
49+
fail_on_error: "false"

0 commit comments

Comments
 (0)