Skip to content

Commit 1acf6d7

Browse files
naman-msftroot
naman-msft
and
root
authoredNov 18, 2024··
Automated testing for exec docs using IE (#242)
Attached github action and its instructions for automated testing of exec docs --------- Co-authored-by: root <root@DESKTOP-MJFFR09>
·
latest
1 parent 15075ee commit 1acf6d7

File tree

2 files changed

+303
-0
lines changed

2 files changed

+303
-0
lines changed
 

‎examples/test-exec-docs/README.md

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Automating Documentation Testing with Innovation Engine
2+
3+
## Overview
4+
5+
This guide explains how to set up and use a GitHub Action that automates testing of your Markdown documentation using Innovation Engine. The action ensures your executable documents are accurate and robust by running tests on them. It will create comments on pull requests for any errors found and, when combined with branch protection rules, can prevent pull requests from merging until those errors are resolved.
6+
7+
## Prerequisites
8+
9+
Before setting up the GitHub Action, you need the following:
10+
11+
- **Azure Credentials**: The action logs into Azure using the Azure CLI, so you need to provide the following repository secrets:
12+
- `AZURE_CLIENT_ID`
13+
- `AZURE_TENANT_ID`
14+
- `AZURE_SUBSCRIPTION_ID`
15+
16+
To obtain these values:
17+
1. Register an application in Azure Active Directory.
18+
2. Assign the necessary permissions to the application.
19+
3. Note the application (client) ID, tenant ID, and subscription ID.
20+
21+
- **GitHub Personal Access Token (PAT)**: The action interacts with the GitHub API to create comments on pull requests. Store your PAT in a repository secret named `PAT`.
22+
23+
## GitHub Action Workflow
24+
25+
The GitHub Action performs the following steps:
26+
27+
1. **Trigger**: Runs on pull request events when a pull request is opened or updated.
28+
29+
2. **Checkout Repository**: Uses `actions/checkout@v3` to clone the repository with full history for accurate diffs.
30+
31+
3. **Determine Commit SHAs**:
32+
- Identifies the base and head commits of the pull request.
33+
- Calculates the number of commits between the base and head.
34+
- Sets output variables for use in subsequent steps.
35+
36+
4. **Check for Changed Markdown Files**:
37+
38+
- Compares changes between the base and head commits.
39+
- Generates a list of changed Markdown files to be tested.
40+
- If no Markdown files have changed, the workflow exits early.
41+
42+
5. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v2` (only if Markdown files have changed).
43+
44+
6. **Set Up Python**: Sets up a Python 3.12 environment using `actions/setup-python@v4` (only if Markdown files have changed).
45+
46+
7. **Install Dependencies**: Installs required Python packages (only if Markdown files have changed):
47+
48+
```bash
49+
pip install --upgrade pip==24.3.1
50+
pip install setuptools wheel
51+
pip install PyGithub==2.1.1 pyyaml==6.0.2
52+
```
53+
54+
8. **Run Tests and Handle Pull Request**: Executes a Python script that (only if Markdown files have changed):
55+
56+
- **Innovation Engine Installation**: Installs `ie` (Innovation Engine) if not already installed.
57+
- **Repository Information**: Retrieves repository details such as owner, name, and pull request number.
58+
- **Testing Markdown Files**:
59+
- Iterates over the list of changed Markdown files.
60+
- Runs `ie execute` on each file to test the executable documentation.
61+
- **On Test Failure**:
62+
- Extracts error logs from the `ie.log` file.
63+
- Creates a comment on the pull request with the error details.
64+
- Records the failure in the test results.
65+
- **On Test Success**:
66+
- Records the success in the test results.
67+
- **Reporting Results**:
68+
- Posts the test results as a comment on the pull request.
69+
- If any failures occurred, exits with a non-zero status to fail the workflow.
70+
71+
## Customization
72+
73+
You can modify the GitHub Action to suit your project's needs:
74+
75+
- **Event Triggers**: Adjust the `on` section to change when the action runs. Currently, it triggers on pull request events (`opened` and `synchronize`).
76+
77+
- **Azure Login Step**: If your documentation doesn't require Azure resources, you can remove the Azure login step.
78+
79+
- **Python Version and Dependencies**:
80+
- Update the `python-version` in the **Set Up Python** step to match your project's requirements.
81+
- In the **Install Dependencies** step, specify the versions of `pip`, `PyGithub`, and `pyyaml` as needed.
82+
83+
- **Testing Script**: Update the Python script to change how tests are executed, how errors are handled, or how results are reported. For example, you might customize the error extraction logic or modify the comment creation process.
84+
85+
- **Secrets Management**: Ensure all necessary secrets (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`, `PAT`) are correctly named and stored in your repository's settings under **Settings > Secrets and variables > Actions**.
86+
87+
## Conclusion
88+
89+
By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and, combined with branch protection rules, prevents broken documentation from being merged into your main branch.

‎examples/test-exec-docs/test-exec-docs.yml

+214
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
# Name of the workflow
2+
name: Innovation Engine Tests
3+
4+
# Set permissions required for the workflow
5+
permissions:
6+
contents: write # Allows writing repository contents
7+
pull-requests: write # Allows creating and modifying pull requests
8+
id-token: write # Allows generating tokens for authentication
9+
issues: write # Allows creating and modifying issues
10+
11+
# Define the events that trigger the workflow
12+
on:
13+
pull_request:
14+
types: [opened, synchronize] # Trigger on PR creation and updates
15+
16+
jobs:
17+
run-tests:
18+
runs-on: ubuntu-latest # Run the job on the latest Ubuntu runner
19+
20+
env:
21+
CREATE_PR: 'false' # No need to create PR as it's already a PR event
22+
23+
steps:
24+
- name: Checkout repository
25+
uses: actions/checkout@v3
26+
with:
27+
fetch-depth: 0 # Fetch all history for accurate diff
28+
29+
- name: Determine commit SHAs
30+
id: commits
31+
run: |
32+
# Set BASE_SHA to the base of the PR
33+
BASE_SHA="${{ github.event.pull_request.base.sha }}"
34+
# Set HEAD_SHA to the head of the PR
35+
HEAD_SHA="${{ github.event.pull_request.head.sha }}"
36+
37+
echo "Base SHA: $BASE_SHA"
38+
echo "Head SHA: $HEAD_SHA"
39+
40+
# Count the number of commits between base and head
41+
COMMIT_COUNT=$(git rev-list --count "${BASE_SHA}..${HEAD_SHA}")
42+
echo "Commit count since base: $COMMIT_COUNT"
43+
44+
if [ "$COMMIT_COUNT" -eq 1 ]; then
45+
FINAL_BASE_SHA="$BASE_SHA"
46+
echo "This is the first commit on the branch."
47+
else
48+
# Ensure HEAD_SHA has a parent commit
49+
PARENT_SHA=$(git rev-parse "${HEAD_SHA}^")
50+
51+
if [ $? -ne 0 ] || [ -z "$PARENT_SHA" ]; then
52+
echo "Error: Unable to find parent commit of HEAD_SHA."
53+
exit 1
54+
fi
55+
56+
FINAL_BASE_SHA="$PARENT_SHA"
57+
echo "This is not the first commit on the branch."
58+
echo "Parent SHA: $PARENT_SHA"
59+
fi
60+
61+
# Set outputs for subsequent steps
62+
echo "final_base_sha=$FINAL_BASE_SHA" >> $GITHUB_OUTPUT
63+
echo "head_sha=$HEAD_SHA" >> $GITHUB_OUTPUT
64+
65+
- name: Check for changed Markdown files
66+
id: check_changes
67+
run: |
68+
FINAL_BASE_SHA="${{ steps.commits.outputs.final_base_sha }}"
69+
HEAD_SHA="${{ steps.commits.outputs.head_sha }}"
70+
71+
# Ensure FINAL_BASE_SHA and HEAD_SHA are set
72+
if [ -z "${FINAL_BASE_SHA}" ] || [ -z "${HEAD_SHA}" ]; then
73+
echo "Error: FINAL_BASE_SHA or HEAD_SHA is not set."
74+
exit 1
75+
fi
76+
77+
echo "Final Base SHA: $FINAL_BASE_SHA"
78+
echo "Head SHA: $HEAD_SHA"
79+
80+
CHANGED_FILES=$(git diff --name-only "${FINAL_BASE_SHA}" "${HEAD_SHA}" --)
81+
82+
echo "CHANGED_FILES<<EOF" >> $GITHUB_ENV
83+
echo "$CHANGED_FILES" >> $GITHUB_ENV
84+
echo "EOF" >> $GITHUB_ENV
85+
86+
CHANGED_MARKDOWN_FILES=$(echo "$CHANGED_FILES" | grep -E '\.md$' || true)
87+
88+
echo "Markdown files to test:"
89+
echo "$CHANGED_MARKDOWN_FILES"
90+
91+
if [ -n "$CHANGED_MARKDOWN_FILES" ]; then
92+
echo "changed=true" >> $GITHUB_OUTPUT
93+
echo "changed_files<<EOF" >> $GITHUB_OUTPUT
94+
echo "$CHANGED_MARKDOWN_FILES" >> $GITHUB_OUTPUT
95+
echo "EOF" >> $GITHUB_OUTPUT
96+
else
97+
echo "changed=false" >> $GITHUB_OUTPUT
98+
echo "No Markdown files changed. Exiting." >&2
99+
exit 0
100+
fi
101+
102+
- name: Azure CLI Login
103+
if: steps.check_changes.outputs.changed == 'true'
104+
uses: azure/login@v2
105+
with:
106+
client-id: ${{ secrets.AZURE_CLIENT_ID }} # Azure Client ID from secrets
107+
tenant-id: ${{ secrets.AZURE_TENANT_ID }} # Azure Tenant ID from secrets
108+
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} # Azure Subscription ID from secrets
109+
110+
- name: Set up Python
111+
if: steps.check_changes.outputs.changed == 'true'
112+
uses: actions/setup-python@v4
113+
with:
114+
python-version: '3.12' # Specify Python version 3.x
115+
116+
- name: Install dependencies
117+
if: steps.check_changes.outputs.changed == 'true'
118+
run: |
119+
# Install required packages with pinned versions
120+
pip install --upgrade pip==24.3.1
121+
pip install setuptools wheel
122+
pip install PyGithub==2.1.1 pyyaml==6.0.2
123+
124+
- name: Run tests and handle PR
125+
if: steps.check_changes.outputs.changed == 'true'
126+
env:
127+
GITHUB_TOKEN: ${{ secrets.PAT }} # Personal Access Token from secrets
128+
CHANGED_MARKDOWN_FILES: ${{ steps.check_changes.outputs.changed_files }} # List of changed Markdown files
129+
CREATE_PR: ${{ env.CREATE_PR }}
130+
PR_NUMBER: ${{ github.event.pull_request.number }} # Add PR_NUMBER environment variable
131+
run: |
132+
python <<EOF
133+
import os
134+
import subprocess
135+
import shutil
136+
from github import Github
137+
138+
# Get GitHub token from environment variable
139+
GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
140+
PR_NUMBER = os.getenv('PR_NUMBER') # Retrieve PR_NUMBER
141+
142+
if not PR_NUMBER:
143+
print("Error: PR_NUMBER is not set.")
144+
exit(1)
145+
146+
try:
147+
pr_number = int(PR_NUMBER)
148+
except ValueError:
149+
print(f"Error: Invalid PR_NUMBER '{PR_NUMBER}'.")
150+
exit(1)
151+
152+
# Authenticate with GitHub using PyGithub
153+
g = Github(GITHUB_TOKEN)
154+
155+
# Check if 'ie' (Innovation Engine) is installed, install if not
156+
if not shutil.which("ie"):
157+
# Update package lists and install unzip
158+
subprocess.run("sudo apt-get update && sudo apt-get install -y unzip", shell=True, check=True)
159+
160+
# Install Innovation Engine CLI
161+
INNOVATION_ENGINE_VERSION = "v0.2.3"
162+
subprocess.run(f"curl -Lks https://aka.ms/install-ie | /bin/bash -s {INNOVATION_ENGINE_VERSION}", shell=True, check=True)
163+
164+
# Retrieve repository information
165+
repo_full_name = os.getenv('GITHUB_REPOSITORY')
166+
repo = g.get_repo(repo_full_name)
167+
168+
# Get the list of changed Markdown files
169+
changed_files_env = os.getenv('CHANGED_MARKDOWN_FILES', '')
170+
changed_files = [f.strip() for f in changed_files_env.split('\n') if f.strip().endswith('.md')]
171+
172+
# Initialize a list to store test results
173+
results = ["**Test Results**\n\n==========\n\n"]
174+
175+
any_failures = False
176+
177+
# Iterate over changed Markdown files
178+
for file_path in changed_files:
179+
# Execute the Innovation Engine on the Markdown file
180+
result = subprocess.run(['ie', 'execute', file_path, '--environment', 'github-action'])
181+
if result.returncode != 0:
182+
any_failures = True
183+
# If execution fails, extract error log from 'ie.log'
184+
error_log = ''
185+
186+
try:
187+
with open('ie.log', 'r') as log_file:
188+
lines = log_file.readlines()
189+
# Filter out error lines
190+
error_lines = [line for line in lines if "level=error" in line]
191+
if error_lines:
192+
error_log = error_lines[-1] # Get the last error line
193+
else:
194+
error_log = "No error message found in ie.log"
195+
except FileNotFoundError:
196+
error_log = "ie.log not found."
197+
198+
# Append failure result to the results list
199+
results.append(f"**{file_path}**: Tests failed.\n\nError Details:\n\n***{error_log}***\n\n==========\n\n")
200+
else:
201+
# Append success result to the results list
202+
results.append(f"**{file_path}**: Tests passed successfully.\n\n==========\n\n")
203+
204+
# Post the test results as a comment on the pull request
205+
pr = repo.get_pull(pr_number)
206+
pr.create_issue_comment('\n'.join(results))
207+
208+
# If any failures occurred, exit with a non-zero status to fail the workflow
209+
if any_failures:
210+
exit(1)
211+
else:
212+
exit(0)
213+
214+
EOF

0 commit comments

Comments
 (0)