From e368809dc565d5ce7dc3685084da2f748b97aa0d Mon Sep 17 00:00:00 2001
From: root <root@DESKTOP-MJFFR09>
Date: Tue, 12 Nov 2024 14:14:30 -0800
Subject: [PATCH 1/9] added instructions on the github action for automated
 testing of exec docs using IE and the github action itself

---
 examples/test-exec-doc.yml    |  0
 examples/test-instructions.md | 70 +++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+)
 create mode 100644 examples/test-exec-doc.yml
 create mode 100644 examples/test-instructions.md

diff --git a/examples/test-exec-doc.yml b/examples/test-exec-doc.yml
new file mode 100644
index 00000000..e69de29b
diff --git a/examples/test-instructions.md b/examples/test-instructions.md
new file mode 100644
index 00000000..a211f8b4
--- /dev/null
+++ b/examples/test-instructions.md
@@ -0,0 +1,70 @@
+# Automating Documentation Testing with Innovation Engine
+
+## Overview
+
+This guide explains how to set up and use a GitHub Action that automates testing of your Markdown documentation using Innovation Engine. The action ensures your executable documents are accurate and robust by running tests on them. It will create issues for any errors found and prevent pull requests from merging until those errors are resolved.
+
+## Prerequisites
+
+Before setting up the GitHub Action, you need the following:
+
+- **Azure Credentials**: The action logs into Azure using the Azure CLI, so you need to provide the following repository secrets:
+  - `AZURE_CLIENT_ID`
+  - `AZURE_TENANT_ID`
+  - `AZURE_SUBSCRIPTION_ID`
+
+  To obtain these values:
+  1. Register an application in Azure Active Directory.
+  2. Assign the necessary permissions to the application.
+  3. Note the application (client) ID, tenant ID, and subscription ID.
+
+- **GitHub Personal Access Token (PAT)**: The action interacts with the GitHub API to create issues and pull requests. Store your PAT in a repository secret named `PAT`.
+
+## GitHub Action Workflow
+
+The GitHub Action performs the following steps:
+
+1. **Trigger**: Runs on push or pull request events affecting Markdown files, or when manually triggered.
+
+2. **Checkout Repository**: Uses `actions/checkout@v2` to clone the repository.
+
+3. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v1`.
+
+4. **Set Up Python**: Sets up a Python 3.x environment using `actions/setup-python@v2`.
+
+5. **Install Dependencies**: Installs required Python packages:
+   ```bash
+   pip install --upgrade pip
+   pip install PyGithub pyyaml
+   ```
+
+6. **Run Tests and Handle PRs**: Executes a Python script that:
+   - Installs Innovation Engine (`ie`) if not already installed.
+   - Retrieves repository information.
+   - If on a branch other than `main`, it creates or updates a pull request to merge changes into `main`.
+   - Iterates over all Markdown files in the repository:
+     - Runs `ie execute` on each file to test the executable documentation.
+     - If tests fail:
+       - Extracts error logs from the ie.log file, which contains logs of Innovation Engine execution.
+       - Creates or updates a GitHub issue with the error details.
+       - Records the failure in the test results.
+     - If tests pass, records the success.
+   - Posts the test results as a comment on the pull request or outputs them in the workflow logs.
+
+## Customization
+
+You can modify the GitHub Action to suit your project's needs:
+
+- **Event Triggers**: Adjust the `on` section to change when the action runs (e.g., on different branches or file types).
+
+- **Azure Login Step**: If your documentation doesn't require Azure resources, you can remove the Azure login step.
+
+- **Dependencies**: Add or remove Python packages in the `Install dependencies` step as needed for your tests.
+
+- **Testing Script**: Update the Python script to change how tests are executed, how errors are handled, or how results are reported.
+
+- **Secrets Management**: Ensure all necessary secrets are correctly named and stored in your repository's settings.
+
+## Conclusion
+
+By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and prevents broken documentation from being merged into your main branch.
\ No newline at end of file

From 1943aa42ef92929830cb209cee9f7b701a79b679 Mon Sep 17 00:00:00 2001
From: root <root@DESKTOP-MJFFR09>
Date: Tue, 12 Nov 2024 14:17:30 -0800
Subject: [PATCH 2/9] added comments to the gh action to make it easier for
 users to understand it from the file itself

---
 examples/test-exec-doc.yml | 153 +++++++++++++++++++++++++++++++++++++
 1 file changed, 153 insertions(+)

diff --git a/examples/test-exec-doc.yml b/examples/test-exec-doc.yml
index e69de29b..dbd57bad 100644
--- a/examples/test-exec-doc.yml
+++ b/examples/test-exec-doc.yml
@@ -0,0 +1,153 @@
+# Name of the workflow
+name: Run Tests on Push to Main
+
+# Set permissions required for the workflow
+permissions:
+  contents: write           # Allows writing repository contents
+  pull-requests: write      # Allows creating and modifying pull requests
+  id-token: write           # Allows generating tokens for authentication
+  issues: write             # Allows creating and modifying issues
+
+# Define the events that trigger the workflow
+on:
+  push:
+    paths:
+      - '**/*.md'           # Trigger on push to any Markdown file
+  pull_request:
+    paths:
+      - '**/*.md'           # Trigger on pull requests affecting Markdown files
+  workflow_dispatch:        # Allows manual triggering of the workflow
+
+jobs:
+  run-tests:
+    runs-on: ubuntu-latest  # Run the job on the latest Ubuntu runner
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+      with:
+        fetch-depth: 0      # Fetch all history for all branches and tags
+
+    - name: Azure CLI Login
+      uses: azure/login@v1
+      with:
+        client-id: ${{ secrets.AZURE_CLIENT_ID }}             # Azure Client ID from secrets
+        tenant-id: ${{ secrets.AZURE_TENANT_ID }}             # Azure Tenant ID from secrets
+        subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} # Azure Subscription ID from secrets
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x' # Specify Python version 3.x
+
+    - name: Install dependencies
+      run: |
+        pip install --upgrade pip   # Upgrade pip
+        pip install PyGithub pyyaml # Install required Python packages
+
+    - name: Run tests and handle PR
+      env:
+        GITHUB_TOKEN: ${{ secrets.PAT }} # Personal Access Token from secrets
+      run: |
+        python <<EOF
+        import os
+        import subprocess
+        import shutil
+        import github
+
+        # Get GitHub token from environment variable
+        GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
+        # Authenticate with GitHub using PyGithub
+        g = github.Github(GITHUB_TOKEN)
+
+        # Check if 'ie' (Innovation Engine) is installed, install if not
+        if not shutil.which("ie"):
+            # Update package lists and install unzip
+            subprocess.run("sudo apt-get update && sudo apt-get install -y unzip", shell=True, check=True)
+            # Install Innovation Engine CLI
+            subprocess.run("curl -Lks https://aka.ms/install-ie | /bin/bash", shell=True, check=True)
+
+        # Retrieve repository information
+        result = subprocess.run(['git', 'config', '--get', 'remote.origin.url'], capture_output=True, text=True)
+        repo_url = result.stdout.strip()
+        parts = repo_url.split('/')
+        owner = parts[-2]                     # Repository owner
+        repo_name = parts[-1].replace('.git', '') # Repository name
+        # Access the repository using GitHub API
+        repo = g.get_repo(f"{owner}/{repo_name}")
+
+        # Get branch name from environment variable
+        BRANCH_NAME = os.environ['GITHUB_REF'].split('/')[-1]
+        if BRANCH_NAME != "main":
+            # Prepare to create or update a pull request to 'main'
+            PR_TITLE = f"Merge Exec Doc from {BRANCH_NAME} to main"
+            PR_BODY = "This PR attempts to publish your Exec Doc. Running tests on it now using Innovation Engine..."
+
+            # Check if a PR with the same title already exists
+            existing_pulls = repo.get_pulls(state='open', head=f"{owner}:{BRANCH_NAME}", base='main')
+            pr = None
+            for pull in existing_pulls:
+                if pull.title == PR_TITLE:
+                    pr = pull
+                    pr.edit(body=PR_BODY) # Update PR body if it exists
+                    break
+            if pr is None:
+                # Create a new pull request if none exists
+                pr = repo.create_pull(title=PR_TITLE, body=PR_BODY, head=f"{owner}:{BRANCH_NAME}", base='main')
+            pr_number = pr.number
+        else:
+            pr_number = None  # No PR needed if on 'main' branch
+
+        # Initialize a list to store test results
+        results = ["**Test Results**\n\n==========\n\n"]
+        # Walk through all files in the repository
+        for root, dirs, files in os.walk('.'):
+            for file in files:
+                if file.endswith('.md'):  # Process only Markdown files
+                    file_path = os.path.join(root, file)
+                    # Execute the Innovation Engine on the Markdown file
+                    result = subprocess.run(['ie', 'execute', file_path, '--environment', 'github-action'])
+                    if result.returncode != 0:
+                        # If execution fails, extract error log from 'ie.log'
+                        error_log = ''
+                        with open('ie.log', 'r') as log_file:
+                            lines = log_file.readlines()
+                            # Filter out error lines
+                            error_lines = [line for line in lines if "level=error" in line]
+                            error_log = ''.join(error_lines)
+                        # Prepare issue details
+                        doc_path = '/'.join(file_path.split('/')[1:])
+                        issue_title = f"DOC FAILING TESTS: {doc_path}"
+                        issue_body = f"Hey! Your executable document is not working. Please fix the errors given below.\n\nLink to Doc: https://github.com/{owner}/{repo_name}/blob/main/{doc_path}\n\n### Error Details\n\n***{error_log}***"
+                        issues = repo.get_issues(state='open')
+                        # Check if an issue with the same title already exists
+                        existing_issue = next((issue for issue in issues if issue.title == issue_title), None)
+                        if existing_issue:
+                            # Add a reminder comment to the existing issue
+                            existing_issue.create_comment(f"Reminder: Please address the issues in {doc_path}.")
+                            issue_url = existing_issue.html_url
+                            issue_number = existing_issue.number
+                        else:
+                            # Create a new issue if none exists
+                            try:
+                                new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee=owner)
+                            except:
+                                # Assign to default user if assigning to owner fails
+                                new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee='naman-msft')
+                            issue_url = new_issue.html_url
+                            issue_number = new_issue.number
+                        # Append failure result to the results list
+                        results.append(f"**{file_path}**: Tests failed. Issue created.\n\nIssue Details: [Issue #{issue_number}]({issue_url})\n\n==========\n\n")
+                    else:
+                        # Append success result to the results list
+                        results.append(f"**{file_path}**: Tests passed successfully.\n\n==========\n\n")
+
+        # Post the test results as a comment on the pull request
+        if pr_number:
+            pr = repo.get_pull(pr_number)
+            pr.create_issue_comment('\n'.join(results))
+        else:
+            # Print the test results to the workflow log if no PR
+            print('No PR created. Test results:\n' + '\n'.join(results))
+
+        EOF
\ No newline at end of file

From 34ed78b25ac7f4ed5ff9ffc94f2f336e69556799 Mon Sep 17 00:00:00 2001
From: root <root@DESKTOP-MJFFR09>
Date: Tue, 12 Nov 2024 23:32:53 -0800
Subject: [PATCH 3/9] updated exec doc testing action and instruction
 addressing the PR review

---
 examples/test-exec-doc.yml    | 170 ++++++++++++++++++++++++----------
 examples/test-instructions.md |   6 +-
 2 files changed, 122 insertions(+), 54 deletions(-)

diff --git a/examples/test-exec-doc.yml b/examples/test-exec-doc.yml
index dbd57bad..c4fb8f59 100644
--- a/examples/test-exec-doc.yml
+++ b/examples/test-exec-doc.yml
@@ -13,9 +13,6 @@ on:
   push:
     paths:
       - '**/*.md'           # Trigger on push to any Markdown file
-  pull_request:
-    paths:
-      - '**/*.md'           # Trigger on pull requests affecting Markdown files
   workflow_dispatch:        # Allows manual triggering of the workflow
 
 jobs:
@@ -26,28 +23,88 @@ jobs:
     - name: Checkout repository
       uses: actions/checkout@v2
       with:
-        fetch-depth: 0      # Fetch all history for all branches and tags
+        fetch-depth: 0        # Fetch all history for accurate diff
 
+    - name: Check for changed Markdown files
+      id: check_changes
+      run: |
+        # Check if this is the initial commit (all zeros SHA)
+        if [ "${{ github.event.before }}" = "0000000000000000000000000000000000000000" ]; then
+          echo "Initial commit detected."
+
+          # Determine the default branch (e.g., main)
+          if git show-ref --verify --quiet refs/remotes/origin/main; then
+            DEFAULT_BRANCH="main"
+          elif git show-ref --verify --quiet refs/remotes/origin/master; then
+            DEFAULT_BRANCH="master"
+          else
+            echo "Default branch is not 'main' or 'master'. Getting all Markdown files."
+            CHANGED_MARKDOWN_FILES=$(git ls-files '*.md')
+            echo "Markdown files to test:"
+            echo "$CHANGED_MARKDOWN_FILES"
+            echo "changed=true" >> $GITHUB_OUTPUT
+            echo "changed_files<<EOF" >> $GITHUB_OUTPUT
+            echo "$CHANGED_MARKDOWN_FILES" >> $GITHUB_OUTPUT
+            echo "EOF" >> $GITHUB_OUTPUT
+            exit 0
+          fi
+          echo "Default branch is $DEFAULT_BRANCH"
+          
+          # Get the last commit SHA of the default branch
+          UPSTREAM_SHA=$(git rev-parse origin/$DEFAULT_BRANCH)
+          echo "Upstream SHA: $UPSTREAM_SHA"
+          
+          # List changed Markdown files by comparing with the upstream branch
+          CHANGED_MARKDOWN_FILES=$(git diff --name-only $UPSTREAM_SHA ${{ github.sha }} | grep -E '\.md$' || true)
+
+        else
+          # Compare the latest commit with the previous one
+          CHANGED_FILES=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }})
+          echo "Changed files: $CHANGED_FILES"
+          # Filter changed Markdown files
+          CHANGED_MARKDOWN_FILES=$(echo "$CHANGED_FILES" | grep -E '\.md$' || true)
+        fi
+
+        echo "Markdown files to test:"
+        echo "$CHANGED_MARKDOWN_FILES"
+
+        if [ ! -z "$CHANGED_MARKDOWN_FILES" ]; then
+          echo "changed=true" >> $GITHUB_OUTPUT
+          # Set multiline output using the '<<EOF' syntax
+          echo "changed_files<<EOF" >> $GITHUB_OUTPUT
+          echo "$CHANGED_MARKDOWN_FILES" >> $GITHUB_OUTPUT
+          echo "EOF" >> $GITHUB_OUTPUT
+        else
+          echo "changed=false" >> $GITHUB_OUTPUT
+          echo "No Markdown files changed. Exiting."
+          exit 0
+        fi
+        
     - name: Azure CLI Login
-      uses: azure/login@v1
+      if: steps.check_changes.outputs.changed == 'true'
+      uses: azure/login@v2
       with:
         client-id: ${{ secrets.AZURE_CLIENT_ID }}             # Azure Client ID from secrets
         tenant-id: ${{ secrets.AZURE_TENANT_ID }}             # Azure Tenant ID from secrets
         subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} # Azure Subscription ID from secrets
 
     - name: Set up Python
+      if: steps.check_changes.outputs.changed == 'true'
       uses: actions/setup-python@v2
       with:
-        python-version: '3.x' # Specify Python version 3.x
+        python-version: '3.x'  # Specify Python version 3.x
 
     - name: Install dependencies
+      if: steps.check_changes.outputs.changed == 'true'
       run: |
-        pip install --upgrade pip   # Upgrade pip
-        pip install PyGithub pyyaml # Install required Python packages
+        pip install --upgrade pip==24.3.1 || pip install --upgrade pip  # Try pinned version, fallback to latest
+        pip install PyGithub==2.1.1 pyyaml==5.4.1 || pip install PyGithub pyyaml # Install required Python packages
 
     - name: Run tests and handle PR
+      if: steps.check_changes.outputs.changed == 'true'
       env:
-        GITHUB_TOKEN: ${{ secrets.PAT }} # Personal Access Token from secrets
+        GITHUB_TOKEN: ${{ secrets.PAT }}  # Personal Access Token from secrets
+        CHANGED_MARKDOWN_FILES: ${{ steps.check_changes.outputs.changed_files }}  # List of changed Markdown files
       run: |
         python <<EOF
         import os
@@ -64,8 +121,10 @@ jobs:
         if not shutil.which("ie"):
             # Update package lists and install unzip
             subprocess.run("sudo apt-get update && sudo apt-get install -y unzip", shell=True, check=True)
+
             # Install Innovation Engine CLI
-            subprocess.run("curl -Lks https://aka.ms/install-ie | /bin/bash", shell=True, check=True)
+            INNOVATION_ENGINE_VERSION = "v0.2.3"
+            subprocess.run(f"curl -Lks https://aka.ms/install-ie | /bin/bash -s {INNOVATION_ENGINE_VERSION}", shell=True, check=True)
 
         # Retrieve repository information
         result = subprocess.run(['git', 'config', '--get', 'remote.origin.url'], capture_output=True, text=True)
@@ -98,49 +157,58 @@ jobs:
         else:
             pr_number = None  # No PR needed if on 'main' branch
 
+        # Get the list of changed Markdown files
+        changed_files_env = os.getenv('CHANGED_MARKDOWN_FILES', '')
+        changed_files = [f.strip() for f in changed_files_env.split('\n') if f.strip().endswith('.md')]
+
+        if not changed_files:
+            print("No Markdown files to test.")
+            exit(0)
+
         # Initialize a list to store test results
         results = ["**Test Results**\n\n==========\n\n"]
-        # Walk through all files in the repository
-        for root, dirs, files in os.walk('.'):
-            for file in files:
-                if file.endswith('.md'):  # Process only Markdown files
-                    file_path = os.path.join(root, file)
-                    # Execute the Innovation Engine on the Markdown file
-                    result = subprocess.run(['ie', 'execute', file_path, '--environment', 'github-action'])
-                    if result.returncode != 0:
-                        # If execution fails, extract error log from 'ie.log'
-                        error_log = ''
-                        with open('ie.log', 'r') as log_file:
-                            lines = log_file.readlines()
-                            # Filter out error lines
-                            error_lines = [line for line in lines if "level=error" in line]
-                            error_log = ''.join(error_lines)
-                        # Prepare issue details
-                        doc_path = '/'.join(file_path.split('/')[1:])
-                        issue_title = f"DOC FAILING TESTS: {doc_path}"
-                        issue_body = f"Hey! Your executable document is not working. Please fix the errors given below.\n\nLink to Doc: https://github.com/{owner}/{repo_name}/blob/main/{doc_path}\n\n### Error Details\n\n***{error_log}***"
-                        issues = repo.get_issues(state='open')
-                        # Check if an issue with the same title already exists
-                        existing_issue = next((issue for issue in issues if issue.title == issue_title), None)
-                        if existing_issue:
-                            # Add a reminder comment to the existing issue
-                            existing_issue.create_comment(f"Reminder: Please address the issues in {doc_path}.")
-                            issue_url = existing_issue.html_url
-                            issue_number = existing_issue.number
-                        else:
-                            # Create a new issue if none exists
-                            try:
-                                new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee=owner)
-                            except:
-                                # Assign to default user if assigning to owner fails
-                                new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee='naman-msft')
-                            issue_url = new_issue.html_url
-                            issue_number = new_issue.number
-                        # Append failure result to the results list
-                        results.append(f"**{file_path}**: Tests failed. Issue created.\n\nIssue Details: [Issue #{issue_number}]({issue_url})\n\n==========\n\n")
-                    else:
-                        # Append success result to the results list
-                        results.append(f"**{file_path}**: Tests passed successfully.\n\n==========\n\n")
+        # Iterate over changed Markdown files
+        for file_path in changed_files:
+            # Execute the Innovation Engine on the Markdown file
+            result = subprocess.run(['ie', 'execute', file_path, '--environment', 'github-action'])
+            if result.returncode != 0:
+                # If execution fails, extract error log from 'ie.log'
+                error_log = ''
+                try:
+                    with open('ie.log', 'r') as log_file:
+                        lines = log_file.readlines()
+                        # Filter out error lines
+                        error_lines = [line for line in lines if "level=error" in line]
+                        error_log = ''.join(error_lines)
+                except FileNotFoundError:
+                    error_log = "ie.log not found."
+
+                # Prepare issue details
+                doc_path = file_path
+                issue_title = f"DOC FAILING TESTS: {doc_path}"
+                issue_body = f"Hey! Your executable document is not working. Please fix the errors given below.\n\nLink to Doc: https://github.com/{owner}/{repo_name}/blob/main/{doc_path}\n\n### Error Details\n\n***{error_log}***"
+                issues = repo.get_issues(state='open')
+                # Check if an issue with the same title already exists
+                existing_issue = next((issue for issue in issues if issue.title == issue_title), None)
+                if existing_issue:
+                    # Add a reminder comment to the existing issue
+                    existing_issue.create_comment(f"Reminder: Please address the issues in {doc_path}.")
+                    issue_url = existing_issue.html_url
+                    issue_number = existing_issue.number
+                else:
+                    # Create a new issue if none exists
+                    try:
+                        new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee=owner)
+                    except:
+                        # Assign to default user if assigning to owner fails
+                        new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee='naman-msft')
+                    issue_url = new_issue.html_url
+                    issue_number = new_issue.number
+                # Append failure result to the results list
+                results.append(f"**{file_path}**: Tests failed. Issue created.\n\nIssue Details: [Issue #{issue_number}]({issue_url})\n\n==========\n\n")
+            else:
+                # Append success result to the results list
+                results.append(f"**{file_path}**: Tests passed successfully.\n\n==========\n\n")
 
         # Post the test results as a comment on the pull request
         if pr_number:
diff --git a/examples/test-instructions.md b/examples/test-instructions.md
index a211f8b4..8e37dc45 100644
--- a/examples/test-instructions.md
+++ b/examples/test-instructions.md
@@ -2,7 +2,7 @@
 
 ## Overview
 
-This guide explains how to set up and use a GitHub Action that automates testing of your Markdown documentation using Innovation Engine. The action ensures your executable documents are accurate and robust by running tests on them. It will create issues for any errors found and prevent pull requests from merging until those errors are resolved.
+This guide explains how to set up and use a GitHub Action that automates testing of your Markdown documentation using Innovation Engine. The action ensures your executable documents are accurate and robust by running tests on them. It will create issues for any errors found and, when combined with branch protection rules, can prevent pull requests from merging until those errors are resolved.
 
 ## Prerequisites
 
@@ -28,7 +28,7 @@ The GitHub Action performs the following steps:
 
 2. **Checkout Repository**: Uses `actions/checkout@v2` to clone the repository.
 
-3. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v1`.
+3. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v2`.
 
 4. **Set Up Python**: Sets up a Python 3.x environment using `actions/setup-python@v2`.
 
@@ -67,4 +67,4 @@ You can modify the GitHub Action to suit your project's needs:
 
 ## Conclusion
 
-By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and prevents broken documentation from being merged into your main branch.
\ No newline at end of file
+By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and, combine with branch protection rules, prevents broken documentation from being merged into your main branch.
\ No newline at end of file

From 7954838eee50142c86c8c9287fd896fc99478f18 Mon Sep 17 00:00:00 2001
From: naman-msft <namanparikh@microsoft.com>
Date: Tue, 12 Nov 2024 23:36:12 -0800
Subject: [PATCH 4/9] finalized action

---
 examples/test-instructions.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/examples/test-instructions.md b/examples/test-instructions.md
index 8e37dc45..23a179e3 100644
--- a/examples/test-instructions.md
+++ b/examples/test-instructions.md
@@ -67,4 +67,4 @@ You can modify the GitHub Action to suit your project's needs:
 
 ## Conclusion
 
-By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and, combine with branch protection rules, prevents broken documentation from being merged into your main branch.
\ No newline at end of file
+By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and, combined with branch protection rules, prevents broken documentation from being merged into your main branch.
\ No newline at end of file

From c46a4f7a95be054a37ef44677cff62bcb689259b Mon Sep 17 00:00:00 2001
From: naman-msft <namanparikh@microsoft.com>
Date: Wed, 13 Nov 2024 10:05:51 -0800
Subject: [PATCH 5/9] updated action and instructions based on PR review

---
 examples/test-exec-doc.yml    |  8 ++---
 examples/test-instructions.md | 60 ++++++++++++++++++++++++-----------
 2 files changed, 43 insertions(+), 25 deletions(-)

diff --git a/examples/test-exec-doc.yml b/examples/test-exec-doc.yml
index c4fb8f59..83670a8e 100644
--- a/examples/test-exec-doc.yml
+++ b/examples/test-exec-doc.yml
@@ -92,7 +92,7 @@ jobs:
       if: steps.check_changes.outputs.changed == 'true'
       uses: actions/setup-python@v2
       with:
-        python-version: '3.x'  # Specify Python version 3.x
+        python-version: '3.12'  # Specify Python version 3.x
 
     - name: Install dependencies
       if: steps.check_changes.outputs.changed == 'true'
@@ -161,10 +161,6 @@ jobs:
         changed_files_env = os.getenv('CHANGED_MARKDOWN_FILES', '')
         changed_files = [f.strip() for f in changed_files_env.split('\n') if f.strip().endswith('.md')]
 
-        if not changed_files:
-            print("No Markdown files to test.")
-            exit(0)
-
         # Initialize a list to store test results
         results = ["**Test Results**\n\n==========\n\n"]
         # Iterate over changed Markdown files
@@ -201,7 +197,7 @@ jobs:
                         new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee=owner)
                     except:
                         # Assign to default user if assigning to owner fails
-                        new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee='naman-msft')
+                        new_issue = repo.create_issue(title=issue_title, body=issue_body)
                     issue_url = new_issue.html_url
                     issue_number = new_issue.number
                 # Append failure result to the results list
diff --git a/examples/test-instructions.md b/examples/test-instructions.md
index 23a179e3..2a25e360 100644
--- a/examples/test-instructions.md
+++ b/examples/test-instructions.md
@@ -24,32 +24,52 @@ Before setting up the GitHub Action, you need the following:
 
 The GitHub Action performs the following steps:
 
-1. **Trigger**: Runs on push or pull request events affecting Markdown files, or when manually triggered.
+1. **Trigger**: Runs on push events affecting Markdown files or when manually triggered.
 
-2. **Checkout Repository**: Uses `actions/checkout@v2` to clone the repository.
+2. **Checkout Repository**: Uses `actions/checkout@v2` to clone the repository with full history for accurate diffs.
 
-3. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v2`.
+3. **Check for Changed Markdown Files**:
 
-4. **Set Up Python**: Sets up a Python 3.x environment using `actions/setup-python@v2`.
+   - **Initial Commit**:
+     - If this is the initial commit and the default branch is `main` or `master`, it compares changes with the default branch.
+     - If the default branch is not `main` or `master`, it considers all Markdown files in the repository.
+   - **Subsequent Commits**:
+     - Compares changes between the latest commit and the previous one.
+   - **Outputs**:
+     - Sets an output variable indicating whether Markdown files have changed.
+     - Lists the changed Markdown files to be tested.
+
+4. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v2` (only if Markdown files have changed).
+
+5. **Set Up Python**: Sets up a Python 3.12 environment using `actions/setup-python@v2` (only if Markdown files have changed).
+
+6. **Install Dependencies**: Installs required Python packages (only if Markdown files have changed):
 
-5. **Install Dependencies**: Installs required Python packages:
    ```bash
-   pip install --upgrade pip
-   pip install PyGithub pyyaml
+   pip install --upgrade pip==24.3.1 || pip install --upgrade pip  # Try specific pip version, fallback to latest
+   pip install PyGithub==2.1.1 pyyaml==5.4.1 || pip install PyGithub pyyaml  # Try specific package versions, fallback to latest
    ```
 
-6. **Run Tests and Handle PRs**: Executes a Python script that:
-   - Installs Innovation Engine (`ie`) if not already installed.
-   - Retrieves repository information.
-   - If on a branch other than `main`, it creates or updates a pull request to merge changes into `main`.
-   - Iterates over all Markdown files in the repository:
+7. **Run Tests and Handle PRs**: Executes a Python script that (only if Markdown files have changed):
+
+   - **Innovation Engine Installation**: Installs `ie` (Innovation Engine) if not already installed.
+   - **Repository Information**: Retrieves repository details such as owner, name, and branch.
+   - **Pull Request Handling**:
+     - If on a branch other than `main`, it creates or updates a pull request to merge changes into `main`.
+     - Checks for existing pull requests with the same title to avoid duplicates.
+   - **Testing Markdown Files**:
+     - Iterates over the list of changed Markdown files.
      - Runs `ie execute` on each file to test the executable documentation.
-     - If tests fail:
-       - Extracts error logs from the ie.log file, which contains logs of Innovation Engine execution.
+     - **On Test Failure**:
+       - Extracts error logs from the ie.log file.
        - Creates or updates a GitHub issue with the error details.
        - Records the failure in the test results.
-     - If tests pass, records the success.
-   - Posts the test results as a comment on the pull request or outputs them in the workflow logs.
+       - Does not assign the issue to any user to avoid permission issues.
+     - **On Test Success**:
+       - Records the success in the test results.
+   - **Reporting Results**:
+     - Posts the test results as a comment on the pull request.
+     - If no pull request exists (i.e., on the `main` branch), outputs the results in the workflow logs.
 
 ## Customization
 
@@ -59,11 +79,13 @@ You can modify the GitHub Action to suit your project's needs:
 
 - **Azure Login Step**: If your documentation doesn't require Azure resources, you can remove the Azure login step.
 
-- **Dependencies**: Add or remove Python packages in the `Install dependencies` step as needed for your tests.
+- **Python Version and Dependencies**:
+  - Update the `python-version` in the `Set Up Python` step to match your project's requirements.
+  - In the `Install Dependencies` step, specify the versions of `pip`, `PyGithub`, and `pyyaml` as needed. The action tries to install specific versions first and falls back to the latest versions if the specified versions are not available.
 
-- **Testing Script**: Update the Python script to change how tests are executed, how errors are handled, or how results are reported.
+- **Testing Script**: Update the Python script to change how tests are executed, how errors are handled, or how results are reported. For example, you might customize the error extraction logic or modify the issue creation process.
 
-- **Secrets Management**: Ensure all necessary secrets are correctly named and stored in your repository's settings.
+- **Secrets Management**: Ensure all necessary secrets (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`, `PAT`) are correctly named and stored in your repository's settings under **Settings > Secrets and variables > Actions**.
 
 ## Conclusion
 

From a2cdd70a5fa6d2cdaf6dd1dd969b7ad6b729cfd8 Mon Sep 17 00:00:00 2001
From: naman-msft <namanparikh@microsoft.com>
Date: Wed, 13 Nov 2024 20:44:50 -0800
Subject: [PATCH 6/9] updated action and instructions following PR review

---
 .../test-exec-docs.yml}                       | 25 +++++++++++++++----
 .../{ => test-exec-docs}/test-instructions.md |  2 ++
 2 files changed, 22 insertions(+), 5 deletions(-)
 rename examples/{test-exec-doc.yml => test-exec-docs/test-exec-docs.yml} (92%)
 rename examples/{ => test-exec-docs}/test-instructions.md (96%)

diff --git a/examples/test-exec-doc.yml b/examples/test-exec-docs/test-exec-docs.yml
similarity index 92%
rename from examples/test-exec-doc.yml
rename to examples/test-exec-docs/test-exec-docs.yml
index 83670a8e..c1d761d5 100644
--- a/examples/test-exec-doc.yml
+++ b/examples/test-exec-docs/test-exec-docs.yml
@@ -14,11 +14,19 @@ on:
     paths:
       - '**/*.md'           # Trigger on push to any Markdown file
   workflow_dispatch:        # Allows manual triggering of the workflow
+    inputs:
+      create_pr:
+        description: 'Set to true to enable PR creation logic'
+        required: true
+        default: 'false'
 
 jobs:
   run-tests:
     runs-on: ubuntu-latest  # Run the job on the latest Ubuntu runner
-
+    
+    env:
+      # Set CREATE_PR to true if the event is a push, otherwise use the value from workflow_dispatch input or default to false
+      CREATE_PR: ${{ github.event_name == 'push' && 'true' || github.event.inputs.create_pr || 'false' }}
     steps:
     - name: Checkout repository
       uses: actions/checkout@v2
@@ -97,14 +105,19 @@ jobs:
     - name: Install dependencies
       if: steps.check_changes.outputs.changed == 'true'
       run: |
-        pip install --upgrade pip==24.3.1 || pip install --upgrade pip  # Try pinned version, fallback to latest
-        pip install PyGithub==2.1.1 pyyaml==5.4.1 || pip install PyGithub pyyaml # Install required Python packages
+        # Try installing pinned versions
+        pip install --upgrade pip==24.3.1 || pip install --upgrade pip
+        pip install setuptools wheel
+
+        # Try installing required packages with pinned versions
+        pip install PyGithub==2.1.1 pyyaml==5.4.1 || pip install PyGithub pyyaml
 
     - name: Run tests and handle PR
       if: steps.check_changes.outputs.changed == 'true'
       env:
         GITHUB_TOKEN: ${{ secrets.PAT }}  # Personal Access Token from secrets
         CHANGED_MARKDOWN_FILES: ${{ steps.check_changes.outputs.changed_files }}  # List of changed Markdown files
+        CREATE_PR: ${{ env.CREATE_PR }}
       run: |
         python <<EOF
         import os
@@ -114,6 +127,8 @@ jobs:
 
         # Get GitHub token from environment variable
         GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
+        CREATE_PR = os.getenv('CREATE_PR', 'false').lower() == 'true'
+
         # Authenticate with GitHub using PyGithub
         g = github.Github(GITHUB_TOKEN)
 
@@ -150,10 +165,10 @@ jobs:
                     pr = pull
                     pr.edit(body=PR_BODY) # Update PR body if it exists
                     break
-            if pr is None:
+            if pr is None and CREATE_PR:
                 # Create a new pull request if none exists
                 pr = repo.create_pull(title=PR_TITLE, body=PR_BODY, head=f"{owner}:{BRANCH_NAME}", base='main')
-            pr_number = pr.number
+            pr_number = pr.number if pr else None
         else:
             pr_number = None  # No PR needed if on 'main' branch
 
diff --git a/examples/test-instructions.md b/examples/test-exec-docs/test-instructions.md
similarity index 96%
rename from examples/test-instructions.md
rename to examples/test-exec-docs/test-instructions.md
index 2a25e360..f160a01e 100644
--- a/examples/test-instructions.md
+++ b/examples/test-exec-docs/test-instructions.md
@@ -57,6 +57,8 @@ The GitHub Action performs the following steps:
    - **Pull Request Handling**:
      - If on a branch other than `main`, it creates or updates a pull request to merge changes into `main`.
      - Checks for existing pull requests with the same title to avoid duplicates.
+     - **CREATE_PR Environment Variable**:
+       - The `CREATE_PR` variable is set to `true` if the event is a push, otherwise it uses the value from `workflow_dispatch` input or defaults to `false`.
    - **Testing Markdown Files**:
      - Iterates over the list of changed Markdown files.
      - Runs `ie execute` on each file to test the executable documentation.

From aa3f250ac8790e0e0d37a127430d3de829809911 Mon Sep 17 00:00:00 2001
From: naman-msft <namanparikh@microsoft.com>
Date: Thu, 14 Nov 2024 09:33:49 -0800
Subject: [PATCH 7/9] updated dependencies version numbers

---
 examples/test-exec-docs/test-exec-docs.yml | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/examples/test-exec-docs/test-exec-docs.yml b/examples/test-exec-docs/test-exec-docs.yml
index c1d761d5..1f23b694 100644
--- a/examples/test-exec-docs/test-exec-docs.yml
+++ b/examples/test-exec-docs/test-exec-docs.yml
@@ -105,12 +105,10 @@ jobs:
     - name: Install dependencies
       if: steps.check_changes.outputs.changed == 'true'
       run: |
-        # Try installing pinned versions
-        pip install --upgrade pip==24.3.1 || pip install --upgrade pip
-        pip install setuptools wheel
-
         # Try installing required packages with pinned versions
-        pip install PyGithub==2.1.1 pyyaml==5.4.1 || pip install PyGithub pyyaml
+        pip install --upgrade pip==24.3.1 
+        pip install setuptools wheel
+        pip install PyGithub==2.1.1 pyyaml==6.0.2 
 
     - name: Run tests and handle PR
       if: steps.check_changes.outputs.changed == 'true'

From 0f2d8505f0f5f00d4e7fce2836b7df1d18bf9179 Mon Sep 17 00:00:00 2001
From: naman-msft <namanparikh@microsoft.com>
Date: Mon, 18 Nov 2024 07:45:35 -0800
Subject: [PATCH 8/9] updated action and instructions

---
 examples/test-exec-docs/README.md            |  89 ++++++++
 examples/test-exec-docs/test-exec-docs.yml   | 208 +++++++++----------
 examples/test-exec-docs/test-instructions.md |  94 ---------
 3 files changed, 185 insertions(+), 206 deletions(-)
 create mode 100644 examples/test-exec-docs/README.md
 delete mode 100644 examples/test-exec-docs/test-instructions.md

diff --git a/examples/test-exec-docs/README.md b/examples/test-exec-docs/README.md
new file mode 100644
index 00000000..4ef5b20f
--- /dev/null
+++ b/examples/test-exec-docs/README.md
@@ -0,0 +1,89 @@
+# Automating Documentation Testing with Innovation Engine
+
+## Overview
+
+This guide explains how to set up and use a GitHub Action that automates testing of your Markdown documentation using Innovation Engine. The action ensures your executable documents are accurate and robust by running tests on them. It will create comments on pull requests for any errors found and, when combined with branch protection rules, can prevent pull requests from merging until those errors are resolved.
+
+## Prerequisites
+
+Before setting up the GitHub Action, you need the following:
+
+- **Azure Credentials**: The action logs into Azure using the Azure CLI, so you need to provide the following repository secrets:
+  - `AZURE_CLIENT_ID`
+  - `AZURE_TENANT_ID`
+  - `AZURE_SUBSCRIPTION_ID`
+
+  To obtain these values:
+  1. Register an application in Azure Active Directory.
+  2. Assign the necessary permissions to the application.
+  3. Note the application (client) ID, tenant ID, and subscription ID.
+
+- **GitHub Personal Access Token (PAT)**: The action interacts with the GitHub API to create comments on pull requests. Store your PAT in a repository secret named `PAT`.
+
+## GitHub Action Workflow
+
+The GitHub Action performs the following steps:
+
+1. **Trigger**: Runs on pull request events when a pull request is opened or updated.
+
+2. **Checkout Repository**: Uses `actions/checkout@v3` to clone the repository with full history for accurate diffs.
+
+3. **Determine Commit SHAs**:
+   - Identifies the base and head commits of the pull request.
+   - Calculates the number of commits between the base and head.
+   - Sets output variables for use in subsequent steps.
+
+4. **Check for Changed Markdown Files**:
+
+   - Compares changes between the base and head commits.
+   - Generates a list of changed Markdown files to be tested.
+   - If no Markdown files have changed, the workflow exits early.
+
+5. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v2` (only if Markdown files have changed).
+
+6. **Set Up Python**: Sets up a Python 3.12 environment using `actions/setup-python@v4` (only if Markdown files have changed).
+
+7. **Install Dependencies**: Installs required Python packages (only if Markdown files have changed):
+
+   ```bash
+   pip install --upgrade pip==24.3.1 
+   pip install setuptools wheel
+   pip install PyGithub==2.1.1 pyyaml==6.0.2 
+   ```
+
+8. **Run Tests and Handle Pull Request**: Executes a Python script that (only if Markdown files have changed):
+
+   - **Innovation Engine Installation**: Installs `ie` (Innovation Engine) if not already installed.
+   - **Repository Information**: Retrieves repository details such as owner, name, and pull request number.
+   - **Testing Markdown Files**:
+     - Iterates over the list of changed Markdown files.
+     - Runs `ie execute` on each file to test the executable documentation.
+     - **On Test Failure**:
+       - Extracts error logs from the `ie.log` file.
+       - Creates a comment on the pull request with the error details.
+       - Records the failure in the test results.
+     - **On Test Success**:
+       - Records the success in the test results.
+   - **Reporting Results**:
+     - Posts the test results as a comment on the pull request.
+     - If any failures occurred, exits with a non-zero status to fail the workflow.
+
+## Customization
+
+You can modify the GitHub Action to suit your project's needs:
+
+- **Event Triggers**: Adjust the `on` section to change when the action runs. Currently, it triggers on pull request events (`opened` and `synchronize`).
+
+- **Azure Login Step**: If your documentation doesn't require Azure resources, you can remove the Azure login step.
+
+- **Python Version and Dependencies**:
+  - Update the `python-version` in the **Set Up Python** step to match your project's requirements.
+  - In the **Install Dependencies** step, specify the versions of `pip`, `PyGithub`, and `pyyaml` as needed.
+
+- **Testing Script**: Update the Python script to change how tests are executed, how errors are handled, or how results are reported. For example, you might customize the error extraction logic or modify the comment creation process.
+
+- **Secrets Management**: Ensure all necessary secrets (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`, `PAT`) are correctly named and stored in your repository's settings under **Settings > Secrets and variables > Actions**.
+
+## Conclusion
+
+By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and, combined with branch protection rules, prevents broken documentation from being merged into your main branch.
\ No newline at end of file
diff --git a/examples/test-exec-docs/test-exec-docs.yml b/examples/test-exec-docs/test-exec-docs.yml
index 1f23b694..21ff3ad2 100644
--- a/examples/test-exec-docs/test-exec-docs.yml
+++ b/examples/test-exec-docs/test-exec-docs.yml
@@ -1,5 +1,5 @@
 # Name of the workflow
-name: Run Tests on Push to Main
+name: Innovation Engine Tests
 
 # Set permissions required for the workflow
 permissions:
@@ -10,81 +10,92 @@ permissions:
 
 # Define the events that trigger the workflow
 on:
-  push:
-    paths:
-      - '**/*.md'           # Trigger on push to any Markdown file
-  workflow_dispatch:        # Allows manual triggering of the workflow
-    inputs:
-      create_pr:
-        description: 'Set to true to enable PR creation logic'
-        required: true
-        default: 'false'
+  pull_request:
+    types: [opened, synchronize]  # Trigger on PR creation and updates
 
 jobs:
   run-tests:
     runs-on: ubuntu-latest  # Run the job on the latest Ubuntu runner
     
     env:
-      # Set CREATE_PR to true if the event is a push, otherwise use the value from workflow_dispatch input or default to false
-      CREATE_PR: ${{ github.event_name == 'push' && 'true' || github.event.inputs.create_pr || 'false' }}
+      CREATE_PR: 'false'  # No need to create PR as it's already a PR event
+    
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
       with:
         fetch-depth: 0        # Fetch all history for accurate diff
 
+    - name: Determine commit SHAs
+      id: commits
+      run: |
+        # Set BASE_SHA to the base of the PR
+        BASE_SHA="${{ github.event.pull_request.base.sha }}"
+        # Set HEAD_SHA to the head of the PR
+        HEAD_SHA="${{ github.event.pull_request.head.sha }}"
+        
+        echo "Base SHA: $BASE_SHA"
+        echo "Head SHA: $HEAD_SHA"
+
+        # Count the number of commits between base and head
+        COMMIT_COUNT=$(git rev-list --count "${BASE_SHA}..${HEAD_SHA}")
+        echo "Commit count since base: $COMMIT_COUNT"
+
+        if [ "$COMMIT_COUNT" -eq 1 ]; then
+          FINAL_BASE_SHA="$BASE_SHA"
+          echo "This is the first commit on the branch."
+        else
+          # Ensure HEAD_SHA has a parent commit
+          PARENT_SHA=$(git rev-parse "${HEAD_SHA}^")
+          
+          if [ $? -ne 0 ] || [ -z "$PARENT_SHA" ]; then
+            echo "Error: Unable to find parent commit of HEAD_SHA."
+            exit 1
+          fi
+
+          FINAL_BASE_SHA="$PARENT_SHA"
+          echo "This is not the first commit on the branch."
+          echo "Parent SHA: $PARENT_SHA"
+        fi
+
+        # Set outputs for subsequent steps
+        echo "final_base_sha=$FINAL_BASE_SHA" >> $GITHUB_OUTPUT
+        echo "head_sha=$HEAD_SHA" >> $GITHUB_OUTPUT
+
     - name: Check for changed Markdown files
       id: check_changes
       run: |
-        # Check if this is the initial commit (all zeros SHA)
-        if [ "${{ github.event.before }}" = "0000000000000000000000000000000000000000" ]; then
-          echo "Initial commit detected."
-
-          # Determine the default branch (e.g., main)
-          if git show-ref --verify --quiet refs/remotes/origin/main; then
-            DEFAULT_BRANCH="main"
-          elif git show-ref --verify --quiet refs/remotes/origin/master; then
-            DEFAULT_BRANCH="master"
-          else
-            echo "Default branch is not 'main' or 'master'. Getting all Markdown files."
-            CHANGED_MARKDOWN_FILES=$(git ls-files '*.md')
-            echo "Markdown files to test:"
-            echo "$CHANGED_MARKDOWN_FILES"
-            echo "changed=true" >> $GITHUB_OUTPUT
-            echo "changed_files<<EOF" >> $GITHUB_OUTPUT
-            echo "$CHANGED_MARKDOWN_FILES" >> $GITHUB_OUTPUT
-            echo "EOF" >> $GITHUB_OUTPUT
-            exit 0
-          fi
-          echo "Default branch is $DEFAULT_BRANCH"
-          
-          # Get the last commit SHA of the default branch
-          UPSTREAM_SHA=$(git rev-parse origin/$DEFAULT_BRANCH)
-          echo "Upstream SHA: $UPSTREAM_SHA"
-          
-          # List changed Markdown files by comparing with the upstream branch
-          CHANGED_MARKDOWN_FILES=$(git diff --name-only $UPSTREAM_SHA ${{ github.sha }} | grep -E '\.md$' || true)
+        FINAL_BASE_SHA="${{ steps.commits.outputs.final_base_sha }}"
+        HEAD_SHA="${{ steps.commits.outputs.head_sha }}"
 
-        else
-          # Compare the latest commit with the previous one
-          CHANGED_FILES=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }})
-          echo "Changed files: $CHANGED_FILES"
-          # Filter changed Markdown files
-          CHANGED_MARKDOWN_FILES=$(echo "$CHANGED_FILES" | grep -E '\.md$' || true)
+        # Ensure FINAL_BASE_SHA and HEAD_SHA are set
+        if [ -z "${FINAL_BASE_SHA}" ] || [ -z "${HEAD_SHA}" ]; then
+          echo "Error: FINAL_BASE_SHA or HEAD_SHA is not set."
+          exit 1
         fi
 
+        echo "Final Base SHA: $FINAL_BASE_SHA"
+        echo "Head SHA: $HEAD_SHA"
+
+        CHANGED_FILES=$(git diff --name-only "${FINAL_BASE_SHA}" "${HEAD_SHA}" --)
+
+        echo "CHANGED_FILES<<EOF" >> $GITHUB_ENV
+        echo "$CHANGED_FILES" >> $GITHUB_ENV
+        echo "EOF" >> $GITHUB_ENV
+        
+        CHANGED_MARKDOWN_FILES=$(echo "$CHANGED_FILES" | grep -E '\.md$' || true)
+
         echo "Markdown files to test:"
         echo "$CHANGED_MARKDOWN_FILES"
 
-        if [ ! -z "$CHANGED_MARKDOWN_FILES" ]; then
+        if [ -n "$CHANGED_MARKDOWN_FILES" ]; then
           echo "changed=true" >> $GITHUB_OUTPUT
-          # Set multiline output using the '<<EOF' syntax
           echo "changed_files<<EOF" >> $GITHUB_OUTPUT
           echo "$CHANGED_MARKDOWN_FILES" >> $GITHUB_OUTPUT
           echo "EOF" >> $GITHUB_OUTPUT
         else
           echo "changed=false" >> $GITHUB_OUTPUT
-          echo "No Markdown files changed. Exiting."
+          echo "No Markdown files changed. Exiting." >&2
           exit 0
         fi
         
@@ -98,14 +109,14 @@ jobs:
 
     - name: Set up Python
       if: steps.check_changes.outputs.changed == 'true'
-      uses: actions/setup-python@v2
+      uses: actions/setup-python@v4
       with:
         python-version: '3.12'  # Specify Python version 3.x
 
     - name: Install dependencies
       if: steps.check_changes.outputs.changed == 'true'
       run: |
-        # Try installing required packages with pinned versions
+        # Install required packages with pinned versions
         pip install --upgrade pip==24.3.1 
         pip install setuptools wheel
         pip install PyGithub==2.1.1 pyyaml==6.0.2 
@@ -116,19 +127,30 @@ jobs:
         GITHUB_TOKEN: ${{ secrets.PAT }}  # Personal Access Token from secrets
         CHANGED_MARKDOWN_FILES: ${{ steps.check_changes.outputs.changed_files }}  # List of changed Markdown files
         CREATE_PR: ${{ env.CREATE_PR }}
+        PR_NUMBER: ${{ github.event.pull_request.number }}  # Add PR_NUMBER environment variable
       run: |
         python <<EOF
         import os
         import subprocess
         import shutil
-        import github
+        from github import Github
 
         # Get GitHub token from environment variable
         GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
-        CREATE_PR = os.getenv('CREATE_PR', 'false').lower() == 'true'
+        PR_NUMBER = os.getenv('PR_NUMBER')  # Retrieve PR_NUMBER
+
+        if not PR_NUMBER:
+            print("Error: PR_NUMBER is not set.")
+            exit(1)
+
+        try:
+            pr_number = int(PR_NUMBER)
+        except ValueError:
+            print(f"Error: Invalid PR_NUMBER '{PR_NUMBER}'.")
+            exit(1)
 
         # Authenticate with GitHub using PyGithub
-        g = github.Github(GITHUB_TOKEN)
+        g = Github(GITHUB_TOKEN)
 
         # Check if 'ie' (Innovation Engine) is installed, install if not
         if not shutil.which("ie"):
@@ -140,35 +162,8 @@ jobs:
             subprocess.run(f"curl -Lks https://aka.ms/install-ie | /bin/bash -s {INNOVATION_ENGINE_VERSION}", shell=True, check=True)
 
         # Retrieve repository information
-        result = subprocess.run(['git', 'config', '--get', 'remote.origin.url'], capture_output=True, text=True)
-        repo_url = result.stdout.strip()
-        parts = repo_url.split('/')
-        owner = parts[-2]                     # Repository owner
-        repo_name = parts[-1].replace('.git', '') # Repository name
-        # Access the repository using GitHub API
-        repo = g.get_repo(f"{owner}/{repo_name}")
-
-        # Get branch name from environment variable
-        BRANCH_NAME = os.environ['GITHUB_REF'].split('/')[-1]
-        if BRANCH_NAME != "main":
-            # Prepare to create or update a pull request to 'main'
-            PR_TITLE = f"Merge Exec Doc from {BRANCH_NAME} to main"
-            PR_BODY = "This PR attempts to publish your Exec Doc. Running tests on it now using Innovation Engine..."
-
-            # Check if a PR with the same title already exists
-            existing_pulls = repo.get_pulls(state='open', head=f"{owner}:{BRANCH_NAME}", base='main')
-            pr = None
-            for pull in existing_pulls:
-                if pull.title == PR_TITLE:
-                    pr = pull
-                    pr.edit(body=PR_BODY) # Update PR body if it exists
-                    break
-            if pr is None and CREATE_PR:
-                # Create a new pull request if none exists
-                pr = repo.create_pull(title=PR_TITLE, body=PR_BODY, head=f"{owner}:{BRANCH_NAME}", base='main')
-            pr_number = pr.number if pr else None
-        else:
-            pr_number = None  # No PR needed if on 'main' branch
+        repo_full_name = os.getenv('GITHUB_REPOSITORY')
+        repo = g.get_repo(repo_full_name)
 
         # Get the list of changed Markdown files
         changed_files_env = os.getenv('CHANGED_MARKDOWN_FILES', '')
@@ -176,55 +171,44 @@ jobs:
 
         # Initialize a list to store test results
         results = ["**Test Results**\n\n==========\n\n"]
+
+        any_failures = False
+
         # Iterate over changed Markdown files
         for file_path in changed_files:
             # Execute the Innovation Engine on the Markdown file
             result = subprocess.run(['ie', 'execute', file_path, '--environment', 'github-action'])
             if result.returncode != 0:
+                any_failures = True
                 # If execution fails, extract error log from 'ie.log'
                 error_log = ''
+                
                 try:
                     with open('ie.log', 'r') as log_file:
                         lines = log_file.readlines()
                         # Filter out error lines
                         error_lines = [line for line in lines if "level=error" in line]
-                        error_log = ''.join(error_lines)
+                        if error_lines:
+                            error_log = error_lines[-1]  # Get the last error line
+                        else:
+                            error_log = "No error message found in ie.log"
                 except FileNotFoundError:
                     error_log = "ie.log not found."
 
-                # Prepare issue details
-                doc_path = file_path
-                issue_title = f"DOC FAILING TESTS: {doc_path}"
-                issue_body = f"Hey! Your executable document is not working. Please fix the errors given below.\n\nLink to Doc: https://github.com/{owner}/{repo_name}/blob/main/{doc_path}\n\n### Error Details\n\n***{error_log}***"
-                issues = repo.get_issues(state='open')
-                # Check if an issue with the same title already exists
-                existing_issue = next((issue for issue in issues if issue.title == issue_title), None)
-                if existing_issue:
-                    # Add a reminder comment to the existing issue
-                    existing_issue.create_comment(f"Reminder: Please address the issues in {doc_path}.")
-                    issue_url = existing_issue.html_url
-                    issue_number = existing_issue.number
-                else:
-                    # Create a new issue if none exists
-                    try:
-                        new_issue = repo.create_issue(title=issue_title, body=issue_body, assignee=owner)
-                    except:
-                        # Assign to default user if assigning to owner fails
-                        new_issue = repo.create_issue(title=issue_title, body=issue_body)
-                    issue_url = new_issue.html_url
-                    issue_number = new_issue.number
                 # Append failure result to the results list
-                results.append(f"**{file_path}**: Tests failed. Issue created.\n\nIssue Details: [Issue #{issue_number}]({issue_url})\n\n==========\n\n")
+                results.append(f"**{file_path}**: Tests failed.\n\nError Details:\n\n***{error_log}***\n\n==========\n\n")
             else:
                 # Append success result to the results list
                 results.append(f"**{file_path}**: Tests passed successfully.\n\n==========\n\n")
 
         # Post the test results as a comment on the pull request
-        if pr_number:
-            pr = repo.get_pull(pr_number)
-            pr.create_issue_comment('\n'.join(results))
+        pr = repo.get_pull(pr_number)
+        pr.create_issue_comment('\n'.join(results))
+
+        # If any failures occurred, exit with a non-zero status to fail the workflow
+        if any_failures:
+            exit(1)
         else:
-            # Print the test results to the workflow log if no PR
-            print('No PR created. Test results:\n' + '\n'.join(results))
+            exit(0)
 
         EOF
\ No newline at end of file
diff --git a/examples/test-exec-docs/test-instructions.md b/examples/test-exec-docs/test-instructions.md
deleted file mode 100644
index f160a01e..00000000
--- a/examples/test-exec-docs/test-instructions.md
+++ /dev/null
@@ -1,94 +0,0 @@
-# Automating Documentation Testing with Innovation Engine
-
-## Overview
-
-This guide explains how to set up and use a GitHub Action that automates testing of your Markdown documentation using Innovation Engine. The action ensures your executable documents are accurate and robust by running tests on them. It will create issues for any errors found and, when combined with branch protection rules, can prevent pull requests from merging until those errors are resolved.
-
-## Prerequisites
-
-Before setting up the GitHub Action, you need the following:
-
-- **Azure Credentials**: The action logs into Azure using the Azure CLI, so you need to provide the following repository secrets:
-  - `AZURE_CLIENT_ID`
-  - `AZURE_TENANT_ID`
-  - `AZURE_SUBSCRIPTION_ID`
-
-  To obtain these values:
-  1. Register an application in Azure Active Directory.
-  2. Assign the necessary permissions to the application.
-  3. Note the application (client) ID, tenant ID, and subscription ID.
-
-- **GitHub Personal Access Token (PAT)**: The action interacts with the GitHub API to create issues and pull requests. Store your PAT in a repository secret named `PAT`.
-
-## GitHub Action Workflow
-
-The GitHub Action performs the following steps:
-
-1. **Trigger**: Runs on push events affecting Markdown files or when manually triggered.
-
-2. **Checkout Repository**: Uses `actions/checkout@v2` to clone the repository with full history for accurate diffs.
-
-3. **Check for Changed Markdown Files**:
-
-   - **Initial Commit**:
-     - If this is the initial commit and the default branch is `main` or `master`, it compares changes with the default branch.
-     - If the default branch is not `main` or `master`, it considers all Markdown files in the repository.
-   - **Subsequent Commits**:
-     - Compares changes between the latest commit and the previous one.
-   - **Outputs**:
-     - Sets an output variable indicating whether Markdown files have changed.
-     - Lists the changed Markdown files to be tested.
-
-4. **Azure CLI Login**: Logs into Azure using the provided credentials with `azure/login@v2` (only if Markdown files have changed).
-
-5. **Set Up Python**: Sets up a Python 3.12 environment using `actions/setup-python@v2` (only if Markdown files have changed).
-
-6. **Install Dependencies**: Installs required Python packages (only if Markdown files have changed):
-
-   ```bash
-   pip install --upgrade pip==24.3.1 || pip install --upgrade pip  # Try specific pip version, fallback to latest
-   pip install PyGithub==2.1.1 pyyaml==5.4.1 || pip install PyGithub pyyaml  # Try specific package versions, fallback to latest
-   ```
-
-7. **Run Tests and Handle PRs**: Executes a Python script that (only if Markdown files have changed):
-
-   - **Innovation Engine Installation**: Installs `ie` (Innovation Engine) if not already installed.
-   - **Repository Information**: Retrieves repository details such as owner, name, and branch.
-   - **Pull Request Handling**:
-     - If on a branch other than `main`, it creates or updates a pull request to merge changes into `main`.
-     - Checks for existing pull requests with the same title to avoid duplicates.
-     - **CREATE_PR Environment Variable**:
-       - The `CREATE_PR` variable is set to `true` if the event is a push, otherwise it uses the value from `workflow_dispatch` input or defaults to `false`.
-   - **Testing Markdown Files**:
-     - Iterates over the list of changed Markdown files.
-     - Runs `ie execute` on each file to test the executable documentation.
-     - **On Test Failure**:
-       - Extracts error logs from the ie.log file.
-       - Creates or updates a GitHub issue with the error details.
-       - Records the failure in the test results.
-       - Does not assign the issue to any user to avoid permission issues.
-     - **On Test Success**:
-       - Records the success in the test results.
-   - **Reporting Results**:
-     - Posts the test results as a comment on the pull request.
-     - If no pull request exists (i.e., on the `main` branch), outputs the results in the workflow logs.
-
-## Customization
-
-You can modify the GitHub Action to suit your project's needs:
-
-- **Event Triggers**: Adjust the `on` section to change when the action runs (e.g., on different branches or file types).
-
-- **Azure Login Step**: If your documentation doesn't require Azure resources, you can remove the Azure login step.
-
-- **Python Version and Dependencies**:
-  - Update the `python-version` in the `Set Up Python` step to match your project's requirements.
-  - In the `Install Dependencies` step, specify the versions of `pip`, `PyGithub`, and `pyyaml` as needed. The action tries to install specific versions first and falls back to the latest versions if the specified versions are not available.
-
-- **Testing Script**: Update the Python script to change how tests are executed, how errors are handled, or how results are reported. For example, you might customize the error extraction logic or modify the issue creation process.
-
-- **Secrets Management**: Ensure all necessary secrets (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`, `PAT`) are correctly named and stored in your repository's settings under **Settings > Secrets and variables > Actions**.
-
-## Conclusion
-
-By integrating this GitHub Action into your repository, you automate the testing of your executable Markdown documents using Innovation Engine. This helps maintain documentation accuracy and, combined with branch protection rules, prevents broken documentation from being merged into your main branch.
\ No newline at end of file

From 5778d3332b1083dd4b1945fc2a5242043b5c0d4d Mon Sep 17 00:00:00 2001
From: naman-msft <namanparikh@microsoft.com>
Date: Mon, 25 Nov 2024 15:13:19 -0600
Subject: [PATCH 9/9] updated ie execute to ie test in the exec docs action

---
 examples/test-exec-docs/test-exec-docs.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/examples/test-exec-docs/test-exec-docs.yml b/examples/test-exec-docs/test-exec-docs.yml
index 21ff3ad2..64c7e2f3 100644
--- a/examples/test-exec-docs/test-exec-docs.yml
+++ b/examples/test-exec-docs/test-exec-docs.yml
@@ -177,7 +177,7 @@ jobs:
         # Iterate over changed Markdown files
         for file_path in changed_files:
             # Execute the Innovation Engine on the Markdown file
-            result = subprocess.run(['ie', 'execute', file_path, '--environment', 'github-action'])
+            result = subprocess.run(['ie', 'test', file_path, '--environment', 'github-action'])
             if result.returncode != 0:
                 any_failures = True
                 # If execution fails, extract error log from 'ie.log'