Skip to content

Add optional repository metadata columns (teams, topics, custom properties) #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Add optional repository metadata columns (teams, topics, custom properties) #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
259fb1b
Initial plan for issue
Copilot Jun 13, 2025
4a35d76
Add environment variable flag and update CSV functions for extended r…
Copilot Jun 13, 2025
051f090
Complete enterprise functions and documentation for repo metadata fea…
Copilot Jun 13, 2025
6c917c2
Merge branch 'main' into copilot/fix-61
felickz Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ The list of all available options that can be set as environmental variables is
- `GITHUB_REPORT_SCOPE`: The scope of the report to generate. Valid values are `repository` (default), `organization` or `enterprise`.
- `SCOPE_NAME` or `GITHUB_REPOSITORY`: The name of the repository, organization or enterprise to generate the report for. If `SCOPE_NAME` is not set, the value of `GITHUB_REPOSITORY` is used if it is set. If neither is set, an error occurs.
- `FEATURES`: A comma-separated list of features to include in the report. Valid values are `codescanning`, `secretscanning`, `dependabot` or simply `all`. Default value: `all`.
- `INCLUDE_REPO_METADATA`: Include extended repository metadata (teams, topics, custom properties) in the CSV output. Valid values are `true` or `false`. Default value: `false`. **Warning**: Enabling this feature will significantly increase API calls and execution time at organization or enterprise scope.

The first two are only needed if you're running this in a GitHub Enterprise Server or GitHub AE environment. The last one is useful if you only want to get data on a specific feature. For example, if you only want to get data on secret scanning, you can set `FEATURES` to `secretscanning`. Here's just another example how you would configure this on a GitHub Enterprise Server:

Expand All @@ -92,6 +93,20 @@ The first two are only needed if you're running this in a GitHub Enterprise Serv
FEATURES: "secretscanning,codescanning"
```

To include extended repository metadata (teams, topics, and custom properties) in the CSV output:

```yaml
- name: CSV export with extended metadata
uses: advanced-security/ghas-to-csv@v3
env:
GITHUB_PAT: ${{ secrets.PAT }}
GITHUB_REPORT_SCOPE: "organization"
SCOPE_NAME: "org-name-goes-here"
INCLUDE_REPO_METADATA: "true"
```

⚠️ **Note**: The `INCLUDE_REPO_METADATA` feature will make additional API calls for each repository to fetch teams, topics, and custom properties. This can significantly increase execution time and API usage when used at organization or enterprise scope.

## Reporting

| | GitHub Enterprise Cloud | GitHub Enterprise Server (3.5+) | GitHub AE (M2) | Notes |
Expand Down
22 changes: 12 additions & 10 deletions main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@
report_scope = os.getenv("GITHUB_REPORT_SCOPE", "repository")
scope_name = os.getenv("SCOPE_NAME", os.getenv("GITHUB_REPOSITORY"))
requested_features = os.getenv("FEATURES")
# Flag to enable extended repository metadata (teams, topics, properties)
include_repo_metadata = os.getenv("INCLUDE_REPO_METADATA", "false").lower() == "true"
if (requested_features is None) or (requested_features == "all"):
features = FEATURES
else:
Expand All @@ -53,7 +55,7 @@
if "secretscanning" in features:
try:
secrets_list = secret_scanning.get_enterprise_ss_alerts(api_endpoint, github_pat, scope_name)
secret_scanning.write_enterprise_ss_list(secrets_list)
secret_scanning.write_enterprise_ss_list(secrets_list, include_repo_metadata, api_endpoint, github_pat)
except Exception as e:
if any(x in str(e).lower() for x in secret_scanning_disabled_strings):
print("Skipping Secret Scanning as it is not enabled.")
Expand All @@ -69,15 +71,15 @@
if version.startswith("3.5") or version.startswith("3.6"):
repo_list = enterprise.get_repo_report(url, github_pat)
cs_list = code_scanning.list_enterprise_server_cs_alerts(api_endpoint, github_pat, repo_list)
code_scanning.write_enterprise_server_cs_list(cs_list)
code_scanning.write_enterprise_server_cs_list(cs_list, include_repo_metadata, api_endpoint, github_pat)
else:
cs_list = code_scanning.list_enterprise_cloud_cs_alerts(api_endpoint, github_pat, scope_name)
code_scanning.write_enterprise_cloud_cs_list(cs_list)
code_scanning.write_enterprise_cloud_cs_list(cs_list, include_repo_metadata, api_endpoint, github_pat)
# dependabot alerts
if "dependabot" in features:
try:
dependabot_list = dependabot.list_enterprise_dependabot_alerts(api_endpoint, github_pat, scope_name)
dependabot.write_org_or_enterprise_dependabot_list(dependabot_list)
dependabot.write_org_or_enterprise_dependabot_list(dependabot_list, include_repo_metadata, api_endpoint, github_pat)
except Exception as e:
if any(x in str(e).lower() for x in dependabot_disabled_strings):
print("Skipping Dependabot as it is not enabled.")
Expand All @@ -89,12 +91,12 @@
# code scanning
if "codescanning" in features:
cs_list = code_scanning.list_org_cs_alerts(api_endpoint, github_pat, scope_name)
code_scanning.write_org_cs_list(cs_list)
code_scanning.write_org_cs_list(cs_list, include_repo_metadata, api_endpoint, github_pat)
# dependabot alerts
if "dependabot" in features:
try:
dependabot_list = dependabot.list_org_dependabot_alerts(api_endpoint, github_pat, scope_name)
dependabot.write_org_or_enterprise_dependabot_list(dependabot_list)
dependabot.write_org_or_enterprise_dependabot_list(dependabot_list, include_repo_metadata, api_endpoint, github_pat)
except Exception as e:
if any(x in str(e).lower() for x in dependabot_disabled_strings):
print("Skipping Dependabot as it is not enabled.")
Expand All @@ -105,7 +107,7 @@
if "secretscanning" in features:
try:
secrets_list = secret_scanning.get_org_ss_alerts(api_endpoint, github_pat, scope_name)
secret_scanning.write_org_ss_list(secrets_list)
secret_scanning.write_org_ss_list(secrets_list, include_repo_metadata, api_endpoint, github_pat)
except Exception as e:
if any(x in str(e).lower() for x in secret_scanning_disabled_strings):
print("Skipping Secret Scanning as it is not enabled.")
Expand All @@ -117,12 +119,12 @@
# code scanning
if "codescanning" in features:
cs_list = code_scanning.list_repo_cs_alerts(api_endpoint, github_pat, scope_name)
code_scanning.write_repo_cs_list(cs_list)
code_scanning.write_repo_cs_list(cs_list, include_repo_metadata, api_endpoint, github_pat, scope_name)
# dependabot alerts
if "dependabot" in features:
try:
dependabot_list = dependabot.list_repo_dependabot_alerts(api_endpoint, github_pat, scope_name)
dependabot.write_repo_dependabot_list(dependabot_list)
dependabot.write_repo_dependabot_list(dependabot_list, include_repo_metadata, api_endpoint, github_pat, scope_name)
except Exception as e:
if any(x in str(e).lower() for x in dependabot_disabled_strings):
print("Skipping Dependabot as it is not enabled.")
Expand All @@ -133,7 +135,7 @@
if "secretscanning" in features:
try:
secrets_list = secret_scanning.get_repo_ss_alerts(api_endpoint, github_pat, scope_name)
secret_scanning.write_repo_ss_list(secrets_list)
secret_scanning.write_repo_ss_list(secrets_list, include_repo_metadata, api_endpoint, github_pat, scope_name)
except Exception as e:
if any(x in str(e).lower() for x in secret_scanning_disabled_strings):
print("Skipping Secret Scanning as it is not enabled.")
Expand Down
64 changes: 63 additions & 1 deletion src/api_helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,66 @@
while "next" in response.links.keys():
response = requests.get(response.links["next"]["url"], headers=headers)
response_json.extend(response.json())
return response_json
return response_json


def get_repo_metadata(api_endpoint, github_pat, repo_name):
"""
Get extended repository metadata including teams, topics, and custom properties.
Inputs:
- API endpoint (for GHES/GHAE compatibility)
- PAT of appropriate scope
- Repository name (owner/repo format)
Outputs:
- Dictionary with teams, topics, and custom_properties
"""
metadata = {
"teams": [],
"topics": [],
"custom_properties": {}
}

try:
# Get repository teams
teams_url = f"{api_endpoint}/repos/{repo_name}/teams?per_page=100&page=1"
teams = make_api_call(teams_url, github_pat)
metadata["teams"] = [team["name"] for team in teams]
except Exception as e:
print(f"Warning: Could not fetch teams for {repo_name}: {e}")

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.

Copilot Autofix

AI 4 months ago

To address the issue, we will sanitize the log messages to avoid exposing sensitive information. Specifically:

  1. Replace the repository name (repo_name) with a generic placeholder or omit it entirely.
  2. Avoid logging the full exception message (e) and instead log a generic error message or a sanitized version of the exception.

We will modify the print statements in src/api_helpers.py to ensure no sensitive data is logged. This involves replacing the repository name and exception details with non-sensitive placeholders.

Suggested changeset 1
src/api_helpers.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/src/api_helpers.py b/src/api_helpers.py
--- a/src/api_helpers.py
+++ b/src/api_helpers.py
@@ -42,3 +42,3 @@
     except Exception as e:
-        print(f"Warning: Could not fetch teams for {repo_name}: {e}")
+        print("Warning: Could not fetch teams for the repository. Please check the logs for more details.")
         metadata["teams"] = []
@@ -51,3 +51,3 @@
     except Exception as e:
-        print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
+        print("Warning: Could not fetch repository details. Please check the logs for more details.")
         metadata["topics"] = []
@@ -60,3 +60,3 @@
     except Exception as e:
-        print(f"Warning: Could not fetch custom properties for {repo_name}: {e}")
+        print("Warning: Could not fetch custom properties for the repository. Please check the logs for more details.")
         metadata["custom_properties"] = {}
EOF
@@ -42,3 +42,3 @@
except Exception as e:
print(f"Warning: Could not fetch teams for {repo_name}: {e}")
print("Warning: Could not fetch teams for the repository. Please check the logs for more details.")
metadata["teams"] = []
@@ -51,3 +51,3 @@
except Exception as e:
print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
print("Warning: Could not fetch repository details. Please check the logs for more details.")
metadata["topics"] = []
@@ -60,3 +60,3 @@
except Exception as e:
print(f"Warning: Could not fetch custom properties for {repo_name}: {e}")
print("Warning: Could not fetch custom properties for the repository. Please check the logs for more details.")
metadata["custom_properties"] = {}
Copilot is powered by AI and may make mistakes. Always verify output.
metadata["teams"] = []

try:
# Get repository details (includes topics)
repo_url = f"{api_endpoint}/repos/{repo_name}"
repo_data = make_single_api_call(repo_url, github_pat)
metadata["topics"] = repo_data.get("topics", [])
except Exception as e:
print(f"Warning: Could not fetch repository details for {repo_name}: {e}")

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.

Copilot Autofix

AI 4 months ago

To fix the issue, we will sanitize the logging statements to ensure that sensitive data is not exposed. Specifically:

  1. Replace the direct logging of repo_name and e with a generic warning message that does not include sensitive details.
  2. If additional context is needed for debugging, log only non-sensitive information or use a secure logging mechanism that restricts access to sensitive logs.

For the flagged line in src/api_helpers.py, we will modify the print statement to exclude repo_name and e. Instead, we will log a generic warning message indicating that fetching repository details failed.

Suggested changeset 1
src/api_helpers.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/src/api_helpers.py b/src/api_helpers.py
--- a/src/api_helpers.py
+++ b/src/api_helpers.py
@@ -51,3 +51,3 @@
     except Exception as e:
-        print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
+        print("Warning: Could not fetch repository details. Please check the logs for more information.")
         metadata["topics"] = []
EOF
@@ -51,3 +51,3 @@
except Exception as e:
print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
print("Warning: Could not fetch repository details. Please check the logs for more information.")
metadata["topics"] = []
Copilot is powered by AI and may make mistakes. Always verify output.
metadata["topics"] = []

try:
# Get custom properties
properties_url = f"{api_endpoint}/repos/{repo_name}/properties"
properties = make_single_api_call(properties_url, github_pat)
metadata["custom_properties"] = properties
except Exception as e:
print(f"Warning: Could not fetch custom properties for {repo_name}: {e}")

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.

Copilot Autofix

AI 4 months ago

To fix the issue, we will sanitize the logging statements to avoid exposing sensitive information. Specifically:

  1. Replace the logging of repo_name with a generic placeholder or a sanitized version.
  2. Avoid logging the full exception details (e) and instead log a generic error message or a sanitized version of the exception.

We will modify the logging statements in src/api_helpers.py to redact sensitive information while preserving the utility of the logs for debugging purposes.

Suggested changeset 1
src/api_helpers.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/src/api_helpers.py b/src/api_helpers.py
--- a/src/api_helpers.py
+++ b/src/api_helpers.py
@@ -42,3 +42,3 @@
     except Exception as e:
-        print(f"Warning: Could not fetch teams for {repo_name}: {e}")
+        print(f"Warning: Could not fetch teams for the repository. Error: {str(e).splitlines()[0]}")
         metadata["teams"] = []
@@ -51,3 +51,3 @@
     except Exception as e:
-        print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
+        print(f"Warning: Could not fetch repository details. Error: {str(e).splitlines()[0]}")
         metadata["topics"] = []
@@ -60,3 +60,3 @@
     except Exception as e:
-        print(f"Warning: Could not fetch custom properties for {repo_name}: {e}")
+        print(f"Warning: Could not fetch custom properties for the repository. Error: {str(e).splitlines()[0]}")
         metadata["custom_properties"] = {}
EOF
@@ -42,3 +42,3 @@
except Exception as e:
print(f"Warning: Could not fetch teams for {repo_name}: {e}")
print(f"Warning: Could not fetch teams for the repository. Error: {str(e).splitlines()[0]}")
metadata["teams"] = []
@@ -51,3 +51,3 @@
except Exception as e:
print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
print(f"Warning: Could not fetch repository details. Error: {str(e).splitlines()[0]}")
metadata["topics"] = []
@@ -60,3 +60,3 @@
except Exception as e:
print(f"Warning: Could not fetch custom properties for {repo_name}: {e}")
print(f"Warning: Could not fetch custom properties for the repository. Error: {str(e).splitlines()[0]}")
metadata["custom_properties"] = {}
Copilot is powered by AI and may make mistakes. Always verify output.
metadata["custom_properties"] = {}

return metadata


def make_single_api_call(url, github_pat):
"""
Make a single API call without pagination.
"""
headers = {
"Authorization": "token {}".format(github_pat),
"Accept": "application/vnd.github.v3+json",
}
response = requests.get(url, headers=headers)
if not response.ok:
raise Exception(response.status_code, response.text)
return response.json()
Loading
Loading