Skip to content

Conversation

@bio-boris
Copy link
Contributor

  • I incremented the version in the VERSION file
  • I updated CHANGELOG.md
  • I added test coverage for any changes

slebras and others added 30 commits March 4, 2021 14:59
* Add prefix to alias name for generic index

The alias for the generic indexes is missing the prefix.

* Update version and changelog
Add an option to skip triggering a narrative reindex.
This is desireable in an admin scenario where you don't
want to amplify indexing.
This adds two configurable options to deal with large workspace.
- A skip_workspaces configurable (comma-seperated list)
- A cutoff for large workspaces that skips the narrative reindex (default: 500)
Need to convert the type for the logic to work.
This add an option (SKIP_ES) to completely disable elastic
pieces.  It also add an RE specific config.
Add options for large workspaces

Dakota approves.
Changing samples releng indexer to use '_from' and '_to' properties
Copilot AI review requested due to automatic review settings October 8, 2025 16:39
Comment on lines +7 to +11
uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
with:
name: '${{ github.event.repository.name }}-develop'
tags: br-${{ github.ref_name }}
secrets: inherit

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix this problem, we should add a permissions block to the workflow file, .github/workflows/manual-build.yml. Since the workflow's main activity is building and pushing, and it uses a reusable workflow, we should grant only the minimal permissions required for the workflow to operate. If we are uncertain what permissions are needed for the reusable workflow, a good minimal starting point (as recommended by CodeQL) is to set all permissions to read. You can define the permissions block at the top level, so it applies to all jobs, unless jobs set their own restrictions.
Changes needed:

  • Add a permissions: block before the jobs: key in .github/workflows/manual-build.yml.
  • The block should specify contents: read as the baseline least privilege.
    No additional methods, definitions, or imports are needed—the change is the addition of a simple YAML block.
Suggested changeset 1
.github/workflows/manual-build.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/manual-build.yml b/.github/workflows/manual-build.yml
--- a/.github/workflows/manual-build.yml
+++ b/.github/workflows/manual-build.yml
@@ -1,5 +1,7 @@
 ---
 name: Manual Build & Push
+permissions:
+  contents: read
 on:
  workflow_dispatch:
 jobs:
EOF
@@ -1,5 +1,7 @@
---
name: Manual Build & Push
permissions:
contents: read
on:
workflow_dispatch:
jobs:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +16 to +19
if: github.base_ref == 'develop' && github.event.pull_request.merged == false
uses: kbase/.github/.github/workflows/reusable_build.yml@main
secrets: inherit
build-develop-merge:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix the issue, add a permissions block at the top level of the workflow (global to all jobs), specifying the minimal set of permissions required by the jobs. If most jobs only need to read repository contents, you can set contents: read; if you know a subset of jobs (such as those pushing code or tags) require additional permissions (contents: write or pull-requests: write), you should consider setting those on those jobs individually, or broaden at the root if all need them. Based on the file's described jobs (building, tagging, pushing, scanning), the minimal starting point can be contents: read globally, but review or testing may require adding contents: write, pull-requests: write, or similar on jobs that push code, images, or create tags. In this fix, introduce at the workflow root:

permissions:
  contents: read

You may wish to expand this as you confirm job-level requirements.

The change needed is:

  • Insert a permissions: block after the workflow name (line 2).
    No new packages or imports are required; the fix is made only in .github/workflows/pr_build.yml.
Suggested changeset 1
.github/workflows/pr_build.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/pr_build.yml b/.github/workflows/pr_build.yml
--- a/.github/workflows/pr_build.yml
+++ b/.github/workflows/pr_build.yml
@@ -1,5 +1,7 @@
 ---
 name: Pull Request Build, Tag, & Push
+permissions:
+  contents: read
 on:
   pull_request:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Pull Request Build, Tag, & Push
permissions:
contents: read
on:
pull_request:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +20 to +26
if: github.base_ref == 'develop' && github.event.pull_request.merged == true
uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
with:
name: '${{ github.event.repository.name }}-develop'
tags: pr-${{ github.event.number }},latest
secrets: inherit
build-main-open:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix this problem, add an explicit permissions block to the workflow. This can be done at the root level (so all jobs inherit it), or individually per job if some jobs require more or fewer privileges. Since the jobs here primarily build, tag, push, and scan on pull requests, it's safest to set a minimal contents: read at the workflow level, and expand only if any jobs require more privilege (for example, if pushing to releases, writing commit statuses, or posting comments on pull requests). As a starting point, set:

permissions:
  contents: read

at the top after the name: field. This gives all jobs access only to read repository contents. If a job (such as one that needs to comment on a pull request) needs more, you would expand only that job's permissions.

Files/regions/lines to change:

  • Edit .github/workflows/pr_build.yml
  • Add the permissions: block just below the name: on line 2 (i.e., as line 3), pushing the rest of the content down by one.

What is needed:

  • A single YAML block.
  • No imports or other code changes are needed.

Suggested changeset 1
.github/workflows/pr_build.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/pr_build.yml b/.github/workflows/pr_build.yml
--- a/.github/workflows/pr_build.yml
+++ b/.github/workflows/pr_build.yml
@@ -1,5 +1,7 @@
 ---
 name: Pull Request Build, Tag, & Push
+permissions:
+  contents: read
 on:
   pull_request:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Pull Request Build, Tag, & Push
permissions:
contents: read
on:
pull_request:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +27 to +33
if: (github.base_ref == 'main' || github.base_ref == 'master') && github.event.pull_request.merged == false
uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
with:
name: '${{ github.event.repository.name }}'
tags: pr-${{ github.event.number }}
secrets: inherit
build-main-merge:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix this issue, add a permissions key at the top level of .github/workflows/pr_build.yml (just below the name: and before on:) to restrict the permissions of the GITHUB_TOKEN that runs this workflow. Since this workflow is triggered by pull requests and delegates actual work to reusable workflows, it's safest to start with the minimal recommended permissions: contents: read only. This allows the workflow to read repository contents, but not push, create, or modify anything. If this workflow (including the reusable workflows it calls) needs to create or modify pull requests (for example, update statuses or post comments), you could add pull-requests: write. Otherwise, stay as restrictive as possible.

Region to change:

  • File: .github/workflows/pr_build.yml
  • Add the following block near the top, below name: and before on:

No imports, definitions, or methods required; just add the YAML block.

Suggested changeset 1
.github/workflows/pr_build.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/pr_build.yml b/.github/workflows/pr_build.yml
--- a/.github/workflows/pr_build.yml
+++ b/.github/workflows/pr_build.yml
@@ -1,5 +1,7 @@
 ---
 name: Pull Request Build, Tag, & Push
+permissions:
+  contents: read
 on:
   pull_request:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Pull Request Build, Tag, & Push
permissions:
contents: read
on:
pull_request:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +34 to +40
if: (github.base_ref == 'main' || github.base_ref == 'master') && github.event.pull_request.merged == true
uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
with:
name: '${{ github.event.repository.name }}'
tags: pr-${{ github.event.number }},latest-rc
secrets: inherit
trivy-scans:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix the problem, explicitly add a permissions block at the top level of the workflow so that all jobs launched in this workflow (unless they declare their own permissions) inherit these minimal privileges. The safest starting point is to grant only read access to repository contents with:

permissions:
  contents: read

This ensures any job in the workflow or in a downstream called reusable workflow will not receive write access to the repository from the GITHUB_TOKEN, unless they explicitly request more in their own scope.
The change should be made by adding the permissions block directly below the name key, and above the on key, at the top of the file.


Suggested changeset 1
.github/workflows/pr_build.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/pr_build.yml b/.github/workflows/pr_build.yml
--- a/.github/workflows/pr_build.yml
+++ b/.github/workflows/pr_build.yml
@@ -1,5 +1,7 @@
 ---
 name: Pull Request Build, Tag, & Push
+permissions:
+  contents: read
 on:
   pull_request:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Pull Request Build, Tag, & Push
permissions:
contents: read
on:
pull_request:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +41 to +43
if: (github.base_ref == 'develop' || github.base_ref == 'main' || github.base_ref == 'master' ) && github.event.pull_request.merged == false
uses: kbase/.github/.github/workflows/reusable_trivy-scans.yml@main
secrets: inherit

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix this problem, a permissions block should be added — ideally at the top-level of the workflow YAML file so it applies to all jobs, unless specific jobs require additional write permissions. The default should be as restrictive as possible, ideally contents: read, which is sufficient for most build/test workflows triggered by pull requests. If certain jobs require elevated permissions (such as pull-requests: write or contents: write to push changes), they can have a more permissive block on that job only. The fix is to insert a top-level permissions block after the name field, specifying the minimal required access; based on the current jobs, contents: read should be enough. If subsequent investigation shows write is needed for releases, tags, or pull requests, that can be expanded.

Suggested changeset 1
.github/workflows/pr_build.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/pr_build.yml b/.github/workflows/pr_build.yml
--- a/.github/workflows/pr_build.yml
+++ b/.github/workflows/pr_build.yml
@@ -1,5 +1,7 @@
 ---
 name: Pull Request Build, Tag, & Push
+permissions:
+  contents: read
 on:
   pull_request:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Pull Request Build, Tag, & Push
permissions:
contents: read
on:
pull_request:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +11 to +14
uses: kbase/.github/.github/workflows/reusable_validate-branch.yml@main
with:
build_branch: '${{ github.event.release.target_commitish }}'
validate-release-tag:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix the problem, introduce a permissions block at the workflow root (just after the name: line) in .github/workflows/release-main.yml. To observe the least-privilege principle, set contents: read, which is the minimal permission needed for most workflows, unless additional permissions are required by the jobs. Since all jobs here are simple uses calls to reusable workflows and there is no evidence they require higher privileges, starting with contents: read is appropriate. If any of the called reusable workflows need additional or write permissions, specify these as finely as possible.

Steps:

  • Insert
    permissions:
      contents: read
    after the name: line (after line 2).

No other code or dependency changes are needed.


Suggested changeset 1
.github/workflows/release-main.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/release-main.yml b/.github/workflows/release-main.yml
--- a/.github/workflows/release-main.yml
+++ b/.github/workflows/release-main.yml
@@ -1,5 +1,7 @@
 ---
 name: Release - Build & Push Image
+permissions:
+  contents: read
 on:
   release:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Release - Build & Push Image
permissions:
contents: read
on:
release:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +15 to +19
needs: check-source-branch
uses: kbase/.github/.github/workflows/reusable_validate-release-tag.yml@main
with:
release_tag: '${{ github.event.release.tag_name }}'
build-push:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix the problem, add an explicit permissions key at the root of the workflow file, directly under the name: or on: fields. This should grant only the minimal required permissions for the workflow to function correctly. Since the workflow uses reusable workflows for validating a branch, validating a release tag, and building/pushing an image, it's likely that read access to the repository contents (contents: read) is sufficient for most steps. If any step requires additional permissions (such as packages: write for Docker or npm registry, or pull-requests: write), these should be explicitly added at the root or per-job basis as needed.

As a minimal starting point, and unless further knowledge about the needs of the downstream reusable workflows is provided, supplying contents: read as the default is the safest option. You can adjust and expand as necessary if errors arise due to permission issues.

You need to:

  • Insert the following block after the name: field (preferably) in .github/workflows/release-main.yml:
    permissions:
      contents: read
Suggested changeset 1
.github/workflows/release-main.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/release-main.yml b/.github/workflows/release-main.yml
--- a/.github/workflows/release-main.yml
+++ b/.github/workflows/release-main.yml
@@ -1,5 +1,7 @@
 ---
 name: Release - Build & Push Image
+permissions:
+  contents: read
 on:
   release:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Release - Build & Push Image
permissions:
contents: read
on:
release:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +20 to +25
needs: validate-release-tag
uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
with:
name: '${{ github.event.repository.name }}'
tags: '${{ github.event.release.tag_name }},latest'
secrets: inherit

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 2 months ago

To fix this issue, add a permissions block at the root of the workflow (just below name: and before on:), specifying the minimum required permissions for the workflow. Since this workflow only involves running checks and triggering builds/pushes (presumably via reusable workflows), it's most secure to start with contents: read. If any job or called workflow needs greater or more specific permissions (pull-requests: write, etc.), those should be explicitly set at the job or reusable workflow level. The change should be a single, simple edit near the top of the workflow file.

  • How:
    Add

    permissions:
      contents: read

    immediately after the name: line, and before the on: block.

  • What:
    Only edit .github/workflows/release-main.yml. No function/method changes are required; this is a metadata-only change in the workflow YAML file.

  • Where:
    After line 2, before line 3.


Suggested changeset 1
.github/workflows/release-main.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/release-main.yml b/.github/workflows/release-main.yml
--- a/.github/workflows/release-main.yml
+++ b/.github/workflows/release-main.yml
@@ -1,5 +1,7 @@
 ---
 name: Release - Build & Push Image
+permissions:
+  contents: read
 on:
   release:
     branches:
EOF
@@ -1,5 +1,7 @@
---
name: Release - Build & Push Image
permissions:
contents: read
on:
release:
branches:
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +22 to +60
runs-on: ubuntu-latest

env:
WORKSPACE_TOKEN: ${{ secrets.WORKSPACE_TOKEN }}
RE_API_TOKEN: ${{ secrets.RE_API_TOKEN }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.9.19

- name: Pip installation
run: python -m pip install --upgrade pip poetry

- name: Poetry installation
run: poetry install

- name: Run unit tests
run: scripts/run_unit_tests

- name: Run integration tests
run: |
echo "WORKSPACE_TOKEN=$WORKSPACE_TOKEN" > .env
echo "RE_API_TOKEN=$RE_API_TOKEN" >> .env
scripts/run_integration_tests

- name: Print Docker Compose logs
if: failure()
run: docker compose logs

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: true

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium test

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI about 2 months ago

The recommended fix is to add a minimal permissions block specifying the least privilege necessary for the job to function. Since the workflow as shown does not require write access to the repository's contents and only performs read operations (checking out code, running tests, uploading coverage), the safest default is to set contents: read. Add a permissions: section at the top level of the workflow, immediately below the name: field, or at the job level (inside the test: job block); for workflows with simple or single jobs, top-level is preferred and more DRY. This requires editing the .github/workflows/test.yml file, inserting:

permissions:
  contents: read

immediately before the on: declaration.

No changes to imports, definitions, or other configuration are required.


Suggested changeset 1
.github/workflows/test.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -2,6 +2,8 @@
 # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
 
 name: Index Runner test
+permissions:
+  contents: read
 
 on:
   pull_request:
EOF
@@ -2,6 +2,8 @@
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Index Runner test
permissions:
contents: read

on:
pull_request:
Copilot is powered by AI and may make mistakes. Always verify output.
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR migrates the project from Docker Compose v1 to v2 commands, updates Python dependencies, and adds new configuration options for skipping specific workspaces and controlling narrative reindexing behavior.

Reviewed Changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/wait_for_deps.py Updated docker-compose command to docker compose
tests/unit/test_config.py Added new test file for configuration validation
tests/unit/index_runner/test_main.py Added tests for workspace skipping and reindex limits
tests/unit/index_runner/releng/test_samples.py Fixed edge property names from "from"/"to" to "_from"/"_to"
src/utils/kafka.py Added poll timeout configuration and optimized config calls
src/utils/config.py Added new configuration options for workspace skipping and narrative reindexing
src/index_runner/releng/samples.py Updated edge properties and removed extra whitespace
src/index_runner/releng/import_obj.py Added debug logging for releng indexer
src/index_runner/main.py Added workspace skipping logic and ES skip configuration
src/index_runner/es_indexer.py Fixed alias creation with proper prefix
spec/re_config.yaml Added minimal RE-only configuration file
scripts/* Updated Docker Compose commands and enhanced test coverage
pyproject.toml Updated Python version and dependencies
docker-compose.yaml Removed version specification and added platform comments
VERSION Incremented version to 1.9.21
README.md Enhanced documentation with dependencies and configuration
Dockerfile Updated Python version and build dependencies
CHANGELOG.md Added release notes for recent versions
.github/workflows/* Added comprehensive CI/CD workflows
.github/dependabot.yml Added dependency management configuration
.github/codeql.yml Added security scanning workflow
.coveragerc Added test coverage configuration

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

---
name: Manual Build & Push
on:
workflow_dispatch:
Copy link

Copilot AI Oct 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing space in YAML syntax - should be 'workflow_dispatch:' with proper indentation.

Suggested change
workflow_dispatch:
workflow_dispatch:

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants