-
Notifications
You must be signed in to change notification settings - Fork 4
D->M #274
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
* Add prefix to alias name for generic index The alias for the generic indexes is missing the prefix. * Update version and changelog
Closes issue206
Add an option to skip triggering a narrative reindex. This is desireable in an admin scenario where you don't want to amplify indexing.
This adds two configurable options to deal with large workspace. - A skip_workspaces configurable (comma-seperated list) - A cutoff for large workspaces that skips the narrative reindex (default: 500)
Need to convert the type for the logic to work.
This add an option (SKIP_ES) to completely disable elastic pieces. It also add an RE specific config.
Optional narrative reindex
Add options for large workspaces Dakota approves.
Changing samples releng indexer to use '_from' and '_to' properties
trigger build
…o dev-add_workflows
SECURITY-51: Add GHA workflows
| uses: kbase/.github/.github/workflows/reusable_build-push.yml@main | ||
| with: | ||
| name: '${{ github.event.repository.name }}-develop' | ||
| tags: br-${{ github.ref_name }} | ||
| secrets: inherit |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix this problem, we should add a permissions block to the workflow file, .github/workflows/manual-build.yml. Since the workflow's main activity is building and pushing, and it uses a reusable workflow, we should grant only the minimal permissions required for the workflow to operate. If we are uncertain what permissions are needed for the reusable workflow, a good minimal starting point (as recommended by CodeQL) is to set all permissions to read. You can define the permissions block at the top level, so it applies to all jobs, unless jobs set their own restrictions.
Changes needed:
- Add a
permissions:block before thejobs:key in.github/workflows/manual-build.yml. - The block should specify
contents: readas the baseline least privilege.
No additional methods, definitions, or imports are needed—the change is the addition of a simple YAML block.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Manual Build & Push | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| workflow_dispatch: | ||
| jobs: |
| if: github.base_ref == 'develop' && github.event.pull_request.merged == false | ||
| uses: kbase/.github/.github/workflows/reusable_build.yml@main | ||
| secrets: inherit | ||
| build-develop-merge: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix the issue, add a permissions block at the top level of the workflow (global to all jobs), specifying the minimal set of permissions required by the jobs. If most jobs only need to read repository contents, you can set contents: read; if you know a subset of jobs (such as those pushing code or tags) require additional permissions (contents: write or pull-requests: write), you should consider setting those on those jobs individually, or broaden at the root if all need them. Based on the file's described jobs (building, tagging, pushing, scanning), the minimal starting point can be contents: read globally, but review or testing may require adding contents: write, pull-requests: write, or similar on jobs that push code, images, or create tags. In this fix, introduce at the workflow root:
permissions:
contents: readYou may wish to expand this as you confirm job-level requirements.
The change needed is:
- Insert a
permissions:block after the workflow name (line 2).
No new packages or imports are required; the fix is made only in.github/workflows/pr_build.yml.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Pull Request Build, Tag, & Push | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| pull_request: | ||
| branches: |
| if: github.base_ref == 'develop' && github.event.pull_request.merged == true | ||
| uses: kbase/.github/.github/workflows/reusable_build-push.yml@main | ||
| with: | ||
| name: '${{ github.event.repository.name }}-develop' | ||
| tags: pr-${{ github.event.number }},latest | ||
| secrets: inherit | ||
| build-main-open: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix this problem, add an explicit permissions block to the workflow. This can be done at the root level (so all jobs inherit it), or individually per job if some jobs require more or fewer privileges. Since the jobs here primarily build, tag, push, and scan on pull requests, it's safest to set a minimal contents: read at the workflow level, and expand only if any jobs require more privilege (for example, if pushing to releases, writing commit statuses, or posting comments on pull requests). As a starting point, set:
permissions:
contents: readat the top after the name: field. This gives all jobs access only to read repository contents. If a job (such as one that needs to comment on a pull request) needs more, you would expand only that job's permissions.
Files/regions/lines to change:
- Edit
.github/workflows/pr_build.yml - Add the
permissions:block just below thename:on line 2 (i.e., as line 3), pushing the rest of the content down by one.
What is needed:
- A single YAML block.
- No imports or other code changes are needed.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Pull Request Build, Tag, & Push | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| pull_request: | ||
| branches: |
| if: (github.base_ref == 'main' || github.base_ref == 'master') && github.event.pull_request.merged == false | ||
| uses: kbase/.github/.github/workflows/reusable_build-push.yml@main | ||
| with: | ||
| name: '${{ github.event.repository.name }}' | ||
| tags: pr-${{ github.event.number }} | ||
| secrets: inherit | ||
| build-main-merge: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix this issue, add a permissions key at the top level of .github/workflows/pr_build.yml (just below the name: and before on:) to restrict the permissions of the GITHUB_TOKEN that runs this workflow. Since this workflow is triggered by pull requests and delegates actual work to reusable workflows, it's safest to start with the minimal recommended permissions: contents: read only. This allows the workflow to read repository contents, but not push, create, or modify anything. If this workflow (including the reusable workflows it calls) needs to create or modify pull requests (for example, update statuses or post comments), you could add pull-requests: write. Otherwise, stay as restrictive as possible.
Region to change:
- File:
.github/workflows/pr_build.yml - Add the following block near the top, below
name:and beforeon:
No imports, definitions, or methods required; just add the YAML block.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Pull Request Build, Tag, & Push | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| pull_request: | ||
| branches: |
| if: (github.base_ref == 'main' || github.base_ref == 'master') && github.event.pull_request.merged == true | ||
| uses: kbase/.github/.github/workflows/reusable_build-push.yml@main | ||
| with: | ||
| name: '${{ github.event.repository.name }}' | ||
| tags: pr-${{ github.event.number }},latest-rc | ||
| secrets: inherit | ||
| trivy-scans: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix the problem, explicitly add a permissions block at the top level of the workflow so that all jobs launched in this workflow (unless they declare their own permissions) inherit these minimal privileges. The safest starting point is to grant only read access to repository contents with:
permissions:
contents: readThis ensures any job in the workflow or in a downstream called reusable workflow will not receive write access to the repository from the GITHUB_TOKEN, unless they explicitly request more in their own scope.
The change should be made by adding the permissions block directly below the name key, and above the on key, at the top of the file.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Pull Request Build, Tag, & Push | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| pull_request: | ||
| branches: |
| if: (github.base_ref == 'develop' || github.base_ref == 'main' || github.base_ref == 'master' ) && github.event.pull_request.merged == false | ||
| uses: kbase/.github/.github/workflows/reusable_trivy-scans.yml@main | ||
| secrets: inherit |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix this problem, a permissions block should be added — ideally at the top-level of the workflow YAML file so it applies to all jobs, unless specific jobs require additional write permissions. The default should be as restrictive as possible, ideally contents: read, which is sufficient for most build/test workflows triggered by pull requests. If certain jobs require elevated permissions (such as pull-requests: write or contents: write to push changes), they can have a more permissive block on that job only. The fix is to insert a top-level permissions block after the name field, specifying the minimal required access; based on the current jobs, contents: read should be enough. If subsequent investigation shows write is needed for releases, tags, or pull requests, that can be expanded.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Pull Request Build, Tag, & Push | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| pull_request: | ||
| branches: |
| uses: kbase/.github/.github/workflows/reusable_validate-branch.yml@main | ||
| with: | ||
| build_branch: '${{ github.event.release.target_commitish }}' | ||
| validate-release-tag: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix the problem, introduce a permissions block at the workflow root (just after the name: line) in .github/workflows/release-main.yml. To observe the least-privilege principle, set contents: read, which is the minimal permission needed for most workflows, unless additional permissions are required by the jobs. Since all jobs here are simple uses calls to reusable workflows and there is no evidence they require higher privileges, starting with contents: read is appropriate. If any of the called reusable workflows need additional or write permissions, specify these as finely as possible.
Steps:
- Insert
after the
permissions: contents: read
name:line (after line 2).
No other code or dependency changes are needed.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Release - Build & Push Image | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| release: | ||
| branches: |
| needs: check-source-branch | ||
| uses: kbase/.github/.github/workflows/reusable_validate-release-tag.yml@main | ||
| with: | ||
| release_tag: '${{ github.event.release.tag_name }}' | ||
| build-push: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix the problem, add an explicit permissions key at the root of the workflow file, directly under the name: or on: fields. This should grant only the minimal required permissions for the workflow to function correctly. Since the workflow uses reusable workflows for validating a branch, validating a release tag, and building/pushing an image, it's likely that read access to the repository contents (contents: read) is sufficient for most steps. If any step requires additional permissions (such as packages: write for Docker or npm registry, or pull-requests: write), these should be explicitly added at the root or per-job basis as needed.
As a minimal starting point, and unless further knowledge about the needs of the downstream reusable workflows is provided, supplying contents: read as the default is the safest option. You can adjust and expand as necessary if errors arise due to permission issues.
You need to:
- Insert the following block after the
name:field (preferably) in.github/workflows/release-main.yml:permissions: contents: read
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Release - Build & Push Image | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| release: | ||
| branches: |
| needs: validate-release-tag | ||
| uses: kbase/.github/.github/workflows/reusable_build-push.yml@main | ||
| with: | ||
| name: '${{ github.event.repository.name }}' | ||
| tags: '${{ github.event.release.tag_name }},latest' | ||
| secrets: inherit |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
To fix this issue, add a permissions block at the root of the workflow (just below name: and before on:), specifying the minimum required permissions for the workflow. Since this workflow only involves running checks and triggering builds/pushes (presumably via reusable workflows), it's most secure to start with contents: read. If any job or called workflow needs greater or more specific permissions (pull-requests: write, etc.), those should be explicitly set at the job or reusable workflow level. The change should be a single, simple edit near the top of the workflow file.
-
How:
Addpermissions: contents: read
immediately after the
name:line, and before theon:block. -
What:
Only edit.github/workflows/release-main.yml. No function/method changes are required; this is a metadata-only change in the workflow YAML file. -
Where:
After line 2, before line 3.
-
Copy modified lines R3-R4
| @@ -1,5 +1,7 @@ | ||
| --- | ||
| name: Release - Build & Push Image | ||
| permissions: | ||
| contents: read | ||
| on: | ||
| release: | ||
| branches: |
| runs-on: ubuntu-latest | ||
|
|
||
| env: | ||
| WORKSPACE_TOKEN: ${{ secrets.WORKSPACE_TOKEN }} | ||
| RE_API_TOKEN: ${{ secrets.RE_API_TOKEN }} | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: 3.9.19 | ||
|
|
||
| - name: Pip installation | ||
| run: python -m pip install --upgrade pip poetry | ||
|
|
||
| - name: Poetry installation | ||
| run: poetry install | ||
|
|
||
| - name: Run unit tests | ||
| run: scripts/run_unit_tests | ||
|
|
||
| - name: Run integration tests | ||
| run: | | ||
| echo "WORKSPACE_TOKEN=$WORKSPACE_TOKEN" > .env | ||
| echo "RE_API_TOKEN=$RE_API_TOKEN" >> .env | ||
| scripts/run_integration_tests | ||
|
|
||
| - name: Print Docker Compose logs | ||
| if: failure() | ||
| run: docker compose logs | ||
|
|
||
| - name: Upload coverage to Codecov | ||
| uses: codecov/codecov-action@v5 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| fail_ci_if_error: true |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium test
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 2 months ago
The recommended fix is to add a minimal permissions block specifying the least privilege necessary for the job to function. Since the workflow as shown does not require write access to the repository's contents and only performs read operations (checking out code, running tests, uploading coverage), the safest default is to set contents: read. Add a permissions: section at the top level of the workflow, immediately below the name: field, or at the job level (inside the test: job block); for workflows with simple or single jobs, top-level is preferred and more DRY. This requires editing the .github/workflows/test.yml file, inserting:
permissions:
contents: readimmediately before the on: declaration.
No changes to imports, definitions, or other configuration are required.
-
Copy modified lines R5-R6
| @@ -2,6 +2,8 @@ | ||
| # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions | ||
|
|
||
| name: Index Runner test | ||
| permissions: | ||
| contents: read | ||
|
|
||
| on: | ||
| pull_request: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR migrates the project from Docker Compose v1 to v2 commands, updates Python dependencies, and adds new configuration options for skipping specific workspaces and controlling narrative reindexing behavior.
Reviewed Changes
Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/wait_for_deps.py | Updated docker-compose command to docker compose |
| tests/unit/test_config.py | Added new test file for configuration validation |
| tests/unit/index_runner/test_main.py | Added tests for workspace skipping and reindex limits |
| tests/unit/index_runner/releng/test_samples.py | Fixed edge property names from "from"/"to" to "_from"/"_to" |
| src/utils/kafka.py | Added poll timeout configuration and optimized config calls |
| src/utils/config.py | Added new configuration options for workspace skipping and narrative reindexing |
| src/index_runner/releng/samples.py | Updated edge properties and removed extra whitespace |
| src/index_runner/releng/import_obj.py | Added debug logging for releng indexer |
| src/index_runner/main.py | Added workspace skipping logic and ES skip configuration |
| src/index_runner/es_indexer.py | Fixed alias creation with proper prefix |
| spec/re_config.yaml | Added minimal RE-only configuration file |
| scripts/* | Updated Docker Compose commands and enhanced test coverage |
| pyproject.toml | Updated Python version and dependencies |
| docker-compose.yaml | Removed version specification and added platform comments |
| VERSION | Incremented version to 1.9.21 |
| README.md | Enhanced documentation with dependencies and configuration |
| Dockerfile | Updated Python version and build dependencies |
| CHANGELOG.md | Added release notes for recent versions |
| .github/workflows/* | Added comprehensive CI/CD workflows |
| .github/dependabot.yml | Added dependency management configuration |
| .github/codeql.yml | Added security scanning workflow |
| .coveragerc | Added test coverage configuration |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| --- | ||
| name: Manual Build & Push | ||
| on: | ||
| workflow_dispatch: |
Copilot
AI
Oct 8, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing space in YAML syntax - should be 'workflow_dispatch:' with proper indentation.
| workflow_dispatch: | |
| workflow_dispatch: |
VERSIONfileCHANGELOG.md