-
Notifications
You must be signed in to change notification settings - Fork 10
Rework environment setup #78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 50 commits
Commits
Show all changes
60 commits
Select commit
Hold shift + click to select a range
3ca3465
Introduce sandboxing and new conda environment to replace LCG stack d…
tvoigtlaender 2243654
Add local testing script for CROWN executables that works with sandbo…
tvoigtlaender 5bba7bb
Add CROWN as KingMaker submodule
tvoigtlaender 19dda97
Simplify compile scripts. Move setup_sandbox script to processor dir.
tvoigtlaender b664756
Remove old init.sh script from tasks as it is no longer needed at run…
tvoigtlaender 821c662
Remove os dependent image usage.
tvoigtlaender 1cedbc3
black
tvoigtlaender bd42cdf
black
tvoigtlaender 0aebdd2
move sandboxing to common class
tvoigtlaender 861cddb
Bugfix
tvoigtlaender c83356b
Replace HTCondor docker setup with cvmfs unpacked apptainer image. Up…
tvoigtlaender 3fa1c8d
bugfix
tvoigtlaender fefcc04
Move container info from submodule to ordinary directory as setup has…
tvoigtlaender 9c3396e
Adjust setup script to env file location change. Temporarily add King…
tvoigtlaender e8c4868
Update CROWN submodule
tvoigtlaender e8e2e0d
Cleanup after local cvmfs env was patched
tvoigtlaender 4f71047
more cleanup
tvoigtlaender 8b15c93
Replace jit calls with proper dict parsing lib
tvoigtlaender 83498b6
Move default ccache dir to CROWN/.cache/ccache
tvoigtlaender c3252f5
Generalize sandboxing. Allows for nosandboxing and run in local envir…
tvoigtlaender 5558b71
Black
tvoigtlaender 28dde3d
Generalize ccache location for when non-CROWN workflow uses sandboxin…
tvoigtlaender a4306d6
Generalize sandbox_pre_setup_cmds setup for easier adding of env vari…
tvoigtlaender d113b3f
Add helper function for sandbox setup
tvoigtlaender 540fbca
Update CROWN github CI checks. Update container initilization. Update…
tvoigtlaender 2a6c040
Update KingMakerStandalone env file with pinned versions
tvoigtlaender 91a1292
Bugfix
tvoigtlaender 63a1cb3
Add lxplus variant
tvoigtlaender b2e9899
Allow for mixed compilation/execution between local and container
tvoigtlaender 69b3a4a
Update CROWN. Remove tedious shell check from setup.
tvoigtlaender 3ab2880
Clean up config files
tvoigtlaender 2ca6f3b
Merge pull request #79 from KIT-CMS/lxplus_compatible
tvoigtlaender dd0ecf5
propagate CROWN changes
tvoigtlaender 9e01aa9
Fix lxplus sandboxing by setting forward_law to False in [singularity…
34b5600
Use local scheduler for lxplus as central scheduler is inconsistent.
227f6cf
Update container path to cvmfs unpacked. Add apptainer to base KingMa…
tvoigtlaender 575b96f
Add missing dep.
tvoigtlaender 830de26
black
tvoigtlaender fd80635
Move 'ANA_NAME' env variable export to sandbox to defaults
tvoigtlaender bfc67ea
Apply the same python formatting check as for CROWN
tvoigtlaender 7704322
Add black to Kingmaker env
tvoigtlaender 1d175a9
Apply python formatting
tvoigtlaender 8d72cff
Re-add ignore for samples.txt and similar
tvoigtlaender e0b0b2e
Update CROWN
tvoigtlaender d09a285
Resolve conflict
tvoigtlaender 86d7df3
fast-forward CROWN changes
tvoigtlaender 8b8a64b
Resolve merge conflicts
tvoigtlaender d4f1aa7
Update README
tvoigtlaender 40a46dd
Merge pull request #84 from KIT-CMS/main
tvoigtlaender 0839d4c
Allow to run setup script from non KingMaker dir
tvoigtlaender cb07c76
Fix path to CROWN init.sh script
tvoigtlaender 8f71dcd
Address comments
tvoigtlaender 44704bc
fix version number
tvoigtlaender bf30034
replace http with ssh adress in submodules
tvoigtlaender ca2bbe5
synchronize setup option names with CROWN
nshadskiy 15499d5
synchronize setup option names with CROWN 2
nshadskiy 9597bce
Cleanup
tvoigtlaender e67383b
update to new general kingmakerimages repo
nshadskiy aa35bbb
Merge branch 'main' into sandboxing_newenv
nshadskiy 947bda8
black
nshadskiy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,6 @@ | |
| [submodule "sample_database"] | ||
| path = sample_database | ||
| url = [email protected]:KIT-CMS/KingMaker_sample_database.git | ||
| [submodule "kingmaker-images"] | ||
| path = kingmaker-images | ||
| url = git@github.com:KIT-CMS/kingmaker-images.git | ||
| [submodule "CROWN"] | ||
| path = CROWN | ||
| url = https://github.com/KIT-CMS/CROWN.git | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,13 @@ | ||
| # KingMaker | ||
|
|
||
|
|
||
| [](https://github.com/KIT-CMS/kingmaker-images/actions/workflows/deploy-base-images.yml) | ||
|
|
||
| KingMaker is the workflow management for producing ntuples with the [CROWN](https://github.com/KIT-CMS/CROWN) framework. The workflow management is based on [law](https://github.com/riga/law), which is using [luigi](https://github.com/spotify/luigi) as backend. | ||
|
|
||
| **⚠ Important: A detailed description of the KingMaker workflow to produce NTuples can be found in the [CROWN documentation](https://crown.readthedocs.io/en/latest/kingmaker.html#).** | ||
|
|
||
|
|
||
| ## 🛠 Infrastructure & Containers | ||
|
|
||
| KingMaker can runs within container environments to ensure reproducibility. | ||
|
|
||
| * **Container Images and Environments**: Dockerfiles and Conda environment specifications are located in the [`/containers`](./containers) directory. | ||
| * **Usage**: For instructions on building custom images or using existing ones from CVMFS, see the [Container Documentation](./containers/README.md). | ||
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Configuration | ||
| APPLY_FIXES=false | ||
|
|
||
| # Parse CLI arguments | ||
| while [[ "$#" -gt 0 ]]; do | ||
| case $1 in | ||
| --apply) APPLY_FIXES=true ;; | ||
| *) echo "Unknown parameter: $1"; exit 1 ;; | ||
| esac | ||
| shift | ||
| done | ||
|
|
||
| echo "🔍 Finding Python files ..." | ||
|
|
||
| # Find tracked python files | ||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| FILES=$(git ls-files | grep -E '\.py$' | grep -vEf "$SCRIPT_DIR/format_ignore.txt") | ||
|
|
||
| if [[ -z "$FILES" ]]; then | ||
| echo "No Python files found." | ||
| exit 0 | ||
| fi | ||
|
|
||
| if [ "$APPLY_FIXES" = true ]; then | ||
| echo "🛠 Running black to fix formatting..." | ||
| # Black formats in-place by default | ||
| echo "$FILES" | xargs black | ||
| echo "✅ Python formatting complete." | ||
| else | ||
| echo "🧪 Checking Python formatting (Dry Run)..." | ||
|
|
||
| # --check: return exit code 1 if files need formatting | ||
| # --diff: show what would change | ||
| # --quiet: reduce noise, we only want the diffs/errors | ||
| if echo "$FILES" | xargs black --check --diff; then | ||
| echo "✨ Python code is looking good!" | ||
| exit 0 | ||
| else | ||
| echo "------------------------------------------------------" | ||
| echo "❌ ERROR: Python formatting violations found." | ||
| echo "Run this script with --apply to fix them." | ||
| echo "------------------------------------------------------" | ||
| exit 1 | ||
| fi | ||
| fi |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.