-
Notifications
You must be signed in to change notification settings - Fork 52
Add compression comparison functionality #386
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
| /external/* | ||
|
|
||
| # Exclude results tracking files | ||
| green_code_tracking.txt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| green_code_tracking.txt | |
| *.txt | |
| *.npz | |
| *.dat | |
| *.png | |
| *.root | |
| *.jpg | |
| *.jpeg | |
| *.log |
Are there any .txt files we would want to track? Maybe a catch-all for results/data/log files would be better
| helper.create_new_project(workspace_name, project_name, verbose) | ||
| elif mode == "train": | ||
| perform_training(output_path=output_path, config=config, verbose=verbose) | ||
| perform_training(output_path, config, project_name, verbose) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure project_name is always provided? Do we have a default? Just thinking this would break compatibility with an old script if there isn't a default, and this isn't defined
baler/modules/compare.py
Outdated
| @@ -0,0 +1,380 @@ | |||
| # Copyright 2022 Baler Contributors | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| # Copyright 2022 Baler Contributors | |
| # Copyright 2022-2025 Baler Contributors |
And similar for other files
|
Hi @BradleyBooth , nice PR! Left a couple small comments, but I have a bigger one - there's a lot of refactoring into new methods and classes - are we sure this doesn't affect functionality? Is there any validation to look at, e.g. running a bundled example like CMS or CFD and seeing that they give the same output? |
Added
New autoencoder model in models.py - AE_float32
When Baler compressed float32 data using the default AE model, it resulted in compressed files larger than the original. This was due to the layers all being hardcoded to float64. Using this model with float32 data avoids the issue.
dtype=torch.float32c.float_dtype = "float32"to the project config file.Lossy Compression Comparison functionality (compare.py)
New baler operating mode defined in baler.py to benchmark baler performance on the current project against a selection of lossy compression approaches
--mode compare