Skip to content

Caching integration #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
maxfischer2781 wants to merge 159 commits into
base: master
Choose a base branch
from
Open

Caching integration #72

maxfischer2781 wants to merge 159 commits into from

Conversation

@maxfischer2781
Copy link
Member

@maxfischer2781 maxfischer2781 commented Nov 26, 2019

This PR merges the current caching draft into master. Changes include:

  • model storage and caches attached to pools/drones,
  • ...

tfesenbecker and others added 30 commits October 31, 2019 16:54
@tfesenbecker
added storage object to represent caches and associated readout funct…
9215d64 
…ionality
@tfesenbecker
extended CLI to support storage files
0a46da7
@tfesenbecker
extended simulator to support storage files
e727b69
@tfesenbecker
added new drone attribute sitename connecting drones and storage elem…
d64954d 
…ents
@tfesenbecker
added file provider object connecting storage objects and jobs
5597434
@tfesenbecker
added different caching/cache cleaning/walltime recalculation algorithms
bb1dcbe
@tfesenbecker
renamed storage readout
8d6db96
@tfesenbecker
fixed debug output
fb150db
@tfesenbecker
renamed storage input reader
69072ae
@tfesenbecker
updated Job class
53ebec5
@tfesenbecker
replaced function modifying walltime by function with property decorator
f997223
@tfesenbecker
Revert "replaced function modifying walltime by function with propert…
2e2c06f 
…y decorator"

This reverts commit f997223.
@tfesenbecker
replaced function modifying walltime by function with property decorator
110b3e9
@tfesenbecker
resolving PEP8 issues
b032a0d
@tfesenbecker
Merge branch 'master' of https://github.com/MatterMiners/lapis into c…
7753d0d 
…achingextension
@tfesenbecker
fixed file provider bug (wrong inputfiles dictionary)
5123034
@tfesenbecker @eileen-kuehn
Update lapis/cli/simulate.py
1c2fe9f 
Co-Authored-By: Eileen Kuehn <[email protected]>
@tfesenbecker
renamed function get_used_storage to _calculate_used_storage
8739ce9
@tfesenbecker
Merge branch 'cachingextension' of https://github.com/tfesenbecker/lapis
0b5a922 
 into cachingextension
@tfesenbecker
attached fileprovider to drone instead of job and passed it via make_…
855242a 
…drone instead of scheduler
@tfesenbecker
reworked file coverage function to return a score
bfadacb
@tfesenbecker
added proper __repr__ function
3f30c58
@tfesenbecker
added file classes
2b214aa
@tfesenbecker
moved caching algorithm and associated cache cleanup to it's own class
2bd91d7
@tfesenbecker
Redesign of the storage class and associated changes
29576eb
@tfesenbecker
put walltime getter and walltime recalculation back in seperate methods
146fbe3
@tfesenbecker
added parallel treatment of jobs input files in file provider
7ef8dd9
@tfesenbecker
fixed failed unit test that were caused by Drone without file provide…
b94ab82 
…r, one unit test fails because job has no drone, ticket already exists
@tfesenbecker
Merge branch 'master' of https://github.com/MatterMiners/lapis into c…
1e9e795 
…achingextension
@tfesenbecker
changed scoring to take filesizes into consideration
191df2b
eileen-kuehn and others added 27 commits November 29, 2019 12:15
@eileen-kuehn
added docstrings to storage interface
c7af203
@eileen-kuehn
extended tests
0019583
@eileen-kuehn
removed public update method from storage and made update private
01efb93
@tfesenbecker
added unit conversion for storageelement connection and remote connec…
a7177f7 
…tion throughput_limit, filesize, usedsize
@tfesenbecker
reformated debug output
d684dc8
@tfesenbecker
added debug output and fixed wrong function call
328e6dc
@tfesenbecker
added debug output and fixed wrong function call
0e49c32
@tfesenbecker
Merge branch 'feature/caching' of https://github.com/MatterMiners/lapis…
98dbba1 
… into feature/caching
@tfesenbecker
Merge branch 'master' of https://github.com/MatterMiners/lapis into f…
93589ba 
…eature/caching
@tfesenbecker
reduced debug output, added default value for job_repr
2351f51
@tfesenbecker
Added unittests to test functionality of Connection class with Hitrat…
9e3a446 
…eStorage storage elements
@tfesenbecker
extended type hints
5cdcf35
@tfesenbecker
fixed error in _usedstorage calculation
2747bce
@tfesenbecker
storage.py now catches exception caused by not specified storage cont…
4f6132f 
…ent config
@tfesenbecker
fixed test_storage unit tests
fa0e98f
@tfesenbecker
added new unit tests for hitrate based caching
d01ba77
@tfesenbecker
removed forgotten debug output
5477344
@tfesenbecker
storage_content_reader() handles empty files correctly now
2dc77ff
@tfesenbecker
fixed line length
c573ac5
@tfesenbecker
fixed line length
f6fec85
@maxfischer2781
Monitor.run now properly closes the sampling aiter
2fb8afe
@maxfischer2781
made the linter happy
a5d13de
@tfesenbecker
added default value for calculation_efficiency
d277354
@tfesenbecker
remove calculation_efficiency default value
a165249
@tfesenbecker
corrected wrong attribute names
691d523
@tfesenbecker
fixed usage of transfer() interface and debug outputs
b8ae864
@tfesenbecker
fixed debug output leading to failing unit test
73d6352
@tfesenbecker tfesenbecker marked this pull request as ready for review December 3, 2019 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eileen-kuehn eileen-kuehn eileen-kuehn left review comments

@tfesenbecker tfesenbecker tfesenbecker left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@maxfischer2781 @pep8speaks @tfesenbecker @eileen-kuehn