fixes needed to run Harvester at NERSC #72
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
doesn't the worker maker create the needed directories already. If the directories don't exist wouldn't it be better to issue an error with the shared file messenger than just create the missing directory. |
tmaeno
reviewed
Feb 24, 2018
| @@ -285,6 +285,8 @@ def feed_jobs(self, workspec, jobspec_list): | |||
| pandaIDs = [] | |||
| for jobSpec in jobspec_list: | |||
| accessPoint = self.get_access_point(workspec, jobSpec.PandaID) | |||
| if not os.path.exists(accessPoint): | |||
There was a problem hiding this comment.
setup_access_points() does the same
and it is called before feed_jobs() in submitter
So I'm not sure why os.mkdirs() is required here.
|
HI Tadashi, Doug,
setup_access_point(self, workspec_list) require a list of workspec, while in feed_jobs we only know one workspec. Can I construct a single element list from workspec and call setup_access_point()?
regards,
--
Wei Yang | [email protected] | 650-926-3338
From: Tadashi Maeno <[email protected]>
Reply-To: PanDAWMS/panda-harvester <[email protected]>
Date: Saturday, February 24, 2018 at 4:24 AM
To: PanDAWMS/panda-harvester <[email protected]>
Cc: "Yang, Wei" <[email protected]>, Author <[email protected]>
Subject: Re: [PanDAWMS/panda-harvester] fixes needed to run Harvester at NERSC (#72)
@tmaeno commented on this pull request.
________________________________________
In
pandaharvester/harvestermessenger/shared_file_messenger.py <#72 (comment)>:
@@ -285,6 +285,8 @@ def feed_jobs(self, workspec, jobspec_list):
pandaIDs = []
for jobSpec in jobspec_list:
accessPoint = self.get_access_point(workspec, jobSpec.PandaID)
+ if not os.path.exists(accessPoint):
setup_access_points() does the same
https://github.com/PanDAWMS/panda-harvester/blob/master/pandaharvester/harvestermessenger/shared_file_messenger.py#L465
and it is called before feed_jobs() in submitter
https://github.com/PanDAWMS/panda-harvester/blob/master/pandaharvester/harvesterbody/submitter.py#L191
So I'm not sure why os.mkdirs() is required here.
—
You are receiving this because you authored the thread.
Reply to this email directly,
view it on GitHub <#72 (review)>, or
mute the thread <https://github.com/notifications/unsubscribe-auth/AE9TAzpEJXWfEIokEngETCTYcZxPUKcTks5tX_99gaJpZM4SR0DW>.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/PanDAWMS/panda-harvester","title":"PanDAWMS/panda-harvester","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/PanDAWMS/panda-harvester"}},"updates":{"snippets":[{"icon":"PERSON","message":"@tmaeno commented on #72"}],"action":{"name":"View Pull Request","url":"#72 (review)"}}}
|
> Can I construct a single element list from workspec and call setup_access_point()? That is possible like https://github.com/PanDAWMS/panda-harvester/blob/master/pandaharvester/harvestertest/submitterTest.py#L51 In any case, setup_access_point() needs to be called independently of feed_jobs() since some workflows like pull and job late-binding don't call feed_jobs() before other methods which require the access point. I've removed the mkdirs stuff to merge your pull request. Please submit another pull request and explain the reason there if you still think that the dir making is required in feed_jobs().
![https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open](https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open){kind=link}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please check if the fix in shared_file_messenger.py is appropriate.