Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for running Galaxy from systemd using Gravity (galaxyctl) #153

Merged
merged 10 commits into from
May 31, 2022
Merged

Support for running Galaxy from systemd using Gravity (galaxyctl) #153

merged 10 commits into from
May 31, 2022

Conversation

natefoo
Copy link
Member

@natefoo natefoo commented Apr 7, 2022

Also, support systemd user scope (systemctl --user ...).

Also fixed #137.

@natefoo natefoo marked this pull request as draft April 7, 2022 22:25
@natefoo
Copy link
Member Author

natefoo commented Apr 7, 2022

Draft until I update the README.

@natefoo
Copy link
Member Author

natefoo commented Apr 7, 2022

galaxy_systemd_mode should probably default to gravity if the version is >= 22.05?

@natefoo
Copy link
Member Author

natefoo commented Apr 8, 2022

galaxy_systemd_mode should probably default to gravity if the version is >= 22.05?

This unfortunately creates a circular dependency - you don't know what the version is until after the clone task, but the clone task notifies the restart handler, the name of which would be templated conditionally based on the version. I haven't been able to come up with a solution.

@guerler
Copy link
Contributor

guerler commented Apr 9, 2022

Is it possible to set a flag upfront (for now) to distinguish the two?

@natefoo
Copy link
Member Author

natefoo commented Apr 11, 2022

Yes, you can currently set it manually, but it will have to be mule (uwsgi) for backwards compatibility unfortunately. Which will completely break under 22.05.

@natefoo
Copy link
Member Author

natefoo commented Apr 15, 2022

Ok I think I got the automatic galaxy_systemd_mode working.

@mvdbeek
Copy link
Member

mvdbeek commented Apr 16, 2022

There's a merge conflict now

@natefoo natefoo marked this pull request as ready for review April 17, 2022 16:24
@natefoo
Copy link
Member Author

natefoo commented Apr 18, 2022

path_join isn't critical, but Ansible 2.9 is end of life as of December 2021, so perhaps we should just update the minimum version to 2.10 rather than make this backwards compatible?

@natefoo
Copy link
Member Author

natefoo commented Apr 22, 2022

I think the restart shortly after starting is the issue here, but I've been unable to come up with a way to avoid it so far.

@guerler guerler requested a review from hexylena April 27, 2022 18:37
@mvdbeek
Copy link
Member

mvdbeek commented May 2, 2022

Can you dump the beat logs ?

@mvdbeek
Copy link
Member

mvdbeek commented May 4, 2022

@guerler and I had a look at this, the celery-beat seemingly fails because of permission issues:

__    -    ... __   -        _
LocalTime -> 2022-05-04 16:36:21
Configuration ->
    . broker -> sqlalchemy+sqlite:////srv/galaxy/var/universe.sqlite
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%DEBUG
    . maxinterval -> 5.00 minutes (300s)
No Galaxy config file found, running from current working directory: /srv/galaxy/server
[2022-05-04 16:36:27,286: DEBUG/MainProcess] Setting default socket timeout to 30
[2022-05-04 16:36:27,286: INFO/MainProcess] beat: Starting...
[2022-05-04 16:36:27,288: ERROR/MainProcess] Removing corrupted schedule file 'celerybeat-schedule': error(13, 'Permission denied')
Traceback (most recent call last):
  File "/srv/galaxy/venv/lib/python3.8/site-packages/celery/beat.py", line 533, in setup_schedule
    self._store = self._open_schedule()
  File "/srv/galaxy/venv/lib/python3.8/site-packages/celery/beat.py", line 523, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python3.8/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python3.8/shelve.py", line 227, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python3.8/dbm/__init__.py", line 95, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 13] Permission denied: 'celerybeat-schedule'
[2022-05-04 16:36:27,289: CRITICAL/MainProcess] beat raised exception <class '_gdbm.error'>: error(13, 'Permission denied')
Traceback (most recent call last):
  File "/srv/galaxy/venv/lib/python3.8/site-packages/celery/beat.py", line 533, in setup_schedule
    self._store = self._open_schedule()
  File "/srv/galaxy/venv/lib/python3.8/site-packages/celery/beat.py", line 523, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python3.8/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python3.8/shelve.py", line 227, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python3.8/dbm/__init__.py", line 95, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 13] Permission denied: 'celerybeat-schedule'

I think the schedule file is going to be created in ${{galaxy_root}}, and that fails with the privsep setup.

@mvdbeek mvdbeek mentioned this pull request May 6, 2022
Merged
5 tasks
@mvdbeek
Copy link
Member

mvdbeek commented May 6, 2022

Looks like the test timed out while Galaxy was still auto-installing Conda. Should we just disable that for the test ?

@guerler
Copy link
Contributor

guerler commented May 10, 2022

@natefoo is it possible to increase the timeout here?

@mvdbeek
Copy link
Member

mvdbeek commented May 10, 2022

it's the Conda auto install, we should just turn that off

@natefoo
Copy link
Member Author

natefoo commented May 11, 2022

Yeah, we do not need Galaxy to autoinstall conda. Most of the time it's not desirable in a production deployment where we will preinstall it e.g. with galaxyproject.miniconda.

@natefoo natefoo merged commit c15de9f into main May 31, 2022
@nsoranzo nsoranzo deleted the gravity branch June 5, 2022 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hexylena hexylena Awaiting requested review from hexylena

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

__galaxy_user_group is undefined if galaxy_manage_paths is not enabled
3 participants
@natefoo @guerler @mvdbeek