galaxy-celery-beat.service fails to start (Galaxy 23)
#187
Comments
kysrpex
changed the title
galaxy-celery-beat.service fails to startgalaxy-celery-beat.service fails to start (Galaxy 23)
|
The issue is directly related to this change committed yesterday. |
|
Interestingly running the |
|
After having a closer look at everything it seems that systemd is being used (obvious from the log I pasted 😅) rather than supervisor, but nevertheless I believe at least an instance of the class |
You can find extra details about my playbook below. @natefoo Do you think it makes sense to include a task that creates ansible.cfg [defaults]
interpreter_python = /usr/bin/python3
inventory = hosts
stdout_callback = yaml
bin_ansible_callbacks = True
vault_password_file = .vault-password.txt
[ssh_connection]
pipelining = true
[diff]
always = Truegalaxy.yml ---
- hosts: dbservers
become: true
become_user: root
pre_tasks:
- name: Install dependencies
package:
name: 'acl'
roles:
- galaxyproject.postgresql
- role: galaxyproject.postgresql_objects
become: true
become_user: postgres
- hosts: galaxyservers
become: true
become_user: root
vars_files:
- group_vars/secret.yml
pre_tasks:
- name: Install dependencies
package:
name:
- acl
- bzip2
- git
- make
- tar
- python3-venv
- python3-setuptools
roles:
- galaxyproject.galaxy
- role: galaxyproject.miniconda
become: true
become_user: "{{ galaxy_user_name }}"hosts [galaxyservers]
gat-2.eu.training.galaxyproject.eu ansible_connection=local ansible_user=ubuntu
[dbservers:children]
galaxyserversrequirements.yml - src: galaxyproject.galaxy
version: 0.10.11
- src: galaxyproject.nginx
version: 0.7.1
- src: galaxyproject.postgresql
version: 1.1.2
- src: galaxyproject.postgresql_objects
version: 1.2.0
- src: galaxyproject.miniconda
version: 0.3.1
- src: usegalaxy_eu.certbot
version: 0.1.7group_vars/all.yml ---
# Python 3 support
pip_virtualenv_command: /usr/bin/python3 -m venv # usegalaxy_eu.certbot, usegalaxy_eu.tiaas2, galaxyproject.galaxy
certbot_virtualenv_package_name: python3-venv # usegalaxy_eu.certbot
# Common variables needed by all hosts
galaxy_user_name: galaxy
galaxy_db_name: galaxygroup_vars/dbservers.yml ---
# PostgreSQL
postgresql_objects_users:
- name: "{{ galaxy_user_name }}"
postgresql_objects_databases:
- name: "{{ galaxy_db_name }}"
owner: "{{ galaxy_user_name }}"group_vars/galaxyservers.yml ---
galaxy_create_user: true
galaxy_separate_privileges: true
galaxy_manage_paths: true
galaxy_layout: root-dir
galaxy_root: /srv/galaxy
galaxy_user:
name: "galaxy"
shell: /bin/bash
galaxy_commit_id: release_23.0
galaxy_force_checkout: true
miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
miniconda_version: 4.12.0
miniconda_channels:
- conda-forge
- defaults
galaxy_config:
galaxy:
admin_users:
- "[email protected]"
brand: "Cool Galaxy 🌌"
database_connection: postgresql:///galaxy?host=/var/run/postgresql
tool_data_path: "{{ galaxy_mutable_data_dir }}/tool-data"
object_store_store_by: uuid
id_secret: "{{ vault_id_secret }}"
gravity:
process_manager: systemd
galaxy_root: "{{ galaxy_root }}/server"
galaxy_user: "{{ galaxy_user_name }}"
virtualenv: "{{ galaxy_venv_dir }}"
gunicorn:
bind: "unix:{{ galaxy_mutable_config_dir }}/gunicorn.sock"
workers: 2
extra_args: '--forwarded-allow-ips="*"'
preload: true
celery:
concurrency: 2
loglevel: DEBUG
handlers:
handler:
processes: 2
pools:
- job-handlers
- workflow-schedulers |
|
This is a Gravity bug, I will see if I can fix it quickly to prevent this going forward. The SupervisorProcessManager is instantiated because all ProcessManagers have to be whenever |
|
This should be fixed by galaxyproject/gravity#105 |
I just checked out this commit from git-gat (that according to the README corresponds to 63f37692cd on training-material) on a fresh VM ( As you say, the service seems to be working ubuntu@gat-5:~/git-gat$ sudo galaxyctl status
UNIT LOAD ACTIVE SUB DESCRIPTION
galaxy-celery-beat.service loaded active running Galaxy celery-beat
galaxy-celery.service loaded active running Galaxy celery
galaxy-gunicorn.service loaded active running Galaxy gunicorn
[email protected] loaded active running Galaxy handler (process 0)
[email protected] loaded active running Galaxy handler (process 1)
galaxy.target loaded inactive dead Galaxy
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
6 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.but the galaxy target is not reached and the logs show a restart loop for |
Yes, that PR fixes it, thanks! root@gat-5:~# su galaxy
galaxy@gat-5:/root$ source /srv/galaxy/venv/bin/activate
(venv) galaxy@gat-5:/root$ pip install git+https://github.com/galaxyproject/gravity.git@22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Collecting git+https://github.com/galaxyproject/gravity.git@22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Cloning https://github.com/galaxyproject/gravity.git (to revision 22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b) to /tmp/pip-req-build-adwr2sko
Running command git clone --filter=blob:none --quiet https://github.com/galaxyproject/gravity.git /tmp/pip-req-build-adwr2sko
Running command git rev-parse -q --verify 'sha^22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b'
Running command git fetch -q https://github.com/galaxyproject/gravity.git 22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Resolved https://github.com/galaxyproject/gravity.git to commit 22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests-unixsocket in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (0.3.0)
Requirement already satisfied: pyyaml in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (6.0)
Requirement already satisfied: requests in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (2.28.2)
Requirement already satisfied: pydantic in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (1.10.4)
Requirement already satisfied: jsonref in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (1.1.0)
Requirement already satisfied: supervisor in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (4.2.5)
Requirement already satisfied: Click in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (8.1.3)
Requirement already satisfied: packaging in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (21.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /srv/galaxy/venv/lib/python3.10/site-packages (from packaging->gravity==1.0.2) (3.0.9)
Requirement already satisfied: typing-extensions>=4.2.0 in /srv/galaxy/venv/lib/python3.10/site-packages (from pydantic->gravity==1.0.2) (4.4.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (1.26.14)
Requirement already satisfied: charset-normalizer<4,>=2 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (2022.12.7)
Requirement already satisfied: setuptools in /srv/galaxy/venv/lib/python3.10/site-packages (from supervisor->gravity==1.0.2) (66.1.1)
(venv) galaxy@gat-5:/root$
exit
root@gat-5:~# ls -lah /srv/galaxy/var/gravity
ls: cannot access '/srv/galaxy/var/gravity': No such file or directory
root@gat-5:~# galaxyctl start
UNIT LOAD ACTIVE SUB DESCRIPTION
galaxy-celery-beat.service loaded active running Galaxy celery-beat
galaxy-celery.service loaded active running Galaxy celery
galaxy-gunicorn.service loaded active running Galaxy gunicorn
[email protected] loaded active running Galaxy handler (process 0)
[email protected] loaded active running Galaxy handler (process 1)
galaxy.target loaded active active Galaxy
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
6 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
root@gat-5:~# ls -lah /srv/galaxy/var/gravity
total 12K
drwxr-xr-x 3 root root 4.0K Apr 13 08:21 .
drwxr-xr-x 14 galaxy galaxy 4.0K Apr 13 08:21 ..
drwxr-xr-x 3 root root 4.0K Apr 13 08:21 supervisor |
Short version
Issue #103 in gravity repository prevents
galaxy-celery-beat.servicefrom starting because it causes the directory{{ galaxy_gravity_state_dir }}(by default/srv/galaxy/var/gravity/) to be created with the wrong ownership.Ultimately this is not a problem with the Galaxy role but Gravity, nevertheless I think it makes sense to have it reported here for anybody experiencing the same issue (even if the issue is closed inmediately, which is probably the right way to proceed, it can be read later).
Long version
I am following the tutorial "Galaxy Installation with Ansible" from the Galaxy
training-materialrepository (commit 37599dd37674e4d2679c4a5d0f2c9bb818e0006b).When I reached the part "Hands-on: Status Check", I noticed that galaxy.target is not reached.
Invoking
galaxyctl followyields the following output, that repeats in a loop.Everything seems to boil down to not having permission to write on
/srv/galaxy/var/gravity/celery-beat-schedule. The directory/srv/galaxy/var/gravityseems in fact to be owned by root instead of by the galaxy user.root@gat-2:/home/ubuntu/galaxy# ls -lah /srv/galaxy/var/gravity/ total 12K drwxr-xr-x 3 root root 4.0K Apr 11 15:16 . drwxr-xr-x 14 galaxy galaxy 4.0K Apr 11 15:16 .. drwxr-xr-x 3 root root 4.0K Apr 11 15:16 supervisorFor the default configuration of the role,
/srv/galaxy/var/gravitycorresponds togalaxy_gravity_state_dir.I am invoking the role with the group variable
galaxy_commit_idset torelease_23.0. The checked out commit in/srv/galaxy/serveris a1bdffa43f5f23769dc32d33909cc4631aa01ca5.Based on this, I believe the reason for
/srv/galaxy/var/gravity/being owned byrootinstead ofgalaxyare the Gravity handlers for Galaxy 23.0 (see main.yml as well).In fact, the handlers call
galaxyctlwith the argumentsupdateandgraceful. Both involve calling the process manager. I assume the actual process manager being spawned isSupervisorProcessManager, because of the folder structureroot@gat-2:/srv/galaxy/server# tree /srv/galaxy/var/gravity/ /srv/galaxy/var/gravity/ └── supervisor └── supervisord.conf.d 2 directories, 0 filesand the paths defined in the
__init__method.This line must be creating not only
/srv/galaxy/var/gravity/supervisor/supervisord.conf.d, but also the top-level directory/srv/galaxy/var/gravity. I assumegalaxyctlis being run then as root, which is what is expected to happen because I also setgravity/process_managertosystemdingroup_vars/galaxyservers.yml(as instructed in the tutorial) and by defaultgalaxy_systemd_rootis set totrue.This does not make much sense now, because systemd should be thus being used instead of supervisor. This may be a bug, and there is already an issue in the gravity repository.
Therefore, ultimately this is not a problem with the Galaxy role but Gravity, nevertheless I think it makes sense to have it reported here for anybody experiencing the same issue (even if the issue is closed inmediately, which is probably the right way to proceed, it can be read later).
The text was updated successfully, but these errors were encountered: