PermissionError on Docker when using a non default UID

[giorgiotobia]

Hello

I'm using airflow 2.2.5 also to run some workload on Docker Swarm.
Everything works fine using standard user airflow (uid 50000).
Now i want to setup a new worker running with a different UID.
I modified the entrypoint script in order to add the user "default" to the docker group, as i previously did statically in the Dockerfile that customize my airflow image.

The worker container now runs with user "default", with the desired UID and with docker as additional group, listening on "myqueue" queue.

default@6beee633f9c1:/opt/airflow$ id
uid=498238284(default) gid=0(root) groups=0(root),99521(docker)

If i run a dockerswarmoperator dag on the default worker it works. Groups are the same.

airflow@3780a225bbbd:/opt/airflow$ id
uid=50000(airflow) gid=0(root) groups=0(root),99521(docker)

If I add queue='myqueue' to the dag definition, the task fails

I checked connection and it seems fine to me.

default@6beee633f9c1:/opt/airflow$ python
Python 3.7.13 (default, Apr 7 2022, 05:52:19)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import docker
client = docker.APIClient(base_url='unix://var/run/docker.sock')
client.version()
{'Version': '1.13.1', 'ApiVersion': '1.26', 'MinAPIVersion': '1.12', 'GitCommit': '64e9980/1.13.1', 'GoVersion': 'go1.10.3', 'Os': 'linux', 'Arch': 'amd64', 'KernelVersion': '3.10.0-1160.59.1.el7.x86_64', 'Experimental': True, 'BuildTime': '2020-06-22T03:20:20.124401455-04:00', 'PkgVersion': 'docker-1.13.1-162.git64e9980.el7_8.x86_64'}

client.version(api_version=False)["ApiVersion"]
'1.26'

The task fails with these messages:

[2022-07-05, 12:50:01 UTC] {taskinstance.py:1270} INFO - Executing <Task(DockerSwarmOperator): task1> on 2022-07-05 12:40:00+00:00
[2022-07-05, 12:50:01 UTC] {standard_task_runner.py:52} INFO - Started process 5374 to run task
[2022-07-05, 12:50:01 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'doSomethingWithDocker', 'task1', 'scheduled__2022-07-05T12:40:00+00:00', '--job-id', '13615', '--raw', '--subdir', 'DAGS_FOLDER/bbl_dosomething.py', '--cfg-path', '/tmp/tmpsoe18k7n', '--error-file', '/tmp/tmp40bwce6x']
[2022-07-05, 12:50:01 UTC] {standard_task_runner.py:80} INFO - Job 13615: Subtask task1
[2022-07-05, 12:50:01 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: doSomethingWithDocker.task1 scheduled__2022-07-05T12:40:00+00:00 [running]> on host 6beee633f9c1
[2022-07-05, 12:50:01 UTC] {taskinstance.py:1448} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=default
AIRFLOW_CTX_DAG_ID=doSomethingWithDocker
AIRFLOW_CTX_TASK_ID=task1
AIRFLOW_CTX_EXECUTION_DATE=2022-07-05T12:40:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-07-05T12:40:00+00:00
[2022-07-05, 12:50:01 UTC] {taskinstance.py:1774} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 450, in send
timeout=timeout
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in get
return self.request('GET', url, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker_swarm.py", line 116, in execute
self.cli = self._get_cli()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 390, in _get_cli
return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config)
File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in init
self._version = self._retrieve_server_version()
File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
f'Error while fetching server API version: {e}'
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[2022-07-05, 12:50:02 UTC] {taskinstance.py:1288} INFO - Marking task as FAILED. dag_id=doSomethingWithDocker, task_id=task1, execution_date=20220705T124000, start_date=20220705T125001, end_date=20220705T125002
[2022-07-05, 12:50:02 UTC] {standard_task_runner.py:98} ERROR - Failed to execute job 13615 for task task1 (Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied')); 5374)

[giorgiotobia]

I can even run the dag manually from command line and it succeed.

airflow dags test docker_swarm_dag $(date +%Y-%m-%d)

[giorgiotobia]

Solved, i need to update docker-compose in order to use group_add.
I didn't find any other way to make the shell in which airflow is running aware of the supplementary docker group

[reyibf624]

Solved, i need to: sudo chmod 666 /var/run/docker.sock

[GraphtyLove]

Works on dev set-up but should not be used in production. This is the same has giving sudo power to the user.

[tseth92]

Then what should be followed in production @GraphtyLove ?