Skip to content

Misiug/multi node docker containers #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
misiugodfrey wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Misiug/multi node docker containers #134

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0ad9ed7
Added support for multiple gpu workers on the same node
misiugodfrey Nov 19, 2025
a496b61
Changed scripts to use services instead of all
misiugodfrey Nov 24, 2025
4a21359
fixed worker id
misiugodfrey Nov 24, 2025
bf7ac7d
Merge branch 'main' into misiug/MultiNodeDockerContainers
misiugodfrey Nov 24, 2025
0d33647
Rewrite to dynamically generate multi-worker dockerfile
misiugodfrey Dec 11, 2025
52a4755
Rewrite to dynamically generate multi-worker dockerfile
misiugodfrey Dec 11, 2025
ae18cea
Removed port change
misiugodfrey Dec 11, 2025
d265e0b
removed unnecessary code
misiugodfrey Dec 11, 2025
6753aea
generate all native-gpu dockerfiles
misiugodfrey Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion presto/docker/config/template/etc_worker/config_native.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# This node is a worker; it executes tasks but doesn't coordinate queries.
coordinator=false
# Worker REST/HTTP port for internal and admin endpoints.
http-server.http.port=8080
http-server.http.port=8081
Copy link
Contributor Author

@misiugodfrey misiugodfrey Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This port should be distinct from the port used to connect to the presto-coordinator if they are on the same machine.

Copy link
Contributor

@paul-aiyedun paul-aiyedun Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please expand on why this is required now? This should be running as a separate service on the docker network.

Copy link
Contributor Author

@misiugodfrey misiugodfrey Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found it to be necessary when working in Slurm environments, but that is because there is no docker network. I'll remove this as it isn't a change that is required for the scope of this PR.

# Address of the coordinator's discovery service for registration.
discovery.uri=http://presto-coordinator:8080

Expand Down
41 changes: 41 additions & 0 deletions presto/docker/docker-compose.native-gpu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,44 @@ services:
- ./config/generated/gpu/etc_common:/opt/presto-server/etc
- ./config/generated/gpu/etc_worker/node.properties:/opt/presto-server/etc/node.properties
- ./config/generated/gpu/etc_worker/config_native.properties:/opt/presto-server/etc/config.properties

# These workers are available to run on one node with a single GPU pinned to each.
Copy link
Contributor Author

@misiugodfrey misiugodfrey Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would have been nice to de-duplicate this code, but AFAIK we can't meta-program services with different NVIDIA_VISIBLE_DEVICES.

presto-native-worker-gpu-0:
Copy link
Contributor

@paul-aiyedun paul-aiyedun Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We seem to be hardcoding and deploying a fixed number of workers?

extends:
service: presto-native-worker-gpu
container_name: presto-native-worker-gpu-0
environment:
- NVIDIA_VISIBLE_DEVICES=0
volumes:
- ./config/generated/gpu/etc_worker_0/node.properties:/opt/presto-server/etc/node.properties
- ./config/generated/gpu/etc_worker_0/config_native.properties:/opt/presto-server/etc/config.properties

presto-native-worker-gpu-1:
extends:
service: presto-native-worker-gpu
container_name: presto-native-worker-gpu-1
environment:
- NVIDIA_VISIBLE_DEVICES=1
volumes:
- ./config/generated/gpu/etc_worker_1/node.properties:/opt/presto-server/etc/node.properties
- ./config/generated/gpu/etc_worker_1/config_native.properties:/opt/presto-server/etc/config.properties

presto-native-worker-gpu-2:
extends:
service: presto-native-worker-gpu
container_name: presto-native-worker-gpu-2
environment:
- NVIDIA_VISIBLE_DEVICES=2
volumes:
- ./config/generated/gpu/etc_worker_2/node.properties:/opt/presto-server/etc/node.properties
- ./config/generated/gpu/etc_worker_2/config_native.properties:/opt/presto-server/etc/config.properties

presto-native-worker-gpu-3:
extends:
service: presto-native-worker-gpu
container_name: presto-native-worker-gpu-3
environment:
- NVIDIA_VISIBLE_DEVICES=3
volumes:
- ./config/generated/gpu/etc_worker_3/node.properties:/opt/presto-server/etc/node.properties
- ./config/generated/gpu/etc_worker_3/config_native.properties:/opt/presto-server/etc/config.properties
29 changes: 27 additions & 2 deletions presto/scripts/start_presto_helper.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,19 +66,22 @@ function conditionally_add_build_target() {
echo "Added $2 to the list of services to build because the '$BUILD_TARGET' build target was specified"
BUILD_TARGET_ARG+=($2)
fi
}
}

conditionally_add_build_target $COORDINATOR_IMAGE $COORDINATOR_SERVICE "coordinator|c"

if [[ "$VARIANT_TYPE" == "java" ]]; then
DOCKER_COMPOSE_FILE="java"
conditionally_add_build_target $JAVA_WORKER_IMAGE $JAVA_WORKER_SERVICE "worker|w"
WORKERS="$JAVA_WORKER_SERVICE"
Copy link
Contributor Author

@misiugodfrey misiugodfrey Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are no longer guaranteed to want all the services in a docker-compose file to run, we need to specify which default worker service we want to run.

elif [[ "$VARIANT_TYPE" == "cpu" ]]; then
DOCKER_COMPOSE_FILE="native-cpu"
conditionally_add_build_target $CPU_WORKER_IMAGE $CPU_WORKER_SERVICE "worker|w"
WORKERS="$CPU_WORKER_SERVICE"
elif [[ "$VARIANT_TYPE" == "gpu" ]]; then
DOCKER_COMPOSE_FILE="native-gpu"
conditionally_add_build_target $GPU_WORKER_IMAGE $GPU_WORKER_SERVICE "worker|w"
WORKERS="$GPU_WORKER_SERVICE"
else
echo "Internal error: unexpected VARIANT_TYPE value: $VARIANT_TYPE"
fi
Expand Down Expand Up @@ -130,4 +133,26 @@ if (( ${#BUILD_TARGET_ARG[@]} )); then
${BUILD_TARGET_ARG[@]}
fi

docker compose -f $DOCKER_COMPOSE_FILE_PATH up -d
function duplicate_worker_configs() {
Copy link
Contributor Author

@misiugodfrey misiugodfrey Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is based on the way we do multiple configs for the Slurm clusters right now. This will probably be changed in the future.

local worker_config="../docker/config/generated/gpu/etc_worker_${1}"
rm -rf ${worker_config}
cp -r ../docker/config/generated/gpu/etc_worker ${worker_config}

sed -i "s+http-server\.http\.port.*+http-server\.http\.port=808${1}+g" \
${worker_config}/config_native.properties
sed -i "s+single-node-execution-enabled.*+single-node-execution-enabled=false+g" \
${worker_config}/config_native.properties

# Give each worker a unique id.
sed -i "s+node\.id.*+node\.id=worker_${1}+g" ${worker_config}/node.properties
}

if [[ -n "$NUM_WORKERS" && "$VARIANT_TYPE" == "gpu" ]]; then
WORKERS=""
for i in $( seq 0 $(( $NUM_WORKERS - 1 )) ); do
WORKERS="$WORKERS ${GPU_WORKER_SERVICE}-${i}"
duplicate_worker_configs $i
done
fi

docker compose -f $DOCKER_COMPOSE_FILE_PATH up -d $WORKERS
Loading