From 229df11ea0593d84e0e538a9bdd180fc600f5e70 Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Fri, 26 Dec 2025 17:24:12 +0000 Subject: [PATCH 01/10] Rewords the introduction. --- docs/source/user-guide/introduction.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/source/user-guide/introduction.md b/docs/source/user-guide/introduction.md index 3f153de326..9bebd15cad 100644 --- a/docs/source/user-guide/introduction.md +++ b/docs/source/user-guide/introduction.md @@ -21,9 +21,10 @@ Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache DataFusion. -Ballista has a scheduler and an executor process that are standard Rust executables and can be executed directly, but -Dockerfiles are provided to build images for use in containerized environments, such as Docker, Docker Compose, and -Kubernetes. See the [deployment guide](deployment.md) for more information +Ballista has both scheduler and an executor component processes that are standard Rust executables. + +Dockerfiles are also provided to build images for use in containerized environments, such as Docker, Docker Compose, +and Kubernetes. See the [deployment guide](deployment.md) for more information SQL and DataFrame queries can be submitted from Python and Rust, and SQL queries can be submitted via the Arrow Flight SQL JDBC driver, supporting your favorite JDBC compliant tools such as [DataGrip](datagrip) @@ -31,7 +32,7 @@ or [tableau](tableau). For setup instructions, please see the [FlightSQL guide]( ## How does this compare to Apache Spark? -Although Ballista is largely inspired by Apache Spark, there are some key differences. +Although Ballista is largely inspired by Apache Spark, there are some key differences: - The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead of GC pauses. From f164fd29b1f2696bc7a01def954021816cb45428 Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Sat, 27 Dec 2025 11:41:47 +0000 Subject: [PATCH 02/10] Rewords the cargo install section. --- docs/source/user-guide/deployment/cargo-install.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user-guide/deployment/cargo-install.md b/docs/source/user-guide/deployment/cargo-install.md index 782073fde6..9bd99e67a5 100644 --- a/docs/source/user-guide/deployment/cargo-install.md +++ b/docs/source/user-guide/deployment/cargo-install.md @@ -19,7 +19,7 @@ # Deploying a standalone Ballista cluster using cargo install -A simple way to start a local cluster for testing purposes is to use cargo to install +Another simple way to start a local cluster for testing purposes is to use cargo to install the scheduler and executor crates. ```bash From 9ce7b0f51d49d1ae443732f85aa2f4c87b5fbfe0 Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Sat, 27 Dec 2025 17:13:11 +0000 Subject: [PATCH 03/10] Rewords the docker section and updates builder Dockerfile. --- dev/docker/ballista-builder.Dockerfile | 3 +-- docs/source/user-guide/deployment/docker.md | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/dev/docker/ballista-builder.Dockerfile b/dev/docker/ballista-builder.Dockerfile index 2347f1cd03..e2c8dd8768 100644 --- a/dev/docker/ballista-builder.Dockerfile +++ b/dev/docker/ballista-builder.Dockerfile @@ -31,8 +31,7 @@ RUN curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \ apt-get install -y nodejs && \ npm install -g yarn -# create build user with same UID as -RUN adduser -q -u $EXT_UID builder --home /home/builder && \ +RUN adduser -q builder --home /home/builder && \ mkdir -p /home/builder/workspace USER builder diff --git a/docs/source/user-guide/deployment/docker.md b/docs/source/user-guide/deployment/docker.md index 971d1780e5..a054237749 100644 --- a/docs/source/user-guide/deployment/docker.md +++ b/docs/source/user-guide/deployment/docker.md @@ -27,7 +27,7 @@ Run the following commands to download the [official Docker image](https://githu docker pull ghcr.io/apache/datafusion-ballista-standalone:latest ``` -Altenatively run the following commands to clone the source repository and build the Docker images from source: +Alternatively run the following commands to clone the source repository and build the Docker images from source: ```bash git clone git@github.com:apache/datafusion-ballista.git From b168ec056ba37c80d2ec8c50ea90907efb7f7dad Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Mon, 29 Dec 2025 12:49:54 +0000 Subject: [PATCH 04/10] Rewords the docker compose, kubernetes, and scheduler sections. --- .../user-guide/deployment/docker-compose.md | 22 +-------------- .../user-guide/deployment/kubernetes.md | 28 +++---------------- docs/source/user-guide/scheduler.md | 2 +- 3 files changed, 6 insertions(+), 46 deletions(-) diff --git a/docs/source/user-guide/deployment/docker-compose.md b/docs/source/user-guide/deployment/docker-compose.md index 3c7eb4f7bf..7d6c041807 100644 --- a/docs/source/user-guide/deployment/docker-compose.md +++ b/docs/source/user-guide/deployment/docker-compose.md @@ -23,27 +23,7 @@ Docker Compose is a convenient way to launch a cluster when testing locally. ## Build Docker Images -Run the following commands to download the [official Docker image](https://github.com/apache/datafusion-ballista/pkgs/container/datafusion-ballista-standalone): - -```bash -docker pull ghcr.io/apache/datafusion-ballista-standalone:latest -``` - -Altenatively run the following commands to clone the source repository and build the Docker images from source: - -```bash -git clone git@github.com:apache/datafusion-ballista.git -b latest -cd datafusion-ballista -./dev/build-ballista-docker.sh -``` - -This will create the following images: - -- `apache/datafusion-ballista-benchmarks:latest` -- `apache/datafusion-ballista-cli:latest` -- `apache/datafusion-ballista-executor:latest` -- `apache/datafusion-ballista-scheduler:latest` -- `apache/datafusion-ballista-standalone:latest` +To create the required Docker images please refer to the previous page. ## Start a Cluster diff --git a/docs/source/user-guide/deployment/kubernetes.md b/docs/source/user-guide/deployment/kubernetes.md index f2b7e5ad61..047652ae67 100644 --- a/docs/source/user-guide/deployment/kubernetes.md +++ b/docs/source/user-guide/deployment/kubernetes.md @@ -41,27 +41,7 @@ microk8s enable dns ## Build Docker Images -Run the following commands to download the [official Docker image](https://github.com/apache/datafusion-ballista/pkgs/container/datafusion-ballista-standalone): - -```bash -docker pull ghcr.io/apache/datafusion-ballista-standalone:0.12.0-rc4 -``` - -Altenatively run the following commands to clone the source repository and build the Docker images from source: - -```bash -git clone git@github.com:apache/datafusion-ballista.git -b 0.12.0 -cd datafusion-ballista -./dev/build-ballista-docker.sh -``` - -This will create the following images: - -- `apache/datafusion-ballista-benchmarks:0.12.0` -- `apache/datafusion-ballista-cli:0.12.0` -- `apache/datafusion-ballista-executor:0.12.0` -- `apache/datafusion-ballista-scheduler:0.12.0` -- `apache/datafusion-ballista-standalone:0.12.0` +To create the required Docker images please refer to the Deployment/Docker page. ## Publishing Docker Images @@ -267,9 +247,9 @@ kubectl delete -f cluster.yaml ## Autoscaling Executors -Ballista supports autoscaling for executors through [Keda](http://keda.sh). Keda allows scaling a deployment -through custom metrics which are exposed through the Ballista scheduler, and it can even scale the number of -executors down to 0 if there is no activity in the cluster. +Ballista supports autoscaling for executors through [Keda](http://keda.sh). Keda allows for the scaling of a +deployment through custom metrics which are exposed through the Ballista scheduler, and it +can even scale the number of executors down to 0 if there is no activity in the cluster. Keda can be installed in your kubernetes cluster through a single command line: diff --git a/docs/source/user-guide/scheduler.md b/docs/source/user-guide/scheduler.md index be608ce6d4..81f7aecac6 100644 --- a/docs/source/user-guide/scheduler.md +++ b/docs/source/user-guide/scheduler.md @@ -23,7 +23,7 @@ The scheduler also provides a REST API that allows jobs to be monitored. -> This is optional scheduler feature which should be enabled with `rest-api` feature +> This is optional scheduler feature which should be enabled with the `rest-api` feature | API | Method | Description | | ------------------------------------ | ------ | ----------------------------------------------------------------- | From 035291a7ef0d1e5da8594d22ac2df869b062c7df Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Mon, 29 Dec 2025 14:06:06 +0000 Subject: [PATCH 05/10] Rewords the docker compose, kubernetes, and scheduler sections. --- docs/source/user-guide/deployment/docker-compose.md | 4 ++-- docs/source/user-guide/deployment/kubernetes.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/user-guide/deployment/docker-compose.md b/docs/source/user-guide/deployment/docker-compose.md index 7d6c041807..fcec5bdcc6 100644 --- a/docs/source/user-guide/deployment/docker-compose.md +++ b/docs/source/user-guide/deployment/docker-compose.md @@ -23,8 +23,8 @@ Docker Compose is a convenient way to launch a cluster when testing locally. ## Build Docker Images -To create the required Docker images please refer to the previous page. - +To create the required Docker images please refer to the [docker deployment page](docker.md). +[](docker.md) ## Start a Cluster Using the [docker-compose.yml](https://github.com/apache/datafusion-ballista/blob/main/docker-compose.yml) from the diff --git a/docs/source/user-guide/deployment/kubernetes.md b/docs/source/user-guide/deployment/kubernetes.md index 047652ae67..b1a1aa12e7 100644 --- a/docs/source/user-guide/deployment/kubernetes.md +++ b/docs/source/user-guide/deployment/kubernetes.md @@ -41,7 +41,7 @@ microk8s enable dns ## Build Docker Images -To create the required Docker images please refer to the Deployment/Docker page. +To create the required Docker images please refer to the [docker deployment page](docker.md). ## Publishing Docker Images From b475d8c2c0745074d16d36c63c08c7f19a889037 Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Mon, 29 Dec 2025 14:48:09 +0000 Subject: [PATCH 06/10] Adds a link to the docker section for the docker compose section. --- docs/source/user-guide/deployment/docker-compose.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/source/user-guide/deployment/docker-compose.md b/docs/source/user-guide/deployment/docker-compose.md index fcec5bdcc6..87d41c8241 100644 --- a/docs/source/user-guide/deployment/docker-compose.md +++ b/docs/source/user-guide/deployment/docker-compose.md @@ -22,9 +22,8 @@ Docker Compose is a convenient way to launch a cluster when testing locally. ## Build Docker Images - To create the required Docker images please refer to the [docker deployment page](docker.md). -[](docker.md) + ## Start a Cluster Using the [docker-compose.yml](https://github.com/apache/datafusion-ballista/blob/main/docker-compose.yml) from the From c88043b3e005c4fbf207e3a7d30ea25d6d41ef86 Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Mon, 29 Dec 2025 14:49:39 +0000 Subject: [PATCH 07/10] Adds a full stop --- docs/source/user-guide/introduction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user-guide/introduction.md b/docs/source/user-guide/introduction.md index 9bebd15cad..9fa01b0557 100644 --- a/docs/source/user-guide/introduction.md +++ b/docs/source/user-guide/introduction.md @@ -24,7 +24,7 @@ Ballista is a distributed compute platform primarily implemented in Rust, and po Ballista has both scheduler and an executor component processes that are standard Rust executables. Dockerfiles are also provided to build images for use in containerized environments, such as Docker, Docker Compose, -and Kubernetes. See the [deployment guide](deployment.md) for more information +and Kubernetes. See the [deployment guide](deployment.md) for more information. SQL and DataFrame queries can be submitted from Python and Rust, and SQL queries can be submitted via the Arrow Flight SQL JDBC driver, supporting your favorite JDBC compliant tools such as [DataGrip](datagrip) From 88440587439c2ad35f5cbe0080629dd0ffb7a4c5 Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Mon, 29 Dec 2025 14:51:00 +0000 Subject: [PATCH 08/10] Adds full stop --- docs/source/user-guide/scheduler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user-guide/scheduler.md b/docs/source/user-guide/scheduler.md index 81f7aecac6..26581f8cb4 100644 --- a/docs/source/user-guide/scheduler.md +++ b/docs/source/user-guide/scheduler.md @@ -23,7 +23,7 @@ The scheduler also provides a REST API that allows jobs to be monitored. -> This is optional scheduler feature which should be enabled with the `rest-api` feature +> This is optional scheduler feature which should be enabled with the `rest-api` feature. | API | Method | Description | | ------------------------------------ | ------ | ----------------------------------------------------------------- | From 6161b963913005e62abcf15224f0c55122be1934 Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Mon, 29 Dec 2025 18:48:32 +0000 Subject: [PATCH 09/10] Prettifies the docs. --- docs/source/user-guide/deployment/docker-compose.md | 1 + docs/source/user-guide/deployment/kubernetes.md | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/source/user-guide/deployment/docker-compose.md b/docs/source/user-guide/deployment/docker-compose.md index 87d41c8241..67f40b7a52 100644 --- a/docs/source/user-guide/deployment/docker-compose.md +++ b/docs/source/user-guide/deployment/docker-compose.md @@ -22,6 +22,7 @@ Docker Compose is a convenient way to launch a cluster when testing locally. ## Build Docker Images + To create the required Docker images please refer to the [docker deployment page](docker.md). ## Start a Cluster diff --git a/docs/source/user-guide/deployment/kubernetes.md b/docs/source/user-guide/deployment/kubernetes.md index b1a1aa12e7..d3062ed843 100644 --- a/docs/source/user-guide/deployment/kubernetes.md +++ b/docs/source/user-guide/deployment/kubernetes.md @@ -247,8 +247,8 @@ kubectl delete -f cluster.yaml ## Autoscaling Executors -Ballista supports autoscaling for executors through [Keda](http://keda.sh). Keda allows for the scaling of a -deployment through custom metrics which are exposed through the Ballista scheduler, and it +Ballista supports autoscaling for executors through [Keda](http://keda.sh). Keda allows for the scaling of a +deployment through custom metrics which are exposed through the Ballista scheduler, and it can even scale the number of executors down to 0 if there is no activity in the cluster. Keda can be installed in your kubernetes cluster through a single command line: From 8467d8a6a56f6445f7f1bac940cc987c9a87d63d Mon Sep 17 00:00:00 2001 From: Louis Burke Date: Thu, 1 Jan 2026 11:08:34 +0000 Subject: [PATCH 10/10] Rolls back change to builder Dockerfile. --- dev/docker/ballista-builder.Dockerfile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/dev/docker/ballista-builder.Dockerfile b/dev/docker/ballista-builder.Dockerfile index e2c8dd8768..2347f1cd03 100644 --- a/dev/docker/ballista-builder.Dockerfile +++ b/dev/docker/ballista-builder.Dockerfile @@ -31,7 +31,8 @@ RUN curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \ apt-get install -y nodejs && \ npm install -g yarn -RUN adduser -q builder --home /home/builder && \ +# create build user with same UID as +RUN adduser -q -u $EXT_UID builder --home /home/builder && \ mkdir -p /home/builder/workspace USER builder