KAFKA-15445: Add JVM Docker image (apache#14552)

This PR aims to add Apache Kafka JVM Docker image as per the following KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka

Reviewers:  Ismael Juma <[email protected]>, Ashwin Pankaj <[email protected]>, Manikumar Reddy <[email protected]>, Sanjay Awatramani <[email protected]>, 
Nikita Konev

.github/workflows/docker_build_and_test.yml

@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Docker Build Test
+
+on:
+  workflow_dispatch:
+    inputs:
+      image_type:
+        type: choice
+        description: Docker image type to build and test
+        options: 
+          - "jvm"
+      kafka_url:
+        description: Kafka url to be used to build the docker image
+        required: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.10"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r docker/requirements.txt
+    - name: Build image and run tests
+      working-directory: ./docker
+      run: |
+        python docker_build_test.py kafka/test -tag=test -type=${{ github.event.inputs.image_type }} -u=${{ github.event.inputs.kafka_url }}
+    - name: Run CVE scan
+      uses: aquasecurity/trivy-action@master
+      with:
+        image-ref: 'kafka/test:test'
+        format: 'table'
+        severity: 'CRITICAL,HIGH'
+        output: scan_report_${{ github.event.inputs.image_type }}.txt
+        exit-code: '1'
+    - name: Upload test report
+      if: always()
+      uses: actions/upload-artifact@v3
+      with:
+        name: report_${{ github.event.inputs.image_type }}.html
+        path: docker/test/report_${{ github.event.inputs.image_type }}.html
+    - name: Upload CVE scan report
+      if: always()
+      uses: actions/upload-artifact@v3
+      with:
+        name: scan_report_${{ github.event.inputs.image_type }}.txt
+        path: scan_report_${{ github.event.inputs.image_type }}.txt

.gitignore

@@ -59,3 +59,6 @@ jmh-benchmarks/src/main/generated
 **/src/generated-test
 
 storage/kafka-tiered-storage/
+
+docker/test/report_*.html
+__pycache__

build.gradle

@@ -207,7 +207,9 @@ if (repo != null) {
         'streams/streams-scala/logs/*',
         'licenses/*',
         '**/generated/**',
-        'clients/src/test/resources/serializedData/*'
+        'clients/src/test/resources/serializedData/*',
+        'docker/resources/utility/go.sum',
+        'docker/test/fixtures/secrets/*'
     ])
   }
 } else {

docker/README.md

@@ -0,0 +1,102 @@
+Docker Images
+=============
+
+This directory contains scripts to build, test, push and promote docker image for kafka.
+
+Local Setup
+-----------
+Make sure you have python (>= 3.7.x) and java (>= 17) (java needed only for running tests) installed before running the tests and scripts.
+
+Run `pip install -r requirements.txt` to get all the requirements for running the scripts.
+
+Make sure you have docker installed with support for buildx enabled. (For pushing multi-architecture image to docker registry)
+
+Bulding image and running tests locally
+---------------------------------------
+- `docker_build_test.py` script builds and tests the docker image.
+- kafka binary tarball url along with image name, tag and type is needed to build the image. For detailed usage description check `python docker_build_test.py --help`.
+- Sanity tests for the docker image are present in test/docker_sanity_test.py.
+- By default image will be built and tested, but if you only want to build the image, pass `--build` (or `-b`) flag and if you only want to test the given image pass `--test` (or `-t`) flag.
+- An html test report will be generated after the tests are executed containing the results.
+
+Example command:-
+To build and test an image named test under kafka namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used
+```
+python docker_build_test.py kafka/test --image-tag=3.6.0 --image-type=jvm --kafka-url=https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
+```
+
+Bulding image and running tests using github actions
+----------------------------------------------------
+This is the recommended way to build, test and get a CVE report for the docker image.
+Just choose the image type and provide kafka url to `Docker Build Test` workflow. It will generate a test report and CVE report that can be shared with the community.
+
+kafka-url - This is the url to download kafka tarball from. For example kafka tarball url from (https://archive.apache.org/dist/kafka). For building RC image this will be an RC tarball url.
+
+image-type - This is the type of image that we intend to build. This will be dropdown menu type selection in the workflow. `jvm` image type is for official docker image (to be hosted on apache/kafka) as described in [KIP-975](https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka)
+
+Example command:-
+To build and test a jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following inputs in github actions workflow are recommended.
+```
+image_type: jvm
+kafka_url: https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
+```
+
+Creating a release
+------------------
+- `docker_release.py` script builds a multi-architecture image and pushes it to provided docker registry.
+- Ensure you are logged in to the docker registry before triggering the script.
+- kafka binary tarball url along with image name (in the format `<registry>/<namespace>/<image_name>:<image_tag>`) and type is needed to build the image. For detailed usage description check `python docker_release.py --help`.
+
+Example command:-
+To push an image named test under kafka dockerhub namespace with 3.6.0 tag and jvm image type ensuring kafka to be containerised should be https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz (it is recommended to use scala 2.13 binary tarball), following command can be used. (Make sure you have push access to the docker repo)
+```
+# kafka/test is an example repo. Please replace with the docker hub repo you have push access to.
+
+python docker_release.py kafka/test:3.6.0 --kafka-url https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz
+```
+
+Please note that we use docker buildx for preparing the multi-architecture image and pushing it to docker registry. It's possible to encounter build failures because of buildx. Please retry the command in case some buildx related error occurs.
+
+Promoting a release
+-------------------
+`docker_promote.py` provides an interactive way to pull an RC Docker image and promote it to required dockerhub repo.
+
+Using the image in a docker container
+-------------------------------------
+- The image uses the kafka downloaded from provided kafka url
+- The image can be run in a container in default mode by running
+`docker run -p 9092:9092 <image-name:tag>`
+- Default configs run kafka in kraft mode with plaintext listners on 9092 port.
+- Once user provided config properties are provided default configs will get replaced.
+- User can provide kafka configs following two ways:-
+    - By mounting folder containing property files
+        - Mount the folder containing kafka property files to `/mnt/shared/config`
+        - These files will replace the default config files
+    - Using environment variables
+        - Kafka properties defined via env variables will override properties defined in file input
+        - If properties are provided via environment variables only, default configs will be replaced by user provided properties
+        - Input format for env variables:-
+            - Replace . with _
+            - Replace _ with __(double underscore)
+            - Replace - with ___(triple underscore)
+            - Prefix the result with KAFKA_
+            - Examples:
+                - For abc.def, use KAFKA_ABC_DEF
+                - For abc-def, use KAFKA_ABC___DEF
+                - For abc_def, use KAFKA_ABC__DEF
+- Hence order of precedence of properties is the following:-
+    - Env variable (highest)
+    - File input
+    - Default configs (only when there is no user provided config)
+- Any env variable that is commonly used in starting kafka(for example, CLUSTER_ID) can be supplied to docker container and it will be available when kafka starts
+
+Steps to release docker image
+-----------------------------
+- Make sure you have executed `release.py` script to prepare RC tarball in apache sftp server.
+- Use the RC tarball url (make sure you choose scala 2.13 version) as input kafka url to build docker image and run sanity tests.
+- Trigger github actions workflow using the RC branch, provide RC tarball url as kafka url.
+- This will generate test report and CVE report for docker images.
+- If the reports look fine, RC docker image can be built and published.
+- Execute `docker_release.py` script to build and publish RC docker image in your dockerhub account.
+- Share the RC docker image, test report and CVE report with the community in RC vote email.
+- Once approved and ready, take help from someone in PMC to trigger `docker_promote.py` script and promote the RC docker image to apache/kafka dockerhub repo

docker/common.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import subprocess
+import tempfile
+import os
+from distutils.dir_util import copy_tree
+import shutil
+
+def execute(command):
+    if subprocess.run(command).returncode != 0:
+        raise SystemError("Failure in executing following command:- ", " ".join(command))
+
+def get_input(message):
+    value = input(message)
+    if value == "":
+        raise ValueError("This field cannot be empty")
+    return value
+
+def jvm_image(command):
+    temp_dir_path = tempfile.mkdtemp()
+    current_dir = os.path.dirname(os.path.realpath(__file__))
+    copy_tree(f"{current_dir}/jvm", f"{temp_dir_path}/jvm")
+    copy_tree(f"{current_dir}/resources", f"{temp_dir_path}/jvm/resources")
+    command = command.replace("$DOCKER_FILE", f"{temp_dir_path}/jvm/Dockerfile")
+    command = command.replace("$DOCKER_DIR", f"{temp_dir_path}/jvm")
+    try:
+        execute(command.split())
+    except:
+        raise SystemError("Docker Image Build failed")
+    finally:
+        shutil.rmtree(temp_dir_path)

docker/docker_build_test.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Python script to build and test a docker image
+This script is used to generate a test report
+
+Usage:
+    docker_build_test.py --help
+        Get detailed description of each option
+
+    Example command:-
+        docker_build_test.py <image_name> --image-tag <image_tag> --image-type <image_type> --kafka-url <kafka_url>
+
+        This command will build an image with <image_name> as image name, <image_tag> as image_tag (it will be latest by default),
+        <image_type> as image type (jvm by default), <kafka_url> for the kafka inside the image and run tests on the image.
+        -b can be passed as additional argument if you just want to build the image.
+        -t can be passed if you just want to run tests on the image.
+"""
+
+from datetime import date
+import argparse
+from distutils.dir_util import copy_tree
+import shutil
+from test.docker_sanity_test import run_tests
+from common import execute, jvm_image
+import tempfile
+import os
+
+def build_jvm(image, tag, kafka_url):
+    image = f'{image}:{tag}'
+    jvm_image(f"docker build -f $DOCKER_FILE -t {image} --build-arg kafka_url={kafka_url} --build-arg build_date={date.today()} $DOCKER_DIR")
+
+def run_jvm_tests(image, tag, kafka_url):
+    temp_dir_path = tempfile.mkdtemp()
+    try:
+        current_dir = os.path.dirname(os.path.realpath(__file__))
+        copy_tree(f"{current_dir}/test/fixtures", f"{temp_dir_path}/fixtures")
+        execute(["wget", "-nv", "-O", f"{temp_dir_path}/kafka.tgz", kafka_url])
+        execute(["mkdir", f"{temp_dir_path}/fixtures/kafka"])
+        execute(["tar", "xfz", f"{temp_dir_path}/kafka.tgz", "-C", f"{temp_dir_path}/fixtures/kafka", "--strip-components", "1"])
+        failure_count = run_tests(f"{image}:{tag}", "jvm", temp_dir_path)
+    except:
+        raise SystemError("Failed to run the tests")
+    finally:
+        shutil.rmtree(temp_dir_path)
+    test_report_location_text = f"To view test report please check {current_dir}/test/report_jvm.html"
+    if failure_count != 0:
+        raise SystemError(f"{failure_count} tests have failed. {test_report_location_text}")
+    else:
+        print(f"All tests passed successfully. {test_report_location_text}")
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument("image", help="Image name that you want to keep for the Docker image")
+    parser.add_argument("--image-tag", "-tag", default="latest", dest="tag", help="Image tag that you want to add to the image")
+    parser.add_argument("--image-type", "-type", choices=["jvm"], default="jvm", dest="image_type", help="Image type you want to build")
+    parser.add_argument("--kafka-url", "-u", dest="kafka_url", help="Kafka url to be used to download kafka binary tarball in the docker image")
+    parser.add_argument("--build", "-b", action="store_true", dest="build_only", default=False, help="Only build the image, don't run tests")
+    parser.add_argument("--test", "-t", action="store_true", dest="test_only", default=False, help="Only run the tests, don't build the image")
+    args = parser.parse_args()
+
+    if args.image_type == "jvm" and (args.build_only or not (args.build_only or args.test_only)):
+        if args.kafka_url:
+            build_jvm(args.image, args.tag, args.kafka_url)
+        else:
+            raise ValueError("--kafka-url is a required argument for jvm image")
+
+    if args.image_type == "jvm" and (args.test_only or not (args.build_only or args.test_only)):
+        run_jvm_tests(args.image, args.tag, args.kafka_url)