Skip to content

docs: using gitlab-ci with kubernetes through kubedock guide #2943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+124 −3
Open

docs: using gitlab-ci with kubernetes through kubedock guide #2943

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b96efd8
[Docs](gitlab-ci) add: kubedock for kubernetes executors
amirsalarsafaei Jan 19, 2025
cfbbfc0
[Docs](gitlab-ci) fix: code block language should be yaml
amirsalarsafaei Jan 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 124 additions & 3 deletions docs/system_requirements/ci/gitlab_ci.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ See below for an example runner configuration:

Please also include the following in your GitlabCI pipeline definitions (`.gitlab-ci.yml`) that use Testcontainers:

```yml
```yaml
variables:
TESTCONTAINERS_HOST_OVERRIDE: "host.docker.internal"
```
Expand All @@ -39,9 +39,9 @@ So edit your `.gitlab-ci.yml` to include the [Docker-In-Docker service](https://

Caveat: Current Docker releases (verified for 20.10.9) intentionally delay the startup, if the Docker API is bound to a network address but not TLS protected. To avoid this delay, the Docker process needs to be started with the argument `--tls=false`. Otherwise jobs which access the Docker API at the very beginning might fail.

Here is a sample `.gitlab-ci.yml` that executes test with gradle:
Here is a sample `.gitlab-ci.yml` that executes go test:

```yml
```yaml
# DinD service is required for Testcontainers
services:
- name: docker:dind
Expand All @@ -61,3 +61,124 @@ test:
stage: test
script: go test ./... -v
```

## Example using Kubedock

This applies if your executor is `kubernetes` and you don't want to use DinD. One option is to use [kubedock](https://github.com/joyrex2001/kubedock). This library is a minimal implementation of the Docker API that will orchestrate containers on a Kubernetes cluster.

Here is the example Kubernetes configuration you must create:
Copy link
Member

@mdelapenya mdelapenya Jan 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amirsalarsafaei can you include the location of this k8s configuration file? 🙏

I'm currently reproducing this guide into Gitlab, and I don't know where to put that file.

Thanks!

Copy link
Author

@amirsalarsafaei amirsalarsafaei Jan 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should apply the configuration into the Kubernetes namespace that you are using to run your pipelines, there is no need to create any files. If you think that need clarification in guides, let me know.

Copy link
Member

@mdelapenya mdelapenya Jan 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amirsalarsafaei so we first need to create the k8s cluster in gitlab, right? Please forgive me but my k8s kungfu is very limited

Copy link
Author

@amirsalarsafaei amirsalarsafaei Jan 31, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's OK :D. I'd be happy to explain. So gitlab gives you the option to run your gitlab runners in Kubernetes, you can see the guide on how to do that alongside some useful information here

gitlab's kubernetes executer guide

You will need access to a Gitlab server (I believe hosted privately) and a Kubernetes namespace to do achieve this. Although I think you can test this with a minikube (it's a version of kubernetes that can run on your local machine).

Context:
Kubernetes uses containerd to spawn up docker images (kind of like docker swarm but way different). and because of this architecture there is no docker socket and server to use in runner environment unless you explicitly have a docker in the image or mount it from the sidecar or etc. The best approach is using kubedock which spawns new pod for each container that testcontainer request just like how gitlab kubernetes executer works.

Copy link
Author

@amirsalarsafaei amirsalarsafaei Jan 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After you created your minikube environment apply the configurations in the gitlab runner guide and the guide I provided.


```yaml
# ServiceAccount for Kubedock
apiVersion: v1
kind: ServiceAccount
metadata:
name: kubedock
namespace: gitlab-runner

# Role for Kubedock
Copy link
Member

@mdelapenya mdelapenya Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug: could you separate all different k8s objects with ---? My Docker Desktop's kind cluster complained with:

Error from server (BadRequest): error when creating "kubedoc.yml": Service in version "v1" cannot be handled as a Service: strict decoding error: unknown field "roleRef", unknown field "rules", unknown field "subjects"

I was able to apply the file once I added them to the file.

Copy link
Author

@amirsalarsafaei amirsalarsafaei Feb 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thats odd cause in k8s you can do that, but I can separate them.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kubedock-role
namespace: gitlab-runner
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "get", "list", "delete", "watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "get", "list", "delete"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create", "get", "list", "delete"]

# RoleBinding for Kubedock
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubedock-rolebinding
namespace: gitlab-runner
subjects:
- kind: User
name: system:serviceaccount:gitlab-runner:kubedock
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: kubedock-role
apiGroup: rbac.authorization.k8s.io

# Deployment for Kubedock Server
apiVersion: apps/v1
kind: Deployment
metadata:
name: kubedock-server
namespace: gitlab-runner
spec:
replicas: 1
selector:
matchLabels:
app: kubedock-server
template:
metadata:
labels:
app: kubedock-server
spec:
serviceAccountName: kubedock
containers:
- name: kubedock-server
image: joyrex2001/kubedock:0.17.0
resources:
limits:
memory: "4Gi"
cpu: "1000m"
requests:
memory: "2Gi"
cpu: "200m"
ports:
- containerPort: 2475
args: [
# Configuration options described here:
# https://github.com/joyrex2001/kubedock/blob/master/config.md
"server",
"--namespace=gitlab-runner",
"--service-account=kubedock",
"--timeout=20m0s",
"--request-cpu=1",
"--request-memory=2Gi",
"--disable-dind",
"--reverse-proxy",
"--reapmax=60m",
]

# Service for Kubedock
apiVersion: v1
kind: Service
metadata:
name: kubedock-service
namespace: gitlab-runner
spec:
selector:
app: kubedock-server
type: ClusterIP
clusterIP: None
```


Here is a sample `.gitlab-ci.yml` that executes go test:

```yaml
variables:
# Instruct Testcontainers to use the daemon of kubedock to create containers in kubernetes
DOCKER_HOST: "tcp://kubedock-service:2475"
test:
image: golang:1.22
stage: test
script: go test ./... -v
```
Loading