From dff049fe91baa2da8e9a549dca171963d5c3d2d3 Mon Sep 17 00:00:00 2001 From: JessamyT Date: Thu, 3 Apr 2025 13:05:49 -0700 Subject: [PATCH 1/7] 3068firstrundocker --- .../reference/advanced-modules/_index.md | 70 ++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/docs/operate/reference/advanced-modules/_index.md b/docs/operate/reference/advanced-modules/_index.md index 3fefd64fdc..670a5f6956 100644 --- a/docs/operate/reference/advanced-modules/_index.md +++ b/docs/operate/reference/advanced-modules/_index.md @@ -11,7 +11,7 @@ tags: "components", "services", ] -description: "Some usage may require you to define new APIs or deploy custom components using a server on a remote part" +description: "Some usage may require you to define new APIs or deploy custom components in non-standard ways." aliases: - /program/extend/ - /modular-resources/advanced/ @@ -46,6 +46,74 @@ Running {{< glossary_tooltip term_id="modular-resource" text="modular resources" However, if you are unable to use modular resources because you need to host `viam-server` on a non-Linux system or have an issue with compilation, you may need to [implement a custom component and register it on a server configured as a remote](/operate/reference/advanced-modules/custom-components-remotes/) on your machine. +## Package and deploy using Docker + +In rare cases, you may need to package and deploy a module using Docker. +Use cases for this include: + +- Your module has complex system dependencies that cannot be installed on a machine. +- You have a large bundle and want to use layer caching to reduce the size of the download. +- You have specific security requirements that are difficult to meet with the default module deployment. + +If you choose to deploy your module using Docker, we recommend creating a first run script or binary to run any necessary setup steps. + +{{% expand "Click for first run script instructions" %}} + +1. Create a tarball that contains: + + - The module's entrypoint script or binary + - A meta.json + - A first run script or binary that will be executed during the setup phase, for example: + + ```sh {id="terminal-prompt" class="command-line" data-prompt="$"} + #!/usr/bin/env bash + + if [[ -n "$VIAM_TEST_FAIL_RUN_FIRST" ]]; then + echo "Sorry, I've failed you." + exit 1 + fi + + docker pull mongo:6 + + cat << EOF + ------------------------------------- + Congratulations! + The setup script ran successfully! + This message is obnoxiously large for + testing purposes. + Sincerely, + First Run Script + ------------------------------------- + EOF + + exit 0 + ``` + + [This example Makefile on GitHub](https://github.com/viam-labs/wifi-sensor/blob/7823b6ad3edcbbbf20b06c34b3181453f5f3f078/Makefile) builds a module binary and bundles it along with a meta.json and first run script.

+ +1. Edit the meta.json to include a `first_run` field that points to the first run script or binary. + + ```json + { + ... + "first_run": "first_run.sh" + } + ``` + +1. Configure your module on your machine in the same way as you would for a regular module. + The first run script will execute once when `viam-server` receives a new configuration. + It will only execute once per module or per version of the module. + +1. (Optional) After a first run script runs successfully, Viam adds a marker file with a `.first_run_succeeded` suffix in the module’s data directory on disk. + It has the location and form: `/root/.viam/packages/data/module/--.first_run_succeeded`. + If you want to force a first run script to run again without changing the configured module or module version, you can do so by deleting this file. + +1. (Optional) By default, a first run script will timeout after 1 hour. + This can be adjusted by adding a `first_run_timeout` to the module’s configuration. + For example, `"first_run_timeout": "5m"` will lower the script timeout to 5 minutes. + +{{% /expand %}} + ## Design a custom ML model When working with the [ML model service](/dev/reference/apis/services/ml/), you can deploy an [existing model](/data-ai/ai/deploy/) or [train your own model](/data-ai/ai/train/). From bb29e8325bbaef14ed4fe4556560979e7bffa90a Mon Sep 17 00:00:00 2001 From: Jessamy Taylor <75634662+JessamyT@users.noreply.github.com> Date: Thu, 3 Apr 2025 13:28:15 -0700 Subject: [PATCH 2/7] Apply suggestions from code review --- docs/operate/reference/advanced-modules/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/operate/reference/advanced-modules/_index.md b/docs/operate/reference/advanced-modules/_index.md index 670a5f6956..b267dcc6e2 100644 --- a/docs/operate/reference/advanced-modules/_index.md +++ b/docs/operate/reference/advanced-modules/_index.md @@ -51,11 +51,11 @@ However, if you are unable to use modular resources because you need to host `vi In rare cases, you may need to package and deploy a module using Docker. Use cases for this include: -- Your module has complex system dependencies that cannot be installed on a machine. +- Your module has complex system dependencies that cannot be easily installed on a machine. - You have a large bundle and want to use layer caching to reduce the size of the download. - You have specific security requirements that are difficult to meet with the default module deployment. -If you choose to deploy your module using Docker, we recommend creating a first run script or binary to run any necessary setup steps. +If you choose to deploy your module using Docker, we recommend creating a "first run" script or binary to run any necessary setup steps. {{% expand "Click for first run script instructions" %}} From 2ee751bf04a2f798154ef5f4dd3bbe9e876c8d0f Mon Sep 17 00:00:00 2001 From: Jessamy Taylor <75634662+JessamyT@users.noreply.github.com> Date: Fri, 4 Apr 2025 10:43:25 -0700 Subject: [PATCH 3/7] Apply suggestions from code review --- docs/operate/reference/advanced-modules/_index.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/operate/reference/advanced-modules/_index.md b/docs/operate/reference/advanced-modules/_index.md index b267dcc6e2..927e52b781 100644 --- a/docs/operate/reference/advanced-modules/_index.md +++ b/docs/operate/reference/advanced-modules/_index.md @@ -77,12 +77,7 @@ If you choose to deploy your module using Docker, we recommend creating a "first cat << EOF ------------------------------------- - Congratulations! The setup script ran successfully! - This message is obnoxiously large for - testing purposes. - Sincerely, - First Run Script ------------------------------------- EOF From 3776863dbcd834555de01c484863a5b85cf3f6a6 Mon Sep 17 00:00:00 2001 From: JessamyT Date: Fri, 4 Apr 2025 11:02:59 -0700 Subject: [PATCH 4/7] Address feedback --- docs/operate/reference/advanced-modules/_index.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/operate/reference/advanced-modules/_index.md b/docs/operate/reference/advanced-modules/_index.md index 927e52b781..3caa105482 100644 --- a/docs/operate/reference/advanced-modules/_index.md +++ b/docs/operate/reference/advanced-modules/_index.md @@ -56,6 +56,7 @@ Use cases for this include: - You have specific security requirements that are difficult to meet with the default module deployment. If you choose to deploy your module using Docker, we recommend creating a "first run" script or binary to run any necessary setup steps. +Note this is _not_ recommended for modules that do not use Docker, it adds unnecessary complexity. {{% expand "Click for first run script instructions" %}} @@ -68,11 +69,6 @@ If you choose to deploy your module using Docker, we recommend creating a "first ```sh {id="terminal-prompt" class="command-line" data-prompt="$"} #!/usr/bin/env bash - if [[ -n "$VIAM_TEST_FAIL_RUN_FIRST" ]]; then - echo "Sorry, I've failed you." - exit 1 - fi - docker pull mongo:6 cat << EOF From a4a501c926b80946b44e8c4ae511a8e15c366e37 Mon Sep 17 00:00:00 2001 From: Jessamy Taylor <75634662+JessamyT@users.noreply.github.com> Date: Wed, 9 Apr 2025 12:03:26 -0700 Subject: [PATCH 5/7] Apply suggestions from code review Co-authored-by: Naomi Pentrel <5212232+npentrel@users.noreply.github.com> --- docs/operate/reference/advanced-modules/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operate/reference/advanced-modules/_index.md b/docs/operate/reference/advanced-modules/_index.md index 3caa105482..0beb98ff9e 100644 --- a/docs/operate/reference/advanced-modules/_index.md +++ b/docs/operate/reference/advanced-modules/_index.md @@ -52,7 +52,7 @@ In rare cases, you may need to package and deploy a module using Docker. Use cases for this include: - Your module has complex system dependencies that cannot be easily installed on a machine. -- You have a large bundle and want to use layer caching to reduce the size of the download. +- You use a large container image and some layers are already used by your machine which means layer caching can reduce the size of the download. - You have specific security requirements that are difficult to meet with the default module deployment. If you choose to deploy your module using Docker, we recommend creating a "first run" script or binary to run any necessary setup steps. From 1c4fc85e348d418de7b609628cbb73947a81c139 Mon Sep 17 00:00:00 2001 From: JessamyT Date: Wed, 9 Apr 2025 19:13:10 -0700 Subject: [PATCH 6/7] Changes after testing --- .../reference/advanced-modules/_index.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/operate/reference/advanced-modules/_index.md b/docs/operate/reference/advanced-modules/_index.md index 0beb98ff9e..daccada2b0 100644 --- a/docs/operate/reference/advanced-modules/_index.md +++ b/docs/operate/reference/advanced-modules/_index.md @@ -62,7 +62,13 @@ Note this is _not_ recommended for modules that do not use Docker, it adds unnec 1. Create a tarball that contains: - - The module's entrypoint script or binary + - The module's entrypoint script or binary, for example: + + ```sh {id="terminal-prompt" class="command-line" data-prompt="$"} + #!/bin/bash + exec docker run + ``` + - A meta.json - A first run script or binary that will be executed during the setup phase, for example: @@ -93,14 +99,16 @@ Note this is _not_ recommended for modules that do not use Docker, it adds unnec 1. Configure your module on your machine in the same way as you would for a regular module. The first run script will execute once when `viam-server` receives a new configuration. - It will only execute once per module or per version of the module. + It will only execute once per module or per version of the module.

+ +1. (Optional) After a first run script runs successfully, Viam adds a marker file in the module's data directory. + The marker file path format is of the form `unpackedModDir + FirstRunSuccessSuffix`. + For example, `.viam/packages/data/module/abcd1234-abcd-abcd-abcd-abcd12345678-viam-rtsp-0_1_0-linux-amd64/bin.first_run_succeeded`. -1. (Optional) After a first run script runs successfully, Viam adds a marker file with a `.first_run_succeeded` suffix in the module’s data directory on disk. - It has the location and form: `/root/.viam/packages/data/module/--.first_run_succeeded`. - If you want to force a first run script to run again without changing the configured module or module version, you can do so by deleting this file. + If you want to force a first run script to run again without changing the configured module or module version, you can do so by deleting this file.

1. (Optional) By default, a first run script will timeout after 1 hour. - This can be adjusted by adding a `first_run_timeout` to the module’s configuration. + This can be adjusted by adding a `first_run_timeout` to the module's configuration. For example, `"first_run_timeout": "5m"` will lower the script timeout to 5 minutes. {{% /expand %}} From fb370cfcc7d807118231df660ed17c4c5fc3454f Mon Sep 17 00:00:00 2001 From: JessamyT Date: Thu, 10 Apr 2025 17:21:31 -0700 Subject: [PATCH 7/7] Move to sub-page --- .../reference/advanced-modules/_index.md | 67 +---------------- .../advanced-modules/docker-modules.md | 74 +++++++++++++++++++ 2 files changed, 76 insertions(+), 65 deletions(-) create mode 100644 docs/operate/reference/advanced-modules/docker-modules.md diff --git a/docs/operate/reference/advanced-modules/_index.md b/docs/operate/reference/advanced-modules/_index.md index daccada2b0..237f4ce3b1 100644 --- a/docs/operate/reference/advanced-modules/_index.md +++ b/docs/operate/reference/advanced-modules/_index.md @@ -46,72 +46,9 @@ Running {{< glossary_tooltip term_id="modular-resource" text="modular resources" However, if you are unable to use modular resources because you need to host `viam-server` on a non-Linux system or have an issue with compilation, you may need to [implement a custom component and register it on a server configured as a remote](/operate/reference/advanced-modules/custom-components-remotes/) on your machine. -## Package and deploy using Docker +## Deploy a module using Docker -In rare cases, you may need to package and deploy a module using Docker. -Use cases for this include: - -- Your module has complex system dependencies that cannot be easily installed on a machine. -- You use a large container image and some layers are already used by your machine which means layer caching can reduce the size of the download. -- You have specific security requirements that are difficult to meet with the default module deployment. - -If you choose to deploy your module using Docker, we recommend creating a "first run" script or binary to run any necessary setup steps. -Note this is _not_ recommended for modules that do not use Docker, it adds unnecessary complexity. - -{{% expand "Click for first run script instructions" %}} - -1. Create a tarball that contains: - - - The module's entrypoint script or binary, for example: - - ```sh {id="terminal-prompt" class="command-line" data-prompt="$"} - #!/bin/bash - exec docker run - ``` - - - A meta.json - - A first run script or binary that will be executed during the setup phase, for example: - - ```sh {id="terminal-prompt" class="command-line" data-prompt="$"} - #!/usr/bin/env bash - - docker pull mongo:6 - - cat << EOF - ------------------------------------- - The setup script ran successfully! - ------------------------------------- - EOF - - exit 0 - ``` - - [This example Makefile on GitHub](https://github.com/viam-labs/wifi-sensor/blob/7823b6ad3edcbbbf20b06c34b3181453f5f3f078/Makefile) builds a module binary and bundles it along with a meta.json and first run script.

- -1. Edit the meta.json to include a `first_run` field that points to the first run script or binary. - - ```json - { - ... - "first_run": "first_run.sh" - } - ``` - -1. Configure your module on your machine in the same way as you would for a regular module. - The first run script will execute once when `viam-server` receives a new configuration. - It will only execute once per module or per version of the module.

- -1. (Optional) After a first run script runs successfully, Viam adds a marker file in the module's data directory. - The marker file path format is of the form `unpackedModDir + FirstRunSuccessSuffix`. - For example, `.viam/packages/data/module/abcd1234-abcd-abcd-abcd-abcd12345678-viam-rtsp-0_1_0-linux-amd64/bin.first_run_succeeded`. - - If you want to force a first run script to run again without changing the configured module or module version, you can do so by deleting this file.

- -1. (Optional) By default, a first run script will timeout after 1 hour. - This can be adjusted by adding a `first_run_timeout` to the module's configuration. - For example, `"first_run_timeout": "5m"` will lower the script timeout to 5 minutes. - -{{% /expand %}} +If you need to package and deploy a module using Docker, for example if your module relies on complex system dependencies, see [Deploy a module using Docker](/operate/reference/advanced-modules/docker-modules/) for suggestions. ## Design a custom ML model diff --git a/docs/operate/reference/advanced-modules/docker-modules.md b/docs/operate/reference/advanced-modules/docker-modules.md new file mode 100644 index 0000000000..dac70128b8 --- /dev/null +++ b/docs/operate/reference/advanced-modules/docker-modules.md @@ -0,0 +1,74 @@ +--- +title: "Deploy a module using Docker" +linkTitle: "Docker Modules" +weight: 300 +type: "docs" +tags: ["extending viam", "modular resources"] +description: "Deploy a module using Docker." +no_list: true +date: "2025-04-10" +# updated: "" # When the content was last entirely checked +--- + +In rare cases, you may need to package and deploy a module using Docker. +Use cases for this include: + +- Your module has complex system dependencies that cannot be easily installed on a machine. +- You use a large container image and some layers are already used by your machine which means layer caching can reduce the size of the download. +- You have specific security requirements that are difficult to meet with the default module deployment. + +If you choose to deploy your module using Docker, we recommend creating a "first run" script or binary to run any necessary setup steps. +Note this is _not_ recommended for modules that do not use Docker, as it adds unnecessary complexity. + +## Use a `first_run` script or binary + +1. Create a tarball that contains: + + - The module's entrypoint script or binary, for example: + + ```sh {id="terminal-prompt" class="command-line" data-prompt="$"} + #!/bin/bash + exec docker run + ``` + + - A meta.json + - A first run script or binary that will be executed during the setup phase, for example: + + ```sh {id="terminal-prompt" class="command-line" data-prompt="$"} + #!/usr/bin/env bash + + docker pull mongo:6 + + cat << EOF + ------------------------------------- + The setup script ran successfully! + ------------------------------------- + EOF + + exit 0 + ``` + + [This example Makefile on GitHub](https://github.com/viam-labs/wifi-sensor/blob/7823b6ad3edcbbbf20b06c34b3181453f5f3f078/Makefile) builds a module binary and bundles it along with a meta.json and first run script.

+ +1. Edit the meta.json to include a `first_run` field that points to the first run script or binary. + + ```json + { + ... + "first_run": "first_run.sh" + } + ``` + +1. Configure your module on your machine in the same way you would configure a regular module. + The first run script will execute once when `viam-server` receives a new configuration. + It will only execute once per module or per version of the module.

+ +1. (Optional) After a first run script runs successfully, Viam adds a marker file in the module's data directory. + The marker file path format is of the form `unpackedModDir + FirstRunSuccessSuffix`. + For example, `.viam/packages/data/module/abcd1234-abcd-abcd-abcd-abcd12345678-viam-rtsp-0_1_0-linux-amd64/bin.first_run_succeeded`. + + If you want to force a first run script to run again without changing the configured module or module version, you can do so by deleting this file.

+ +1. (Optional) By default, a first run script will timeout after 1 hour. + This can be adjusted by adding a `first_run_timeout` to the module's configuration. + For example, `"first_run_timeout": "5m"` will lower the script timeout to 5 minutes.