diff --git a/tarfetch/README.md b/tarfetch/README.md index 262f749..800f5a0 100644 --- a/tarfetch/README.md +++ b/tarfetch/README.md @@ -1,48 +1,59 @@ # Tarfetch -This extension leverages `kubectl exec` and `tar` to archive files in a Kubernetes container and then extract them to a destination on the local filesystem. +_**Made with the file-freshening properties of tar!**_ -## Overview +This extension leverages `kubectl exec` and `tar` to archive files in a Kubernetes container and extract them to a destination on the local filesystem. -Tarfetch provides a local resource method (`tarfetch`) to perform manually triggered reverse file synchronization (i.e. from pod container to local filesystem). +```mermaid +flowchart LR -Tarfetch resources can be triggered manually either via the Tilt Web UI, or via CLI using `tilt trigger `. + kubectl[kubectl exec] --> tar + archive -. pipe .-> untar + untar --> dest([Destination]) -## Usage + subgraph Container + source([Source Files]) ==> tar + tar ==> archive(((Archived Files))) + end +``` Tarfetch's only requirement is that both the local machine and container have `tar` installed. This is typically a given ([yes, even on Windows](https://docs.microsoft.com/en-us/virtualization/community/team-blog/2017/20171219-tar-and-curl-come-to-windows)). -Import Tarfetch with the following in your Tiltfile: -``` -load('ext://tarfetch', 'tarfetch') +## Example + +Create a local resource called "tarfetch-app" which connects to the first pod of "deploy/frontend" (and the default container) and syncs the contents of "/app/" to local directory "./frontend" while ignoring all directories named "node_modules": + +```starlark +# Import extension +load("ext://tarfetch", "tarfetch") + +# Setup tarfetch to attach the button to a Tilt resource +tarfetch( + "tilt-app-resource", + "deployments/frontend", + "/app/", + "./frontend", + ignore=["node_modules"] +) ``` +## Usage + A `tarfetch` resource can be created with the following parameters: -Required parameters: -* **name (str)**: a name for the local resource -* **k8s_object (str)**: a Kubernetes object identifier (e.g. `deploy/my-deploy`, `job/my-job`, or a pod ID) that Tilt can use to select a pod. As per the behavior of `kubectl exec`, we will act on the first pod of the specified object, using the first container by default +### Required parameters + +* **tilt_resource (str)**: name of Tilt resource to bind button to +* **k8s_resource (str)**: a Kubernetes object identifier (e.g. `deploy/my-deploy`, `job/my-job`, or a pod ID) that Tilt can use to select a pod. As per the behavior of `kubectl exec`, we will act on the first pod of the specified object, using the first container by default * **src_dir (str)**: directory *in the remote container* to sync from. Any `paths`, if specified, should be relative to this dir. This path *must* be a directory and must contain a trailing slash (e.g. `/app/` is acceptable; `/app` is not) -Optional parameters: -* **target_dir (str, optional)**: directory *on the local filesystem* to sync to. Defaults to `'.'` +### Optional parameters + +* **target_dir (str, optional)**: directory *on the local filesystem* to sync to. Defaults to `"."` * **namespace (str, optiona)**: namespace of the desired `k8s_object`, if not `default`. * **container (str, optional)**: name of the container to sync from (by default, the first container) * **ignore (List[str], optional)**: patterns to ignore when syncing, [see `tar --exclude` documentation for details on supported patterns](https://www.gnu.org/software/tar/manual/html_node/exclude.html). * **keep_newer (bool, optional)**: prevents files overwrites when the destination file is newer. Default is true. * **verbose (bool, optional)**: if true, shows tar extract activity. -* **labels (Union[str, List[str]], optional)**: used to group resources in the Web UI. -### Example invocation -Create a local resource called "tarfetch-app" which connects to the first pod of "deploy/frontend" (and the default container) and syncs the contents of "/app/" to local directory "./frontend" while ignoring all directories named "node_modules": - -```python -tarfetch( - 'tarfetch-app', - 'deployments/frontend', - '/app/', - './frontend', - ignore=["node_modules"] -) - ``` diff --git a/tarfetch/Tiltfile b/tarfetch/Tiltfile index 7004418..63685a4 100644 --- a/tarfetch/Tiltfile +++ b/tarfetch/Tiltfile @@ -1,5 +1,4 @@ -# -*- mode: Python -*- - +TARFETCH_SCRIPT = os.path.join(os.getcwd(), "scripts", "tarfetch.sh") DEFAULT_EXCLUDES = [ ".git", ".gitignore", @@ -11,45 +10,57 @@ DEFAULT_EXCLUDES = [ ] def tarfetch( - name, - k8s_object, + tilt_resource, + k8s_resource, src_dir, - target_dir=".", - namespace="default", - container="", - ignore=None, - keep_newer=True, - verbose=False, - labels=tuple() + target_dir = ".", + namespace = "default", + container = "", + ignore = None, + keep_newer = True, + verbose = False, + labels = None, + button_text = None, ): """ - Create a local resource that will (via rsync) sync the specified files - from the specified k8s object to the local filesystem. + Create a sync button on the specified Tilt resource, which will pull files from + a Kubernetes container onto the local filesystem. - :param name (str): name of the created local resource. - :param k8s_object (str): a Kubernetes object identifier (e.g. deploy/my-deploy, + :param tilt_resource: name of Tilt resource to bind button to. + :param k8s_resource: a Kubernetes object identifier (e.g. deploy/my-deploy, job/my-job, or a pod ID) that Tilt can use to select a pod. As per the behavior of `kubectl exec`, we will act on the first pod of the specified object, using the first container by default. - :param src_dir (str): directory IN THE KUBERNETES CONTAINER to sync from. Any + :param src_dir: directory IN THE KUBERNETES CONTAINER to sync from. Any paths specified, if relative, should be relative to this dir. - :param target_dir (str, optional): directory ON THE LOCAL FS to sync to. Defaults to '.' - :param namespace (str, optional): namespace of the desired k8s_object, if not `default`. - :param container (str, optional): name of the container to sync from (by default, + :param target_dir: directory ON THE LOCAL FS to sync to. Defaults to '.' + :param namespace: namespace of the desired k8s_object, if not `default`. + :param container: name of the container to sync from (by default, the first container) - :param ignore (List[str], optional): patterns to ignore when syncing, see + :param ignore: patterns to ignore when syncing, see `tar --exclude` documentation for details on supported patterns. - :param keep_newer (bool, optional): prevents files overwrites when the destination + :param keep_newer: prevents files overwrites when the destination file is newer. Default is true. - :param verbose (bool, optional): if true, shows tar extract activity. - :return: + :param verbose: if true, shows tar extract activity. + :param labels: deprecated argument from when tarfetch was a resource rather than + a UI button. + :param button_text: provide custom text for button label. """ + # Deprecation handling + if tilt_resource.startswith("sync") or tilt_resource.startswith("tarfetch"): + warn( + "[tarfetch] WARNING: The leading positional argument 'tilt_resource' may " + + "have changed in purpose since it was configured for this project. " + + "Please update it to reference the Tilt UI resource to bind the sync " + + "button to." + ) + if labels: + warn("[tarfetch] WARNING: The 'labels' argument has been deprecated and may be removed.") + # Verify inputs if not src_dir.endswith("/"): - fail( - "src_dir must be a directory and have a trailing slash (because of rsync syntax rules)" - ) + fail("[tarfetch] src_dir must be a directory and have a trailing slash") to_exclude = ignore if not ignore: @@ -62,36 +73,48 @@ def tarfetch( # bundle container flag with k8s object specifier if container: - k8s_object = "{obj} -c {container}".format(obj=k8s_object, container=container) + k8s_resource = "{obj} -c {container}".format(obj = k8s_resource, container = container) - destination_path = os.path.realpath(target_dir) - if not os.path.exists(destination_path): - print("Preparing destination path for reverse sync:") + if not os.path.exists(target_dir): + print("[tarfetch] Preparing destination path for reverse sync:") local( - ["mkdir", "-p", destination_path], - command_bat="mkdir {} || ver>nul".format(destination_path), + ["mkdir", "-p", target_dir], + command_bat = "mkdir {} || ver>nul".format(target_dir), quiet = True, ) - local_resource( - name, - ( - "(" + - "kubectl exec -i -n {namespace} {k8s_object} -- " + - "tar -c -f - --atime-preserve=system --directory={src_dir} {exclude} ." + - ") | " + - "tar -x -f - {verbose} {keep_newer} --directory={target_dir} && " + - "echo Done." - ).format( - namespace=namespace, - k8s_object=k8s_object, - exclude=excludes, - src_dir=src_dir, - target_dir=target_dir, - keep_newer="--keep-newer-files" if keep_newer else "", - verbose="--verbose" if verbose else "", + + btn_name = "btn-tarfetch-" + tilt_resource + v1alpha1.ui_button( + name = btn_name, + location = { + "component_type": "Resource", + "component_id": tilt_resource, + }, + text = button_text or "Sync from Container", + icon_name = "cloud_sync", + annotations = {"tilt.dev/resource": tilt_resource}, + ) + + env = { + "namespace": namespace, + "resource_name": k8s_resource, + "exclude": excludes, + "src_dir": src_dir, + "target_dir": target_dir, + "keep_newer": str(bool(keep_newer)).lower(), + "verbose": str(bool(verbose)).lower(), + } + + v1alpha1.cmd( + name = "cmd-tarfetch-" + tilt_resource, + annotations = { + "tilt.dev/resource": tilt_resource, + "tilt.dev/log-span-id": "cmd:tarfetch:" + tilt_resource, + }, + args = [TARFETCH_SCRIPT], + env = ["TARFETCH_%s=%s" % (k.upper(), v) for k, v in env.items()], + start_on = v1alpha1.start_on_spec( + ui_buttons = [btn_name], ), - trigger_mode=TRIGGER_MODE_MANUAL, - auto_init=False, - labels=labels, ) diff --git a/tarfetch/scripts/tarfetch.sh b/tarfetch/scripts/tarfetch.sh new file mode 100755 index 0000000..24f1c0c --- /dev/null +++ b/tarfetch/scripts/tarfetch.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env sh + +set -eu + +PACK_ARGS="--directory=${TARFETCH_SRC_DIR}" +[ -n "$TARFETCH_EXCLUDE" ] && PACK_ARGS="${PACK_ARGS} ${TARFETCH_EXCLUDE}" + +UNPACK_ARGS="--directory=${TARFETCH_TARGET_DIR}" +[ "$TARFETCH_KEEP_NEWER" = "true" ] && UNPACK_ARGS="${UNPACK_ARGS} --keep-newer-files" + +if [ "$TARFETCH_VERBOSE" = "true" ]; then + UNPACK_ARGS="${UNPACK_ARGS} --verbose" + set -x +fi + +pack() { + kubectl exec -n "$TARFETCH_NAMESPACE" "$TARFETCH_RESOURCE_NAME" -- \ + tar -c -f - $PACK_ARGS . +} + +unpack() { + tar -x -f - $UNPACK_ARGS +} + +pack | unpack + +echo '[tarfetch] Done: Sync from container has finished.' diff --git a/tarfetch/test/Tiltfile b/tarfetch/test/Tiltfile new file mode 100644 index 0000000..e598fd1 --- /dev/null +++ b/tarfetch/test/Tiltfile @@ -0,0 +1,20 @@ +load("../Tiltfile", "tarfetch") + +k8s_yaml("k8s/pod-tarfetch.yaml") + +tarfetch( + "tarfetch-example", + "pods/tarfetch-example", + "/app/", + "./files/", + ignore = [ + "**/dont", + "dont.*", + ] +) + +local_resource( + "tarfetch-tests", + cmd = ["./test-tarfetch.sh"], + resource_deps = ["tarfetch-example"] +) diff --git a/tarfetch/test/k8s/pod-tarfetch.yaml b/tarfetch/test/k8s/pod-tarfetch.yaml new file mode 100644 index 0000000..319afbb --- /dev/null +++ b/tarfetch/test/k8s/pod-tarfetch.yaml @@ -0,0 +1,27 @@ +apiVersion: v1 +kind: Pod +metadata: + name: tarfetch-example +spec: + terminationGracePeriodSeconds: 1 + containers: + - name: tarfetch-example + image: busybox:latest + imagePullPolicy: IfNotPresent + workingDir: /app + command: + - sh + - -euc + args: + - | + mkdir do + mkdir dont + + touch do.sync + touch dont.sync + touch do/do.sync + touch do/dont.sync + touch dont/do.sync + touch dont/dont.sync + + sleep infinity diff --git a/tarfetch/test/test-tarfetch.sh b/tarfetch/test/test-tarfetch.sh new file mode 100755 index 0000000..c916e95 --- /dev/null +++ b/tarfetch/test/test-tarfetch.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -eu + +successes=0 +failures=0 + +print_test() { + message="$1" + shift + + if [ $@ ]; then + state="\033[32mPASSED\033[0m" + successes=$((successes + 1)) + else + state="\033[31mFAILED\033[0m" + failures=$((failures + 1)) + fi + + echo -e "${state} - ${message}" +} + +./trigger.sh btn-tarfetch-tarfetch-example +sleep 2 + +echo +echo "Test results:" + +print_test "Test files/ exists" -d files/ +print_test "Test do.sync exists" -f files/do.sync +print_test "Test dont.sync does not exist" ! -f files/dont.sync +print_test "Test do/ exists" -d files/do/ +print_test "Test do/do.sync exists" -f files/do/do.sync +print_test "Test do/dont.sync does not exist" ! -f files/do/dont.sync +print_test "Test dont/ does not exist" ! -d files/dont/ + +echo +echo "Ran $((failures + successes)) tests" +if [ "$failures" = "0" ]; then + echo -e "\033[32mOK\033[0m" +else + echo -e "\033[31mFAILED (failures=${failures}, successes=${successes})\033[0m" +fi +echo diff --git a/tarfetch/test/test.sh b/tarfetch/test/test.sh new file mode 100755 index 0000000..afe240d --- /dev/null +++ b/tarfetch/test/test.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +set -eu + +cleanup() { + find ./files -type f -name '*.do' -delete + find ./files -type f -name '*.dont' -delete +} + +cd "$(dirname "$0")" + +echo "Preparing sync destination..." +if [ -d ./files ]; then + cleanup +else + mkdir ./files +fi + +tilt ci +tilt down + +echo "Cleaning up test files..." +cleanup +rm -r ./files + +echo "Done" diff --git a/tarfetch/test/trigger.sh b/tarfetch/test/trigger.sh new file mode 100755 index 0000000..b9f4797 --- /dev/null +++ b/tarfetch/test/trigger.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# Helper script to trigger a button. +# Source: https://github.com/tilt-dev/tilt-extensions/blob/dd9e9f70/cancel/test/trigger.sh +set -eu + +YAML=$(tilt get uibutton "btn-tarfetch-tarfetch-example" -o yaml) +TIME=$(date '+%FT%T.000000Z') +NEW_YAML=$(echo "$YAML" | sed "s/lastClickedAt.*/lastClickedAt: $TIME/g") + +# Currently, kubectl doesn't support subresource APIs. +# Follow this KEP: +# https://github.com/kubernetes/enhancements/issues/2590 +# For now, we can handle it with curl. +curl -so /dev/null -X PUT -H "Content-Type: application/yaml" -d "$NEW_YAML" \ + "http://localhost:10350/proxy/apis/tilt.dev/v1alpha1/uibuttons/${1}/status"