Skip to content

Latest commit

 

History

History
704 lines (568 loc) · 23.6 KB

configuration.md

File metadata and controls

704 lines (568 loc) · 23.6 KB

Configuration Reference

This document provides comprehensive configuration options for the Temporal Worker Controller.

Table of Contents

  1. Rollout Strategies
  2. Rollback Strategy
  3. Sunset Configuration
  4. Worker Options
  5. Gate Configuration
  6. Advanced Configuration

Rollout Strategies

See the Concepts document for detailed explanations of rollout strategies. Here are the basic configuration patterns:

Manual Strategy (Advanced Use Cases)

rollout:
  strategy: Manual
# Requires manual intervention to promote versions
# Only recommended for special cases requiring full manual control

Use Manual strategy when you need complete control over version promotions, such as:

  • Complex validation processes that require human approval
  • Coordinated deployments across multiple services
  • Special compliance or regulatory requirements

AllAtOnce Strategy

rollout:
  strategy: AllAtOnce
# Immediately routes 100% traffic to new version when healthy

Use AllAtOnce strategy for:

  • Low-risk environments (development, staging)
  • Services where fast deployment is more important than gradual rollout
  • Background processing workers with minimal user impact

Progressive Strategy (Recommended)

rollout:
  strategy: Progressive
  steps:
    # Conservative initial migration settings
    - rampPercentage: 1
      pauseDuration: 10m
    - rampPercentage: 5  
      pauseDuration: 15m
    - rampPercentage: 25
      pauseDuration: 20m
    # Can be optimized to faster ramps after validation:
    # - rampPercentage: 10
    #   pauseDuration: 5m
    # - rampPercentage: 50
    #   pauseDuration: 10m
  gate:
    workflowType: "HealthCheck"  # Optional validation workflow

Progressive strategy is recommended for most production deployments because it:

  • Minimizes risk by gradually increasing traffic to new versions
  • Provides automatic pause points for validation
  • Allows for quick rollback if issues are detected
  • Can be tuned for different risk tolerances

Progressive Rollout Examples

Conservative Production Rollout:

rollout:
  strategy: Progressive
  steps:
    - rampPercentage: 1
      pauseDuration: 15m
    - rampPercentage: 5
      pauseDuration: 30m
    - rampPercentage: 25
      pauseDuration: 45m
    - rampPercentage: 75
      pauseDuration: 30m
  gate:
    workflowType: "ProductionHealthCheck"

Faster Development Environment:

rollout:
  strategy: Progressive
  steps:
    - rampPercentage: 25
      pauseDuration: 2m
    - rampPercentage: 75
      pauseDuration: 3m

Canary-Style Rollout:

rollout:
  strategy: Progressive
  steps:
    - rampPercentage: 1
      pauseDuration: 30m  # Long canary period
    - rampPercentage: 100
      pauseDuration: 0s   # Full rollout after canary validation

Rollback Strategy

The controller automatically detects rollbacks: if the target version was previously the current version within the last hour, it immediately routes 100% of traffic back to that version (AllAtOnce), regardless of the configured rollout strategy. This means setting your target version back to a prior build ID is enough to trigger a rollback — no special action or configuration required.

Rollback behavior is not configurable: it always uses the AllAtOnce strategy with a fixed 1-hour eligibility window, so there is no rollback field in the spec.

Note: Rollback is suppressed when the rollout strategy is Manual, since manual mode implies full user control.

Sunset Configuration

Controls how old versions are scaled down and cleaned up after they're no longer receiving new traffic:

sunset:
  scaledownDelay: 1h    # Wait 1 hour after the version is Drained before scaling to 0
  deleteDelay: 24h      # Wait 24 hours after the version is Drained before deleting

A Draining version keeps its replica count. To run Draining versions at fewer replicas, see Draining version Deployment configuration.

Sunset Configuration Examples

Conservative Cleanup (Recommended for Production):

sunset:
  scaledownDelay: 2h    # Allow time for workflows to complete
  deleteDelay: 48h      # Keep resources for debugging/rollback

Aggressive Cleanup (Development/Staging):

sunset:
  scaledownDelay: 15m   # Quick scaledown
  deleteDelay: 2h       # Minimal retention

Long-Running Workflow Environment:

sunset:
  scaledownDelay: 24h   # Long-running workflows need time
  deleteDelay: 168h     # 1 week retention for analysis

Worker Options

Configure how workers connect to Temporal:

workerOptions:
  connectionRef:
    name: production-temporal     # Reference to Connection
  temporalNamespace: production      # Temporal namespace

Connection Configuration

Reference a Connection resource that defines server details. To share one connection definition across namespaces, use a ClusterConnection instead; it takes the same spec. You can use either mutual TLS (mTLS) or API key authentication, but not both. If the address in hostPort differs from the hostname on the server certificate, set tls.serverName to the certificate hostname. hostPort accepts a host:port (e.g. production.abc123.tmprl.cloud:7233) or a gRPC resolver target URI (e.g. dns:///production.abc123.tmprl.cloud:7233, xds://example.dest).

Using mTLS Authentication:

apiVersion: temporal.io/v1alpha1
kind: Connection
metadata:
  name: production-temporal
spec:
  hostPort: "production.abc123.tmprl.cloud:7233"
  # Optional: override the TLS server name used for certificate verification
  tls:
    serverName: "production.abc123.tmprl.cloud"
  mutualTLSSecretRef: 
    name: temporal-cloud-mtls  # Optional: for mTLS

Creating an mTLS Secret:

The mTLS secret must be of type kubernetes.io/tls and contain tls.crt (certificate) and tls.key (private key) keys.

Show detailed creation steps

Option 1: Using kubectl from existing files:

If you already have your certificate and key files:

kubectl create secret tls temporal-cloud-mtls \
  --cert=/path/to/certificate.pem \
  --key=/path/to/private-key.key \
  --namespace=your-namespace

Option 2: Using kubectl with literal base64-encoded values:

kubectl create secret tls temporal-cloud-mtls \
  --cert=<(echo -n "$CERTIFICATE_CONTENT") \
  --key=<(echo -n "$KEY_CONTENT") \
  --namespace=your-namespace

Option 3: Creating via YAML manifest:

apiVersion: v1
kind: Secret
metadata:
  name: temporal-cloud-mtls
  namespace: your-namespace
type: kubernetes.io/tls
data:
  # Base64-encoded certificate
  tls.crt: LS0tLS1CRUdJTi...
  # Base64-encoded private key
  tls.key: LS0tLS1CRUdJTi...

To generate the base64-encoded values:

# For tls.crt
cat certificate.pem | base64 -w 0

# For tls.key
cat private-key.key | base64 -w 0

Using API Key Authentication:

apiVersion: temporal.io/v1alpha1
kind: Connection
metadata:
  name: production-temporal
spec:
  hostPort: "production.abc123.tmprl.cloud:7233"
  tls:
    # Optional: override the TLS server name used for certificate verification
    serverName: "production.abc123.tmprl.cloud"
    # Optional: trust an additional CA (e.g. a private/internal one) alongside the system
    # trust store. Only needed when the server certificate isn't publicly signed — most
    # Temporal Cloud connections don't need this.
    caCertSecretRef:
      name: temporal-ca-cert  # Name of a Secret containing a `ca.crt` key
  apiKeySecretRef:
    name: temporal-api-key  # Name of the Secret
    key: api-key            # Key within the Secret containing the API key token

Creating an API Key Secret:

The API key secret must be of type kubernetes.io/opaque (or you can omit the type field). The API key token should be stored under a key of your choice.

Show detailed creation steps

Option 1: Using kubectl with literal value:

kubectl create secret generic temporal-api-key \
  --from-literal=api-key=your-api-key-token-here \
  --namespace=your-namespace

Option 2: Using kubectl from a file:

If your API key is stored in a file:

kubectl create secret generic temporal-api-key \
  --from-file=api-key=/path/to/api-key-file.txt \
  --namespace=your-namespace

Option 3: Creating via YAML manifest:

apiVersion: v1
kind: Secret
metadata:
  name: temporal-api-key
  namespace: your-namespace
type: kubernetes.io/opaque
data:
  # Base64-encoded API key token
  # The key name here must match the 'key' field in apiKeySecretRef
  api-key: eW91ci1hcGkta2V5LXRva2VuLWhlcmU=

To generate the base64-encoded value:

echo -n "your-api-key-token-here" | base64

Important Notes:

  • Both secrets must be created in the same Kubernetes namespace as the Connection resource
  • Only one authentication method can be specified per Connection (either mutualTLSSecretRef or apiKeySecretRef)
  • The secret name and key in apiKeySecretRef must match the actual Secret resource and data key
  • tls.serverName affects TLS certificate verification by the controller and is injected into Worker Pods as TEMPORAL_TLS_SERVER_NAME for SDK envconfig users.
  • For mTLS secrets, the keys must be named exactly tls.crt and tls.key
  • tls.caCertSecretRef trusts an extra CA for API-key or no-credentials connections (mTLS already covers this via its own secret's ca.crt key, and cannot be combined with tls.caCertSecretRef). The referenced Secret must have a ca.crt key.

Using Custom Authentication:

Wrapper binaries that embed the controller can customize Temporal client construction at two levels. These are mutually exclusive:

  • Default path — WithDefaultClient: use the built-in client-creation path (parse Secret, build options, dial, health-check). Optionally pass WithCustomizeClientOptions to mutate SDK options (e.g. add an interceptor or a custom HeadersProvider) before the pool dials. The pool still owns Secret parsing, dialing, and the health check.
  • Full control — WithCreateClient: replace the entire client-creation path. The function owns secret parsing, options building, dialing, and health checking; the pool only caches the result. This is the path for wrappers that need to reuse their own client constructor (e.g. one that returns a ready-made client, not options).

Default path (add an interceptor):

r := controller.New(mgr,
    controller.WithDefaultClient(controller.WithCustomizeClientOptions(func(opts sdkclient.Options) sdkclient.Options {
        opts.Interceptors = append(opts.Interceptors, myInterceptor)
        return opts
    })),
)

Full control (reuse a custom client constructor):

r := controller.New(mgr,
    controller.WithCreateClient(func(ctx context.Context, spec v1alpha1.ConnectionSpec, ns, k8sNs, identity string) (controller.CachedClient, error) {
        client, err := myClientConstructor(spec, ns, identity)
        if err != nil {
            return controller.CachedClient{}, err
        }
        return controller.CachedClient{
            Client:  client,
            IsValid: func() bool { return true }, // e.g. check cert/token expiry, not a readiness probe
        }, nil
    }),
)

A Connection using custom auth (no Secret refs):

apiVersion: temporal.io/v1alpha1
kind: Connection
metadata:
  name: production-temporal
spec:
  hostPort: "production.abc123.tmprl.cloud:7233"

Important Notes:

  • Custom auth is derived from the absence of mutualTLSSecretRef and apiKeySecretRef; there is no customAuth field on the Connection CRD.
  • WithCreateClient and WithDefaultClient are mutually exclusive.
  • With WithCustomizeClientOptions (inside WithDefaultClient), the wrapper should wrap, not replace, the pool's HeadersProvider so the temporal-namespace header continues to ride every RPC (including system-level RPCs like GetSystemInfo).
  • The pool owns the client cache and eviction in both cases. Creation success is determined by the CreateClientFunc's error return — if it returns an error, the client is never cached. On a cache hit, the pool calls the cached client's IsValid closure; returning false triggers eviction and re-dial (the built-in path uses this for mTLS cert-expiry invalidation). On transport-class failures the reconciler evicts the cached client as usual so the next reconcile re-dials.

Gate Configuration

Optional validation workflow that must succeed before proceeding with rollout:

rollout:
  strategy: Progressive
  steps:
    - rampPercentage: 10
      pauseDuration: 5m
  gate:
    workflowType: "HealthCheck"
    # Optionally provide input to the gate workflow:
    # 1) Inline arbitrary JSON:
    # input:
    #   thresholds:
    #     errorRate: 0.01
    #     p95LatencyMs: 250
    # 2) Or reference a key from a ConfigMap or Secret containing JSON:
    # inputFrom:
    #   configMapKeyRef:
    #     name: gate-input
    #     key: payload.json
    # inputFrom:
    #   secretKeyRef:
    #     name: gate-input
    #     key: payload.json
    # By default the input is sent as plain JSON. For protobuf and other payload
    # types, see "Gate Input Encoding" below.

Gate workflow input details:

  • Exactly one of input or inputFrom may be set.
  • input accepts any JSON object and is passed as the first parameter to the gate workflow.
  • inputFrom reads a JSON document from the specified ConfigMap or Secret key.
  • The target workflow should declare a single argument matching the JSON shape (e.g., a struct or json.RawMessage).
  • The ConfigMap or Secret is read from the WorkerDeployment's own namespace, using the controller's ServiceAccount. The chart grants it get, list and watch on both.
  • The contents are sent to the gate workflow as its input over the WorkerDeployment's connection. Anyone who can create a WorkerDeployment in a namespace can therefore read any ConfigMap or Secret in that namespace through inputFrom. This is the same trust boundary as a pod template, which can already mount those Secrets, so only let people create WorkerDeployments in namespaces whose ConfigMaps and Secrets they are trusted to read.

Gate Input Encoding

By default the gate input is sent to the workflow as a plain JSON payload. Set encoding when the input is something else — most commonly a protobuf message:

gate:
  workflowType: "HealthCheck"
  encoding: json/protobuf
  messageType: my.package.DeployRequest   # optional for json/protobuf
  input:
    service: checkout
    replicas: 3

The controller does not parse or convert the input. It passes the bytes through unchanged and labels them with the encoding you declare, so the worker knows how to decode them. Declaring an encoding that does not match the actual input causes the gate workflow to fail to start its argument.

Supported values for encoding:

Value Input is Usable with
json/plain A plain JSON document. Same as leaving encoding unset. input, inputFrom
json/protobuf A protobuf message serialized as JSON. input, inputFrom
binary/plain Raw bytes, passed through untouched. inputFrom only
binary/protobuf A protobuf message in binary wire format. inputFrom only

The two binary/* encodings cannot be used with inline input, because a YAML/JSON document cannot carry raw bytes. Use inputFrom with a Secret, or a ConfigMap binaryData key; a binary encoding paired with inline input, or with no input at all, is rejected on apply.

messageType is the fully-qualified protobuf message name. Workers do not need it — they resolve the message type from the gate workflow's own signature — but tools that only see the payload, such as the Temporal UI or a codec server, do:

  • Required for binary/protobuf. The payload is opaque bytes, so nothing else identifies it.
  • Optional for json/protobuf. The payload is already readable JSON; the message name is only a label.
  • Not allowed for the other encodings, which are not protobuf.

Omitting both encoding and messageType preserves the previous behavior exactly, so existing gate configurations need no changes.

Protobuf example. Given this message:

message DeployRequest {
  string service  = 1;
  int32  replicas = 2;
}

its JSON form goes inline, since protobuf-as-JSON is still JSON:

gate:
  workflowType: "VerifyDeploy"
  encoding: json/protobuf
  messageType: my.package.DeployRequest
  input:
    service: checkout
    replicas: 3

and the gate workflow receives it as the declared type:

func VerifyDeploy(ctx workflow.Context, req *DeployRequest) error {
    // req.Service == "checkout", req.Replicas == 3
}

For the binary wire format, put the bytes in a Secret (or a ConfigMap binaryData key) and reference them instead:

gate:
  workflowType: "VerifyDeploy"
  encoding: binary/protobuf
  messageType: my.package.DeployRequest
  inputFrom:
    secretKeyRef:
      name: gate-input
      key: request.bin

Advanced Configuration

Per-version Kubernetes Deployment configuration

You can control the configuration of the Kubernetes Deployment created by Temporal Worker Controller for each WorkerDeploymentVersion (Build ID) by setting the WorkerDeploymentSpec.Deployment field. This field is of type appsv1.DeploymentSpec and supports all the configuration options available to normal Kubernetes Deployments.

Set the spec.deployment.strategy.type field to the desired Kubernetes Deployment strategy -- e.g. RollingUpdate.

To control the maximum number or percent of unavailable Pods during a rollout restart of the Deployment, set the spec.deployment.strategy.rollout.maxUnavailable field.

To control the maximum number or percent of extra Pods with the latest Pod definition during a rollout restart of the Deployment, set the spec.deployment.strategy.rollout.maxSurge field.

spec.deployment.rollout.maxUnavailable and spec.rollout.maxSurgecontrols how Pods are replaced within a single versioned Kubernetes Deployment.

These configuration settings impact the behaviour of Kubernetes when you issue a kubectl rollout restart CLI command.

NOTE: It does not affect Temporal traffic routing across worker versions (see Rollout Strategies).

When omitted, Kubernetes defaults apply maxUnavailable/maxSurge of 25%).

NOTE: The Kubernetes Deployment strategy defaults to RollingUpdate.

On large fleets, set a more conservative strategy if you rely on in-place restarts of Current workers:

spec:
  deployment:
    replicas: 100
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 5%
        maxSurge: 0
  rollout:
    strategy: Progressive

Draining version Deployment configuration

A Draining version receives no new workflow executions. It only serves the executions pinned to it, so it often needs fewer replicas than the Current version. Set spec.drainingDeployment to override settings on the Kubernetes Deployments of Draining versions. Fields left unset keep the behavior that applies without spec.drainingDeployment.

spec:
  deployment:
    replicas: 4
  drainingDeployment:
    replicas: 1
  sunset:
    scaledownDelay: 1h
    deleteDelay: 24h

With this configuration, versions are scaled as follows:

Version status Replicas
Current and Target deployment.replicas (4)
Draining drainingDeployment.replicas (1)
Drained, before scaledownDelay Unchanged (1)
Drained, after scaledownDelay 0

drainingDeployment.replicas:

  • Applies only when the controller manages replicas, that is when spec.deployment.replicas is set. When an autoscaler manages replicas, the field is ignored and the webhook returns a warning. To use a different autoscaler for different version statuses (for example, Draining versions), see Splitting autoscalers by status.
  • Must be at least 1, so a Draining version keeps a poller for its pinned executions. This holds even when deployment.replicas is 0.
  • When unset, the controller leaves Draining versions at their current replica count, and only scales one back up if it reaches 0.

A Draining version that becomes the Target or Current version again, for example after a rollback, scales back to deployment.replicas.

NOTE: Scaling down terminates pods that may be running in-flight tasks, which a versioned rollout otherwise avoids. Before you set drainingDeployment.replicas, make sure workers stop gracefully on SIGTERM, with a stop timeout that fits inside terminationGracePeriodSeconds. Long-running activities should heartbeat so they resume on another pod.

Labels on versioned Deployments

Labels in spec.deployment.template.metadata.labels go on the pods. To label the versioned Kubernetes Deployment objects themselves, set spec.deploymentLabels. Use this for tools that act on the Deployment rather than its pods, such as admission webhooks that opt workloads in or out by label, or cost allocation.

spec:
  deploymentLabels:
    team: payments
    example.com/instrumentation: "false"

The controller keeps these labels in sync on the Deployments of every version it manages, including Draining ones. A change is applied in place: it doesn't create a new Build ID or restart pods. Removing a key from spec.deploymentLabels removes it from the Deployments. Labels that other tools add to the Deployments are left alone; the controller tracks the keys it applied in the temporal.io/managed-deployment-labels annotation.

temporal.io/build-id and temporal.io/deployment-name are managed by the controller and cannot be set. Keys and values must be valid Kubernetes labels; the validating webhook rejects invalid ones.

Environment-Specific Configurations

Production Configuration:

apiVersion: temporal.io/v1alpha1
kind: WorkerDeployment
metadata:
  name: order-processor
  namespace: production
spec:
  deployment:
    replicas: 5
  workerOptions:
    connectionRef:
      name: production-temporal
    temporalNamespace: production
  rollout:
    strategy: Progressive
    steps:
      - rampPercentage: 1
        pauseDuration: 15m
      - rampPercentage: 10
        pauseDuration: 30m
      - rampPercentage: 50
        pauseDuration: 45m
    gate:
      workflowType: "ProductionHealthCheck"
  sunset:
    scaledownDelay: 2h
    deleteDelay: 48h

Staging Configuration:

apiVersion: temporal.io/v1alpha1
kind: WorkerDeployment
metadata:
  name: order-processor
  namespace: staging
spec:
  deployment:
    replicas: 2
  workerOptions:
    connectionRef:
      name: staging-temporal
    temporalNamespace: staging
  rollout:
    strategy: Progressive
    steps:
      - rampPercentage: 25
        pauseDuration: 5m
      - rampPercentage: 100
        pauseDuration: 0s
  sunset:
    scaledownDelay: 30m
    deleteDelay: 4h

Configuration Validation

The controller validates configuration and will report errors in the resource status:

# Check for configuration errors
kubectl describe workerdeployment my-worker

# Look for validation errors in status
kubectl get workerdeployment my-worker -o yaml

Common validation errors:

  • Invalid ramp percentages (must be 1-100)
  • Invalid duration formats (use Go duration format: "5m", "1h", "30s")
  • Missing required fields (connection, temporalNamespace)
  • Invalid strategy combinations

For more examples and patterns, see the Migration Guide and Concepts documentation.