ClickHouse URL expansion fixed for Kubernetes deployments

ThullyoCunha

·Apr 27, 2026·#3449fix(helm): expand CLICKHOUSE_PASSWORD in webapp CLICKHOUSE_URL via kubelet

Self-hosted Trigger.dev deployments using an external ClickHouse database no longer crash on startup. Switching from shell to Kubernetes variable expansion ensures database passwords interpolate correctly, preventing migration script failures.

Self-hosted deployments using an external ClickHouse database no longer crash on startup. The fix addresses a conflict in environment variable injection that caused the database migration tool to interpret a literal string instead of the actual password.

By offloading password expansion to Kubernetes rather than relying on the container's shell, the database URL reaches the application fully formed. Administrators can successfully boot the web app and connect to external ClickHouse clusters without modifying container images or entrypoint scripts.

The changes are confined to the Helm chart templates, specifically targeting deployments that utilize existing Kubernetes secrets for database credentials.

View Original GitHub Description

Summary

When the official Helm chart is deployed with an external ClickHouse and clickhouse.external.existingSecret set — the documented path for not committing secrets to values.yaml — the webapp pod crash-loops on startup:

goose run: parse "http://default:${CLICKHOUSE_PASSWORD}@<host>:8123?secure=false": net/url: invalid userinfo

Context in vouch request #3443. Re-opening in draft status per bot policy (previous attempt was #3445, closed by automation because it wasn't draft; no changes to the patch).

Root cause

Two pieces interact:

hosting/k8s/helm/templates/_helpers.tpl renders CLICKHOUSE_URL (and RUN_REPLICATION_CLICKHOUSE_URL) with a shell-style literal ${CLICKHOUSE_PASSWORD} expecting bash expansion at container start.
docker/scripts/entrypoint.sh does export GOOSE_DBSTRING="$CLICKHOUSE_URL" — single-pass POSIX sh substitution, so the inner ${...} survives as literal text and goose rejects it.

Reproduces against the latest published chart (oci://ghcr.io/triggerdotdev/charts/trigger:4.0.5) and main.

Fix

Switch the two helpers (external + existingSecret branch) from shell-style ${CLICKHOUSE_PASSWORD} to Kubernetes' $(CLICKHOUSE_PASSWORD). Kubelet substitutes $(VAR) at pod-creation time from earlier env entries, and the chart already declares CLICKHOUSE_PASSWORD from the Secret immediately before CLICKHOUSE_URL, so the URL reaches the entrypoint with the real password already inlined. No entrypoint change, no image change. The plain-password branch (no existingSecret) is unchanged.

Operator caveat added as template comments: CLICKHOUSE_PASSWORD must be URL-userinfo-safe since kubelet substitutes verbatim without percent-encoding. Hex-encoded passwords (e.g. openssl rand -hex 32) are safe by construction.

Verification

helm template against external.existingSecret now renders value: "http://default:$(CLICKHOUSE_PASSWORD)@<host>:8123?secure=false" (was ${CLICKHOUSE_PASSWORD}).
helm template against the plain-password branch is byte-identical to before.
Deployed end-to-end on a staging EKS cluster (Meistrari platform): webapp container reaches goose: successfully migrated database to version: 6, Node.js ClickHouse client connects at runtime.

Alternatives considered

Change entrypoint.sh to eval / envsubst the URL — larger surface, touches every deployment mode (Docker Compose + k8s) and every container image.
Mirror the Postgres pattern (chart reads the full URL via valueFrom.secretKeyRef, as in trigger-v4.postgres.useSecretUrl) — cleaner long-term but requires a new values.yaml field and a migration path for existing users. Happy to follow up with that as a separate PR if the minimal fix here isn't the preferred direction.

Changeset

None added — the Helm chart isn't versioned through @changesets/cli (docs/chart-only PRs historically merge without a changeset, e.g. #2671). Happy to add one if the policy changed.

Closes #3443.