From 87e59e78fa2ec6eec81b8dfd1a101437be9f4b47 Mon Sep 17 00:00:00 2001 From: risk-alt Date: Mon, 10 Aug 2026 14:32:33 +0200 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9B(helm)=20show=20the=20database=20er?= =?UTF-8?q?ror=20while=20jobs=20wait=20for=20it=20to=20be=20ready?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The migrate and createsuperuser jobs polled `manage.py check` with its output sent to /dev/null, then printed "Database not ready" whatever the reason. Any failure the check reports, a missing setting or a bad credential for instance, looked like a database that had not started yet, and the job looped forever without a single clue in its logs. The check output is now captured and printed on each failed attempt, along with the attempt number, so the reason the job is still waiting is readable with kubectl logs. The retry behaviour is unchanged. Applied to both jobs in the chart values and in the dev and feature helmfile environments. The chart README table is regenerated accordingly. Signed-off-by: risk-alt --- CHANGELOG.md | 1 + src/helm/env.d/dev/values.impress.yaml.gotmpl | 14 +- .../env.d/feature/values.impress.yaml.gotmpl | 14 +- src/helm/impress/README.md | 150 +++++++++--------- src/helm/impress/values.yaml | 14 +- 5 files changed, 106 insertions(+), 87 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7069b03b1..83b3ea289 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ and this project adheres to - 🐛(backend) ignore CSPs for API docs in development - 🐛(frontend) export images embedded with a relative url #2573 - 🐛(y-provider) fix sentry init #2579 +- 🐛(helm) show the database error while jobs wait for it to be ready #2578 ## [v5.4.1] - 2026-07-09 diff --git a/src/helm/env.d/dev/values.impress.yaml.gotmpl b/src/helm/env.d/dev/values.impress.yaml.gotmpl index 075025784..981f4c84a 100644 --- a/src/helm/env.d/dev/values.impress.yaml.gotmpl +++ b/src/helm/env.d/dev/values.impress.yaml.gotmpl @@ -88,9 +88,12 @@ backend: - "/bin/sh" - "-c" - | - while ! python manage.py check --database default > /dev/null 2>&1 + attempt=0 + until output=$(python manage.py check --database default 2>&1) do - echo "Database not ready" + attempt=$((attempt + 1)) + echo "Database check failed (attempt $attempt), retrying in 2s:" + echo "$output" sleep 2 done @@ -113,9 +116,12 @@ backend: - "/bin/sh" - "-c" - | - while ! python manage.py check --database default > /dev/null 2>&1 + attempt=0 + until output=$(python manage.py check --database default 2>&1) do - echo "Database not ready" + attempt=$((attempt + 1)) + echo "Database check failed (attempt $attempt), retrying in 2s:" + echo "$output" sleep 2 done diff --git a/src/helm/env.d/feature/values.impress.yaml.gotmpl b/src/helm/env.d/feature/values.impress.yaml.gotmpl index 3826b6ab2..e61d74154 100644 --- a/src/helm/env.d/feature/values.impress.yaml.gotmpl +++ b/src/helm/env.d/feature/values.impress.yaml.gotmpl @@ -82,9 +82,12 @@ backend: - "/bin/sh" - "-c" - | - while ! python manage.py check --database default > /dev/null 2>&1 + attempt=0 + until output=$(python manage.py check --database default 2>&1) do - echo "Database not ready" + attempt=$((attempt + 1)) + echo "Database check failed (attempt $attempt), retrying in 2s:" + echo "$output" sleep 2 done @@ -105,9 +108,12 @@ backend: - "/bin/sh" - "-c" - | - while ! python manage.py check --database default > /dev/null 2>&1 + attempt=0 + until output=$(python manage.py check --database default 2>&1) do - echo "Database not ready" + attempt=$((attempt + 1)) + echo "Database check failed (attempt $attempt), retrying in 2s:" + echo "$output" sleep 2 done diff --git a/src/helm/impress/README.md b/src/helm/impress/README.md index c8dcaf434..5e8a97f56 100644 --- a/src/helm/impress/README.md +++ b/src/helm/impress/README.md @@ -87,81 +87,81 @@ ### backend -| Name | Description | Value | -| ----------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `backend.command` | Override the backend container command | `[]` | -| `backend.args` | Override the backend container args | `[]` | -| `backend.replicas` | Amount of backend replicas | `3` | -| `backend.shareProcessNamespace` | Enable share process namespace between containers | `false` | -| `backend.sidecars` | Add sidecars containers to backend deployment | `[]` | -| `backend.migrateJobAnnotations` | Annotations for the migrate job | `{}` | -| `backend.jobs.ttlSecondsAfterFinished` | Period to wait before remove jobs | `30` | -| `backend.jobs.backoffLimit` | Numbers of jobs retries | `2` | -| `backend.securityContext.allowPrivilegeEscalation` | Whether to allow privilege escalation for the backend container | `false` | -| `backend.securityContext.capabilities.drop` | List of capabilities to drop for the backend container | `["ALL"]` | -| `backend.securityContext.runAsNonRoot` | Whether to run the backend container as a non-root user | `true` | -| `backend.securityContext.seccompProfile.type` | Seccomp profile type for the backend container | `RuntimeDefault` | -| `backend.envVars` | Configure backend container environment variables | `undefined` | -| `backend.envVars.BY_VALUE` | Example environment variable by setting value directly | | -| `backend.envVars.FROM_CONFIGMAP.configMapKeyRef.name` | Name of a ConfigMap when configuring env vars from a ConfigMap | | -| `backend.envVars.FROM_CONFIGMAP.configMapKeyRef.key` | Key within a ConfigMap when configuring env vars from a ConfigMap | | -| `backend.envVars.FROM_SECRET.secretKeyRef.name` | Name of a Secret when configuring env vars from a Secret | | -| `backend.envVars.FROM_SECRET.secretKeyRef.key` | Key within a Secret when configuring env vars from a Secret | | -| `backend.django.envVars` | Backend web deployment specific environment variables (not shared with celery worker) | | -| `backend.podAnnotations` | Annotations to add to the backend Pod | `{}` | -| `backend.dpAnnotations` | Annotations to add to the backend Deployment | `{}` | -| `backend.service.type` | backend Service type | `ClusterIP` | -| `backend.service.port` | backend Service listening port | `80` | -| `backend.service.targetPort` | backend container listening port | `8000` | -| `backend.service.annotations` | Annotations to add to the backend Service | `{}` | -| `backend.migrate.command` | backend migrate command | `["/bin/sh","-c","while ! python manage.py check --database default > /dev/null 2>&1\ndo\n echo \"Database not ready\"\n sleep 2\ndone\n\necho \"Database is ready\"\n\npython manage.py migrate --no-input\n"]` | -| `backend.migrate.restartPolicy` | backend migrate job restart policy | `Never` | -| `backend.createsuperuser.command` | backend migrate command | `["/bin/sh","-c","while ! python manage.py check --database default > /dev/null 2>&1\ndo\n echo \"Database not ready\"\n sleep 2\ndone\n\necho \"Database is ready\"\n\npython manage.py createsuperuser --email $DJANGO_SUPERUSER_EMAIL --password $DJANGO_SUPERUSER_PASSWORD\n"]` | -| `backend.createsuperuser.restartPolicy` | backend migrate job restart policy | `Never` | -| `backend.job` | job dedicated to run a random management command, for example after a deployment | | -| `backend.job.name` | The name to use to describe this job | `""` | -| `backend.job.command` | The management command to execute | `[]` | -| `backend.job.restartPolicy` | The restart policy for the job. | `Never` | -| `backend.job.annotations` | Annotations to add to the job [default: argocd.argoproj.io/hook: PostSync] | | -| `backend.cronjobs` | Cronjob name, schedule, command | `[]` | -| `backend.probes.liveness.path` | Configure path for backend HTTP liveness probe | `/__lbheartbeat__` | -| `backend.probes.liveness.targetPort` | Configure port for backend HTTP liveness probe | `nil` | -| `backend.probes.liveness.initialDelaySeconds` | Configure initial delay for backend liveness probe | `10` | -| `backend.probes.liveness.timeoutSeconds` | Configure timeout for backend liveness probe | `nil` | -| `backend.probes.startup.path` | Configure path for backend HTTP startup probe | | -| `backend.probes.startup.targetPort` | Configure port for backend HTTP startup probe | | -| `backend.probes.startup.initialDelaySeconds` | Configure initial delay for backend startup probe | | -| `backend.probes.startup.timeoutSeconds` | Configure timeout for backend startup probe | | -| `backend.probes.readiness.path` | Configure path for backend HTTP readiness probe | `/__heartbeat__` | -| `backend.probes.readiness.targetPort` | Configure port for backend HTTP readiness probe | `nil` | -| `backend.probes.readiness.initialDelaySeconds` | Configure initial delay for backend readiness probe | `10` | -| `backend.probes.readiness.timeoutSeconds` | Configure timeout for backend readiness probe | `nil` | -| `backend.resources` | Resource requirements for the backend container | `{}` | -| `backend.nodeSelector` | Node selector for the backend Pod | `{}` | -| `backend.tolerations` | Tolerations for the backend Pod | `[]` | -| `backend.affinity` | Affinity for the backend Pod | `{}` | -| `backend.persistence` | Additional volumes to create and mount on the backend. Used for debugging purposes | `{}` | -| `backend.persistence.volume-name.size` | Size of the additional volume | | -| `backend.persistence.volume-name.type` | Type of the additional volume, persistentVolumeClaim or emptyDir | | -| `backend.persistence.volume-name.mountPath` | Path where the volume should be mounted to | | -| `backend.extraVolumeMounts` | Additional volumes to mount on the backend. | `[]` | -| `backend.extraVolumes` | Additional volumes to mount on the backend. | `[]` | -| `backend.pdb.enabled` | Enable pdb on backend | `true` | -| `backend.serviceAccountName` | Optional service account name to use for backend pods | `nil` | -| `backend.themeCustomization.enabled` | Enable theme customization | `false` | -| `backend.themeCustomization.file_content` | Content of the theme customization file. Must be a json object. | `""` | -| `backend.themeCustomization.mount_path` | Path where the customization file will be mounted in the backend deployment. | `/app/impress/configuration/theme` | -| `backend.celery.replicas` | Amount of celery replicas | `1` | -| `backend.celery.command` | Override the celery container command | `[]` | -| `backend.celery.args` | Override the celery container args | `["celery","-A","impress.celery_app","worker","-l","INFO","-n","impress@%h"]` | -| `backend.celery.resources` | Resource requirements for the celery container | `{}` | -| `backend.celery.probes.liveness.exec.command` | Override the celery container liveness probe command | `["/bin/sh","-c","celery -A impress.celery_app inspect ping -d impress@$HOSTNAME"]` | -| `backend.celery.probes.liveness.initialDelaySeconds` | Initial delay for the celery container liveness probe | `60` | -| `backend.celery.probes.liveness.timeoutSeconds` | Timeout for the celery container liveness probe | `5` | -| `backend.celery.probes.readiness.exec.command` | Override the celery container readiness probe command | `["/bin/sh","-c","celery -A impress.celery_app inspect ping -d impress@$HOSTNAME"]` | -| `backend.celery.probes.readiness.initialDelaySeconds` | Initial delay for the celery container readiness probe | `15` | -| `backend.celery.probes.readiness.timeoutSeconds` | Timeout for the celery container readiness probe | `5` | -| `backend.celery.envVars` | Celery worker specific environment variables (not shared with the backend web deployment) | | +| Name | Description | Value | +| ----------------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `backend.command` | Override the backend container command | `[]` | +| `backend.args` | Override the backend container args | `[]` | +| `backend.replicas` | Amount of backend replicas | `3` | +| `backend.shareProcessNamespace` | Enable share process namespace between containers | `false` | +| `backend.sidecars` | Add sidecars containers to backend deployment | `[]` | +| `backend.migrateJobAnnotations` | Annotations for the migrate job | `{}` | +| `backend.jobs.ttlSecondsAfterFinished` | Period to wait before remove jobs | `30` | +| `backend.jobs.backoffLimit` | Numbers of jobs retries | `2` | +| `backend.securityContext.allowPrivilegeEscalation` | Whether to allow privilege escalation for the backend container | `false` | +| `backend.securityContext.capabilities.drop` | List of capabilities to drop for the backend container | `["ALL"]` | +| `backend.securityContext.runAsNonRoot` | Whether to run the backend container as a non-root user | `true` | +| `backend.securityContext.seccompProfile.type` | Seccomp profile type for the backend container | `RuntimeDefault` | +| `backend.envVars` | Configure backend container environment variables | `undefined` | +| `backend.envVars.BY_VALUE` | Example environment variable by setting value directly | | +| `backend.envVars.FROM_CONFIGMAP.configMapKeyRef.name` | Name of a ConfigMap when configuring env vars from a ConfigMap | | +| `backend.envVars.FROM_CONFIGMAP.configMapKeyRef.key` | Key within a ConfigMap when configuring env vars from a ConfigMap | | +| `backend.envVars.FROM_SECRET.secretKeyRef.name` | Name of a Secret when configuring env vars from a Secret | | +| `backend.envVars.FROM_SECRET.secretKeyRef.key` | Key within a Secret when configuring env vars from a Secret | | +| `backend.django.envVars` | Backend web deployment specific environment variables (not shared with celery worker) | | +| `backend.podAnnotations` | Annotations to add to the backend Pod | `{}` | +| `backend.dpAnnotations` | Annotations to add to the backend Deployment | `{}` | +| `backend.service.type` | backend Service type | `ClusterIP` | +| `backend.service.port` | backend Service listening port | `80` | +| `backend.service.targetPort` | backend container listening port | `8000` | +| `backend.service.annotations` | Annotations to add to the backend Service | `{}` | +| `backend.migrate.command` | backend migrate command | `["/bin/sh","-c","attempt=0\nuntil output=$(python manage.py check --database default 2>&1)\ndo\n attempt=$((attempt + 1))\n echo \"Database check failed (attempt $attempt), retrying in 2s:\"\n echo \"$output\"\n sleep 2\ndone\n\necho \"Database is ready\"\n\npython manage.py migrate --no-input\n"]` | +| `backend.migrate.restartPolicy` | backend migrate job restart policy | `Never` | +| `backend.createsuperuser.command` | backend migrate command | `["/bin/sh","-c","attempt=0\nuntil output=$(python manage.py check --database default 2>&1)\ndo\n attempt=$((attempt + 1))\n echo \"Database check failed (attempt $attempt), retrying in 2s:\"\n echo \"$output\"\n sleep 2\ndone\n\necho \"Database is ready\"\n\npython manage.py createsuperuser --email $DJANGO_SUPERUSER_EMAIL --password $DJANGO_SUPERUSER_PASSWORD\n"]` | +| `backend.createsuperuser.restartPolicy` | backend migrate job restart policy | `Never` | +| `backend.job` | job dedicated to run a random management command, for example after a deployment | | +| `backend.job.name` | The name to use to describe this job | `""` | +| `backend.job.command` | The management command to execute | `[]` | +| `backend.job.restartPolicy` | The restart policy for the job. | `Never` | +| `backend.job.annotations` | Annotations to add to the job [default: argocd.argoproj.io/hook: PostSync] | | +| `backend.cronjobs` | Cronjob name, schedule, command | `[]` | +| `backend.probes.liveness.path` | Configure path for backend HTTP liveness probe | `/__lbheartbeat__` | +| `backend.probes.liveness.targetPort` | Configure port for backend HTTP liveness probe | `nil` | +| `backend.probes.liveness.initialDelaySeconds` | Configure initial delay for backend liveness probe | `10` | +| `backend.probes.liveness.timeoutSeconds` | Configure timeout for backend liveness probe | `nil` | +| `backend.probes.startup.path` | Configure path for backend HTTP startup probe | | +| `backend.probes.startup.targetPort` | Configure port for backend HTTP startup probe | | +| `backend.probes.startup.initialDelaySeconds` | Configure initial delay for backend startup probe | | +| `backend.probes.startup.timeoutSeconds` | Configure timeout for backend startup probe | | +| `backend.probes.readiness.path` | Configure path for backend HTTP readiness probe | `/__heartbeat__` | +| `backend.probes.readiness.targetPort` | Configure port for backend HTTP readiness probe | `nil` | +| `backend.probes.readiness.initialDelaySeconds` | Configure initial delay for backend readiness probe | `10` | +| `backend.probes.readiness.timeoutSeconds` | Configure timeout for backend readiness probe | `nil` | +| `backend.resources` | Resource requirements for the backend container | `{}` | +| `backend.nodeSelector` | Node selector for the backend Pod | `{}` | +| `backend.tolerations` | Tolerations for the backend Pod | `[]` | +| `backend.affinity` | Affinity for the backend Pod | `{}` | +| `backend.persistence` | Additional volumes to create and mount on the backend. Used for debugging purposes | `{}` | +| `backend.persistence.volume-name.size` | Size of the additional volume | | +| `backend.persistence.volume-name.type` | Type of the additional volume, persistentVolumeClaim or emptyDir | | +| `backend.persistence.volume-name.mountPath` | Path where the volume should be mounted to | | +| `backend.extraVolumeMounts` | Additional volumes to mount on the backend. | `[]` | +| `backend.extraVolumes` | Additional volumes to mount on the backend. | `[]` | +| `backend.pdb.enabled` | Enable pdb on backend | `true` | +| `backend.serviceAccountName` | Optional service account name to use for backend pods | `nil` | +| `backend.themeCustomization.enabled` | Enable theme customization | `false` | +| `backend.themeCustomization.file_content` | Content of the theme customization file. Must be a json object. | `""` | +| `backend.themeCustomization.mount_path` | Path where the customization file will be mounted in the backend deployment. | `/app/impress/configuration/theme` | +| `backend.celery.replicas` | Amount of celery replicas | `1` | +| `backend.celery.command` | Override the celery container command | `[]` | +| `backend.celery.args` | Override the celery container args | `["celery","-A","impress.celery_app","worker","-l","INFO","-n","impress@%h"]` | +| `backend.celery.resources` | Resource requirements for the celery container | `{}` | +| `backend.celery.probes.liveness.exec.command` | Override the celery container liveness probe command | `["/bin/sh","-c","celery -A impress.celery_app inspect ping -d impress@$HOSTNAME"]` | +| `backend.celery.probes.liveness.initialDelaySeconds` | Initial delay for the celery container liveness probe | `60` | +| `backend.celery.probes.liveness.timeoutSeconds` | Timeout for the celery container liveness probe | `5` | +| `backend.celery.probes.readiness.exec.command` | Override the celery container readiness probe command | `["/bin/sh","-c","celery -A impress.celery_app inspect ping -d impress@$HOSTNAME"]` | +| `backend.celery.probes.readiness.initialDelaySeconds` | Initial delay for the celery container readiness probe | `15` | +| `backend.celery.probes.readiness.timeoutSeconds` | Timeout for the celery container readiness probe | `5` | +| `backend.celery.envVars` | Celery worker specific environment variables (not shared with the backend web deployment) | | ### frontend diff --git a/src/helm/impress/values.yaml b/src/helm/impress/values.yaml index a64084647..25ec08736 100644 --- a/src/helm/impress/values.yaml +++ b/src/helm/impress/values.yaml @@ -284,9 +284,12 @@ backend: - /bin/sh - "-c" - | - while ! python manage.py check --database default > /dev/null 2>&1 + attempt=0 + until output=$(python manage.py check --database default 2>&1) do - echo "Database not ready" + attempt=$((attempt + 1)) + echo "Database check failed (attempt $attempt), retrying in 2s:" + echo "$output" sleep 2 done @@ -302,9 +305,12 @@ backend: - "/bin/sh" - "-c" - | - while ! python manage.py check --database default > /dev/null 2>&1 + attempt=0 + until output=$(python manage.py check --database default 2>&1) do - echo "Database not ready" + attempt=$((attempt + 1)) + echo "Database check failed (attempt $attempt), retrying in 2s:" + echo "$output" sleep 2 done