🐛(scalingo) isolate the metrics endpoint in its own gunicorn pool

DeployCenter calls the usage metrics endpoint synchronously while
answering our entitlements requests. Served by the same sync workers,
this callback could queue behind the very requests blocked waiting for
DeployCenter, freezing the whole pool until timeouts fire. A dedicated
single-worker gunicorn behind nginx guarantees the callback always
finds a free worker, making the circular wait impossible.

The main pool is sized to WEB_CONCURRENCY minus the reserved worker so
the container keeps the worker count its size was provisioned for, and
both values are logged at boot to ease diagnosing worker starvation.

bin/scalingo_run_web

@@ -3,8 +3,26 @@
 # Trigger collabora configuration
 python manage.py trigger_wopi_configuration
 
+# Keep the total worker count at WEB_CONCURRENCY: the metrics pool below
+# takes one worker, the main pool gets the rest (at least one).
+MAIN_WORKERS=$((${WEB_CONCURRENCY:-2} - 1))
+if [ "$MAIN_WORKERS" -lt 1 ]; then
+    MAIN_WORKERS=1
+fi
+echo "WEB_CONCURRENCY=${WEB_CONCURRENCY:-unset} MAIN_WORKERS=$MAIN_WORKERS"
+
 # Start the Django backend
-gunicorn -b :8000 drive.wsgi:application --log-file - &
+gunicorn -b :8000 -w "$MAIN_WORKERS" drive.wsgi:application --log-file - &
+
+# Start a dedicated Django backend pool for the usage metrics endpoint.
+# DeployCenter calls it synchronously while Drive itself is blocked waiting
+# on a DeployCenter entitlements response: served by the main pool, this
+# callback would queue behind the very workers waiting for it (deadlock
+# until timeout). An isolated pool guarantees it always gets a worker.
+# Its access log is enabled with a distinctive prefix so the platform
+# logs show which requests were served by this pool.
+gunicorn -b :8001 -w 1 drive.wsgi:application --log-file - \
+    --access-logfile - --access-logformat 'metrics-pool %(h)s "%(r)s" %(s)s' &
 
 # Start the Nginx server
 bin/run &

src/nginx/servers.conf.erb

@@ -5,6 +5,13 @@ upstream backend_server {
 	server localhost:8000 fail_timeout=0;
 }
 
+# Dedicated gunicorn pool for the usage metrics endpoint, so DeployCenter's
+# synchronous callback never competes with user requests blocked on
+# DeployCenter itself (see bin/scalingo_run_web).
+upstream metrics_server {
+	server localhost:8001 fail_timeout=0;
+}
+
 server {
 	
 	listen <%= ENV["PORT"] %>;
@@ -14,6 +21,18 @@ server {
     
     error_page 404 /404.html;
 
+    # Usage metrics, isolated from the main backend pool. Must stay above
+    # the generic api/external_api block: nginx picks the first matching
+    # regex location.
+    location ~ ^/external_api/v[0-9.]+/metrics/ {
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header Host $http_host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+        proxy_redirect off;
+        proxy_pass http://metrics_server;
+    }
+
     # Django rest framework and external API
     location ~ ^/(api|external_api)/ {
         proxy_set_header X-Forwarded-Proto https;