From 854b76addd5be62f06a52b5a60efd8ad62f67488 Mon Sep 17 00:00:00 2001 From: Andrew Nguonly Date: Mon, 14 Apr 2025 15:55:07 -0700 Subject: [PATCH] docs: Update LangGraph Platform autoscaling docs (#4268) --- docs/docs/concepts/langgraph_data_plane.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/docs/concepts/langgraph_data_plane.md b/docs/docs/concepts/langgraph_data_plane.md index fa2368387..5b9f39cf1 100644 --- a/docs/docs/concepts/langgraph_data_plane.md +++ b/docs/docs/concepts/langgraph_data_plane.md @@ -44,16 +44,19 @@ Feature Differences: ### Autoscaling -[`Production` type](../concepts/langgraph_control_plane.md#deployment-types) deployments automatically scale up to 10 containers. Scaling is based on the current request load for a single container. Specifically, the autoscaling implementation scales the deployment so that each container is processing about 10 concurrent requests. For example... +[`Production` type](../concepts/langgraph_control_plane.md#deployment-types) deployments automatically scale up to 10 containers. Scaling is based on 3 metrics: -- If the deployment is processing 20 concurrent requests, the deployment will scale up from 1 container to 2 containers (20 requests / 2 containers = 10 requests per container). -- If a deployment of 2 containers is processing 10 requests, the deployment will scale down from 2 containers to 1 container (10 requests / 1 container = 10 requests per container). +1. CPU utilization +1. Memory utilization +1. Number of pending (in progress) [runs](../concepts/langgraph_server.md#runs) -10 concurrent requests per container is the target threshold. However, 10 concurrent requests per container is not a hard limit. The number of concurrent requests can exceed 10 if there is a sudden burst of requests. +For CPU utilization, the autoscaler targets 75% utilization. This means the autoscaler will scale the number of containers up or down to ensure that CPU utilization is at or near 75%. For memory utilization, the autoscaler targets 75% utilization as well. -Scale down actions are delayed for 30 minutes before any action is taken. In other words, if the autoscaling implementation decides to scale down a deployment, it will first wait for 30 minutes before scaling down. After 30 minutes, the concurrency metric is recomputed and the deployment will scale down if the concurrency metric has met the target threshold. Otherwise, the deployment remains scaled up. This "cool down" period ensures that deployments do not scale up and down too frequently. +For number of pending runs, the autoscaler targets 10 pending runs. For example, if the current number of containers is 1, but the number of pending runs in 20, the autoscaler will scale up the deployment to 2 containers (20 pending runs / 2 containers = 10 pending runs per container). -In the future, the autoscaling implementation may evolve to accommodate other metrics such as background run queue size. +Each metric is computed independently and the autoscaler will determine the scaling action based on the metric that results in the most number of containers. + +Scale down actions are delayed for 30 minutes before any action is taken. In other words, if the autoscaler decides to scale down a deployment, it will first wait for 30 minutes before scaling down. After 30 minutes, the metrics are recomputed and the deployment will scale down if the recomputed metrics result in a lower number of containers than the current number. Otherwise, the deployment remains scaled up. This "cool down" period ensures that deployments do not scale up and down too frequently. ### Static IP Addresses