️(backend) optimize media_auth cpu usage

Once the sql queries improved we have still a bottleneck on large
concurrent requests on this endpoint. We notive in the profiles generated
that lot of time was spent in creating a new s3 client instance on each
request. django_storage use a thread local cache for signed and unsigned
connection, but using uvicorn we have a new thread for each request, so
on each request a new s3 client is generated and it appears to be an
expensive operation. To fix this issue, we cache the client and share it
accross all the thread and requests.
This commit is contained in:
Manuel Raynaud
2026-08-20 17:28:31 +02:00
parent 7372c4610f
commit 4111e4e5ed
5 changed files with 65 additions and 4 deletions
+7 -2
View File
@@ -73,6 +73,7 @@ from core.tasks.mail import send_ask_for_access_mail
from core.utils.analytics import PosthogEventName, posthog_capture
from core.utils.dicts import lowercase_keys
from core.utils.paths import filter_descendants
from core.utils.s3 import get_s3_client
from core.utils.s3_response_stream import content_stream
from core.utils.treebeard import create_tree_node_with_retry
from core.utils.users import users_sharing_documents_with
@@ -2043,8 +2044,12 @@ class DocumentViewSet(
logger.debug("User '%s' lacks permission for attachment", user)
raise drf.exceptions.PermissionDenied()
# Check if the attachment is ready
s3_client = default_storage.connection.meta.client
# Check if the attachment is ready. Use the process-global S3 client
# (see core.utils.s3): django-storages caches its client per thread, so
# relying on default_storage.connection here rebuilds the boto3 client
# on every fresh thread -- profiling showed that client construction,
# not the DB, dominated this endpoint's CPU under load.
s3_client = get_s3_client()
bucket_name = default_storage.bucket_name
try:
head_resp = s3_client.head_object(Bucket=bucket_name, Key=key)