The readiness probe runs every Django check and opens a database
connection. With a 1s timeout every 10s, a slow database took the
pods out of the Service. Give it more time and call it less often.
Fix the probe rows in the README.
The quota field only handled the usage and limit shape of the quota. The
other ones, like an error or a locked organization, were shown as raw
values without labels and an absent quota was not mentioned at all. Every
field of the quota is now labelled, along with the storage counted by
Drive as the backend may compute the usage on its own.
Checking why a user hits the quota required database access. Show the
storage counted in the quota and the effective limit of each user, and
the biggest files in the user form, with a selectable count. Admin
sizes now use format_size to match the app.
The admin formats sizes in base 1024 with Django filesizeformat while
the app and the storage quota use decimal units. Mirror the frontend
formatSize, rounding included, so both sides show the same values.
quota_excluded could not be checked nor fixed without database access.
Show it with the item size in the list, filter on it, and let admins
toggle it on items and their descendants, from the list or the form.
Excluding a folder from the storage quota must cover its whole tree.
The bulk update bypasses Item.save(), so the storage used cache of
every impacted creator is invalidated explicitly.
Emit item_deleted on soft delete and item_hard_deleted on the
hard-delete action to measure how often users empty their trash.
Detaching a restriction is not a deletion and stays silent.
Documents created from a template never go through the upload flow, so
the item_uploaded event misses them. Emit item_created_from_template
from the shared helper to cover both root and children creation.
File content was streamed inline on the application origin, with no
fallback when the item has no mimetype. WOPI clients fetch it server
side, so serving it as a sandboxed download costs them nothing and
keeps stored bytes from ever rendering in a browser.
The trash purge and the stale pending items cleaning only ran through
the cron jobs of the Helm chart, so any other deployment had to recreate
them or never clean anything. Celery beat already runs next to the app,
so it can schedule them wherever Drive is deployed.
Each command has its own setting holding a crontab line followed by the
command arguments, so operators can tune both in one place. They are
empty by default: scheduling a command that already runs as a cron job
would run it twice. They are validated at startup, arguments included,
because celery only logs errors raised while registering periodic tasks
and a typo would then silently never run the command.
The command deleted the row of an abandoned upload but left its object
in the storage. The row being the only pointer to that object, nothing
could find it afterwards and it stayed there forever, out of any quota.
Stale pending items are now only marked as hard deleted, leaving the
purge_deleted_items command remove the object then the row as it does
for any other item. An upload that never reached the storage has no
object to remove, which the purge already tolerates.
The command listed the stale pending items then deleted them one by one
without checking again. An upload ended between the listing and the
deletion was still removed, along with its finalized object reference.
Each item is now locked and its pending state checked again before it is
deleted.
The number of accesses of each child was counted by joining the
accesses, which read the whole access table and grouped the query. A
grouped query cannot be counted directly, so the pagination count
wrapped the whole query, computing again every annotation of every
child: 294ms of the children listing of a large production folder.
Counting the direct accesses in a correlated subquery returns the same
numbers without grouping the query, and the pagination count becomes a
plain count of the children.
The children of an item were found among all its descendants, then
filtered on their depth: listing the 9 children of a production folder
holding 62k items read the whole subtree through the GiST index, 1.2s,
and counting the children of each of them read their own subtrees,
another 0.8s.
The item table is now indexed on the path of each item's parent, so
that both are exact lookups returning only the children: 0.7ms and
4.6ms on the same folder. The index is built concurrently, about 20
seconds and 900MB for 4M items, and matches the previous definition of
children for every item, including those whose path is inconsistent.
Queries that are fast on a development database can take seconds on an
instance with millions of items, and nothing explained how to check a
change on realistic data.
This page explains how to generate a production sized dataset and
profile the endpoints, why JIT should be disabled on production
databases, and why the GiST index on paths should be avoided in queries
that run for every request or every item.
Each listed item read its number of accesses from the cache with its
own round trip: 350ms for a page of 200 items.
The item list serializer now reads them all with a single get_many
before serializing the items, when its fields include the number of
accesses. Items missing from the cache still compute it when it is
read, through the permissions backend as before.
To tell whether each listed file can be opened with WOPI, the serializer
fetched the whole WOPI configuration from the cache, file by file. The
configuration is large and unpickling it for every file was most of the
serialization time: 315ms for a page of 200 items.
It is now fetched once per serialization and passed down to the WOPI
support check. Other callers keep fetching it themselves.
The number of children and of folder children of each listed item is
counted by two subqueries, each one scanning the GiST index of paths to
find all the descendants of the item: about 1ms per subquery on a large
database, so 2 x 200 scans for a page of items.
The descendants are now found with a new path__in_subtree lookup, a
faster equivalent of path__descendants: it reads the subtree as a range
of the unique btree index on paths, like subtrees in recents, and
returns the same rows about 20 times faster. The depth is kept as a
difference with the outer item: with an equality on the depth, Postgres
would combine the path range with the depth index, matching a whole
level of all the trees.
Recents and favorites filter the items under the roots the user can
reach with one "path is a descendant of this root" condition per root,
OR-ed together. Postgres resolved them with one GiST index scan per
root: about 1s for a user reaching 1700 roots, paid twice with the
pagination count.
ltree compares labels byte by byte and the hyphen is the lowest byte
allowed in a label, so the subtree of "a.b", and only it, sorts between
"a.b" and "a.b-". Each subtree is thus a range of the unique btree index
on paths. The ids of all the subtrees are computed at once from these
ranges, then items are looked up by primary key: 35ms instead of 1s for
the same user.
Unlike recents and favorites, the item list did not give the ancestors
links of its items to its serializer. Each listed item that is not at
the root of its tree, like a folder shared from inside another tree,
computed them with its own query when computing its abilities: 66
queries for a user reaching many shared folders.
Now that they are computed in one query for all the items, the list
computes them too.
The ancestors links of listed items, and of the items traced by link
when filtering accessible items, were computed with one query per tree:
more than 200 queries for a user with many link traces, each one an
ancestors lookup on the GiST index of paths.
The paths of all the ancestors are known from the items paths, so they
are now fetched at once through the unique index on paths. The mapping
only covers the parents of the items and their ancestors, the only paths
it is read at, so that root items need no query at all.
Accessible items were filtered with "id in the accesses of the user OR
id in the items traced by link". Postgres cannot resolve such an OR
with an index and scanned the whole item table instead, as soon as the
user had one link trace: about 1s on every list, recents and favorites
request with 4M items, whatever the number of items of the user.
Merging both sets in a single UNION subquery lets Postgres look items
up by primary key: the same query now takes a few milliseconds.
effective_accesses and roles_at found the accesses applying to an item
with an ancestors lookup on the item paths, resolved on their GiST index
at about 10ms per item. Counting the accesses of 200 items took 2.4s,
which happens when their number of accesses is missing from the cache.
Filtering accesses on the ids found in the item path uses the access
item index instead and returns the same accesses: the same 200 counts
now take 85ms.
The trashbin owner filter, the reachable restrictions hidden from the
item list and the contact filter used the same join on the ancestors of
each item as the user roles did, with the same cost growing with the
number of accesses or link traces of the user. They now rely on IdInPath
to look accesses, link traces and ancestors up by id.
The user roles subquery joined the accesses of the user on their items
and kept those whose path is an ancestor of the listed item. Postgres
went through all the accesses of the user for each listed item: a user
holding 1500 accesses and listing 200 items meant 305k index lookups and
more than 800ms.
An item path is made of the ids of its ancestors and its own id, so the
accesses applying to an item are those whose item id is one of its path
labels. The IdInPath expression looks them up on the access index
directly, without joining items: the same page now takes 5ms.
Measuring the effect of query changes needs more than the query counts
asserted by tests. This command calls the item and user endpoints as a
given user and reports their response times, their queries and,
optionally, the execution plan of their slowest query. The JIT can be
disabled for the session to measure its cost, and the endpoints can be
narrowed down to iterate faster.
Slow queries reported in production could not be reproduced locally:
the demo command only creates a handful of items through factories.
This command fills an empty database with a dataset shaped like
production: 50k users, 4M items spread over deep trees, accesses,
link traces, favorites and trashed items. Rows are generated with set
based SQL so that it runs in minutes, and no file is written to the
object storage. The development user gets a small drive to reproduce
slow requests for users with few items.
The La Gaufre services url of the anct theme carried the siret of a
single organization, so every user was served the services list of
that organization. The operator alone is enough to scope the request.
The cunningham tokens are regenerated accordingly.
The selection bar button and the modal confirm button share the same
name. When the locator resolved before the modal was rendered, it kept
targeting the bar button hidden behind the backdrop and the click timed
out. It made the chromium shard fail on unrelated branches.
Django applies DATA_UPLOAD_MAX_MEMORY_SIZE to the request body too, so a
10 bytes limit rejected the JSON payload of the test itself before the
serializer could check the reserved size. The CI failed on it.
A pending item already in the trash, directly or through an ancestor,
made soft_delete raise and aborted the whole command, leaving every
following item in place. Each item is now cleaned in its own
transaction and a failure is logged and counted instead of stopping
the run.
Creating, adding a child and duplicating now share a per-user burst and
sustained budget, whatever the item type, so creation is bounded in rate as
well as in volume.
Both rates are settings so an operator can tune them, and the defaults sit
well above what the interface produces: uploads are serial and a folder drop
sends one request per entry, so a real user never reaches them.
Integrators copying this snippet would now get a 400: creating a file item
requires a size reservation, and the returned policy signs Content-Length,
so the raw file must be sent rather than a multipart form.
Item creation now rejects a file without a reservation, so both session
scenarios would fail at the upload step. The size is read from the file the
scenario is about to send, keeping the generated traffic realistic.
The reservation only holds if the signed Content-Length, MinIO and the quota
agree, which unit tests cannot show since they stub the storage. The e2e
stack now runs the local entitlements backend so admission is really
exercised, and a dedicated account keeps the quota assertions independent
from the items other specs leave behind.
Clients now declare the size when creating a file item. The item is created
with that reservation and the entitlement decision is applied in the same
transaction, so a refusal rolls back the creation before any authorization
is returned.
The presigned URL signs Content-Length and finalization verifies the stored
object against the reservation. Storage usage stays a plain sum of size,
with no double counting once the upload ends, and a pending upload is
accounted for from the moment it is created.
