The S3 persistence plugin records the version id of the object it wrote
and names that version when it deletes it. On a versioned bucket - what
a deployment runs - a delete that names no version deletes nothing: it
writes a delete marker and keeps every version underneath. Each
compaction supersedes the blobs of the one before, so what was kept was
every version of every document ever written, a document someone asked
to erase included, still readable by anyone who can list versions. On
AWS this needs s3:DeleteObjectVersion, which a policy granting
s3:DeleteObject alone does not cover.
Blobs are written to the bucket for every branch of a document.
YHUB_S3_PERSISTENCE now governs only whether new blobs are written
there. The plugin itself is attached whenever the YHUB_S3_* settings
name a bucket, on or off, because reading is the half that must never
be taken away: a row pointing at an object is unreadable without the
plugin that wrote it, and yhub reports such a version as having no
content rather than as an error. Turning the toggle off stops the
writing and leaves the reading alone; it is the settings, not the
toggle, that a deployment whose bucket holds anything must keep. Half a
configuration is a startup error naming what is missing, as before.
The dev stack keeps the toggle off and creates its bucket versioned, so
flipping it on exercises what a deployment runs rather than a simpler
case. Its createbuckets job needed fixing to do so: the folded yaml
block joins its lines with a space, so the trailing backslashes reached
the shell as an escaped space glued to the next word and everything
past the first && silently did nothing - the media bucket never had
versioning enabled either.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
The backend application will have to call the yhub REST API for some
operations. We want to use a dedicated service to do that. This first
commit introduces the shape of this service, it only does the
configuration for now, calling actions will be implemented later.
The CollaborationService was doing nothing since we started the
migration to yhub, all the code using it is now removed. Also the
`can-edit` endpoint and all the safeguard mechanism relying on the
presence of other users connected to the websocket will not be used
anymore, it will be possible to replace all of this with yhub, so all
this code is also removed.
We have performance issue with the media_auth endpoint. We need to
monitor and profile this endpoint to understand what is happening. We
decided to install and configure django-silk. By default django-silk is
not enabled and not accessible, once enabled migration must be run and
the /silk/ endpoint will be accessible only using a super user.
To have relecant profiles and monitor we must have a database comparable
to a production one because we are facing a scaling issue. Unless a
million documents probably nothing is happening, for this we added
several management commands to help us. A first one to anonymize a
database than can be reused without any risk to leak PII data, can be
called using python manage.py anonymize_database with some mandatory
options. An other command to generate a database profile, this command
should be run on a production database, this profile can then be used by
the generate_volumetry to populate a development database with same
volumetry than describe in the profile.
The settings COLLABORATION_WS_NOT_CONNECTED_READY_ONLY contains a typo.
We don't want to see READY_ONLY but READ_ONLY. This commit fix the typo
and change everywhere in the codebase the settings name.
