The yhub database schema have new update and will probably be modified
in the future. We don't want to maintain this sql schema in the Docs
repo, we want to reuse what is directly made in the yhub project. For
this we reuse the existing bin/init-db.js script
Cover both paths off the legacy Django store end to end: the lazy seed on
first access, and the migrate endpoint replaying every S3 version. The tests
need no database — the admin JWT short-circuits document authorization, so a
fixture is an S3 object on a random uuid — and read the timeline through
yhub 0.5.0's `Accept: application/json`, which spares python a lib0 decoder.
CI grows a valkey service and starts a collaboration server alongside the
backend test job; the tests skip themselves when nothing answers on the new
COLLABORATION_API_URL setting, so `make test` without the dev stack still
passes.
Writing them turned up three things worth fixing in the server.
Backend reads now seed too. getAccessType short-circuited on the admin token
before reaching the legacy store, so a server-side read of an unmigrated
document answered with an empty one, and a create-ydoc against it would have
written a second lineage beside the content the first user access was about
to seed in.
Seeding no longer decides access; the backend's answer alone does. A legacy
object that cannot be migrated — it does not decode, or it exceeds the size
we load — opens as a new document instead of denying, since no retry can fix
it and refusing would leave the document unopenable by anyone. The cause is
logged once per attempt with the bucket, key and stack, and every later access
logs that it admitted a caller without migrating.
That made the failure classifier dangerous, so it is inverted. It was an
allowlist of retryable errors — eight socket errnos — which left every way S3
can refuse (AccessDenied on a rotated key, NoSuchBucket, a region redirect)
counting as "this object is unusable". Denying, that was survivable; opening
empty, one misscoped credential would fork every document touched during the
window. Now only a failure raised while interpreting bytes we already hold is
permanent, marked at the throw site, and everything else answers a retryable
503. Guessing wrong that way costs a retry; the other way costs the document.
The admin seed is also fenced to the org and to main, like the user path
above it. The legacy store is branchless — {docid}/file is main — and the
bookkeeping is per document, so seeding ?branch=draft would have written
main's content into an orphan room and left the real one permanently empty.
Signed-off-by: Kevin Jahns <kevin.jahns@protonmail.com>
We add permissions level on workflow to avoid
warning in github action logs.
This is a warning and not an error,
but it is good to have a clean log without warnings.
The psycopg pool config was enabled by default forcing its usage. Using
psycopg pool can be difficult, finding the good configuration take time.
By default its usage should be disable and the maintainer of the
instance should decide to enable it or not.
In order to run the tests we need to increase the max pool size. Only
having 4 connections in the pool is not enough and all the tests using a
transaction are failing with a tiemout error.
We have the same problem running locally so the same value is added to
the postgresql environment file
Update all GitHub Actions to their latest major versions for improved
performance, security patches, and Node.js runtime compatibility.
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
We added a feature to print documents directly
from the browser. The function is called
`window.print()`, this name collides with Python's
`print()` function. To avoid false positives in our
CI when checking for print statements, we limit
the search to only the backend code.
git-lint steps are independant and we would like to have all checks at
once. Using the `if: always()` instruction should ensure all steps
should be run event if the previous fails.
The setup-python action is able to cache the dependencies and reuse this
cache while the pyproject file has not changed. It is easy to setup,
just the package manager used has to be declared in the cache settings.
When we were executing the crowdin_upload workflow,
we were not building the mail template to dispatch it
to the backend. It resulted in the mail not being
totally translated. This commit fixes that issue
by adding the build mail step to the crowdin_upload.
To do so, we added it to the dependencies workflow.
"dependencies" workflow is callable by other
workflows that need a specific job.
All the uploaded files had the content-type set
to `application/octet-stream`. It create issues
when the file is downloaded from the frontend
because the browser doesn't know how to handle
the file.
We now determine the content-type of the file
and set it to the file object.
We were testing the backend with python 3.10.0, but
actually the backend was running with python 3.12.6.
We bump the python version in the backend test to match
the running version of the backend.
Pushing to crowdin from the workflow has some side
effects, if 2 branches are pushing to crowdin it
can cause conflicts and delete translations on
Crowdin side.
Better to push to crowdin manually to keep good
control over the translations.
We can now export our document to a docx file.
This is done by converting the html to a docx
file using the pypandoc and pandoc library.
We added the "format" param to the
generate-document endpoint, "format" accept
"pdf" or "docx" as value.
The tests e2e need the mail template to
send the invitation to the user.
We are now caching the mail template instead
of using a artifact, thanks to that we can
build only the mail template if the cache is
invalidated, so we save time and resources.
The linter in the CI was checking only the
impress directory. We were passing near some lint
issues. Now the linter checks the whole backend
directory.
We want to be able to share a document with a person even if this person
does not have an account in impress yet.
This code is ported from https://github.com/numerique-gouv/people.
Manuel Raynaud