diff --git a/CHANGELOG.md b/CHANGELOG.md index 91d35a0a2..b98ef6e9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,8 @@ and this project adheres to - ✨(frontend) fall back to http polling when the websocket cannot be opened. - 🔧(collaboration) make the version-history granularity configurable through `COLLABORATION_VERSION_GRANULARITY_MS` +- ✨(frontend) keep a local copy of documents, so they open and stay editable + offline ### Changed @@ -43,6 +45,8 @@ and this project adheres to - 🐛(frontend) hide the selection highlight on presenter images #2665 - 🐛(frontend) stop reconnecting to the collaboration server when it has refused the connection for good. +- 🐛(frontend) stop the service worker from caching the collaboration server's + rest api ### Removed diff --git a/documentation/collaboration.md b/documentation/collaboration.md index 6053e59ed..6f986e3a4 100644 --- a/documentation/collaboration.md +++ b/documentation/collaboration.md @@ -96,6 +96,66 @@ Documents are never in conflict either way: both transports publish from the sam Yjs merges. Before the fallback existed, users who could not open a websocket edited a document that was saved wholesale and erased each other's modifications; that is what this removes. +## Working offline + +Every document this browser opens is kept locally, as a Yjs document in IndexedDB +(`IndexeddbPersistence`, one database per document id). Opening a document offline shows what was +written last time instead of an empty editor, and what is written offline survives a reload. + +It lives next to the `Y.Doc`, in `useProviderStore`, and not in the service worker — which is where +it looks like it belongs, since the service worker is already what serves the page and the document +metadata offline. A service worker only sees http, and content that arrives over the websocket +never passes through one, so a worker-side cache would be empty for exactly the clients whose +connection works. Hooking the document instead means it does not matter which transport filled it. + +Nothing extra publishes those changes. Both providers answer the collaboration server's sync step 1 +with everything the server is missing, so the next connection that opens carries whatever was +written offline, and Yjs merges it — no replay queue, and no wholesale save that could erase someone +else's work. + +Three consequences worth knowing: + +- A browser with no `indexedDB` — one told to block site data, some private windows — gets an editor + that behaves exactly as it did before, not a broken one. Local persistence is treated as absent. +- Local content is enough to render, so the editor appears before the socket has finished opening, + online as well as offline. +- A reader's `PATCH /ydoc` is now dropped client-side rather than sent. The collaboration server + enforces read-only differently per transport: the socket drops a reader's document updates and + stays open, while http refuses them with a 403 — and a 4xx is permanent, so the provider would + stop *before* its first `GET`, since the `PATCH` comes first in a round. That never arose while a + reader's document stayed empty until the first `GET` filled it, which is exactly what a local copy + changes. Nothing is lost by dropping it: a reader's content came from the server to begin with. + +### What is kept, and for how long + +A copy is dropped once nobody has opened that document for thirty days, swept once on startup +(`sweepLocalDocs`). `IndexeddbPersistence` has no expiry of its own, so without this every document +ever opened would be kept until the browser evicted it under storage pressure — which it does +without asking and without order. + +When each copy was last opened is tracked in a small IndexedDB database of its own +(`docs-local-index`), not in local storage — so it shares one fate with the copies it tracks. A +browser clears site data and evicts storage per origin, so the index and the documents it indexes +are wiped together or kept together, never one without the other. Where the browser can enumerate +its databases (`indexedDB.databases()` — Chromium and WebKit, never Firefox), the sweep also drops +any document-shaped database the index has lost track of, so an index that was somehow lost cannot +strand copies on disk. + +**Content is not cleared on logout.** A local copy outlives the session that created it, so on a +shared machine the next person to use that browser profile has the documents of the last one. This +is a deliberate gap, not an oversight — clearing on logout, and on a document whose access the +server has revoked, is still to do. + +### The service worker's part + +The collaboration server's rest api is never cached: `ydoc`, `activity`, `changeset` and `rollback` +are all `NetworkOnly`. This is not the default — the collaboration server is on the application's +own origin unless an instance moves it, so without a route of its own a poll for the live document +falls through to the catch-all `StaleWhileRevalidate` and is answered from a cache: a room frozen at +the moment it was first read, and, offline, one the provider would take for a successful round and +report as synced. A version list read from a cache has the same problem, missing every version made +since. + ## Who may share a cursor Presence — the coloured cursors and selections of the other people in a document — is a permission diff --git a/documentation/env.md b/documentation/env.md index 11454b34c..76f786008 100644 --- a/documentation/env.md +++ b/documentation/env.md @@ -38,6 +38,7 @@ These are the environment variables you can set for the `impress-backend` contai | CACHES_SESSION_IGNORE_EXCEPTIONS | Ignoring exception, behave like a missed cache. Highly recommended to set it as False when used with redis (See https://github.com/jazzband/django-redis#memcached-exceptions-behavior) | False | | CACHES_SESSION_SOCKET_CONNECT_TIMEOUT | Timeout for the connection to be established for the session cache (In seconds) | 0.5 | | CACHES_SESSION_SOCKET_TIMEOUT | Timeout for read and write operations after the connection is established for the session cache (In seconds) | 1 | +| COLLABORATION_LOCAL_DOC_RETENTION_DAYS | How many days a browser keeps its local (offline) copy of a document after the last time it was opened. The frontend drops copies older than this on startup. | 30 | | COLLABORATION_VERSION_GRANULARITY_MS | How coarse the document version history is, in milliseconds: edits closer together than this are shown as one version, and none spans more than this. Sent to the collaboration server as the grouping window and re-applied in the browser. | 60000 | | COLLABORATION_WS_INACTIVITY_TIMEOUT | Timeout (in seconds) after which the user is considered inactive when there is no activity. The WebSocket is closed after this inactivity period. `None` means disabled. | None | | COLLABORATION_WS_URL | Collaboration websocket url | | diff --git a/src/backend/core/api/viewsets.py b/src/backend/core/api/viewsets.py index 4280e4660..2e17f5a53 100644 --- a/src/backend/core/api/viewsets.py +++ b/src/backend/core/api/viewsets.py @@ -2954,6 +2954,7 @@ class ConfigView(drf.views.APIView): "AI_FEATURE_BLOCKNOTE_ENABLED", "AI_FEATURE_LEGACY_ENABLED", "API_USERS_SEARCH_QUERY_MIN_LENGTH", + "COLLABORATION_LOCAL_DOC_RETENTION_DAYS", "COLLABORATION_VERSION_GRANULARITY_MS", "COLLABORATION_WS_URL", "COLLABORATION_WS_INACTIVITY_TIMEOUT", diff --git a/src/backend/core/tests/test_api_config.py b/src/backend/core/tests/test_api_config.py index 8bf4b5579..3472db697 100644 --- a/src/backend/core/tests/test_api_config.py +++ b/src/backend/core/tests/test_api_config.py @@ -24,6 +24,7 @@ pytestmark = pytest.mark.django_db AI_FEATURE_BLOCKNOTE_ENABLED=False, AI_FEATURE_LEGACY_ENABLED=False, API_USERS_SEARCH_QUERY_MIN_LENGTH=6, + COLLABORATION_LOCAL_DOC_RETENTION_DAYS=15, COLLABORATION_VERSION_GRANULARITY_MS=45000, COLLABORATION_WS_URL="http://testcollab/", COLLABORATION_WS_INACTIVITY_TIMEOUT=300, @@ -55,6 +56,7 @@ def test_api_config(is_authenticated): "AI_FEATURE_BLOCKNOTE_ENABLED": False, "AI_FEATURE_LEGACY_ENABLED": False, "API_USERS_SEARCH_QUERY_MIN_LENGTH": 6, + "COLLABORATION_LOCAL_DOC_RETENTION_DAYS": 15, "COLLABORATION_VERSION_GRANULARITY_MS": 45000, "COLLABORATION_WS_URL": "http://testcollab/", "COLLABORATION_WS_INACTIVITY_TIMEOUT": 300, diff --git a/src/backend/impress/settings.py b/src/backend/impress/settings.py index cf98a1f68..0027bf64a 100755 --- a/src/backend/impress/settings.py +++ b/src/backend/impress/settings.py @@ -546,6 +546,14 @@ class Base(Configuration): environ_name="COLLABORATION_VERSION_GRANULARITY_MS", environ_prefix=None, ) + # How long a browser keeps its local (offline) copy of a document after the + # last time it was opened, in days. The frontend drops copies older than this + # on startup. + COLLABORATION_LOCAL_DOC_RETENTION_DAYS = values.IntegerValue( + 30, + environ_name="COLLABORATION_LOCAL_DOC_RETENTION_DAYS", + environ_prefix=None, + ) # Base url of the collaboration server's REST api, including its route # prefix (e.g. "http://yhub:3002/collaboration"). Server-to-server only: # used with an admin JWT to migrate legacy documents and, later, to kick diff --git a/src/frontend/apps/e2e/__tests__/app-impress/utils-common.ts b/src/frontend/apps/e2e/__tests__/app-impress/utils-common.ts index ea81b63bf..44dc5a8bd 100644 --- a/src/frontend/apps/e2e/__tests__/app-impress/utils-common.ts +++ b/src/frontend/apps/e2e/__tests__/app-impress/utils-common.ts @@ -18,6 +18,7 @@ export const CONFIG = { AI_FEATURE_BLOCKNOTE_ENABLED: false, AI_FEATURE_LEGACY_ENABLED: true, API_USERS_SEARCH_QUERY_MIN_LENGTH: 3, + COLLABORATION_LOCAL_DOC_RETENTION_DAYS: 30, COLLABORATION_VERSION_GRANULARITY_MS: 60000, COLLABORATION_WS_INACTIVITY_TIMEOUT: 15, COLLABORATION_WS_URL: process.env.COLLABORATION_WS_URL, diff --git a/src/frontend/apps/impress/package.json b/src/frontend/apps/impress/package.json index 3a682831c..b2002316f 100644 --- a/src/frontend/apps/impress/package.json +++ b/src/frontend/apps/impress/package.json @@ -39,7 +39,7 @@ "@dnd-kit/modifiers": "9.0.0", "@emoji-mart/data": "1.2.1", "@emoji-mart/react": "1.1.1", - "@floating-ui/react": "^0.27.19", + "@floating-ui/react": "0.27.19", "@fontsource-variable/inter": "5.3.0", "@fontsource-variable/material-symbols-outlined": "5.3.3", "@fontsource/material-icons": "5.3.0", @@ -82,6 +82,7 @@ "styled-components": "6.5.3", "use-debounce": "10.1.1", "uuid": "14.0.2", + "y-indexeddb": "9.0.12", "y-prosemirror": "1.3.7", "y-protocols": "1.0.7", "y-websocket": "3.1.0", diff --git a/src/frontend/apps/impress/src/core/config/ConfigProvider.tsx b/src/frontend/apps/impress/src/core/config/ConfigProvider.tsx index 85391b37c..63227438a 100644 --- a/src/frontend/apps/impress/src/core/config/ConfigProvider.tsx +++ b/src/frontend/apps/impress/src/core/config/ConfigProvider.tsx @@ -6,6 +6,7 @@ import { useTranslation } from 'react-i18next'; import { Box } from '@/components'; import { useCunninghamTheme } from '@/cunningham'; +import { sweepLocalDocs } from '@/docs/doc-management/localDocs'; import { useAuthQuery } from '@/features/auth'; import { useCustomTranslations, @@ -65,6 +66,16 @@ export const ConfigProvider = ({ children }: PropsWithChildren) => { setTheme(conf.FRONTEND_THEME); }, [conf?.FRONTEND_THEME, setTheme]); + /** + * Offline local document sweep based on retention days. + */ + useEffect(() => { + if (!conf?.COLLABORATION_LOCAL_DOC_RETENTION_DAYS) { + return; + } + void sweepLocalDocs(conf.COLLABORATION_LOCAL_DOC_RETENTION_DAYS); + }, [conf?.COLLABORATION_LOCAL_DOC_RETENTION_DAYS]); + useEffect(() => { if (!conf?.POSTHOG_KEY || !conf?.POSTHOG_HOST) { return; diff --git a/src/frontend/apps/impress/src/core/config/api/useConfig.tsx b/src/frontend/apps/impress/src/core/config/api/useConfig.tsx index b309c0764..46b3dbfbf 100644 --- a/src/frontend/apps/impress/src/core/config/api/useConfig.tsx +++ b/src/frontend/apps/impress/src/core/config/api/useConfig.tsx @@ -48,6 +48,7 @@ export interface ConfigResponse { AI_FEATURE_BLOCKNOTE_ENABLED?: boolean; AI_FEATURE_LEGACY_ENABLED?: boolean; API_USERS_SEARCH_QUERY_MIN_LENGTH?: number; + COLLABORATION_LOCAL_DOC_RETENTION_DAYS?: number; COLLABORATION_VERSION_GRANULARITY_MS?: number; COLLABORATION_WS_URL?: string; COLLABORATION_WS_INACTIVITY_TIMEOUT?: number | null; diff --git a/src/frontend/apps/impress/src/core/config/hooks/useCollaborationUrl.tsx b/src/frontend/apps/impress/src/core/config/hooks/useCollaborationUrl.tsx index 9a9ef6284..c5d66e40b 100644 --- a/src/frontend/apps/impress/src/core/config/hooks/useCollaborationUrl.tsx +++ b/src/frontend/apps/impress/src/core/config/hooks/useCollaborationUrl.tsx @@ -6,7 +6,7 @@ import { useConfig } from '../api'; * Where the collaboration server's rooms live, independent of which document is * being opened. Kept apart so the two hooks below cannot answer differently. */ -const useCollaborationBaseUrl = () => { +export const useCollaborationUrl = () => { const { data: conf } = useConfig(); return ( @@ -17,17 +17,6 @@ const useCollaborationBaseUrl = () => { ); }; -export const useCollaborationUrl = (room?: string) => { - const baseUrl = useCollaborationBaseUrl(); - - if (!room) { - return; - } - - // The room is appended to the base URL by the provider (y-websocket) - return baseUrl; -}; - /** * y/hub serves the same rooms over two transports, mounted side by side under one prefix: * `{prefix}/ws/v1/{org}/{docid}` for the websocket and `{prefix}/ydoc/v1/{org}/{docid}` over @@ -65,7 +54,7 @@ export const collaborationHttpTarget = (wsUrl: string) => { * serves. */ export const useCollaborationTarget = (): CollaborationTarget | undefined => { - const baseUrl = useCollaborationBaseUrl(); + const baseUrl = useCollaborationUrl(); return baseUrl ? collaborationHttpTarget(baseUrl) : undefined; }; diff --git a/src/frontend/apps/impress/src/features/docs/doc-editor/hook/useCollaboration.tsx b/src/frontend/apps/impress/src/features/docs/doc-editor/hook/useCollaboration.tsx index ddad61886..9e5a02e99 100644 --- a/src/frontend/apps/impress/src/features/docs/doc-editor/hook/useCollaboration.tsx +++ b/src/frontend/apps/impress/src/features/docs/doc-editor/hook/useCollaboration.tsx @@ -16,7 +16,7 @@ import { useBroadcastStore } from '@/stores/useBroadcastStore'; * been permitted to send — but never looser. */ export const useCollaboration = (room: string, readOnly = false) => { - const collaborationUrl = useCollaborationUrl(room); + const collaborationUrl = useCollaborationUrl(); const { addTask } = useBroadcastStore(); const queryClient = useQueryClient(); const { data: config } = useConfig(); diff --git a/src/frontend/apps/impress/src/features/docs/doc-header/components/AlertOffline.tsx b/src/frontend/apps/impress/src/features/docs/doc-header/components/AlertOffline.tsx new file mode 100644 index 000000000..f0d912444 --- /dev/null +++ b/src/frontend/apps/impress/src/features/docs/doc-header/components/AlertOffline.tsx @@ -0,0 +1,36 @@ +import { useTranslation } from 'react-i18next'; + +import { Box, Card, Icon, Text } from '@/components'; +import { useCunninghamTheme } from '@/cunningham'; + +export const AlertOffline = () => { + const { t } = useTranslation(); + const { spacingsTokens } = useCunninghamTheme(); + + return ( + + + + + {t( + "You're offline. You can keep editing, and your changes will sync automatically once you're back online.", + )} + + + + ); +}; diff --git a/src/frontend/apps/impress/src/features/docs/doc-header/components/DocHeader.tsx b/src/frontend/apps/impress/src/features/docs/doc-header/components/DocHeader.tsx index a89446185..5f9335de8 100644 --- a/src/frontend/apps/impress/src/features/docs/doc-header/components/DocHeader.tsx +++ b/src/frontend/apps/impress/src/features/docs/doc-header/components/DocHeader.tsx @@ -12,7 +12,9 @@ import { useDocTitleUpdate, useDocUtils, } from '@/docs/doc-management'; +import { useIsOffline } from '@/features/service-worker/hooks/useOffline'; +import { AlertOffline } from './AlertOffline'; import { AlertRestore } from './AlertRestore'; import { DocHeaderInfo } from './DocHeaderInfo'; import { DocTitle } from './DocTitle'; @@ -24,9 +26,11 @@ interface DocHeaderProps { export const DocHeader = ({ doc }: DocHeaderProps) => { const { t } = useTranslation(); const isDeletedDoc = !!doc.deleted_at; - // Emoji Management + const isOffline = useIsOffline((state) => state.isOffline); + const { emoji } = getEmojiAndTitle(doc.title ?? ''); const { updateDocEmoji } = useDocTitleUpdate(); + const { isTopRoot } = useDocUtils(doc); const displayEmojiButton = doc.abilities.partial_update && !isTopRoot; const latestTitleRef = useRef(doc.title ?? ''); @@ -64,6 +68,7 @@ export const DocHeader = ({ doc }: DocHeaderProps) => { }} > {isDeletedDoc && } + {isOffline && } diff --git a/src/frontend/apps/impress/src/features/docs/doc-management/__tests__/localDocs.test.tsx b/src/frontend/apps/impress/src/features/docs/doc-management/__tests__/localDocs.test.tsx new file mode 100644 index 000000000..a466d3ae5 --- /dev/null +++ b/src/frontend/apps/impress/src/features/docs/doc-management/__tests__/localDocs.test.tsx @@ -0,0 +1,170 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { rememberLocalDoc, sweepLocalDocs } from '../localDocs'; + +// stands in for the instance's COLLABORATION_LOCAL_DOC_RETENTION_DAYS +const RETENTION_DAYS = 30; +const sweep = (days: number = RETENTION_DAYS) => sweepLocalDocs(days); + +/** + * A Map-backed stand-in for the index database. jsdom has no `indexedDB`, and + * the module only needs `get` / `put` / `delete` / `getAllKeys` / `close` off + * one store. + */ +const store = new Map(); + +vi.mock('idb', () => ({ + openDB: vi.fn(() => + Promise.resolve({ + get: (_s: string, key: string) => Promise.resolve(store.get(key)), + getAll: (_s: string) => Promise.resolve([...store.values()]), + getAllKeys: (_s: string) => Promise.resolve([...store.keys()]), + put: (_s: string, value: unknown, key: string) => { + store.set(key, value); + return Promise.resolve(key); + }, + delete: (_s: string, key: string) => { + store.delete(key); + return Promise.resolve(); + }, + close: () => undefined, + }), + ), +})); + +const mockedClearDocument = vi.fn().mockResolvedValue(undefined); + +vi.mock('y-indexeddb', () => ({ + clearDocument: (name: string) => mockedClearDocument(name), +})); + +const uuid = (n: number) => + `0000000${n}-0000-4000-8000-000000000000`.slice(-36); + +const daysAgo = (days: number) => Date.now() - days * 24 * 60 * 60 * 1000; + +const seedIndex = (entries: Record) => { + for (const [id, at] of Object.entries(entries)) { + store.set(id, at); + } +}; + +/** Make `indexedDB.databases()` report these names present on the origin. */ +const stubDatabases = (names: string[]) => + vi.stubGlobal('indexedDB', { + databases: () => Promise.resolve(names.map((name) => ({ name }))), + }); + +describe('localDocs', () => { + beforeEach(() => { + store.clear(); + // most tests are about the index alone; the enumeration path is opt-in + vi.stubGlobal('indexedDB', undefined); + }); + + afterEach(() => { + vi.clearAllMocks(); + vi.unstubAllGlobals(); + }); + + it('remembers a document that was opened', async () => { + await rememberLocalDoc(uuid(1)); + + expect(store.get(uuid(1))).toBeGreaterThan(daysAgo(1)); + }); + + it('moves a document back out of reach of the sweep when reopened', async () => { + seedIndex({ [uuid(1)]: daysAgo(90) }); + + await rememberLocalDoc(uuid(1)); + + expect(await sweep()).toEqual([]); + expect(mockedClearDocument).not.toHaveBeenCalled(); + }); + + it('drops the copies of documents nobody has opened for a month', async () => { + seedIndex({ + [uuid(1)]: daysAgo(1), + [uuid(2)]: daysAgo(29), + [uuid(3)]: daysAgo(31), + [uuid(4)]: daysAgo(400), + }); + + expect(await sweep()).toEqual([uuid(3), uuid(4)]); + expect(mockedClearDocument).toHaveBeenCalledWith(uuid(3)); + expect(mockedClearDocument).toHaveBeenCalledWith(uuid(4)); + expect([...store.keys()]).toEqual([uuid(1), uuid(2)]); + }); + + it('keeps a document just inside the limit until it passes it', async () => { + seedIndex({ + [uuid(1)]: daysAgo(RETENTION_DAYS) + 60_000, + }); + + expect(await sweep()).toEqual([]); + }); + + it('uses the retention window it is given', async () => { + seedIndex({ [uuid(1)]: daysAgo(3), [uuid(2)]: daysAgo(10) }); + + expect(await sweep(7)).toEqual([uuid(2)]); + expect([...store.keys()]).toEqual([uuid(1)]); + }); + + it('leaves a copy it could not drop in the index, to retry next time', async () => { + mockedClearDocument.mockRejectedValueOnce(new Error('quota')); + seedIndex({ [uuid(1)]: daysAgo(90) }); + vi.spyOn(console, 'error').mockImplementation(() => undefined); + + await sweep(); + + expect([...store.keys()]).toEqual([uuid(1)]); + }); + + it('does nothing when this browser holds no local copy', async () => { + expect(await sweep()).toEqual([]); + expect(mockedClearDocument).not.toHaveBeenCalled(); + }); + + describe('enumeration, where the browser supports it', () => { + it('drops a copy on disk that the index never knew about', async () => { + stubDatabases([uuid(1), 'api-docs-db', 'docs-local-index']); + + // the index is empty; only the enumeration sees uuid(1) + expect(await sweep()).toEqual([uuid(1)]); + expect(mockedClearDocument).toHaveBeenCalledWith(uuid(1)); + // the app's own databases are left alone + expect(mockedClearDocument).toHaveBeenCalledTimes(1); + }); + + it('keeps an enumerated copy that the index still vouches for', async () => { + seedIndex({ [uuid(1)]: daysAgo(1) }); + stubDatabases([uuid(1)]); + + expect(await sweep()).toEqual([]); + }); + + it('sweeps the index and the orphans in one pass', async () => { + seedIndex({ [uuid(1)]: daysAgo(1), [uuid(2)]: daysAgo(90) }); + stubDatabases([uuid(1), uuid(2), uuid(3)]); + + expect((await sweep()).sort()).toEqual([uuid(2), uuid(3)]); + }); + }); + + it('does not hang on a delete the browser blocks', async () => { + vi.useFakeTimers(); + seedIndex({ [uuid(1)]: daysAgo(90) }); + // a blocked deleteDatabase never resolves + mockedClearDocument.mockReturnValueOnce(new Promise(() => undefined)); + vi.spyOn(console, 'error').mockImplementation(() => undefined); + + const swept = sweep(); + await vi.advanceTimersByTimeAsync(5000); + + expect(await swept).toEqual([]); + // still in the index, for the next startup to retry + expect([...store.keys()]).toEqual([uuid(1)]); + vi.useRealTimers(); + }); +}); diff --git a/src/frontend/apps/impress/src/features/docs/doc-management/index.ts b/src/frontend/apps/impress/src/features/docs/doc-management/index.ts index a30fbe251..29e0f91e9 100644 --- a/src/frontend/apps/impress/src/features/docs/doc-management/index.ts +++ b/src/frontend/apps/impress/src/features/docs/doc-management/index.ts @@ -1,6 +1,7 @@ export * from './api'; export * from './components'; export * from './hooks'; +export * from './localDocs'; export * from './stores'; export * from './types'; export * from './utils'; diff --git a/src/frontend/apps/impress/src/features/docs/doc-management/localDocs.ts b/src/frontend/apps/impress/src/features/docs/doc-management/localDocs.ts new file mode 100644 index 000000000..eee528a5e --- /dev/null +++ b/src/frontend/apps/impress/src/features/docs/doc-management/localDocs.ts @@ -0,0 +1,134 @@ +/** + * Drop the local copy of every document that has not been opened for + * `retentionDays` (the instance's `COLLABORATION_LOCAL_DOC_RETENTION_DAYS`), + * plus any copy on disk that the index has lost track of where the browser lets + * us enumerate. + * + * `IndexeddbPersistence` has no expiry of its own, so without this every + * document ever opened is kept until the browser evicts it under storage + * pressure — which it does without asking and without order. + * + * Run once on startup rather than on a timer: it must not run while this tab has + * a document open — it would delete the database that document's provider is + * writing to — and a document opened today is not a candidate anyway. + * + * Returns the ids it dropped, which is what the tests read. + */ + +import { openDB } from 'idb'; +import { validate as uuidValidate } from 'uuid'; +import { clearDocument } from 'y-indexeddb'; + +const DAY_MS = 24 * 60 * 60 * 1000; + +const INDEX_DB = 'docs-local-index'; +const INDEX_STORE = 'opened'; + +const openIndex = () => + openDB(INDEX_DB, 1, { + upgrade: (db) => { + db.createObjectStore(INDEX_STORE); + }, + }); + +/** + * `indexedDB.deleteDatabase` blocks silently while a connection is open — a copy + * of this document held by another tab — and never resolves. The sweep must not + * hang on one, so a delete that has not returned in a few seconds is abandoned + * and left for the next startup, by when that tab has likely gone. + */ +const DELETE_TIMEOUT_MS = 4000; + +const drop = (docId: string) => + Promise.race([ + clearDocument(docId), + new Promise((_, reject) => + setTimeout( + () => reject(new Error(`deleting ${docId} timed out`)), + DELETE_TIMEOUT_MS, + ), + ), + ]); + +/** + * Record that a document has just been opened, which is what keeps its local + * copy alive. + * + * Best effort: if this write is lost, the sweep's enumeration still finds the + * copy, and the worst case is that a document opened once and never again is + * kept an extra cycle rather than dropped on time. + */ +export const rememberLocalDoc = async (docId: string) => { + try { + const db = await openIndex(); + await db.put(INDEX_STORE, Date.now(), docId); + db.close(); + } catch (error) { + console.error( + 'Failed to record the local copy of a document', + docId, + error, + ); + } +}; + +export const sweepLocalDocs = async (retentionDays: number) => { + let db; + try { + db = await openIndex(); + } catch (error) { + console.error('Failed to open the local document index', error); + return []; + } + + const expiry = Date.now() - retentionDays * DAY_MS; + + // id -> last opened, from the index. `getAllKeys` and `getAll` return in the + // same order, so they zip. + const opened = new Map(); + const keys = await db.getAllKeys(INDEX_STORE); + const times = await db.getAll(INDEX_STORE); + keys.forEach((id, i) => { + if (typeof id === 'string' && typeof times[i] === 'number') { + opened.set(id, times[i]); + } + }); + + const present = new Set(); + if (typeof indexedDB !== 'undefined' && 'databases' in indexedDB) { + try { + for (const { name } of await indexedDB.databases()) { + if (name && uuidValidate(name)) { + present.add(name); + } + } + } catch (error) { + console.error('Failed to enumerate the local documents', error); + } + } + + const dropped: string[] = []; + + for (const id of new Set([...opened.keys(), ...present])) { + const at = opened.get(id); + + // known to the index and still fresh + if (at !== undefined && at > expiry) { + continue; + } + + try { + await drop(id); + if (opened.has(id)) { + await db.delete(INDEX_STORE, id); + } + dropped.push(id); + } catch (error) { + // left as it is, so the next startup tries again + console.error('Failed to drop the local copy of a document', id, error); + } + } + + db.close(); + return dropped; +}; diff --git a/src/frontend/apps/impress/src/features/docs/doc-management/stores/__tests__/useProviderStore.test.tsx b/src/frontend/apps/impress/src/features/docs/doc-management/stores/__tests__/useProviderStore.test.tsx index 61aa273d8..77d38c8e1 100644 --- a/src/frontend/apps/impress/src/features/docs/doc-management/stores/__tests__/useProviderStore.test.tsx +++ b/src/frontend/apps/impress/src/features/docs/doc-management/stores/__tests__/useProviderStore.test.tsx @@ -74,8 +74,28 @@ class FakeHttpProvider { } } +/** + * A stand-in for `IndexeddbPersistence`. What the store asks of it is that it exists, that its + * `synced` reaches `isReady` - local content is enough to render an editor - and that it is + * detached with the document. + */ +class FakePersistence { + public destroy = vi.fn().mockResolvedValue(undefined); + + private listeners: Record void)[]> = {}; + + on(event: string, listener: (...args: unknown[]) => void) { + (this.listeners[event] ??= []).push(listener); + } + + emit(event: string, ...args: unknown[]) { + this.listeners[event]?.forEach((listener) => listener(...args)); + } +} + let provider: FakeProvider; let httpProvider: FakeHttpProvider; +let persistence: FakePersistence; let stopFallback: ReturnType; vi.mock('y-websocket', () => ({ @@ -85,6 +105,27 @@ vi.mock('y-websocket', () => ({ }), })); +const { IndexeddbPersistenceMock } = vi.hoisted(() => ({ + IndexeddbPersistenceMock: vi.fn(function (..._args: unknown[]) { + return undefined as never; + }), +})); + +vi.mock('y-indexeddb', () => ({ + IndexeddbPersistence: IndexeddbPersistenceMock, + clearDocument: vi.fn().mockResolvedValue(undefined), +})); + +// its own IndexedDB plumbing is exercised in localDocs.test — here we only +// check that opening a document records it +const { rememberLocalDocMock } = vi.hoisted(() => ({ + rememberLocalDocMock: vi.fn().mockResolvedValue(undefined), +})); + +vi.mock('../../localDocs', () => ({ + rememberLocalDoc: rememberLocalDocMock, +})); + /** * Stands in for `createWebsocketFallback`, emulating the one reaction the store has to order * itself against: on `closed` the real helper starts the http provider, so the store's own @@ -123,7 +164,15 @@ describe('useProviderStore', () => { vi.useFakeTimers(); provider = new FakeProvider(); httpProvider = new FakeHttpProvider(); + persistence = new FakePersistence(); stopFallback = vi.fn(); + // jsdom has none, and the store treats its absence as "no local copy" + vi.stubGlobal('indexedDB', {}); + IndexeddbPersistenceMock.mockClear(); + IndexeddbPersistenceMock.mockImplementation(function () { + return persistence as never; + }); + rememberLocalDocMock.mockClear(); createWebsocketFallback.mockClear(); HttpProviderMock.mockClear(); // the store is a module-level singleton: put it back to its defaults, or @@ -137,9 +186,12 @@ describe('useProviderStore', () => { afterEach(() => { vi.useRealTimers(); + vi.unstubAllGlobals(); }); it('keeps reconnecting when the connection is merely lost', () => { + // the socket had opened before it dropped + provider.emit('status', { status: 'connected' }); closeWith(1006); vi.runAllTimers(); @@ -153,6 +205,33 @@ describe('useProviderStore', () => { expect(stopFallback).not.toHaveBeenCalled(); }); + it('does not refetch the document while a socket that never opened retries', () => { + // a network that blocks websocket upgrades: the socket never connects, and + // `connection-close` fires on every failed attempt + closeWith(1006); + closeWith(1006); + vi.runAllTimers(); + + expect(provider.shouldConnect).toBe(true); + expect(useProviderStore.getState().isPermanentlyClosed).toBe(false); + // no refetch storm on the retry cadence — the http fallback carries the doc + expect(useProviderStore.getState().hasLostConnection).toBe(false); + expect(stopFallback).not.toHaveBeenCalled(); + }); + + it('does not re-render subscribers of the store on a repeat retry that changes nothing', () => { + // components that read the store without a selector (most of them, here) get a new + // object on every `set()` — even a same-value one — so a redundant `set()` on this + // retry loop would flicker every one of them, forever + closeWith(1006); + const listener = vi.fn(); + useProviderStore.subscribe(listener); + + closeWith(1006); + + expect(listener).not.toHaveBeenCalled(); + }); + it.each([ ['a deleted document', 4404], ['a revoked access', 4401], @@ -283,6 +362,83 @@ describe('useProviderStore', () => { expect(provider.destroy).toHaveBeenCalled(); expect(provider.awareness.destroy).toHaveBeenCalled(); expect(provider.doc.destroy).toHaveBeenCalled(); + // detached before the document is, so the last updates are written + expect(persistence.destroy).toHaveBeenCalled(); expect(useProviderStore.getState().httpProvider).toBeUndefined(); + expect(useProviderStore.getState().persistence).toBeUndefined(); + }); + + it('keeps a local copy of the document, under its own id', () => { + expect(IndexeddbPersistenceMock).toHaveBeenCalledTimes(1); + expect(IndexeddbPersistenceMock.mock.calls[0][0]).toBe('doc-id'); + expect(useProviderStore.getState().persistence).toBe(persistence); + }); + + it('renders as soon as the local copy is loaded, without waiting for a connection', () => { + expect(useProviderStore.getState().isReady).toBe(false); + + persistence.emit('synced'); + + expect(useProviderStore.getState().isReady).toBe(true); + // nothing was connected: this is the offline path + expect(useProviderStore.getState().isConnected).toBe(false); + }); + + it('remembers the document, so the sweep leaves its copy alone', () => { + expect(rememberLocalDocMock).toHaveBeenCalledWith('doc-id'); + }); + + it("drops a reader's http writes instead of letting the server refuse them", async () => { + useProviderStore.getState().destroyProvider(); + HttpProviderMock.mockClear(); + const realFetch = vi.fn().mockResolvedValue(new Response(null)); + vi.stubGlobal('fetch', realFetch); + + useProviderStore + .getState() + .createProvider( + 'ws://localhost/collaboration/ws/v1/docs', + 'doc-id', + undefined, + { + readOnly: true, + }, + ); + + const { fetch: providerFetch } = HttpProviderMock.mock.calls[0][3] as { + fetch: (input: string, init?: RequestInit) => Promise; + }; + + /** + * A reader's PATCH would take a 403, and a 4xx stops the provider for good - + * before its first GET, since the PATCH comes first in a round. The socket + * drops a reader's updates and stays open; this makes http agree. + */ + const patched = await providerFetch('http://collab/ydoc/v1/docs/doc-id', { + method: 'PATCH', + }); + + expect(patched.status).toBe(204); + expect(realFetch).not.toHaveBeenCalled(); + + // reading is what a reader is allowed to do, and still goes to the network + await providerFetch('http://collab/ydoc/v1/docs/doc-id'); + + expect(realFetch).toHaveBeenCalledTimes(1); + }); + + it('still builds an editor in a browser that has no indexeddb', () => { + useProviderStore.getState().destroyProvider(); + vi.stubGlobal('indexedDB', undefined); + IndexeddbPersistenceMock.mockClear(); + + useProviderStore + .getState() + .createProvider('ws://localhost/collaboration/ws/v1/docs', 'doc-id'); + + expect(IndexeddbPersistenceMock).not.toHaveBeenCalled(); + expect(useProviderStore.getState().persistence).toBeUndefined(); + // the connection still drives the editor, exactly as before + expect(useProviderStore.getState().provider).toBe(provider); }); }); diff --git a/src/frontend/apps/impress/src/features/docs/doc-management/stores/useProviderStore.tsx b/src/frontend/apps/impress/src/features/docs/doc-management/stores/useProviderStore.tsx index bd63c7a8d..fc4e356a7 100644 --- a/src/frontend/apps/impress/src/features/docs/doc-management/stores/useProviderStore.tsx +++ b/src/frontend/apps/impress/src/features/docs/doc-management/stores/useProviderStore.tsx @@ -1,4 +1,5 @@ import { HttpProvider, createWebsocketFallback } from '@y/yhub-http-fallback'; +import { IndexeddbPersistence } from 'y-indexeddb'; import { WebsocketProvider } from 'y-websocket'; import * as Y from 'yjs'; import { create } from 'zustand'; @@ -6,6 +7,8 @@ import { create } from 'zustand'; import { collaborationHttpTarget } from '@/core/config/hooks/useCollaborationUrl'; import { Base64 } from '@/docs/doc-management'; +import { rememberLocalDoc } from '../localDocs'; + /** * `readOnly` decides whether this client may publish presence. It has to be known * when the providers are built, not merely when the editor renders: the http @@ -29,6 +32,7 @@ export interface UseCollaborationStore { resumeFromInactivity: () => void; provider: WebsocketProvider | undefined; httpProvider: HttpProvider | undefined; + persistence: IndexeddbPersistence | undefined; isConnected: boolean; isReady: boolean; isSynced: boolean; @@ -42,6 +46,7 @@ export interface UseCollaborationStore { const defaultValues = { provider: undefined, httpProvider: undefined, + persistence: undefined, isConnected: false, isReady: false, isSynced: false, @@ -89,6 +94,35 @@ const suspendFallback = (httpProvider: HttpProvider | undefined) => { httpProvider?.disconnect(); }; +/** + * What a reader's `PATCH /ydoc` becomes: dropped here, and reported as accepted. + * A reader may not write. + */ +const readerWriteDropped = () => + Promise.resolve(new Response(null, { status: 204 })); + +/** + * Keep a local copy of the document, when the browser lets us. + * + * `indexedDB` is absent more often than it looks - a browser told to block site + * data, some private windows, and every non-browser context this module is + * imported into. Local persistence is a convenience, so a browser without it + * gets an editor that works exactly as it did before rather than no editor: + * `undefined` here, and every caller treats that as "no local copy". + */ +const createPersistence = (storeId: string, doc: Y.Doc) => { + if (typeof indexedDB === 'undefined') { + return undefined; + } + + try { + return new IndexeddbPersistence(storeId, doc); + } catch (error) { + console.error('Failed to open the local copy of the document', error); + return undefined; + } +}; + export const useProviderStore = create((set, get) => ({ ...defaultValues, createProvider: (wsUrl, storeId, initialDoc, { readOnly = false } = {}) => { @@ -100,6 +134,23 @@ export const useProviderStore = create((set, get) => ({ Y.applyUpdate(doc, Buffer.from(initialDoc, 'base64')); } + /** + * Used for the offline mode, it keeps a local copy of the document in IndexedDB so that + * the editor can display the last known state even when the network is unavailable. + */ + const persistence = createPersistence(storeId, doc); + + if (persistence) { + // Record the local copy immediately to prevent it from being swept as an orphan. + void rememberLocalDoc(storeId); + + /** + * The editor waits on `isReady` (see `DocEditor`), and local content is enough to render: + * whatever the connection then brings merges into what is already on screen. + */ + persistence.on('synced', () => set({ isReady: true })); + } + const provider = new WebsocketProvider(wsUrl, storeId, doc, { // BroadcastChannel would bypass server auth disableBc: true, @@ -149,7 +200,9 @@ export const useProviderStore = create((set, get) => ({ gc: true, // the session cookie is the credential here too, exactly as on the ws upgrade fetch: (input, init) => - fetch(input, { ...init, credentials: 'include' }), + readOnly && init?.method === 'PATCH' + ? readerWriteDropped() + : fetch(input, { ...init, credentials: 'include' }), }, ) : undefined; @@ -185,9 +238,21 @@ export const useProviderStore = create((set, get) => ({ return; } - // The editor renders from the last snapshot, and the http fallback takes over, while - // y-websocket retries - set({ isConnected: false, isReady: true }); + const { isConnected: wasConnected, isReady: wasReady } = get(); + + // This also fires on every failed reconnect attempt - forever, on a network that never + // lets the socket open. Skip the `set()` once these are already at this value, or a + // same-value write still hands every no-selector subscriber a new object to re-render on. + if (wasConnected || !wasReady) { + set({ isConnected: false, isReady: true }); + } + + // Only a connection that had actually opened can have been *lost* in a way that means + // our access changed. A socket that never opens retries forever; refetching the document + // on each attempt would only thrash the query while the http fallback carries it fine. + if (!wasConnected) { + return; + } clearTimeout(lostConnectionTimeout); // Jitter spreading: Math.random() generates a random delay to avoid @@ -230,12 +295,13 @@ export const useProviderStore = create((set, get) => ({ set({ provider, httpProvider, + persistence, }); return provider; }, destroyProvider: () => { - const { provider, httpProvider } = get(); + const { provider, httpProvider, persistence } = get(); stopFallback?.(); stopFallback = undefined; @@ -243,6 +309,9 @@ export const useProviderStore = create((set, get) => ({ // publishes a farewell awareness state, best effort, so the others see us leave httpProvider?.destroy(); + // Destroy the persistence layer, which keeps the local copy of the document. + void persistence?.destroy(); + if (provider) { /** * destroy() emits 'connection-close' synchronously before removing diff --git a/src/frontend/apps/impress/src/features/service-worker/plugins/ApiPlugin.ts b/src/frontend/apps/impress/src/features/service-worker/plugins/ApiPlugin.ts index 31f5453a9..9061e0d00 100644 --- a/src/frontend/apps/impress/src/features/service-worker/plugins/ApiPlugin.ts +++ b/src/frontend/apps/impress/src/features/service-worker/plugins/ApiPlugin.ts @@ -12,8 +12,6 @@ interface OptionsReadonly { type: 'list' | 'item'; } -// TODO(yhub): Used to work offline, we need to implement the patch mechanism -// It will be probably linked to the HTTP fallback mechanism of yhub interface OptionsMutate { type: 'update' | 'delete' | 'create'; } diff --git a/src/frontend/apps/impress/src/features/service-worker/service-worker-api.ts b/src/frontend/apps/impress/src/features/service-worker/service-worker-api.ts index 80c8be8b6..32d38c0a0 100644 --- a/src/frontend/apps/impress/src/features/service-worker/service-worker-api.ts +++ b/src/frontend/apps/impress/src/features/service-worker/service-worker-api.ts @@ -27,6 +27,37 @@ export const isApiUrl = (href: string) => { const isDocumentApiUrl = (url: URL) => isApiUrl(url.href) && /.*\/documents\/([a-z0-9-]+)\/$/g.test(url.href); +const isCollaborationUrl = (url: URL, endpoint: string) => + new RegExp(`/${endpoint}/v1/[^/]+/[^/]+/?$`).test(url.pathname); + +/** + * The collaboration server's rest api: document content (`ydoc`), the editing + * history (`activity`, `changeset`) and the restore it feeds (`rollback`). + * + * `NetworkOnly`, and not by default: the server is on the app's own origin + * unless an instance moves it, so without a route here these fall into the + * catch-all `StaleWhileRevalidate` and get served from cache - a document + * frozen at first read (offline, indistinguishable from a synced round), or a + * version list missing every version since. + * + * Offline content is handled at the doc, not here: the websocket never reaches + * a service worker, so `IndexeddbPersistence` owns it - see `useProviderStore`. + */ +[ + { endpoint: 'ydoc', methods: ['GET', 'PATCH'] as const }, + { endpoint: 'activity', methods: ['GET'] as const }, + { endpoint: 'changeset', methods: ['GET'] as const }, + { endpoint: 'rollback', methods: ['POST'] as const }, +].forEach(({ endpoint, methods }) => { + methods.forEach((method) => { + registerRoute( + ({ url }) => isCollaborationUrl(url, endpoint), + new NetworkOnly({ plugins: [new OfflinePlugin()] }), + method, + ); + }); +}); + /** * API routes */ diff --git a/src/frontend/apps/impress/src/pages/_app.tsx b/src/frontend/apps/impress/src/pages/_app.tsx index 3632c41ff..a11bbc428 100644 --- a/src/frontend/apps/impress/src/pages/_app.tsx +++ b/src/frontend/apps/impress/src/pages/_app.tsx @@ -17,6 +17,7 @@ type AppPropsWithLayout = AppProps & { export default function App({ Component, pageProps }: AppPropsWithLayout) { useSWRegister(); useOffline(); + const getLayout = Component.getLayout ?? ((page) => page); const { t } = useTranslation(); diff --git a/src/frontend/yarn.lock b/src/frontend/yarn.lock index dc7e84666..47932caf1 100644 --- a/src/frontend/yarn.lock +++ b/src/frontend/yarn.lock @@ -2077,7 +2077,7 @@ dependencies: "@floating-ui/dom" "^1.7.6" -"@floating-ui/react@^0.27.18", "@floating-ui/react@^0.27.19": +"@floating-ui/react@0.27.19", "@floating-ui/react@^0.27.18", "@floating-ui/react@^0.27.19": version "0.27.19" resolved "https://registry.yarnpkg.com/@floating-ui/react/-/react-0.27.19.tgz#d8d5d895b7cb97dac370bfbf55f3e630878fdf1f" integrity sha512-31B8h5mm8YxotlE7/AU/PhNAl8eWxAmjL/v2QOxroDNkTFLk3Uu82u63N3b6TXa4EGJeeZLVcd/9AlNlVqzeog== @@ -11726,7 +11726,7 @@ lib0@1.0.0-rc.22: resolved "https://registry.yarnpkg.com/lib0/-/lib0-1.0.0-rc.22.tgz#c154151f5188009afc7f73e1de3f4888011e6fb9" integrity sha512-KNefJloRQIsWncTF2tIcRqQXSQ7bDRYHwVSUhf1lY2P65Rej4WWFnen6L8L+odJQIo1ZNJGVVjK2WzqB9a+B/g== -lib0@^0.2.102, lib0@^0.2.109, lib0@^0.2.99: +lib0@^0.2.102, lib0@^0.2.109, lib0@^0.2.74, lib0@^0.2.99: version "0.2.117" resolved "https://registry.yarnpkg.com/lib0/-/lib0-0.2.117.tgz#6c3f926475d28904af05b590703cbbbc29475716" integrity sha512-DeXj9X5xDCjgKLU/7RR+/HQEVzuuEUiwldwOGsHK/sfAfELGWEyTcf0x+uOvCvK3O2zPmZePXWL85vtia6GyZw== @@ -16220,6 +16220,13 @@ xtend@~4.0.1: resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== +y-indexeddb@9.0.12: + version "9.0.12" + resolved "https://registry.yarnpkg.com/y-indexeddb/-/y-indexeddb-9.0.12.tgz#73657f31d52886d7532256610babf5cca4ad5e58" + integrity sha512-9oCFRSPPzBK7/w5vOkJBaVCQZKHXB/v6SIT+WYhnJxlEC61juqG0hBrAf+y3gmSMLFLwICNH9nQ53uscuse6Hg== + dependencies: + lib0 "^0.2.74" + y-prosemirror@1.3.7: version "1.3.7" resolved "https://registry.yarnpkg.com/y-prosemirror/-/y-prosemirror-1.3.7.tgz#f88e553da4ea33278b114cf0b6a0ea978b154e84"