feat: F6.1 FILIUS files on F: projects from examples and .fls, copies, export

/svc/examples lists FILIUS' examples, which lie beside F in the image (GPL
like this repository, kept out of the dashboard); /svc/projects makes a
project from one; /svc/projects/<id>/import takes a .fls (up to 64 MiB),
/copy copies a project as it is now, /export gives it back as a .fls with its
warnings. Large files go into the blob store on import and come back from it
on export, which is refused when one is missing.

Found on the way: Node's request body is a Buffer, and Yjs takes no Buffer
into a document, so every .fls with binary files would have failed; and
fflate, now in the bundle, imports createRequire itself, which the build's
banner declared again, so F did not start. The banner names it otherwise and
a test builds and starts the bundle. The proxy's /svc/ takes 70 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGES.md

@@ -286,6 +286,7 @@ Swift-Pfade relativ zu `filius-ipad/ios/FiliusPad/TopologyEditor/` (Stand
 | `src/blobs.ts`, `src/blobRoutes.ts`, `src/simWorker.ts` (Blobs), `src/simRooms.ts` (Blobs), `src/simChannel.ts`, `src/server.ts` (`/blobs`), `src/main.ts`, `test/blobs.test.ts` | Plan §5.1 | neu (F5.5): die Blobs auf F, inhaltsadressiert (`filius_blob`, F rechnet den Namen selbst aus); `GET /blobs/<sha>?project=` mit Berechtigung am Projekt, `POST /blobs?project=` mit Schreibstufe, nicht eingefroren, nur von der Herkunft des Editors, höchstens 16 MiB; CORS für die Herkünfte des Editors; der Raumfaden fragt nach den Blobs seines Dokuments, ein Start wartet auf die gefragten, was F noch nicht hat, wird ein paar Mal neu gefragt und kommt in die laufende Simulation; ein Blob, den die Simulation macht, wird gespeichert, bevor die Änderung, die ihn nennt, angewandt wird. Noch nicht: Blobs vergessen, die kein Projekt mehr nennt |
 | `src/editorFiles.ts`, `src/precompress.ts`, `src/precompressMain.ts`, `src/server.ts` (Editor), `src/main.ts` (`FILIUS_EDITOR_DIR`), `test/editorFiles.test.ts`, `editor/browser/fromF.spec.ts` | Plan §8.2, §8.3 | neu (F5.6b): F liefert die Seite und das Bundle des Editors selbst aus (Ordner `editor/` neben `main.mjs`), damit der Editor von Fs Herkunft kommt und sie als seinen Server nimmt; nur GET und HEAD, nur Dateien im Ordner (jedes `..`, auch kodiert, wird aufgelöst und außerhalb abgewiesen); Fs eigene Wege gehen vor; `assets/` (Vite benennt nach dem Inhalt) ein Jahr im Cache, alles andere jedes Mal neu gefragt; der Build legt `.br` und `.gz` daneben, wo sie kleiner sind, F schickt die bessere, die der Browser nimmt (das Bundle 695 KB → 212 KB mit brotli). Ohne Ordner beantwortet F nur seine Wege und meldet es im Protokoll |
 | `src/ticket.ts` (`TICKET_AHEAD_SECONDS`), `test/ticket.test.ts`, `tools/ticket-oracle` (zwei Fälle) | itsdangerous 2.2.0 `TimestampSigner.unsign` | geändert (F5.6c): eine Karte darf bis zu 5 s vor Fs Uhr signiert sein — itsdangerous lehnt schon eine Sekunde ab. Gemessen auf test-filius: Fs Uhr 886 ms hinter der des Dashboard-Rechners wies fast neun von zehn Karten ab, mit NTP 8 ms noch eine von 125 (eine Karte trägt ganze Sekunden). Das Orakel hält fest, dass Python die Karten 1 s und 5 s voraus ablehnt und 6 s voraus wie F |
+| `src/transfer.ts`, `src/service.ts` (Beispiele, Import, Kopie, Export), `src/server.ts`, `src/main.ts` (`FILIUS_EXAMPLES_DIR`), `test/transfer.test.ts`, `test/bundle.test.ts`, `package.json` (Banner) | Plan §11, F6 | neu (F6.1): FILIUS-Dateien über den Dienstweg — `GET /svc/examples` (die Java-Beispiele liegen neben F im Image, GPL wie dieses Repo, nicht im Dashboard), `POST /svc/projects` mit `example`, `POST /svc/projects/<id>/import` (die `.fls` als Rumpf, höchstens 64 MiB), `POST /svc/projects/<id>/copy` (der Stand jetzt, auch während jemand arbeitet), `GET /svc/projects/<id>/export` (die `.fls`, Warnungen in `X-Filius-Warnings` als base64url-JSON); große Dateien gehen beim Import zuerst in den Blob-Speicher, der Export nimmt ihre Bytes von dort und wird ohne sie nicht gemacht. **Fehler vorher gefunden:** der Rumpf einer Anfrage ist in Node ein `Buffer`, die Dateien einer `.fls` wären Stücke davon, und Yjs nimmt keinen `Buffer` ins Dokument — jede `.fls` mit Binärdateien wäre gescheitert. **Und beim Bau des Image:** `fflate` (über `fls` jetzt im Bundle) importiert `createRequire` selbst, das Banner des Builds deklarierte es ein zweites Mal — F startete nicht; das Banner nennt es jetzt anders, ein Test baut und startet den Bundle |
 | `test/editorConnection.test.ts` | — | neu (F5.3b), erweitert (F5.4: Identität von F, die anderen über die Awareness, auch eine lesende Person, weg mit dem Schließen): `connectSharedDoc` des Editors gegen einen echten F — bereit mit Schreib- und Lesestufe, eingefroren und aufgetaut (neues Dokument, die verirrte Änderung ist weg, Schreiben geht wieder), eingefroren von Beginn an, Absage, Widerruf, Löschen, Loslassen |
 | `test/sim.test.ts`, `test/harness.ts` (Faden) | — | neu (F5.3a): echte Sockets, der echte Faden (esbuild bündelt ihn wie der Build), Beispiel internetwork_komplett: ein Start für alle mit Namen, Ping in einer Befehlszeile bei der anderen, Zurückschreiben ins Dokument und eine Datei aus dem Dokument im Raum, Lesestufe, Absagen, Nachlauf, Einfrieren, Löschen, Widerruf, Wahl des Fadens, scheiternder Faden, Herunterfahren, Ping |
 | `test/ticket.test.ts`, `test/tickets.ts`, `test/tickets.python.json` | `tools/ticket-oracle` | neu (F5.2): jede Karte des Python-Orakels liest Node wie Python (Nutzlast oder derselbe Grund der Ablehnung); in Node signierte Karten tragen dasselbe JSON zum selben Zeitpunkt (Nodes zlib packt anders, ungepackt byte-gleich); die Ansprüche |

README.md

@@ -31,7 +31,7 @@ die Herkunft je Paket. Lizenz dieses Repos: **GPL-3.0** (`LICENSE`).
 | `packages/protocol` | die eine Nachrichtenliste zwischen Editor und Simulationsraum (`/sim`) |
 | `packages/sim` | Simulationsraum (für Server F und den Web Worker) und `LocalSimHost` |
 | `packages/editor` | Oberfläche im Iframe des Dashboards (React, Vite), auf einem Projekt von F (`?project=<uuid>`) oder mit einem Raum in der eigenen Seite (Entwicklung): Fläche, Palette, Werkzeuge des Entwurfs, Geräte-Blatt, Aktionsmodus (Desktops, Anwendungen, Datenaustausch), Dokumentationsmodus (Text, Gliederungsfläche, Bild exportieren, Bericht als PDF); `npm run dev -w @mdg-filius/editor` öffnet die Entwicklungsseite |
-| `packages/server` | Filius-Dienst auf Server F: gemeinsame Dokumente über Hocuspocus (`/sync`), gespeichert in Postgres (Stand und Aktualisierungen, verdichtet), Hausmeister; Anmeldung mit der Eintrittskarte des Dashboards (`/__auth`, Keks, Lese- und Schreibstufe); Dienstweg des Dashboards (`/svc/*`: Projekte anlegen, löschen, einfrieren, Zugänge widerrufen); Simulationsräume auf `/sim` in `worker_threads`, einer je Projekt für alle; `/healthz`; liefert Seite und Bundle des Editors aus (`editor/` neben `main.mjs`, vorab gepackt); `npm run build -w @mdg-filius/server` bündelt ihn nach `dist/main.mjs` und den Faden der Räume nach `dist/simWorker.mjs` |
+| `packages/server` | Filius-Dienst auf Server F: gemeinsame Dokumente über Hocuspocus (`/sync`), gespeichert in Postgres (Stand und Aktualisierungen, verdichtet), Hausmeister; Anmeldung mit der Eintrittskarte des Dashboards (`/__auth`, Keks, Lese- und Schreibstufe); Dienstweg des Dashboards (`/svc/*`: Projekte anlegen, löschen, einfrieren, Zugänge widerrufen); Simulationsräume auf `/sim` in `worker_threads`, einer je Projekt für alle; `/healthz`; liefert Seite und Bundle des Editors aus (`editor/` neben `main.mjs`, vorab gepackt); FILIUS-Dateien über `/svc/*`: Projekte aus den Java-Beispielen (`examples/` neben `main.mjs`) und aus `.fls`, Kopie, Export; `npm run build -w @mdg-filius/server` bündelt ihn nach `dist/main.mjs` und den Faden der Räume nach `dist/simWorker.mjs` |
 | `deploy` | Image (`Dockerfile`), ein Stack je Umgebung (`docker-compose.yml`, `*.env.example`), Eintrag für den externen OpenResty (`proxy.conf`) |
 
 Plan und Entscheidungen: `knowledge/plans/FILIUS_WEB_PLAN.md` im Dashboard.

deploy/Dockerfile

@@ -33,6 +33,8 @@ ENV NODE_ENV=production
 WORKDIR /app
 COPY --from=build /src/packages/server/dist/main.mjs /src/packages/server/dist/simWorker.mjs ./
 COPY --from=build /src/packages/editor/dist ./editor
+# FILIUS' examples, from which the dashboard makes projects (/svc/examples).
+COPY corpus/java-examples ./examples
 COPY LICENSE README.md CHANGES.md ./
 # Not root: F needs no privilege, and a hole in it should not get one.
 USER node

deploy/proxy.conf

@@ -185,14 +185,36 @@ server {
         proxy_send_timeout   120s;
     }
 
+    # ── The dashboard's service route: /svc/ ─────────────────────────
+    #
+    # Guarded by `FILIUS_SERVICE_KEY`; the dashboard reaches F under this
+    # name, so if this proxy is down, creating, importing and deleting
+    # projects is down too. A FILIUS file the dashboard imports comes here
+    # whole, with its pictures (`/svc/projects/<id>/import`, F takes up to
+    # 64 MiB, `IMPORT_BYTES`); nginx' default of 1 MB would refuse a class's
+    # project with a few screenshots in it. F limits everything else under
+    # `/svc/` to 64 KiB itself. The limit is again above F's, so that F
+    # answers a file too large itself (see `/blobs`).
+    location /svc/ {
+        proxy_pass http://$filius_upstream;
+        proxy_http_version 1.1;
+
+        proxy_set_header Host              $host;
+        proxy_set_header X-Forwarded-Host  $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto https;
+
+        client_max_body_size 70m;
+        proxy_connect_timeout 10s;
+        proxy_read_timeout   120s;
+        proxy_send_timeout   120s;
+    }
+
     # ── Everything else ──────────────────────────────────────────────
     #
-    # The editor's page and bundle, `/__auth` (ticket in, cookie out),
+    # The editor's page and bundle, `/__auth` (ticket in, cookie out) and
-    # `/svc/*` (the dashboard's service route, guarded by
+    # `/healthz`; nginx's default of 1 MB for a body stays.
-    # `FILIUS_SERVICE_KEY`; the dashboard reaches F under this name, so if
-    # this proxy is down, creating and deleting projects is down too) and
-    # `/healthz`. F limits the bodies of `/svc/*` to 64 KiB itself, so
-    # nginx's default of 1 MB stays.
     location / {
         proxy_pass http://$filius_upstream;
         proxy_http_version 1.1;
@@ -219,4 +241,5 @@ server {
 # | Editor opens, but "connection lost" right away | `/sync` or `/sim` gets no `Upgrade`, or `FILIUS_ORIGINS` does not name this host (F logs `sync.foreignOrigin`). |
 # | Sign-in in the dashboard's frame fails silently | The stack runs with `FILIUS_INSECURE_COOKIE=1`, or the dashboard of the other environment issued the ticket. |
 # | Large picture: "upload failed" without a reason | `client_max_body_size` in `/blobs` is below 16 MiB. |
+# | Importing a FILIUS file fails with 413 (an HTML page, not F's JSON) | `client_max_body_size` in `/svc/` is below 64 MiB. |
 # | Editor in the dashboard stays a white box | Something sets `X-Frame-Options` for this host. `curl -sI https://filius.hub.mdg-hamburg.de/` |

packages/editor/browser/serverF.ts

@@ -53,7 +53,7 @@ export async function startServerF(editorBase: string, editorFolder?: string): P
     outdir: out,
     outExtension: { '.js': '.mjs' },
     external: ['pg-native'],
-    banner: { js: "import { createRequire } from 'node:module'; const require = createRequire(import.meta.url);" },
+    banner: { js: "import { createRequire as createBundleRequire } from 'node:module'; const require = createBundleRequire(import.meta.url);" },
     logLevel: 'error',
   });
   const port = 20_000 + Math.floor(Math.random() * 20_000);

packages/server/package.json

@@ -8,7 +8,7 @@
     ".": "./src/index.ts"
   },
   "scripts": {
-    "build": "esbuild src/main.ts src/simWorker.ts src/precompressMain.ts --bundle --platform=node --format=esm --target=node20 --outdir=dist --out-extension:.js=.mjs --external:pg-native --banner:js=\"import { createRequire } from 'node:module'; const require = createRequire(import.meta.url);\"",
+    "build": "esbuild src/main.ts src/simWorker.ts src/precompressMain.ts --bundle --platform=node --format=esm --target=node20 --outdir=dist --out-extension:.js=.mjs --external:pg-native --banner:js=\"import { createRequire as createBundleRequire } from 'node:module'; const require = createBundleRequire(import.meta.url);\"",
     "start": "node dist/main.mjs"
   },
   "dependencies": {

packages/server/src/index.ts

@@ -15,3 +15,4 @@ export * from './simRooms.js';
 export * from './store.js';
 export * from './sync.js';
 export * from './ticket.js';
+export * from './transfer.js';

packages/server/src/main.ts

@@ -7,7 +7,8 @@
 // machine without HTTPS). The rooms' thread is the bundle beside this one
 // (dist/simWorker.mjs), the editor's page and bundle the folder editor/
 // beside it (FILIUS_EDITOR_DIR elsewhere); without that folder F answers
-// only its own routes and says so in the log.
+// only its own routes and says so in the log. FILIUS' examples lie in the
+// folder examples/ beside it (FILIUS_EXAMPLES_DIR elsewhere).
 
 import { existsSync } from 'node:fs';
 import { join } from 'node:path';
@@ -37,6 +38,8 @@ await blobs.migrate();
 const editorFolder = process.env.FILIUS_EDITOR_DIR ?? fileURLToPath(new URL('./editor/', import.meta.url));
 const editor = existsSync(join(editorFolder, 'index.html')) ? editorFiles(editorFolder) : undefined;
 if (editor === undefined) log.warn('editor.missing', { folder: editorFolder });
+const examplesFolder = process.env.FILIUS_EXAMPLES_DIR ?? fileURLToPath(new URL('./examples/', import.meta.url));
+if (!existsSync(examplesFolder)) log.warn('examples.missing', { folder: examplesFolder });
 const server = await startServer({
   port: Number(process.env.FILIUS_PORT ?? 8080),
   store,
@@ -49,6 +52,7 @@ const server = await startServer({
   secureCookie,
   workerUrl: new URL('./simWorker.mjs', import.meta.url),
   ...(editor === undefined ? {} : { editor }),
+  examplesFolder,
 });
 for (const signal of ['SIGTERM', 'SIGINT'] as const) {
   process.once(signal, () => {

packages/server/src/server.ts

@@ -54,6 +54,8 @@ export interface ServerOptions extends SyncOptions {
   readonly blobs: BlobStore;
   /** The editor's page and bundle; without them only F's own routes answer (development, tests). */
   readonly editor?: EditorFiles;
+  /** The folder of FILIUS' examples (/svc/examples); none: there are none. */
+  readonly examplesFolder?: string;
   readonly simThreads?: number;
   readonly lingerMilliseconds?: number;
   readonly pingMilliseconds?: number;

packages/server/src/service.ts

@@ -6,24 +6,33 @@
 // a running simulation stops, what it left written back), end a person's
 // grant to a project or all their sessions (whoever is connected, to the
 // document or the room, is let go), and list who is signed in to what (the
-// dashboard checks it against its shares). Snapshots, restoring,
+// dashboard checks it against its shares). FILIUS files (F6.1,
-// copying and the .fls come with the commits (F6).
+// transfer.ts): make a project from one of FILIUS' examples or from a .fls
+// the dashboard sends, copy a project (as it is now, also while people work
+// in it), and give a project as a .fls (with what it could not hold, as
+// warnings in X-Filius-Warnings, base64url JSON). Snapshots and restoring come
+// with the commits (F7).
 
 import { createHash, timingSafeEqual } from 'node:crypto';
 import type { IncomingMessage, ServerResponse } from 'node:http';
 import * as Y from 'yjs';
 import { emptyProject, writeProject } from '@mdg-filius/doc';
 import type { SyncClosing } from '@mdg-filius/protocol';
+import type { BlobStore } from './blobs.js';
 import type { Log } from './log.js';
 import type { SessionStore } from './sessions.js';
 import type { SimRooms } from './simRooms.js';
 import type { DocStore } from './store.js';
 import { accessMessage, type ConnectionContext, type Sync } from './sync.js';
 import { isUuid } from './ticket.js';
+import { archiveOf, exampleArchive, exampleNames, stateOfArchive } from './transfer.js';
 
 export const SERVICE_KEY_HEADER = 'x-filius-service-key';
 /** A request's body at most (the service sends small JSON). */
 export const SERVICE_BODY_BYTES = 64 * 1024;
+/** A FILIUS file to import at most: pictures of up to 16 MiB each (BLOB_MAX_BYTES) may be in it. */
+export const IMPORT_BYTES = 64 * 1024 * 1024;
+export const WARNINGS_HEADER = 'x-filius-warnings';
 /** Hocuspocus' code for a connection ended for want of rights. */
 const FORBIDDEN = 4403;
 
@@ -34,8 +43,14 @@ export interface ServiceOptions {
   readonly sims: SimRooms;
   readonly log: Log;
   readonly serviceKey: string;
+  readonly blobs: BlobStore;
+  /** The folder of FILIUS' examples (beside main.mjs in the image); none: there are none. */
+  readonly examplesFolder?: string;
 }
 
+/** What a route answers: JSON, or a FILIUS file with its warnings. */
+type Answer = readonly [number, object] | { readonly file: Uint8Array; readonly warnings: readonly string[] };
+
 class HttpError extends Error {
   constructor(
     readonly status: number,
@@ -52,17 +67,22 @@ export function serviceKeyMatches(given: string | string[] | undefined, expected
   return expected !== '' && typeof given === 'string' && timingSafeEqual(digest(given), digest(expected));
 }
 
-async function body(request: IncomingMessage): Promise<Record<string, unknown>> {
+async function bytes(request: IncomingMessage, most: number): Promise<Buffer> {
   const parts: Buffer[] = [];
   let size = 0;
   for await (const part of request as AsyncIterable<Buffer>) {
     size += part.length;
-    if (size > SERVICE_BODY_BYTES) throw new HttpError(413, 'tooLarge');
+    if (size > most) throw new HttpError(413, 'tooLarge');
     parts.push(part);
   }
+  return Buffer.concat(parts);
+}
+
+async function body(request: IncomingMessage): Promise<Record<string, unknown>> {
+  const given = await bytes(request, SERVICE_BODY_BYTES);
   let value: unknown;
   try {
-    value = JSON.parse(Buffer.concat(parts).toString('utf8'));
+    value = JSON.parse(given.toString('utf8'));
   } catch {
     throw new HttpError(400, 'notJson');
   }
@@ -88,27 +108,92 @@ export async function handleService(request: IncomingMessage, response: ServerRe
     return;
   }
   try {
-    const [status, value] = await route(request, path, options);
+    const answer = await route(request, path, options);
-    send(status, value);
+    if (!('file' in answer)) send(...answer);
+    else {
+      response.writeHead(200, {
+        'content-type': 'application/octet-stream',
+        'content-length': String(answer.file.length),
+        'cache-control': 'no-store',
+        [WARNINGS_HEADER]: Buffer.from(JSON.stringify(answer.warnings)).toString('base64url'),
+      });
+      response.end(answer.file);
+    }
   } catch (error) {
     if (!(error instanceof HttpError)) throw error;
     send(error.status, { error: error.code });
   }
 }
 
-async function route(request: IncomingMessage, path: string, { store, sessions, sync, sims, log }: ServiceOptions): Promise<[number, object]> {
+/** A FILIUS file as a new project's first state; refused when it is none. */
+async function readArchive(archive: Uint8Array, blobs: BlobStore, log: Log, project: string) {
+  try {
+    return await stateOfArchive(archive, blobs);
+  } catch (error) {
+    log.warn('svc.notAFiliusFile', { project, bytes: archive.length, error: error instanceof Error ? error.message : String(error) });
+    throw new HttpError(422, 'notAFiliusFile');
+  }
+}
+
+async function route(request: IncomingMessage, path: string, { store, sessions, sync, sims, log, blobs, examplesFolder }: ServiceOptions): Promise<Answer> {
   const close = (connections: ReturnType<Sync['connections']>, reason: SyncClosing) => {
     for (const connection of connections) connection.close({ code: FORBIDDEN, reason });
     return connections.length;
   };
-  const project = /^\/svc\/projects\/([^/]+)(\/freeze)?$/.exec(path);
+  /** A project's document as it is now, held while `use` runs (a project no one has open is loaded for it). */
+  const withDocument = async <T>(id: string, use: (doc: Y.Doc) => Promise<T>): Promise<T> => {
+    const direct = await sync.hocuspocus.openDirectConnection(id, { service: true }).catch(() => undefined);
+    if (direct === undefined) throw new HttpError(404, 'unknownProject');
+    try {
+      return await use(direct.document!);
+    } finally {
+      await direct.disconnect();
+    }
+  };
+  const project = /^\/svc\/projects\/([^/]+)(\/freeze|\/import|\/copy|\/export)?$/.exec(path);
+  if (request.method === 'GET' && path === '/svc/examples') return [200, { examples: await exampleNames(examplesFolder) }];
   if (request.method === 'POST' && path === '/svc/projects') {
-    const id = uuid((await body(request)).project, 'notAProject');
+    const given = await body(request);
-    const doc = new Y.Doc();
+    const id = uuid(given.project, 'notAProject');
-    writeProject(doc, emptyProject());
+    let made: { readonly state: Uint8Array; readonly warnings: readonly string[] };
-    if (!(await store.create(id, Y.encodeStateAsUpdate(doc)))) throw new HttpError(409, 'exists');
+    if (given.example === undefined) {
-    log.info('svc.created', { project: id });
+      const doc = new Y.Doc();
-    return [201, { project: id }];
+      writeProject(doc, emptyProject());
+      made = { state: Y.encodeStateAsUpdate(doc), warnings: [] };
+    } else {
+      const archive = await exampleArchive(examplesFolder, given.example);
+      if (archive === undefined) throw new HttpError(404, 'unknownExample');
+      made = await readArchive(archive, blobs, log, id);
+    }
+    if (!(await store.create(id, made.state))) throw new HttpError(409, 'exists');
+    log.info('svc.created', { project: id, ...(given.example === undefined ? {} : { example: given.example }) });
+    return [201, { project: id, warnings: made.warnings }];
+  }
+  if (request.method === 'POST' && project !== null && project[2] === '/import') {
+    const id = uuid(project[1], 'notAProject');
+    const archive = await bytes(request, IMPORT_BYTES);
+    const made = await readArchive(archive, blobs, log, id);
+    if (!(await store.create(id, made.state))) throw new HttpError(409, 'exists');
+    log.info('svc.imported', { project: id, bytes: archive.length, warnings: made.warnings.length });
+    return [201, { project: id, warnings: made.warnings }];
+  }
+  if (request.method === 'POST' && project !== null && project[2] === '/copy') {
+    const id = uuid(project[1], 'notAProject');
+    const to = uuid((await body(request)).to, 'notAProject');
+    const state = await withDocument(id, async (doc) => Y.encodeStateAsUpdate(doc));
+    if (!(await store.create(to, state))) throw new HttpError(409, 'exists');
+    log.info('svc.copied', { project: id, to });
+    return [201, { project: to }];
+  }
+  if (request.method === 'GET' && project !== null && project[2] === '/export') {
+    const id = uuid(project[1], 'notAProject');
+    const made = await withDocument(id, (doc) => archiveOf(doc, blobs));
+    if ('missing' in made) {
+      log.error('svc.exportMissingBlobs', { project: id, missing: made.missing });
+      throw new HttpError(409, 'missingBlobs');
+    }
+    log.info('svc.exported', { project: id, bytes: made.archive.length, warnings: made.warnings.length });
+    return { file: made.archive, warnings: made.warnings };
   }
   if (request.method === 'DELETE' && project !== null && project[2] === undefined) {
     const id = uuid(project[1], 'notAProject');

packages/server/src/transfer.ts

@@ -0,0 +1,61 @@
+// Projects into and out of F as FILIUS files (plan §11, F6.1): a project's
+// first state from a .fls — one of FILIUS' examples, which lie beside F (they
+// are GPL like this repository and stay out of the dashboard), or a file the
+// dashboard sends —, and a project's document as a .fls. A large file of a
+// computer goes into F's blob store before the state that names it (as the
+// editor does with an upload); a .fls takes the blobs' bytes back from the
+// store, and is not made when one of them is missing (it would hold the file
+// empty, without a word).
+
+import { randomUUID } from 'node:crypto';
+import { readdir, readFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import * as Y from 'yjs';
+import { attachBlobs, blobsOf, docArchive, memoryBlobSide, writeProject } from '@mdg-filius/doc';
+import { openProjectArchive } from '@mdg-filius/fls';
+import type { BlobStore } from './blobs.js';
+
+/** A FILIUS file as a project's first state, with what Java's file held that the project cannot (warnings); throws on a file that is none. */
+export async function stateOfArchive(archive: Uint8Array, blobs: BlobStore): Promise<{ readonly state: Uint8Array; readonly warnings: readonly string[] }> {
+  // A plain Uint8Array, not the request's Buffer: the files' bytes are pieces of it, and Yjs takes no
+  // Buffer into a document ("Unexpected content type").
+  const project = openProjectArchive(new Uint8Array(archive), randomUUID);
+  const kept: Uint8Array[] = [];
+  const doc = new Y.Doc();
+  attachBlobs(
+    doc,
+    memoryBlobSide((_sha, bytes) => kept.push(bytes)),
+  );
+  writeProject(doc, project);
+  for (const bytes of kept) await blobs.put(bytes);
+  return { state: Y.encodeStateAsUpdate(doc), warnings: project.warnings };
+}
+
+/** A project's document as a FILIUS file, the blobs' bytes from F's store; the blobs the store does not have, if any. */
+export async function archiveOf(doc: Y.Doc, blobs: BlobStore): Promise<{ readonly archive: Uint8Array; readonly warnings: readonly string[] } | { readonly missing: readonly string[] }> {
+  const wanted = [...blobsOf(doc)];
+  const found = await blobs.get(wanted);
+  const missing = wanted.filter((sha) => !found.has(sha));
+  if (missing.length > 0) return { missing };
+  // A copy with the bytes beside it: the live document has no blob side on F's main thread.
+  const copy = new Y.Doc();
+  const side = memoryBlobSide();
+  side.arrived(found);
+  attachBlobs(copy, side);
+  Y.applyUpdate(copy, Y.encodeStateAsUpdate(doc));
+  return docArchive(copy);
+}
+
+const EXAMPLE_SUFFIX = '.fls';
+
+/** The examples in F's folder, by name (the file's name without .fls; Node lists a folder sorted, libuv sorts it); none without the folder. */
+export async function exampleNames(folder: string | undefined): Promise<string[]> {
+  const files = folder === undefined ? [] : await readdir(folder).catch(() => []);
+  return files.filter((file) => file.endsWith(EXAMPLE_SUFFIX)).map((file) => file.slice(0, -EXAMPLE_SUFFIX.length));
+}
+
+/** An example's file; undefined for anything that is not the name of one of the folder's (never a path). */
+export async function exampleArchive(folder: string | undefined, name: unknown): Promise<Uint8Array | undefined> {
+  if (folder === undefined || !(await exampleNames(folder)).includes(name as string)) return undefined;
+  return new Uint8Array(await readFile(join(folder, `${name as string}${EXAMPLE_SUFFIX}`)));
+}

packages/server/test/access.test.ts

@@ -261,7 +261,7 @@ describe("the dashboard's service", () => {
   it('makes an empty project once', async () => {
     const server = await start();
     const project = randomUUID();
-    expect(await answer(await service(server, 'POST', '/svc/projects', { project }))).toEqual([201, { project }]);
+    expect(await answer(await service(server, 'POST', '/svc/projects', { project }))).toEqual([201, { project, warnings: [] }]);
     expect(await answer(await service(server, 'POST', '/svc/projects', { project }))).toEqual([409, { error: 'exists' }]);
     const doc = new Y.Doc();
     Y.applyUpdate(doc, (await store.load(project))!.state);

packages/server/test/bundle.test.ts

@@ -0,0 +1,23 @@
+// Server F as the image runs it: built by the package's own build script
+// and started with node. Without its environment it stops at once and says
+// what is missing — which it can only say when the bundle loads at all (a
+// bundled library that declared createRequire again kept it from loading).
+
+import { spawnSync } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+
+const server = fileURLToPath(new URL('..', import.meta.url));
+
+describe('the built server', () => {
+  it('loads, and stops for want of its environment', () => {
+    const built = spawnSync('npm', ['run', 'build'], { cwd: server, encoding: 'utf8' });
+    expect(built.status, built.stderr).toBe(0);
+    const started = spawnSync(process.execPath, ['dist/main.mjs'], { cwd: server, encoding: 'utf8', env: { PATH: process.env.PATH }, timeout: 20_000 });
+    expect(started.status, started.stderr).toBe(1);
+    expect(JSON.parse(started.stdout.trim().split('\n').at(-1)!)).toMatchObject({
+      event: 'config.missing',
+      variables: ['DATABASE_URL', 'FILIUS_TICKET_SECRET', 'FILIUS_SERVICE_KEY', 'FILIUS_ORIGINS'],
+    });
+  }, 60_000);
+});

packages/server/test/transfer.test.ts

@@ -0,0 +1,275 @@
+// FILIUS files through the dashboard's service (F6.1): the examples beside
+// F, a project made from one and from a file the dashboard sends — its
+// large pictures in F's blob store —, a project copied while people work in
+// it, a project given back as a FILIUS file (with its warnings, and never
+// with a picture it no longer has), and what is refused.
+//
+// A .fls is compared in a canonical form: read with ids counted from one and
+// written again (a file's bytes depend on the ids a reading gave its parts).
+
+import { randomUUID } from 'node:crypto';
+import { copyFileSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { afterAll, beforeAll, describe, expect, it } from 'vitest';
+import * as Y from 'yjs';
+import { attachBlobs, memoryBlobSide, nodeFiles, readProject, writeProject } from '@mdg-filius/doc';
+import { openProjectArchive, readFlsArchive, saveProjectArchive, writeFlsArchive, type ImportedProject } from '@mdg-filius/fls';
+import { IMPORT_BYTES, PostgresBlobs, PostgresSessions, PostgresStore, SERVICE_KEY_HEADER, shaOf, WARNINGS_HEADER } from '../src/index.js';
+import { testDatabase, type TestDatabase } from './database.js';
+import { enter, SERVICE_KEY, startTestServer, until, type TestServer } from './harness.js';
+
+const corpus = new URL('../../../corpus/', import.meta.url);
+const EXAMPLES = new URL('java-examples/', corpus).pathname;
+const example = (name: string) => new Uint8Array(readFileSync(new URL(`java-examples/${name}.fls`, corpus)));
+
+/**
+ * A FILIUS file as it reads: the same for the same project, whatever ids a
+ * reading gave it. A switch's SSID too, which a file without one gets from
+ * its id when it is read (and keeps from then on).
+ */
+function canonical(archive: Uint8Array): string {
+  let counter = 0;
+  const project = openProjectArchive(archive, () => `00000000-0000-4000-8000-${(++counter).toString(16).padStart(12, '0')}`);
+  const xml = new TextDecoder().decode(readFlsArchive(saveProjectArchive(project).archive).configurationXml);
+  return xml.replace(/(<void property="SSID">\s*<string>)[0-9a-f]{6}(<\/string>)/g, '$1ssid$2');
+}
+
+/** An example whose switch is of a kind FILIUS does not know: read, it warns. */
+function withForeignDevice(): Uint8Array {
+  const contents = readFlsArchive(example('internetwork_komplett'));
+  const xml = new TextDecoder().decode(contents.configurationXml).replace('filius.hardware.knoten.Switch', 'filius.hardware.knoten.Unbekannt');
+  return writeFlsArchive(new TextEncoder().encode(xml), contents.supplementalEntries);
+}
+
+const documents = async () => ((await (await fetch(`${server.base}/healthz`)).json()) as { documents: number }).documents;
+
+let database: TestDatabase;
+let store: PostgresStore;
+let sessions: PostgresSessions;
+let blobs: PostgresBlobs;
+let server: TestServer;
+
+beforeAll(async () => {
+  database = await testDatabase();
+  store = PostgresStore.connect(database.url);
+  sessions = new PostgresSessions(store.pool);
+  blobs = new PostgresBlobs(store.pool);
+  await store.migrate();
+  await sessions.migrate();
+  await blobs.migrate();
+  server = await startTestServer(store, sessions, { examplesFolder: EXAMPLES });
+}, 60_000);
+
+afterAll(async () => {
+  await server?.close();
+  await store?.close();
+  await database?.stop();
+});
+
+const service = (method: string, path: string, body?: string | Uint8Array, json = true) =>
+  fetch(`${server.base}${path}`, { method, headers: { [SERVICE_KEY_HEADER]: SERVICE_KEY, ...(json ? { 'content-type': 'application/json' } : {}) }, ...(body === undefined ? {} : { body }) });
+const exported = async (project: string) => {
+  const response = await service('GET', `/svc/projects/${project}/export`);
+  return { response, archive: new Uint8Array(await response.arrayBuffer()) };
+};
+const warningsOf = (response: Response) => JSON.parse(Buffer.from(response.headers.get(WARNINGS_HEADER) ?? '', 'base64url').toString('utf8')) as string[];
+
+describe("FILIUS' examples", () => {
+  it('lists the files of the folder beside F, by name', async () => {
+    const names = ((await (await service('GET', '/svc/examples')).json()) as { examples: string[] }).examples;
+    expect(names).toHaveLength(18);
+    expect(names).toContain('internetwork_komplett');
+    expect(names).toEqual([...names].sort());
+  });
+
+  it('lists none without a folder, nor when the folder is not there', async () => {
+    const gone = await startTestServer(store, sessions, { examplesFolder: join(tmpdir(), `filius-no-examples-${randomUUID()}`) });
+    try {
+      expect(await (await fetch(`${gone.base}/svc/examples`, { headers: { [SERVICE_KEY_HEADER]: SERVICE_KEY } })).json()).toEqual({ examples: [] });
+    } finally {
+      await gone.close();
+    }
+    const bare = await startTestServer(store, sessions);
+    try {
+      const answer = await fetch(`${bare.base}/svc/examples`, { headers: { [SERVICE_KEY_HEADER]: SERVICE_KEY } });
+      expect(await answer.json()).toEqual({ examples: [] });
+      const made = await fetch(`${bare.base}/svc/projects`, {
+        method: 'POST',
+        headers: { [SERVICE_KEY_HEADER]: SERVICE_KEY },
+        body: JSON.stringify({ project: randomUUID(), example: 'internetwork_komplett' }),
+      });
+      expect([made.status, await made.json()]).toEqual([404, { error: 'unknownExample' }]);
+    } finally {
+      await bare.close();
+    }
+  });
+
+  it("lists only the folder's FILIUS files, and tells what one held that a project cannot", async () => {
+    const folder = mkdtempSync(join(tmpdir(), 'filius-examples-'));
+    // Made out of order: the list comes sorted all the same.
+    writeFileSync(join(folder, 'fremd.fls'), withForeignDevice());
+    writeFileSync(join(folder, 'LIESMICH.txt'), 'keine FILIUS-Datei');
+    copyFileSync(new URL('java-examples/webserver.fls', corpus), join(folder, 'webserver.fls'));
+    const own = await startTestServer(store, sessions, { examplesFolder: folder });
+    try {
+      const ask = (method: string, path: string, body?: string) => fetch(`${own.base}${path}`, { method, headers: { [SERVICE_KEY_HEADER]: SERVICE_KEY }, ...(body === undefined ? {} : { body }) });
+      expect(await (await ask('GET', '/svc/examples')).json()).toEqual({ examples: ['fremd', 'webserver'] });
+      const project = randomUUID();
+      const made = (await (await ask('POST', '/svc/projects', JSON.stringify({ project, example: 'fremd' }))).json()) as { warnings: string[] };
+      expect(made.warnings).toEqual(openProjectArchive(withForeignDevice(), randomUUID).warnings);
+      expect(made.warnings.length).toBeGreaterThan(0);
+    } finally {
+      await own.close();
+    }
+  });
+
+  it('makes a project from one, the same as its file', async () => {
+    const project = randomUUID();
+    const made = await service('POST', '/svc/projects', JSON.stringify({ project, example: 'internetwork_komplett' }));
+    expect([made.status, await made.json()]).toEqual([201, { project, warnings: [] }]);
+    const { response, archive } = await exported(project);
+    expect(response.status).toBe(200);
+    expect(canonical(archive)).toBe(canonical(example('internetwork_komplett')));
+    const person = await enter(server, project);
+    expect(readProject(person.doc).graph.nodes.map((node) => node.displayName)).toContain('Notebook 1.10');
+    person.close();
+  });
+
+  it.each([['nichtda'], ['../../package'], ['java-examples/internetwork_komplett'], [7]])('knows no example %j', async (name) => {
+    const made = await service('POST', '/svc/projects', JSON.stringify({ project: randomUUID(), example: name }));
+    expect([made.status, await made.json()]).toEqual([404, { error: 'unknownExample' }]);
+  });
+});
+
+describe('a FILIUS file the dashboard sends', () => {
+  it('becomes a project, the same as the file', async () => {
+    const project = randomUUID();
+    const made = await service('POST', `/svc/projects/${project}/import`, example('nat_with_mapped_port'), false);
+    expect([made.status, await made.json()]).toEqual([201, { project, warnings: [] }]);
+    expect(canonical((await exported(project)).archive)).toBe(canonical(example('nat_with_mapped_port')));
+    expect(server.log.lines).toContainEqual(expect.objectContaining({ event: 'svc.imported', project }));
+  });
+
+  it("takes a large picture into F's blob store, and gives it back in the file", async () => {
+    const picture = Uint8Array.from({ length: 40_000 }, (_, index) => (index * 7) % 251);
+    const original = openProjectArchive(example('webserver'), randomUUID);
+    const computer = original.graph.nodes.find((node) => original.virtualFileSystems.has(node.id))!;
+    original.virtualFileSystems.get(computer.id)!.writeBinaryFile('/bild.png', picture, 'image/png');
+    const file = saveProjectArchive(original).archive;
+    const project = randomUUID();
+    expect((await service('POST', `/svc/projects/${project}/import`, file, false)).status).toBe(201);
+    expect(Buffer.from((await blobs.get([shaOf(picture)])).get(shaOf(picture)) ?? [])).toEqual(Buffer.from(picture));
+    const person = await enter(server, project);
+    // F gave the parts ids of its own: the computer is found by its name.
+    const node = [...person.doc.getMap<Y.Map<unknown>>('nodes').values()].find((each) => each.get('name') === computer.displayName)!;
+    const entry = (node.get('files') as Y.Map<Y.Map<unknown>>).get('/bild.png')!;
+    expect([entry.get('blob'), entry.get('data')]).toEqual([shaOf(picture), undefined]);
+    person.close();
+    const back = openProjectArchive((await exported(project)).archive, randomUUID);
+    const again = back.graph.nodes.find((each) => each.displayName === computer.displayName)!;
+    const content = back.virtualFileSystems.get(again.id)!.entry('/bild.png').content;
+    expect(content.kind === 'binary' || content.kind === 'image' ? [...content.data] : content).toEqual([...picture]);
+  });
+
+  it('tells what the file held that a project cannot', async () => {
+    const project = randomUUID();
+    const made = (await (await service('POST', `/svc/projects/${project}/import`, withForeignDevice(), false)).json()) as { warnings: string[] };
+    expect(made.warnings).toEqual(openProjectArchive(withForeignDevice(), randomUUID).warnings);
+    expect(made.warnings.length).toBeGreaterThan(0);
+  });
+
+  it('is refused when it is no FILIUS file, too large, or for a project there is', async () => {
+    const garbage = await service('POST', `/svc/projects/${randomUUID()}/import`, new Uint8Array([1, 2, 3]), false);
+    expect([garbage.status, await garbage.json()]).toEqual([422, { error: 'notAFiliusFile' }]);
+    const large = await service('POST', `/svc/projects/${randomUUID()}/import`, new Uint8Array(IMPORT_BYTES + 1), false);
+    expect([large.status, await large.json()]).toEqual([413, { error: 'tooLarge' }]);
+    const project = randomUUID();
+    await service('POST', '/svc/projects', JSON.stringify({ project }));
+    const twice = await service('POST', `/svc/projects/${project}/import`, example('webserver'), false);
+    expect([twice.status, await twice.json()]).toEqual([409, { error: 'exists' }]);
+    const notAProject = await service('POST', '/svc/projects/nope/import', example('webserver'), false);
+    expect([notAProject.status, await notAProject.json()]).toEqual([400, { error: 'notAProject' }]);
+  });
+});
+
+describe('a copy of a project', () => {
+  it('lets go of a project no one has open, once it is copied or given as a file', async () => {
+    const project = randomUUID();
+    await service('POST', '/svc/projects', JSON.stringify({ project, example: 'webserver' }));
+    const before = await documents();
+    expect((await service('POST', `/svc/projects/${project}/copy`, JSON.stringify({ to: randomUUID() }))).status).toBe(201);
+    expect((await exported(project)).response.status).toBe(200);
+    await until(async () => (await documents()) === before);
+  });
+
+  it('is the project as it is now, while someone works in it, and goes its own way after', async () => {
+    const project = randomUUID();
+    await service('POST', '/svc/projects', JSON.stringify({ project, example: 'internetwork_komplett' }));
+    const ben = await enter(server, project);
+    const nodes = ben.doc.getMap<Y.Map<unknown>>('nodes');
+    const [first] = [...nodes.keys()];
+    nodes.get(first!)!.set('name', 'Bens Rechner');
+    await until(() => ben.provider.unsyncedChanges === 0);
+    const copy = randomUUID();
+    const made = await service('POST', `/svc/projects/${project}/copy`, JSON.stringify({ to: copy }));
+    expect([made.status, await made.json()]).toEqual([201, { project: copy }]);
+    const mia = await enter(server, copy);
+    expect(mia.doc.getMap<Y.Map<unknown>>('nodes').get(first!)!.get('name')).toBe('Bens Rechner');
+    mia.doc.getMap<Y.Map<unknown>>('nodes').get(first!)!.set('name', 'Mias Rechner');
+    await until(() => mia.provider.unsyncedChanges === 0);
+    expect(nodes.get(first!)!.get('name')).toBe('Bens Rechner');
+    ben.close();
+    mia.close();
+  });
+
+  it('is refused onto a project there is, and of one there is not — which then can be made', async () => {
+    const project = randomUUID();
+    await service('POST', '/svc/projects', JSON.stringify({ project }));
+    const onto = await service('POST', `/svc/projects/${project}/copy`, JSON.stringify({ to: project }));
+    expect([onto.status, await onto.json()]).toEqual([409, { error: 'exists' }]);
+    const missing = randomUUID();
+    const of = await service('POST', `/svc/projects/${missing}/copy`, JSON.stringify({ to: randomUUID() }));
+    expect([of.status, await of.json()]).toEqual([404, { error: 'unknownProject' }]);
+    expect((await service('POST', '/svc/projects', JSON.stringify({ project: missing }))).status).toBe(201);
+    (await enter(server, missing)).close();
+    const bad = await service('POST', `/svc/projects/${project}/copy`, JSON.stringify({ to: 'nope' }));
+    expect([bad.status, await bad.json()]).toEqual([400, { error: 'notAProject' }]);
+  });
+});
+
+describe('a project as a FILIUS file', () => {
+  it("names what Java's file cannot hold", async () => {
+    const original: ImportedProject = openProjectArchive(example('webserver'), randomUUID);
+    const [model] = original.graph.nodes;
+    original.graph.nodes.push({ ...model!, id: randomUUID(), kind: 'unsupported', displayName: 'Fremdgerät', ports: model!.ports.map((port) => ({ ...port, id: randomUUID() })) });
+    const doc = new Y.Doc();
+    writeProject(doc, original);
+    const project = randomUUID();
+    await store.create(project, Y.encodeStateAsUpdate(doc));
+    const { response } = await exported(project);
+    expect(response.status).toBe(200);
+    expect(warningsOf(response)).toEqual(["node 'Fremdgerät': a kind Java FILIUS does not have, left out"]);
+  });
+
+  it('is not made without a picture it names, and not of a project there is not', async () => {
+    const doc = new Y.Doc();
+    attachBlobs(doc, memoryBlobSide());
+    const original = openProjectArchive(example('webserver'), randomUUID);
+    const computer = original.graph.nodes.find((node) => original.virtualFileSystems.has(node.id))!;
+    original.virtualFileSystems.get(computer.id)!.writeBinaryFile(
+      '/fehlt.png',
+      Uint8Array.from({ length: 30_000 }, (_, index) => index % 13),
+      'image/png',
+    );
+    writeProject(doc, original);
+    expect(nodeFiles(doc, computer.id)?.contains('/fehlt.png')).toBe(true);
+    const project = randomUUID();
+    await store.create(project, Y.encodeStateAsUpdate(doc));
+    const refused = await service('GET', `/svc/projects/${project}/export`);
+    expect([refused.status, await refused.json()]).toEqual([409, { error: 'missingBlobs' }]);
+    expect(server.log.lines).toContainEqual(expect.objectContaining({ event: 'svc.exportMissingBlobs', project }));
+    const none = await service('GET', `/svc/projects/${randomUUID()}/export`);
+    expect([none.status, await none.json()]).toEqual([404, { error: 'unknownProject' }]);
+  });
+});