feat: export completed runs as SQLite

This commit is contained in:
NotoriousRebel
2026-08-16 14:41:19 -04:00
parent f5d8a421f3
commit c91456531c
15 changed files with 335 additions and 35 deletions
+2 -2
View File
@@ -96,7 +96,7 @@
<text x="1000" y="276" class="node-title">Inspect and export</text>
<text x="1000" y="300" class="node-copy-strong">history · assessment · route tabs</text>
<text x="1000" y="320" class="node-copy-strong">screenshots · source outcomes · logs</text>
<text x="1000" y="340" class="node-copy">review child actions · export JSONL</text>
<text x="1000" y="340" class="node-copy">review actions · JSONL / SQLite export</text>
<line x1="40" y1="636" x2="1240" y2="636" stroke="rgba(14,29,34,.14)" stroke-width=".8"/>
<line x1="48" y1="676" x2="88" y2="676" class="connector" marker-end="url(#harvestview-architecture-arrow)"/>
@@ -106,4 +106,4 @@
<rect x="676" y="664" width="20" height="20" rx="4" class="tag-teal"/>
<text x="708" y="680" class="legend">durable evidence boundary</text>
<text x="1240" y="680" class="legend" text-anchor="end">provider credentials remain server-side</text>
</svg>
</svg>

Before

Width:  |  Height:  |  Size: 8.2 KiB

After

Width:  |  Height:  |  Size: 8.2 KiB

+11
View File
@@ -32,6 +32,7 @@ Treat the runtime OpenAPI document as the exact request and response reference.
| `POST /api/v1/runs/{run_id}/cancel` | Cancel queued work or request cancellation of running work. |
| `POST /api/v1/runs/import` | Import a JSONL result file without executing discovery. |
| `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. |
| `GET /api/v1/runs/export-database` | Export all completed run evidence as a portable SQLite database. |
| `GET /api/v1/runs/{run_id}/export` | Export normalized results as JSONL. |
| `GET /api/v1/runs/{run_id}/screenshots/{name}` | Retrieve one managed screenshot. |
@@ -169,6 +170,16 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/import-database?filename=stash.sqlite
The server checks the SQLite header, integrity, schema, and each completed run before copying it. Original run IDs are preserved. Exact duplicates are skipped, while a reused ID with different evidence is rejected. Close the source process or checkpoint its WAL before uploading the database. Screenshot metadata is imported, but screenshot files must be copied separately. The default upload ceiling is 1 GiB and can be changed with `THEHARVESTER_MAX_DATABASE_IMPORT_BYTES`.
Export every completed run as a consistent database that can be imported elsewhere:
```bash
curl -s "http://127.0.0.1:5000/api/v1/runs/export-database" \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
-o theharvester-completed-runs.sqlite
```
The export is rebuilt from canonical completed evidence, so it excludes queue state, cancellation state, worker leases, and legacy observations. It includes screenshot metadata but not screenshot files. The server checkpoints and closes the temporary database before download; no manual WAL handling is required.
Export one normalized result set in the same streamable format:
```bash
+1 -1
View File
@@ -75,7 +75,7 @@ Two operational tables support the API without changing those six evidence conce
## API results
`GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. A `hostname` found through the `vhost` action has native endpoint observations; a `prefix` found through RouteViews has native origin, route, and RPKI observations with fixed external-relationship scope. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run, and SQLite import loads completed runs in bulk. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference.
`GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. A `hostname` found through the `vhost` action has native endpoint observations; a `prefix` found through RouteViews has native origin, route, and RPKI observations with fixed external-relationship scope. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run. SQLite import and `GET /api/v1/runs/export-database` move completed runs in bulk without queue, cancellation, or worker-lease state. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference.
## Handling and sharing