compress parsed data to .json.zst

This commit is contained in:
2026-09-23 18:25:15 +02:00
parent c6b27cd3c9
commit 55f7d453b1
9 changed files with 67 additions and 35 deletions
+25 -17
View File
@@ -51,7 +51,7 @@ POST /uploadSave -> store blob, status "uploaded" -> SaveParsingOperation
(background: parsing,
saving_results, done)
GET /getSaveStatus (polled by the frontend until status is "done" or "error")
GET /getSaveData (filtered parse result as JSON)
GET /getSaveData (filtered parse result as zstd-compressed JSON)
GET /getSaveFile (the uploaded blob, unchanged)
```
@@ -79,11 +79,13 @@ archived save -> tar { "meta", "gamestate", ... } -> zstd -> upload
[SaveArchive.cs](ParadoxSaveParser.WebAPI/SaveArchive.cs), which streams
through `DecompressionStream` and `TarReader`. The whole archive is never
loaded into memory.
- Upload limits come from the `SaveUpload` config section.
`MaxSizeCompressedBytes` is checked against `Content-Length` and then again
while the body is copied, because chunked requests have no length.
`MaxSizePlainTextBytes` is checked against the tar entry size once the upload
has finished. Every rejected upload deletes its database row and file.
- Upload limits come from the `SaveUpload` config section, in MiB
(`MaxSizeCompressedMegabytes`, `MaxSizePlainTextMegabytes`). The code uses
their byte equivalents, the `…Bytes` properties, which are not saved to the
config. The compressed limit is checked against `Content-Length` and then
again while the body is copied, because chunked requests have no length. The
plain text limit is checked against the tar entry size once the upload has
finished. Every rejected upload deletes its database row and file.
### Runtime files
@@ -101,8 +103,11 @@ lands in the project folders.
- **Nothing is deployed and there is no data worth keeping.** Do not write
database migrations or backward compatibility code for old layouts; change
the schema directly.
- Open items are in [TODO.md](TODO.md): compressing parsed results,
re-parsing when the query changes, automatic database cleanup.
- Open items are in [TODO.md](TODO.md): re-parsing when the query changes,
automatic database cleanup.
- Parsed results are stored as `<id>.parsed.json.zstd` and `/getSaveData`
serves that file unchanged as `application/zstd`. The client decompresses
it; the server never does.
- The WebAPI exits with code 1 after Ctrl+C even though it logs "server
stopped". This has not been investigated yet.
@@ -152,17 +157,20 @@ A one-off C# check can be a .NET 10 file-based app in `TEMP/`
### Checking your work
**Always build the solution and run the tests after changing code.** Do not
report a change as finished until both pass.
**Always build the solution after changing code.** `dotnet build
ParadoxSaveParser.sln` must finish with zero errors and no **new** warnings.
Three warnings are known and come from sqlite-net-pcl's
`SQLitePCLRaw.lib.e_sqlite3` dependency (NU1903 twice, NETSDK1206 once).
Everything else is yours.
**Never run tests or benchmarks unless the user asks for it.** This is a
strict ban. It covers `dotnet test`, the benchmarks, and starting the WebAPI
to exercise routes with `curl`. When a change needs checking that way, say
what should be run and wait for the user to ask. When they do:
- `dotnet build ParadoxSaveParser.sln` must finish with zero errors and no
**new** warnings. Three warnings are known and come from
sqlite-net-pcl's `SQLitePCLRaw.lib.e_sqlite3` dependency (NU1903 twice,
NETSDK1206 once). Everything else is yours.
- `dotnet test ParadoxSaveParser.Lib.Tests` must pass completely.
- A WebAPI change is not verified until it has been run: start the server
from `TEMP/` and exercise the route with `curl`, including the error cases.
Stop the server afterwards.
- For a WebAPI check, start the server from `TEMP/`, exercise the route with
`curl` including the error cases, then stop the server.
- For a parser performance change, run the relevant benchmark before and
after, and report both numbers.