compress parsed data to .json.zst
This commit is contained in:
@@ -51,7 +51,7 @@ POST /uploadSave -> store blob, status "uploaded" -> SaveParsingOperation
|
||||
(background: parsing,
|
||||
saving_results, done)
|
||||
GET /getSaveStatus (polled by the frontend until status is "done" or "error")
|
||||
GET /getSaveData (filtered parse result as JSON)
|
||||
GET /getSaveData (filtered parse result as zstd-compressed JSON)
|
||||
GET /getSaveFile (the uploaded blob, unchanged)
|
||||
```
|
||||
|
||||
@@ -79,11 +79,13 @@ archived save -> tar { "meta", "gamestate", ... } -> zstd -> upload
|
||||
[SaveArchive.cs](ParadoxSaveParser.WebAPI/SaveArchive.cs), which streams
|
||||
through `DecompressionStream` and `TarReader`. The whole archive is never
|
||||
loaded into memory.
|
||||
- Upload limits come from the `SaveUpload` config section.
|
||||
`MaxSizeCompressedBytes` is checked against `Content-Length` and then again
|
||||
while the body is copied, because chunked requests have no length.
|
||||
`MaxSizePlainTextBytes` is checked against the tar entry size once the upload
|
||||
has finished. Every rejected upload deletes its database row and file.
|
||||
- Upload limits come from the `SaveUpload` config section, in MiB
|
||||
(`MaxSizeCompressedMegabytes`, `MaxSizePlainTextMegabytes`). The code uses
|
||||
their byte equivalents, the `…Bytes` properties, which are not saved to the
|
||||
config. The compressed limit is checked against `Content-Length` and then
|
||||
again while the body is copied, because chunked requests have no length. The
|
||||
plain text limit is checked against the tar entry size once the upload has
|
||||
finished. Every rejected upload deletes its database row and file.
|
||||
|
||||
### Runtime files
|
||||
|
||||
@@ -101,8 +103,11 @@ lands in the project folders.
|
||||
- **Nothing is deployed and there is no data worth keeping.** Do not write
|
||||
database migrations or backward compatibility code for old layouts; change
|
||||
the schema directly.
|
||||
- Open items are in [TODO.md](TODO.md): compressing parsed results,
|
||||
re-parsing when the query changes, automatic database cleanup.
|
||||
- Open items are in [TODO.md](TODO.md): re-parsing when the query changes,
|
||||
automatic database cleanup.
|
||||
- Parsed results are stored as `<id>.parsed.json.zstd` and `/getSaveData`
|
||||
serves that file unchanged as `application/zstd`. The client decompresses
|
||||
it; the server never does.
|
||||
- The WebAPI exits with code 1 after Ctrl+C even though it logs "server
|
||||
stopped". This has not been investigated yet.
|
||||
|
||||
@@ -152,17 +157,20 @@ A one-off C# check can be a .NET 10 file-based app in `TEMP/`
|
||||
|
||||
### Checking your work
|
||||
|
||||
**Always build the solution and run the tests after changing code.** Do not
|
||||
report a change as finished until both pass.
|
||||
**Always build the solution after changing code.** `dotnet build
|
||||
ParadoxSaveParser.sln` must finish with zero errors and no **new** warnings.
|
||||
Three warnings are known and come from sqlite-net-pcl's
|
||||
`SQLitePCLRaw.lib.e_sqlite3` dependency (NU1903 twice, NETSDK1206 once).
|
||||
Everything else is yours.
|
||||
|
||||
**Never run tests or benchmarks unless the user asks for it.** This is a
|
||||
strict ban. It covers `dotnet test`, the benchmarks, and starting the WebAPI
|
||||
to exercise routes with `curl`. When a change needs checking that way, say
|
||||
what should be run and wait for the user to ask. When they do:
|
||||
|
||||
- `dotnet build ParadoxSaveParser.sln` must finish with zero errors and no
|
||||
**new** warnings. Three warnings are known and come from
|
||||
sqlite-net-pcl's `SQLitePCLRaw.lib.e_sqlite3` dependency (NU1903 twice,
|
||||
NETSDK1206 once). Everything else is yours.
|
||||
- `dotnet test ParadoxSaveParser.Lib.Tests` must pass completely.
|
||||
- A WebAPI change is not verified until it has been run: start the server
|
||||
from `TEMP/` and exercise the route with `curl`, including the error cases.
|
||||
Stop the server afterwards.
|
||||
- For a WebAPI check, start the server from `TEMP/`, exercise the route with
|
||||
`curl` including the error cases, then stop the server.
|
||||
- For a parser performance change, run the relevant benchmark before and
|
||||
after, and report both numbers.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user