Timerix/ParadoxSaveParser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated to accept only .tar.zstd uploads

Browse Source
This commit is contained in:
Timerix
2026-09-23 17:48:42 +02:00
parent cf337483d7
commit c6b27cd3c9
15 changed files with 556 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+256
View File
@@ -0,0 +1,256 @@
# AGENTS.md
Guidance for AI coding agents working in this repository.
## Project
A parser for Paradox game save files, and the backend of a save analytics
website. Only EU4 is supported so far.
- **.NET 10**, C# `latest`, nullable reference types enabled everywhere
- **DTLib** / **DTLib.Web** for filesystem helpers, logging and the HTTP server
- **sqlite-net-pcl** for the upload database, **ZstdSharp.Port** for zstd
Projects in the solution ([ParadoxSaveParser.sln](ParadoxSaveParser.sln)):
- `ParadoxSaveParser.Lib`: the parser itself (`Tokenizer`, `SaveParserEU4`,
`SearchExpression/`)
- `ParadoxSaveParser.Lib.Tests`: NUnit tests for the parser
- `ParadoxSaveParser.CLI`: command line tool, with an interactive mode
- `ParadoxSaveParser.WebAPI`: HTTP backend for the website
- `ParadoxSaveParser.Benchmarks`: BenchmarkDotNet comparison against regex
engines and jq, see [its README](ParadoxSaveParser.Benchmarks/README.md)
## Architecture
### Parser
`SaveParserEU4` reads the plain text game state and produces nested
`Dictionary<string, object>` values. It takes a compiled `SearchExpression`,
a small path query such as `countries.*.technology`. **The query is what
makes the parser fast.** `Tokenizer.SkipBlock` throws away a rejected `{...}`
block by scanning raw bytes for brace depth, so rejected data is never turned
into tokens or strings. Full parse: 1.2 s and 962 MB allocated; the same file
with a query: 50 ms and 1.5 MB. Keep any parser change from bringing back
per-byte virtual calls or eager string decoding. Check with the benchmarks
before and after.
### WebAPI
The frontend is a **separate Next.js project in its own repository**. It calls
this service through a same-origin proxy and expects it at
`http://127.0.0.1:5226` (`BaseUrl` in `config.json`). Routes are documented
in [ParadoxSaveParser.WebAPI/README.md](ParadoxSaveParser.WebAPI/README.md);
keep that file in sync when a route or a response shape changes, because the
frontend is written against it.
The flow is upload, poll, fetch:
```
POST /uploadSave -> store blob, status "uploaded" -> SaveParsingOperation
(background: parsing,
saving_results, done)
GET /getSaveStatus (polled by the frontend until status is "done" or "error")
GET /getSaveData (filtered parse result as JSON)
GET /getSaveFile (the uploaded blob, unchanged)
```
### Save files on the wire
The server is a cheap two core machine with little disk space, so the browser
re-compresses every save before uploading it:
```
plain text save -> tar { "gamestate" } -> zstd -> upload
archived save -> tar { "meta", "gamestate", ... } -> zstd -> upload
```
- `.tar.zst` is **the only accepted format.** Zip handling was removed on
purpose; do not add it back. The original type and name come in as query
params (`originalFileType=text|zip`, `originalFileName`), and they are stored
so the browser can rebuild the original file from `/getSaveFile`. A
rebuilt EU4 zip was checked entry by entry against the original and is byte
identical.
- **The blob is stored byte-for-byte.** Never re-encode it on the server.
- The game data entry name comes from `GameSaveEntryNames.Map` in
[Game.cs](ParadoxSaveParser.WebAPI/Game.cs). That map mirrors the frontend's
`GAME_SAVE_ENTRY_NAMES`. Do not hard-code `"gamestate"`.
- All tar and zstd handling lives in
[SaveArchive.cs](ParadoxSaveParser.WebAPI/SaveArchive.cs), which streams
through `DecompressionStream` and `TarReader`. The whole archive is never
loaded into memory.
- Upload limits come from the `SaveUpload` config section.
`MaxSizeCompressedBytes` is checked against `Content-Length` and then again
while the body is copied, because chunked requests have no length.
`MaxSizePlainTextBytes` is checked against the tar entry size once the upload
has finished. Every rejected upload deletes its database row and file.
### Runtime files
The WebAPI creates everything relative to its **working directory**:
`config.json` (written with defaults if missing), `database.sqlite`, `logs/`,
`temp/` (emptied on start), and `public/data/{saves,parsed}/`. When running it
for a test, start it with `TEMP/` as the working directory, so none of this
lands in the project folders.
### Current status
- **Only EU4 works.** `Game.Stellaris` exists and has an entry name, but it has
no `ISaveDataFilter` and no parser, so uploads for it are rejected with 400.
`SaveParsingOperation` is EU4 specific.
- **Nothing is deployed and there is no data worth keeping.** Do not write
database migrations or backward compatibility code for old layouts; change
the schema directly.
- Open items are in [TODO.md](TODO.md): compressing parsed results,
re-parsing when the query changes, automatic database cleanup.
- The WebAPI exits with code 1 after Ctrl+C even though it logs "server
stopped". This has not been investigated yet.
## Tools and files
**Prefer the editor's own tools over shell commands that do the same thing.**
Reading, editing, writing and searching files have dedicated tools. Using
`cat`, `sed`, `grep` or a throwaway script instead does the same work,
spends more tokens, and may need its own approval each time. Use the shell for
what only the shell can do: building, running tests, git, `curl`, `tar`,
`zstd`, or running the program itself.
**Work inside the project directory.** For temporary files (scratch scripts,
intermediate output, test data, a running WebAPI's working directory), create
a `TEMP/` directory in the project root and keep everything there. It is
gitignored. Delete that directory once the work is done. Ask before reading or
writing anything outside the project directory, including `/tmp` and the home
folder. A save file the user names explicitly may be read, but never modified.
## Terminal
**On Windows, prefer `bash.exe`**, but only a real MSYS2, Cygwin, or
MinGW/Git Bash installation, for example `D:\msys2\usr\bin\bash.exe`.
**Never use `C:\Windows\System32\bash.exe`.** Despite its name, it is not a
POSIX bash but the launcher stub for WSL. Do not start WSL to run ordinary
commands.
If no MSYS/Cygwin/MinGW bash is installed, **use PowerShell** as the terminal.
PowerShell is the fallback, never WSL.
If bash reports `command not found` for `dotnet`, the cause is `PATH`, not a
missing install. MSYS2 sets `MSYS2_PATH_TYPE` to `minimal` by default, which
drops the Windows `PATH`. Setting it to `inherit` fixes it.
## Commands
```sh
dotnet build ParadoxSaveParser.sln # build everything
dotnet test ParadoxSaveParser.Lib.Tests # parser tests
dotnet run --project ParadoxSaveParser.WebAPI # run the backend (mind the working directory)
dotnet build ParadoxSaveParser.Benchmarks -c Release # see its README for running
```
A one-off C# check can be a .NET 10 file-based app in `TEMP/`
(`dotnet run -c Release TEMP/check.cs`); it doesn't need a project.
### Checking your work
**Always build the solution and run the tests after changing code.** Do not
report a change as finished until both pass.
- `dotnet build ParadoxSaveParser.sln` must finish with zero errors and no
**new** warnings. Three warnings are known and come from
sqlite-net-pcl's `SQLitePCLRaw.lib.e_sqlite3` dependency (NU1903 twice,
NETSDK1206 once). Everything else is yours.
- `dotnet test ParadoxSaveParser.Lib.Tests` must pass completely.
- A WebAPI change is not verified until it has been run: start the server
from `TEMP/` and exercise the route with `curl`, including the error cases.
Stop the server afterwards.
- For a parser performance change, run the relevant benchmark before and
after, and report both numbers.
## Version control
This project is managed with **git**.
**Read-only git commands are fine to use without asking.**
You can run `git status`, `git log`, `git diff`, `git show`, `git blame` and
similar inspection commands freely.
**Never run a git command that changes the repository without the user's
explicit permission.**
This is a strict ban. It covers every command that writes to the history, the
index, or the working tree, including `add`, `commit`, `checkout`, `restore`,
`reset`, `stash`, `branch`, `merge`, `rebase`, `push`, and `rm`.
Ask first, and wait for a clear yes before running such a command.
## Conventions
Match the surrounding code exactly:
- File-scoped namespaces, 4 spaces, `_camelCase` private fields, `PascalCase`
for everything else.
- **`SaveFileMetadata` properties are lowercase on purpose.** They are the
JSON wire format that the frontend reads, and the sqlite column names too.
Do not rename them to PascalCase. Enums on the wire serialize in lowercase
through `LowerCaseEnumConverter<T>` (`SavingResults` becomes
`saving_results`).
- `Program.cs` declares global usings, which make `File`, `Directory` and
`Path` refer to **DTLib.Filesystem**, not `System.IO`. Paths are `IOPath`.
When a `System.IO` member is needed, write it out in full, for example
`System.IO.Path.GetFileName`.
- Errors from route handlers go through `ReturnHelper.ResponseError` with an
`ErrorMessage` and the right status code. Query values come from
`RequestHelper.GetQueryValue`. Do not write responses by hand.
- **Line endings:** most `.cs` files are CRLF with a UTF-8 BOM. A new file
written with LF mixes endings once the IDE touches it. Match the files next
to it, or convert with `unix2dos -m`.
**Comment the code you write.** Human programmers who did not write this code
will read it, and a wall of uncommented agent-written code is expensive to
review. Keep comments simple, preferably one line, placed above the code they
describe, and in plain language rather than in terms of the types. Say _why_
the code does what it does: what is non-obvious, and what would break if it
were written the obvious way. Leave out what the code already states plainly.
A tricky constant, a workaround for someone else's bug, or an ordering that
matters always deserves a line.
Write doc comments as XML doc comments (`///`), which the IDE shows on hover.
Plain `//` comments do not appear in tooltips.
## Dependencies
**Search for an existing library before writing your own implementation.**
Formats, protocols and codecs are full of details that only show up against
real data, and a maintained library has already met them. Check the BCL first:
tar (`System.Formats.Tar`), zip, JSON and vectorized span operations are all
built in. .NET 10 has no zstd, which is why ZstdSharp.Port is here.
**Read the library's source before adding it, and judge it on that.** Download
counts say nothing about quality. Look for readable sources, comments that
explain intent, and an API that fits the way the code here is written. Code
that nobody can read cannot be maintained and may hide bugs nobody can find.
Do not depend on it.
If nothing of good quality exists, say so and propose writing an
implementation here, rather than pulling in the least bad option.
Do not add a second library for a job a package already referenced can do. A
thin wrapper that shapes an installed library's API to what the code needs is
better than a new dependency.
## Things that will bite you
- **Changing a route's request or response is a two-repository change.** The
frontend has its own copy of the protocol in `SaveParserBackendService.ts`,
`GAME_SAVE_ENTRY_NAMES` and its save archive code. Say what the frontend
needs changed; do not assume it will follow.
- **`Config.Load` rejects any config whose `Version` differs from
`ActualVersion`.** New config properties with defaults don't need a version
bump, because a missing key keeps the default. Bump the version only for
changes that old files cannot express.
- **`PathHelper.CreateTempFile` deletes the file right after opening it.** The
open stream keeps the data alive until it is disposed; do not reopen it by
path.
- **`SaveParsingOperation.StartAsync` is `async void`.** Every exception must
be caught inside it, or it will crash the process. Errors are recorded on
the metadata row with `SetMetadataError`.
- `Program.IsDebug` is always true for now, so debug logging is always on.