ParadoxSaveParser/AGENTS.md

# AGENTS.md

Guidance for AI coding agents working in this repository.

## Project

A parser for Paradox game save files, and the backend of a save analytics
website. Only EU4 is supported so far.

- **.NET 10**, C# `latest`, nullable reference types enabled everywhere
- **DTLib** / **DTLib.Web** for filesystem helpers, logging and the HTTP server
- **sqlite-net-pcl** for the upload database, **ZstdSharp.Port** for zstd

Projects in the solution ([ParadoxSaveParser.sln](ParadoxSaveParser.sln)):

- `ParadoxSaveParser.Lib`: the parser itself (`Tokenizer`, `SaveParserEU4`,
  `SearchExpression/`)
- `ParadoxSaveParser.Lib.Tests`: NUnit tests for the parser
- `ParadoxSaveParser.CLI`: command line tool, with an interactive mode
- `ParadoxSaveParser.WebAPI`: HTTP backend for the website
- `ParadoxSaveParser.Benchmarks`: BenchmarkDotNet comparison against regex
  engines and jq, see [its README](ParadoxSaveParser.Benchmarks/README.md)

## Architecture

### Parser

`SaveParserEU4` reads the plain text game state and produces nested
`Dictionary<string, object>` values. It takes a compiled `SearchExpression`,
a small path query such as `countries.*.technology`. **The query is what
makes the parser fast.** `Tokenizer.SkipBlock` throws away a rejected `{...}`
block by scanning raw bytes for brace depth, so rejected data is never turned
into tokens or strings. Full parse: 1.2 s and 962 MB allocated; the same file
with a query: 50 ms and 1.5 MB. Keep any parser change from bringing back
per-byte virtual calls or eager string decoding. Check with the benchmarks
before and after.

### WebAPI

The frontend is a **separate Next.js project in its own repository**. It calls
this service through a same-origin proxy and expects it at
`http://127.0.0.1:5226` (`BaseUrl` in `config.json`). Routes are documented
in [ParadoxSaveParser.WebAPI/README.md](ParadoxSaveParser.WebAPI/README.md);
keep that file in sync when a route or a response shape changes, because the
frontend is written against it.

The flow is upload, poll, fetch:

```
POST /uploadSave  ->  store blob, status "uploaded"  ->  SaveParsingOperation
                                                          (background: parsing,
                                                           saving_results, done)
GET  /getSaveStatus  (polled by the frontend until status is "done" or "error")
GET  /getSaveData    (filtered parse result as zstd-compressed JSON)
GET  /getSaveFile    (the uploaded blob, unchanged)
```

### Save files on the wire

The server is a cheap two core machine with little disk space, so the browser
re-compresses every save before uploading it:

```
plain text save  ->  tar { "gamestate" }               -> zstd -> upload
archived save    ->  tar { "meta", "gamestate", ... }  -> zstd -> upload
```

- `.tar.zst` is **the only accepted format.** Zip handling was removed on
  purpose; do not add it back. The original type and name come in as query
  params (`originalFileType=text|zip`, `originalFileName`), and they are stored
  so the browser can rebuild the original file from `/getSaveFile`. A
  rebuilt EU4 zip was checked entry by entry against the original and is byte
  identical.
- **The blob is stored byte-for-byte.** Never re-encode it on the server.
- The game data entry name comes from `GameSaveEntryNames.Map` in
  [Game.cs](ParadoxSaveParser.WebAPI/Game.cs). That map mirrors the frontend's
  `GAME_SAVE_ENTRY_NAMES`. Do not hard-code `"gamestate"`.
- All tar and zstd handling lives in
  [SaveArchive.cs](ParadoxSaveParser.WebAPI/SaveArchive.cs), which streams
  through `DecompressionStream` and `TarReader`. The whole archive is never
  loaded into memory.
- Upload limits come from the `SaveUpload` config section, in MiB
  (`MaxSizeCompressedMegabytes`, `MaxSizePlainTextMegabytes`). The code uses
  their byte equivalents, the `…Bytes` properties, which are not saved to the
  config. The compressed limit is checked against `Content-Length` and then
  again while the body is copied, because chunked requests have no length. The
  plain text limit is checked against the tar entry size once the upload has
  finished. Every rejected upload deletes its database row and file.

### Runtime files

The WebAPI creates everything relative to its **working directory**:
`config.json` (written with defaults if missing), `database.sqlite`, `logs/`,
`temp/` (emptied on start), and `public/data/{saves,parsed}/`. When running it
for a test, start it with `TEMP/` as the working directory, so none of this
lands in the project folders.

### Current status

- **Only EU4 works.** `Game.Stellaris` exists and has an entry name, but it has
  no `ISaveDataFilter` and no parser, so uploads for it are rejected with 400.
  `SaveParsingOperation` is EU4 specific.
- **Nothing is deployed and there is no data worth keeping.** Do not write
  database migrations or backward compatibility code for old layouts; change
  the schema directly.
- Open items are in [TODO.md](TODO.md): re-parsing when the query changes,
  automatic database cleanup.
- Parsed results are stored as `<id>.parsed.json.zstd` and `/getSaveData`
  serves that file unchanged as `application/zstd`. The client decompresses
  it; the server never does.
- The WebAPI exits with code 1 after Ctrl+C even though it logs "server
  stopped". This has not been investigated yet.

## Tools and files

**Prefer the editor's own tools over shell commands that do the same thing.**
Reading, editing, writing and searching files have dedicated tools. Using
`cat`, `sed`, `grep` or a throwaway script instead does the same work,
spends more tokens, and may need its own approval each time. Use the shell for
what only the shell can do: building, running tests, git, `curl`, `tar`,
`zstd`, or running the program itself.

**Work inside the project directory.** For temporary files (scratch scripts,
intermediate output, test data, a running WebAPI's working directory), create
a `TEMP/` directory in the project root and keep everything there. It is
gitignored. Delete that directory once the work is done. Ask before reading or
writing anything outside the project directory, including `/tmp` and the home
folder. A save file the user names explicitly may be read, but never modified.

## Terminal

**On Windows, prefer `bash.exe`**, but only a real MSYS2, Cygwin, or
MinGW/Git Bash installation, for example `D:\msys2\usr\bin\bash.exe`.

**Never use `C:\Windows\System32\bash.exe`.** Despite its name, it is not a
POSIX bash but the launcher stub for WSL. Do not start WSL to run ordinary
commands.

If no MSYS/Cygwin/MinGW bash is installed, **use PowerShell** as the terminal.
PowerShell is the fallback, never WSL.

If bash reports `command not found` for `dotnet`, the cause is `PATH`, not a
missing install. MSYS2 sets `MSYS2_PATH_TYPE` to `minimal` by default, which
drops the Windows `PATH`. Setting it to `inherit` fixes it.

## Commands

```sh
dotnet build ParadoxSaveParser.sln        # build everything
dotnet test ParadoxSaveParser.Lib.Tests   # parser tests
dotnet run --project ParadoxSaveParser.WebAPI   # run the backend (mind the working directory)
dotnet build ParadoxSaveParser.Benchmarks -c Release   # see its README for running
```

A one-off C# check can be a .NET 10 file-based app in `TEMP/`
(`dotnet run -c Release TEMP/check.cs`); it doesn't need a project.

### Checking your work

**Always build the solution after changing code.** `dotnet build
ParadoxSaveParser.sln` must finish with zero errors and no **new** warnings.
Three warnings are known and come from sqlite-net-pcl's
`SQLitePCLRaw.lib.e_sqlite3` dependency (NU1903 twice, NETSDK1206 once).
Everything else is yours.

**Never run tests or benchmarks unless the user asks for it.** This is a
strict ban. It covers `dotnet test`, the benchmarks, and starting the WebAPI
to exercise routes with `curl`. When a change needs checking that way, say
what should be run and wait for the user to ask. When they do:

- `dotnet test ParadoxSaveParser.Lib.Tests` must pass completely.
- For a WebAPI check, start the server from `TEMP/`, exercise the route with
  `curl` including the error cases, then stop the server.
- For a parser performance change, run the relevant benchmark before and
  after, and report both numbers.

## Version control

This project is managed with **git**.

**Read-only git commands are fine to use without asking.**
You can run `git status`, `git log`, `git diff`, `git show`, `git blame` and
similar inspection commands freely.

**Never run a git command that changes the repository without the user's
explicit permission.**
This is a strict ban. It covers every command that writes to the history, the
index, or the working tree, including `add`, `commit`, `checkout`, `restore`,
`reset`, `stash`, `branch`, `merge`, `rebase`, `push`, and `rm`.

Ask first, and wait for a clear yes before running such a command.

## Conventions

Match the surrounding code exactly:

- File-scoped namespaces, 4 spaces, `_camelCase` private fields, `PascalCase`
  for everything else.
- **`SaveFileMetadata` properties are lowercase on purpose.** They are the
  JSON wire format that the frontend reads, and the sqlite column names too.
  Do not rename them to PascalCase. Enums on the wire serialize in lowercase
  through `LowerCaseEnumConverter<T>` (`SavingResults` becomes
  `saving_results`).
- `Program.cs` declares global usings, which make `File`, `Directory` and
  `Path` refer to **DTLib.Filesystem**, not `System.IO`. Paths are `IOPath`.
  When a `System.IO` member is needed, write it out in full, for example
  `System.IO.Path.GetFileName`.
- Errors from route handlers go through `ReturnHelper.ResponseError` with an
  `ErrorMessage` and the right status code. Query values come from
  `RequestHelper.GetQueryValue`. Do not write responses by hand.
- **Line endings:** most `.cs` files are CRLF with a UTF-8 BOM. A new file
  written with LF mixes endings once the IDE touches it. Match the files next
  to it, or convert with `unix2dos -m`.

**Comment the code you write.** Human programmers who did not write this code
will read it, and a wall of uncommented agent-written code is expensive to
review. Keep comments simple, preferably one line, placed above the code they
describe, and in plain language rather than in terms of the types. Say _why_
the code does what it does: what is non-obvious, and what would break if it
were written the obvious way. Leave out what the code already states plainly.
A tricky constant, a workaround for someone else's bug, or an ordering that
matters always deserves a line.

Write doc comments as XML doc comments (`///`), which the IDE shows on hover.
Plain `//` comments do not appear in tooltips.

## Dependencies

**Search for an existing library before writing your own implementation.**
Formats, protocols and codecs are full of details that only show up against
real data, and a maintained library has already met them. Check the BCL first:
tar (`System.Formats.Tar`), zip, JSON and vectorized span operations are all
built in. .NET 10 has no zstd, which is why ZstdSharp.Port is here.

**Read the library's source before adding it, and judge it on that.** Download
counts say nothing about quality. Look for readable sources, comments that
explain intent, and an API that fits the way the code here is written. Code
that nobody can read cannot be maintained and may hide bugs nobody can find.
Do not depend on it.

If nothing of good quality exists, say so and propose writing an
implementation here, rather than pulling in the least bad option.

Do not add a second library for a job a package already referenced can do. A
thin wrapper that shapes an installed library's API to what the code needs is
better than a new dependency.

## Things that will bite you

- **Changing a route's request or response is a two-repository change.** The
  frontend has its own copy of the protocol in `SaveParserBackendService.ts`,
  `GAME_SAVE_ENTRY_NAMES` and its save archive code. Say what the frontend
  needs changed; do not assume it will follow.
- **`Config.Load` rejects any config whose `Version` differs from
  `ActualVersion`.** New config properties with defaults don't need a version
  bump, because a missing key keeps the default. Bump the version only for
  changes that old files cannot express.
- **`PathHelper.CreateTempFile` deletes the file right after opening it.** The
  open stream keeps the data alive until it is disposed; do not reopen it by
  path.
- **`SaveParsingOperation.StartAsync` is `async void`.** Every exception must
  be caught inside it, or it will crash the process. Errors are recorded on
  the metadata row with `SetMetadataError`.
- `Program.IsDebug` is always true for now, so debug logging is always on.