ParadoxSaveParser/AGENTS.md

AGENTS.md

Guidance for AI coding agents working in this repository.

Project

A parser for Paradox game save files, and the backend of a save analytics website. Only EU4 is supported so far.

• .NET 10, C# latest, nullable reference types enabled everywhere
• DTLib / DTLib.Web for filesystem helpers, logging and the HTTP server
• sqlite-net-pcl for the upload database, ZstdSharp.Port for zstd

Projects in the solution (ParadoxSaveParser.sln):

• ParadoxSaveParser.Lib: the parser itself (Tokenizer, SaveParserEU4, SearchExpression/)
• ParadoxSaveParser.Lib.Tests: NUnit tests for the parser
• ParadoxSaveParser.CLI: command line tool, with an interactive mode
• ParadoxSaveParser.WebAPI: HTTP backend for the website
• ParadoxSaveParser.Benchmarks: BenchmarkDotNet comparison against regex engines and jq, see its README

Architecture

Parser

SaveParserEU4 reads the plain text game state and produces nested Dictionary<string, object> values. It takes a compiled SearchExpression, a small path query such as countries.*.technology. The query is what makes the parser fast. Tokenizer.SkipBlock throws away a rejected {...} block by scanning raw bytes for brace depth, so rejected data is never turned into tokens or strings. Full parse: 1.2 s and 962 MB allocated; the same file with a query: 50 ms and 1.5 MB. Keep any parser change from bringing back per-byte virtual calls or eager string decoding. Check with the benchmarks before and after.

WebAPI

The frontend is a separate Next.js project in its own repository. It calls this service through a same-origin proxy and expects it at http://127.0.0.1:5226 (BaseUrl in config.json). Routes are documented in ParadoxSaveParser.WebAPI/README.md; keep that file in sync when a route or a response shape changes, because the frontend is written against it.

The flow is upload, poll, fetch:

POST /uploadSave  ->  store blob, status "uploaded"  ->  SaveParsingOperation
                                                          (background: parsing,
                                                           saving_results, done)
GET  /getSaveStatus  (polled by the frontend until status is "done" or "error")
GET  /getSaveData    (filtered parse result as zstd-compressed JSON)
GET  /getSaveFile    (the uploaded blob, unchanged)

Save files on the wire

The server is a cheap two core machine with little disk space, so the browser re-compresses every save before uploading it:

plain text save  ->  tar { "gamestate" }               -> zstd -> upload
archived save    ->  tar { "meta", "gamestate", ... }  -> zstd -> upload

• .tar.zst is the only accepted format. Zip handling was removed on purpose; do not add it back. The original type and name come in as query params (originalFileType=text|zip, originalFileName), and they are stored so the browser can rebuild the original file from /getSaveFile. A rebuilt EU4 zip was checked entry by entry against the original and is byte identical.
• The blob is stored byte-for-byte. Never re-encode it on the server.
• The game data entry name comes from GameSaveEntryNames.Map in Game.cs. That map mirrors the frontend's GAME_SAVE_ENTRY_NAMES. Do not hard-code "gamestate".
• All tar and zstd handling lives in SaveArchive.cs, which streams through DecompressionStream and TarReader. The whole archive is never loaded into memory.
• Upload limits come from the SaveUpload config section, in MiB (MaxSizeCompressedMegabytes, MaxSizePlainTextMegabytes). The code uses their byte equivalents, the …Bytes properties, which are not saved to the config. The compressed limit is checked against Content-Length and then again while the body is copied, because chunked requests have no length. The plain text limit is checked against the tar entry size once the upload has finished. Every rejected upload deletes its database row and file.

Runtime files

The WebAPI creates everything relative to its working directory: config.json (written with defaults if missing), database.sqlite, logs/, temp/ (emptied on start), and public/data/{saves,parsed}/. When running it for a test, start it with TEMP/ as the working directory, so none of this lands in the project folders.

Current status

• Only EU4 works. Game.Stellaris exists and has an entry name, but it has no ISaveDataFilter and no parser, so uploads for it are rejected with 400. SaveParsingOperation is EU4 specific.
• Nothing is deployed and there is no data worth keeping. Do not write database migrations or backward compatibility code for old layouts; change the schema directly.
• Open items are in TODO.md: re-parsing when the query changes, automatic database cleanup.
• Parsed results are stored as <id>.parsed.json.zstd and /getSaveData serves that file unchanged as application/zstd. The client decompresses it; the server never does.
• The WebAPI exits with code 1 after Ctrl+C even though it logs "server stopped". This has not been investigated yet.

Tools and files

Prefer the editor's own tools over shell commands that do the same thing. Reading, editing, writing and searching files have dedicated tools. Using cat, sed, grep or a throwaway script instead does the same work, spends more tokens, and may need its own approval each time. Use the shell for what only the shell can do: building, running tests, git, curl, tar, zstd, or running the program itself.

Work inside the project directory. For temporary files (scratch scripts, intermediate output, test data, a running WebAPI's working directory), create a TEMP/ directory in the project root and keep everything there. It is gitignored. Delete that directory once the work is done. Ask before reading or writing anything outside the project directory, including /tmp and the home folder. A save file the user names explicitly may be read, but never modified.

Terminal

On Windows, prefer bash.exe, but only a real MSYS2, Cygwin, or MinGW/Git Bash installation, for example D:\msys2\usr\bin\bash.exe.

Never use C:\Windows\System32\bash.exe. Despite its name, it is not a POSIX bash but the launcher stub for WSL. Do not start WSL to run ordinary commands.

If no MSYS/Cygwin/MinGW bash is installed, use PowerShell as the terminal. PowerShell is the fallback, never WSL.

If bash reports command not found for dotnet, the cause is PATH, not a missing install. MSYS2 sets MSYS2_PATH_TYPE to minimal by default, which drops the Windows PATH. Setting it to inherit fixes it.

Commands

dotnet build ParadoxSaveParser.sln        # build everything
dotnet test ParadoxSaveParser.Lib.Tests   # parser tests
dotnet run --project ParadoxSaveParser.WebAPI   # run the backend (mind the working directory)
dotnet build ParadoxSaveParser.Benchmarks -c Release   # see its README for running

A one-off C# check can be a .NET 10 file-based app in TEMP/ (dotnet run -c Release TEMP/check.cs); it doesn't need a project.

Checking your work

Always build the solution after changing code. dotnet build ParadoxSaveParser.sln must finish with zero errors and no new warnings. Three warnings are known and come from sqlite-net-pcl's SQLitePCLRaw.lib.e_sqlite3 dependency (NU1903 twice, NETSDK1206 once). Everything else is yours.

Never run tests or benchmarks unless the user asks for it. This is a strict ban. It covers dotnet test, the benchmarks, and starting the WebAPI to exercise routes with curl. When a change needs checking that way, say what should be run and wait for the user to ask. When they do:

• dotnet test ParadoxSaveParser.Lib.Tests must pass completely.
• For a WebAPI check, start the server from TEMP/, exercise the route with curl including the error cases, then stop the server.
• For a parser performance change, run the relevant benchmark before and after, and report both numbers.

Version control

This project is managed with git.

Read-only git commands are fine to use without asking. You can run git status, git log, git diff, git show, git blame and similar inspection commands freely.

Never run a git command that changes the repository without the user's explicit permission. This is a strict ban. It covers every command that writes to the history, the index, or the working tree, including add, commit, checkout, restore, reset, stash, branch, merge, rebase, push, and rm.

Ask first, and wait for a clear yes before running such a command.

Conventions

Match the surrounding code exactly:

• File-scoped namespaces, 4 spaces, _camelCase private fields, PascalCase for everything else.
• SaveFileMetadata properties are lowercase on purpose. They are the JSON wire format that the frontend reads, and the sqlite column names too. Do not rename them to PascalCase. Enums on the wire serialize in lowercase through LowerCaseEnumConverter<T> (SavingResults becomes saving_results).
• Program.cs declares global usings, which make File, Directory and Path refer to DTLib.Filesystem, not System.IO. Paths are IOPath. When a System.IO member is needed, write it out in full, for example System.IO.Path.GetFileName.
• Errors from route handlers go through ReturnHelper.ResponseError with an ErrorMessage and the right status code. Query values come from RequestHelper.GetQueryValue. Do not write responses by hand.
• Line endings: most .cs files are CRLF with a UTF-8 BOM. A new file written with LF mixes endings once the IDE touches it. Match the files next to it, or convert with unix2dos -m.

Comment the code you write. Human programmers who did not write this code will read it, and a wall of uncommented agent-written code is expensive to review. Keep comments simple, preferably one line, placed above the code they describe, and in plain language rather than in terms of the types. Say why the code does what it does: what is non-obvious, and what would break if it were written the obvious way. Leave out what the code already states plainly. A tricky constant, a workaround for someone else's bug, or an ordering that matters always deserves a line.

Write doc comments as XML doc comments (///), which the IDE shows on hover. Plain // comments do not appear in tooltips.

Dependencies

Search for an existing library before writing your own implementation. Formats, protocols and codecs are full of details that only show up against real data, and a maintained library has already met them. Check the BCL first: tar (System.Formats.Tar), zip, JSON and vectorized span operations are all built in. .NET 10 has no zstd, which is why ZstdSharp.Port is here.

Read the library's source before adding it, and judge it on that. Download counts say nothing about quality. Look for readable sources, comments that explain intent, and an API that fits the way the code here is written. Code that nobody can read cannot be maintained and may hide bugs nobody can find. Do not depend on it.

If nothing of good quality exists, say so and propose writing an implementation here, rather than pulling in the least bad option.

Do not add a second library for a job a package already referenced can do. A thin wrapper that shapes an installed library's API to what the code needs is better than a new dependency.

Things that will bite you

• Changing a route's request or response is a two-repository change. The frontend has its own copy of the protocol in SaveParserBackendService.ts, GAME_SAVE_ENTRY_NAMES and its save archive code. Say what the frontend needs changed; do not assume it will follow.
• Config.Load rejects any config whose Version differs from ActualVersion. New config properties with defaults don't need a version bump, because a missing key keeps the default. Bump the version only for changes that old files cannot express.
• PathHelper.CreateTempFile deletes the file right after opening it. The open stream keeps the data alive until it is disposed; do not reopen it by path.
• SaveParsingOperation.StartAsync is async void. Every exception must be caught inside it, or it will crash the process. Errors are recorded on the metadata row with SetMetadataError.
• Program.IsDebug is always true for now, so debug logging is always on.