ParadoxSaveParser.Benchmarks

Compares the query engine of this repo (SearchExpression + SaveParserEU4) against regex engines and jq on one realistic task:

extract the technology block (adm/dip/mil level) of every country from an EU4 save

engine	query
this repo	`countries.*.technology`
jq	`.countries \| map_values(.technology)`
.NET `Regex` / PCRE.NET	see `Extractors.FlatPattern` / `Extractors.PathAwarePattern`
`Utf8JsonReader` / `JsonDocument`	hand written navigation

Corpus

gamestate — a real 109 MB EU4 1.37 save (text format, 6.0M lines, 2803 country blocks, 1870 of which have a technology block).
gamestate.min.json — 92 MB compact JSON twin, produced from the save by this project's own parser. It exists because jq and the System.Text.Json engines cannot read the Paradox format at all.

The save is not in the repository. Point PSP_BENCH_DATA at the directory holding it, or drop it in ParadoxSaveParser.CLI/bin/Debug/net10.0/ (the default location).

Usage

dotnet build ParadoxSaveParser.Benchmarks -c Release
cd ParadoxSaveParser.Benchmarks/bin/Release/net10.0

./ParadoxSaveParser.Benchmarks.exe prepare   # build the JSON twin (once)
./ParadoxSaveParser.Benchmarks.exe verify    # one run of every engine + correctness check
./ParadoxSaveParser.Benchmarks.exe --filter '*'   # full BenchmarkDotNet run

Every engine returns the checksum countries * 1_000_000 + sum_of_tech_levels, so verify shows immediately when an engine finds something different from the others.

Fairness notes

All in-process engines read from memory; file I/O is excluded. jq is an external process, so its numbers include process start (measured separately by the jq startup only benchmark) and reading the JSON from the OS page cache.
The regex engines work on the save text, which is 109 MB; jq works on 92 MB of JSON.
jq is the only engine here that is a general query language: it can do arithmetic, filtering and reshaping, which neither the regexes nor SearchExpression can.

Results

AMD Ryzen 7 5700X, 16 logical cores, Windows 10 21H2, .NET 10.0.12, jq 1.8.2, 109 MB save / 92 MB JSON twin, BenchmarkDotNet RunStrategy.Monitoring, 5 iterations.

engine	mean	vs this project	allocated	correct?
.NET Regex flat, source generated	14.8 ms	0.02x	1.2 MB	path-blind
.NET Regex flat, compiled	17.0 ms	0.03x	1.2 MB	path-blind
.NET Regex flat, interpreted	19.3 ms	0.03x	1.2 MB	path-blind
PCRE.NET flat, JIT compiled	97.4 ms	0.15x	0.9 MB	path-blind
.NET Regex balanced block + flat	112.3 ms	0.17x	224 MB	yes
.NET Regex path aware, compiled	116.2 ms	0.18x	1.4 MB	1869/1870
PCRE.NET path aware, JIT compiled	122.9 ms	0.19x	0.9 MB	1869/1870
Utf8JsonReader over JSON twin	155.7 ms	0.24x	0 B	yes
JsonDocument over JSON twin	321.4 ms	0.50x	0 B (native)	yes
.NET Regex path aware, NonBacktracking	444.5 ms	0.69x	32 MB	1869/1870
SearchExpression (this project)	645.1 ms	1.00x	1.6 MB	yes
Full parse, then select	1831 ms	2.84x	1153 MB	yes
jq `.countries \| map_values(.technology)`	4340 ms	6.73x	n/a	yes
jq `empty` (parse the file, emit nothing)	4227 ms	6.55x	n/a	n/a
jq `--stream`	17490 ms	27.1x	n/a	n/a
jq process startup only	4.2 ms	0.01x	n/a	n/a

Reading the table

jq is ~6.7x slower than this parser and 97% of that time is JSON parsing, not the query: jq empty on the same file costs 4227 ms of the 4340 ms. Process startup is negligible (4 ms). jq's --stream mode, often recommended for large inputs, is 4x slower still. jq also needs the data converted to JSON first, which this parser has to do anyway — so end to end jq is strictly more expensive here.
The regexes are 5-40x faster, but they are not doing the same job. A regex never parses the structure; it scans bytes for a literal and validates a short window around it. Regex with a literal prefix (technology={) is vectorized, so 109 MB is scanned at several GB/s. That speed is real and the cost is real too: nothing verifies that the hit is under countries, at the right depth, or belongs to the tag matched before it.
Path-aware regexes lose both the speed and the correctness. Forcing the tag into the pattern costs 7x (17 ms -> 116 ms) and still returns 1869 of 1870 countries: the tag class [A-Z0-9]{3} silently drops EU4's --- pseudo-country. Pairing survives here only by luck of the file layout — all 933 tech-less country blocks happen to be grouped at the end of the save, so the lazy gap never runs across one. A save that interleaves them would make the regex report a technology block under the wrong country tag with no error.
RegexOptions.NonBacktracking guarantees linear time but is 26x slower than the compiled backtracking engine on this pattern and allocates 32 MB.
PCRE.NET (the most used non-BCL regex engine in .NET, ~460k downloads) is 6x slower than System.Text.RegularExpressions on the flat pattern, because .NET's vectorized literal prefix search beats PCRE2's JIT here. On the path-aware pattern the two are equal. There is no reason to leave the BCL engine for this workload.
The query is what makes this parser fast, not the parsing. Same parser, same file: 645 ms with countries.*.technology, 1831 ms and 1.1 GB allocated without a query. The search expression prunes ~65% of the work and 99.9% of the allocations.
Against a JSON reader on equivalent data, the parser is 4x slower than Utf8JsonReader and 2x slower than JsonDocument. Those numbers exclude the pdx -> JSON conversion (~4.7 s), so they are a ceiling for the format, not a usable alternative.

Where this parser's time goes

645 ms for 109 MB is ~170 MB/s, or ~13 cycles per input byte. The lexer reads the save one byte at a time through Stream.ReadByte() (SaveParserEU4.LexTextSave), which is a virtual call plus bounds check per byte, and appends char by char into a StringBuilder. Reading into a byte[] buffer and scanning it with ReadOnlySpan<byte>.IndexOfAny would be the first thing to try — the regex numbers above show what the same hardware does when it scans a span instead of a stream.

6.3 KiB Raw Blame History