implemented save file compression in browser

2026-09-23 17:26:40 +02:00
parent 54ddc11b44
commit a022dd00dc
25 changed files with 1331 additions and 142 deletions
@@ -0,0 +1,73 @@
+# Compression levels
+
+Reference for `saveUpload.zstdCompressionLevel` in
+[src/config/AppConfig.ts](src/config/AppConfig.ts). If compression ever turns
+out to be too slow or too memory hungry on some device, pick a different level
+from the table below rather than measuring again.
+
+The level only affects the uploader. Decompression on the parser backend costs
+the same at every level.
+
+## What was measured
+
+A real Europa Universalis IV save, as the game wrote it:
+
+|               |                                                       |
+| ------------- | ----------------------------------------------------- |
+| File          | `Ibevar1475_01_02save.eu4`, a zip archive             |
+| Size on disk  | **11,502,739 bytes** (11.0 MB)                        |
+| Entries       | `gamestate` (87,188,760 bytes), `meta`, `ai`          |
+| Input to zstd | **87,342,592 bytes** — the tar of those three entries |
+
+So the ratios below are against the original 11.5 MB game archive, and zstd
+itself is working on the 87 MB of uncompressed save text inside it.
+
+Each level ran through the project's own `packSave` in a separate process. The
+heap columns are the WebAssembly linear memory the encoder and the decoder grow
+to, read from the instance itself — that is the memory a browser tab spends,
+and it is what to look at when a device runs out. The machine was a desktop;
+**a phone is several times slower**, so treat the times as a lower bound.
+
+Every level round tripped to the same 87,342,592 byte tar, so each row is
+correct output rather than merely fast output.
+
+## The table
+
+|  level |    compressed |     ratio |   compress | decompress | encoder heap | decoder heap | window |
+| -----: | ------------: | --------: | ---------: | ---------: | -----------: | -----------: | -----: |
+|      1 |     6,698,281 |     1.72x |     0.54 s |     0.28 s |         8 MB |         8 MB |   1 MB |
+|      2 |     6,600,451 |     1.74x |     0.54 s |     0.29 s |         8 MB |         8 MB |   1 MB |
+|      3 |     6,094,116 |     1.89x |     0.57 s |     0.28 s |         8 MB |         8 MB |   2 MB |
+|      4 |     6,082,362 |     1.89x |     0.61 s |     0.28 s |         8 MB |         8 MB |   2 MB |
+|      5 |     5,280,090 |     2.18x |     0.92 s |     0.28 s |         8 MB |         8 MB |   2 MB |
+|      6 |     4,943,366 |     2.33x |     1.13 s |     0.27 s |         8 MB |         8 MB |   2 MB |
+|      7 |     4,738,515 |     2.43x |     1.27 s |     0.27 s |         9 MB |         8 MB |   2 MB |
+|      8 |     4,514,853 |     2.55x |     1.52 s |     0.28 s |         9 MB |         8 MB |   2 MB |
+|      9 |     4,445,368 |     2.59x |     1.51 s |     0.26 s |        16 MB |         8 MB |   4 MB |
+|     10 |     4,280,567 |     2.69x |     1.74 s |     0.27 s |        26 MB |         8 MB |   4 MB |
+| **11** | **4,157,684** | **2.77x** | **2.44 s** | **0.32 s** |    **26 MB** |     **8 MB** |   4 MB |
+|     12 |     4,155,668 |     2.77x |     2.39 s |     0.27 s |        46 MB |         8 MB |   4 MB |
+|     13 |     4,257,141 |     2.70x |     3.76 s |     0.27 s |        38 MB |         8 MB |   4 MB |
+|     14 |     4,108,558 |     2.80x |     5.39 s |     0.28 s |        54 MB |         8 MB |   4 MB |
+|     15 |     3,968,262 |     2.90x |     8.13 s |     0.33 s |        70 MB |         8 MB |   4 MB |
+|     16 |     3,862,217 |     2.98x |    11.63 s |     0.26 s |        38 MB |         8 MB |   4 MB |
+|     17 |     3,733,198 |     3.08x |    12.39 s |     0.26 s |        58 MB |        10 MB |   8 MB |
+|     18 |     3,705,141 |     3.10x |    15.11 s |     0.26 s |        59 MB |        10 MB |   8 MB |
+|     19 |     3,449,475 |     3.33x |    27.78 s |     0.28 s |        91 MB |        10 MB |   8 MB |
+|     20 |     3,344,642 |     3.44x |    28.24 s |     0.27 s |       195 MB |        34 MB |  32 MB |
+|     21 |     3,285,836 |     3.50x |    33.91 s |     0.30 s |       387 MB |        66 MB |  64 MB |
+|     22 |     3,245,980 |     3.54x |    49.02 s |     0.30 s |       835 MB |       130 MB | 128 MB |
+
+## Picking a level
+
+**11 is the default**, because it is where the curve flattens. It gives good compression in reasonable time.
+
+If there will be troubles with this level in the future, just change it in config.
+
+## Reproducing
+
+There is no script in the repository for this; it was a one-off measurement.
+To repeat it, stream a save through `packSave` once per level in a separate
+process, and read `memory.buffer.byteLength` off the encoder and decoder
+instances of [@tootallnate/zstd-wasm](https://www.npmjs.com/package/@tootallnate/zstd-wasm)
+for the heap figures.