# Profiling Binary Format The profiling module includes a binary file format for storing sampling profiler data. This document describes the format's structure and the design decisions behind it. The implementation is in [`Modules/_remote_debugging/binary_io_writer.c`](../Modules/_remote_debugging/binary_io_writer.c) and [`Modules/_remote_debugging/binary_io_reader.c`](../Modules/_remote_debugging/binary_io_reader.c), with declarations in [`Modules/_remote_debugging/binary_io.h`](../Modules/_remote_debugging/binary_io.h). ## Overview The sampling profiler can generate enormous amounts of data. A typical profiling session sampling at 1000 Hz for 60 seconds produces 60,000 samples. Each sample contains a full call stack, often 20-50 frames deep, and each frame includes a filename, function name, and line number. In a text-based format like collapsed stacks, this would mean repeating the same long file paths and function names thousands of times. The binary format addresses this through two key strategies: 1. **Deduplication**: Strings and frames are stored once in lookup tables, then referenced by small integer indices. A 100-character file path that appears in 50,000 samples is stored once, not 50,000 times. 2. **Compact encoding**: Variable-length integers (varints) encode small values in fewer bytes. Since most indices are small (under 128), they typically need only one byte instead of four. Together with optional zstd compression, these techniques reduce file sizes by 10-50x compared to text formats while also enabling faster I/O. ## File Layout The file consists of five required sections and one optional extension: ``` +------------------+ Offset 0 | Header | 64 bytes (fixed) +------------------+ Offset 64 | | | Sample Data | Blocks of columns (optionally compressed) | | +------------------+ string_table_offset | String Table | Variable size +------------------+ frame_table_offset | Frame Table | Variable size +------------------+ file_size - 88 (when stats are present) | Profile Stats | 56 bytes (optional) +------------------+ file_size - 32 | Footer | 32 bytes (fixed) +------------------+ file_size ``` The layout is designed for streaming writes during profiling. The profiler cannot know in advance how many unique strings or frames will be encountered, so these tables must be built incrementally and written at the end. The header comes first so readers can quickly validate the file and locate the metadata tables. The sample data follows immediately, allowing the writer to write each block of samples as soon as it is full without buffering the entire dataset in memory. The string and frame tables are placed after sample data because they grow as new unique entries are discovered during profiling. By deferring their output until finalization, the writer avoids the complexity of reserving space or rewriting portions of the file. The footer at the end contains counts needed to allocate arrays before parsing the tables. Placing it at a fixed offset from the end (rather than at a variable offset recorded in the header) means readers can locate it with a single seek to `file_size - 32`, without first reading the header. ## Header ``` Offset Size Type Description +--------+------+---------+----------------------------------------+ | 0 | 4 | uint32 | Magic number (0x54414348 = "TACH") | | 4 | 4 | uint32 | Format version | | 8 | 4 | bytes | Python version (major, minor, micro, | | | | | reserved) | | 12 | 8 | uint64 | Start timestamp (microseconds) | | 20 | 8 | uint64 | Sample interval (microseconds) | | 28 | 8 | uint64 | Total sample count | | 36 | 4 | uint32 | Thread count | | 40 | 8 | uint64 | String table offset | | 48 | 8 | uint64 | Frame table offset | | 56 | 4 | uint32 | Compression type (0=none, 1=zstd) | | 60 | 4 | uint32 | Profiling configuration bit field | +--------+------+---------+----------------------------------------+ ``` The low three configuration bits contain the `_remote_debugging.PROFILING_MODE_*` value plus one. Zero means that the mode was not recorded, which is also the value in binaries written before this field was defined. Bit 3 indicates that capture features are known; when set, bits 4 through 8 respectively record `--all-threads`, `--native`, GC frames, `--opcodes`, and `--blocking`. Remaining bits are reserved for future capture features. The magic number `0x54414348` ("TACH" for Tachyon) identifies the file format and also serves as an **endianness marker**. When read on a system with different byte order than the writer, it appears as `0x48434154`. The reader uses this to detect cross-endian files and automatically byte-swap all multi-byte integer fields. The Python version field records the major, minor, and micro version numbers of the Python interpreter that generated the file. This allows analysis tools to detect version mismatches when replaying data collected on a different Python version, which may have different internal structures or behaviors. The header is written as zeros initially, then overwritten with actual values during finalization. This requires the output stream to be seekable, which is acceptable since the format targets regular files rather than pipes or network streams. ## Sample Data Sample data begins at offset 64 and extends to `string_table_offset`. Samples are grouped into blocks of at most 1 MiB of decoded data. Within a block, samples use delta compression to minimize redundancy when consecutive samples from the same thread have identical or similar call stacks. Each block can be decoded on its own once the string and frame tables are loaded, so a reader only needs to decode one block at a time. ### Block Format A block stores values of the same kind together, in six columns: ``` +------------------------+ | Block header | 21 to 88 bytes +------------------------+ | STREAM_TABLE | (thread_id, interpreter_id) pairs +------------------------+ | SAMPLE_STREAM_IDS | One stream index per sample +------------------------+ | TIMESTAMP_RESIDUALS | One timestamp residual per sample +------------------------+ | STATUS_BITMAPS | One bitmap per varying status flag +------------------------+ | STACK_INSTRUCTIONS | Stack encodings +------------------------+ | FRAME_IDS | Frame table indices +------------------------+ ``` **Block header:** ``` +---------------------+-----------+----------------------------------------+ | magic | 4 bytes | "TBLK" | | sample_count | varint | Number of samples in this block | | base_timestamp | varint | Timestamp of the first sample | | frame_id_width | 1 byte | Bytes per frame index (1 to 4) | | common_status_flags | 1 byte | Status bits set in every sample | | status_bitmap_flags | 1 byte | Status bits stored as bitmaps | | columns | varies | One column descriptor per column | +---------------------+-----------+----------------------------------------+ ``` **Column descriptor:** ``` +---------------------+-----------+----------------------------------------+ | decoded_size | varint | Size of the decoded column | | encoding | 1 byte | 0=raw, 1=zstd, 2=constant | | stored_size | varint | Compressed size (zstd only) | +---------------------+-----------+----------------------------------------+ ``` A raw column is stored as is, and a constant column is a single byte repeated `decoded_size` times, so neither needs a stored size. The number of streams and frame indices in a block follows from the decoded sizes of `STREAM_TABLE` and `FRAME_IDS`. A block holds at most 4,096 streams. There is no separate limit on the number of samples: the stream index and timestamp columns need at least one byte per sample, so the 1 MiB limit already bounds it. ### Stack Encoding Types Each thread's stack is reconstructed from the previous stack of the same thread in the block. The first sample of a thread in a block starts from an empty stack. `STACK_INSTRUCTIONS` holds one of these encodings for every sample not covered by a `REPEAT`: | Code | Name | Description | |------|------|-------------| | 0x00 | REPEAT | RLE: identical stack repeated N times | | 0x01 | REPLACE_SUFFIX | Keeps N outer frames of the previous stack, adds M new frames | | 0x02 | REUSE | Complete stack stored earlier in the block | Stacks hold at most 256 frames. The writer keeps the innermost 256 frames of deeper stacks. ### Stack Encoding Formats All fields are varints. Stacks are written outermost frame first. **REPEAT (0x00) - Run-Length Encoded Identical Stacks:** ``` +-----------------+-----------+----------------------------------------+ | encoding | varint | 0x00 (REPEAT) | | count | varint | Number of samples in this RLE group | +-----------------+-----------+----------------------------------------+ ``` The stack is inherited from this thread's previous sample, for this sample and the thread's next `count - 1` samples. Samples from other threads can appear in between: the reader keeps a remaining count for each thread and reads no encoding for that thread until the count reaches zero. Each sample in the group keeps its own timestamp and status in the other columns. **REPLACE_SUFFIX (0x01) - Shared Outer Frames:** ``` +-----------------+-----------+----------------------------------------+ | encoding | varint | 0x01 (REPLACE_SUFFIX) | | keep_count | varint | Outer frames kept from previous stack | | new_count | varint | Inner frames read from FRAME_IDS | +-----------------+-----------+----------------------------------------+ ``` Used when a call or return changed the inner part of the stack: the outer frames are kept, and `new_count` new frames are read from `FRAME_IDS`. A complete stack, such as the first sample of a thread, uses `keep_count` 0. **REUSE (0x02) - Repeated Complete Stack:** ``` +-----------------+-----------+----------------------------------------+ | encoding | varint | 0x02 (REUSE) | | offset | varint | Index of the first frame in FRAME_IDS | | count | varint | Number of frames in the stack | +-----------------+-----------+----------------------------------------+ ``` Used when a complete stack was already written earlier in the same block, for example by another thread running the same code. The frames are copied from `FRAME_IDS` instead of being stored again. For example, two threads produce four samples: | Sample | Thread | Stack | Encoding | |--------|--------|----------------|----------------------| | 0 | A | `[4, 21, 37]` | `REPLACE_SUFFIX 0 3` | | 1 | B | `[4, 21, 37]` | `REUSE 0 3` | | 2 | A | `[4, 21, 52]` | `REPLACE_SUFFIX 2 1` | | 3 | A | `[4, 21, 52]` | `REPEAT 1` | `FRAME_IDS` holds only `[4, 21, 37, 52]`. ### Thread and Interpreter Identification Each `(thread_id, interpreter_id)` pair used in a block is stored once in `STREAM_TABLE`, as an 8-byte thread ID followed by a 4-byte interpreter ID. `SAMPLE_STREAM_IDS` then holds one varint per sample: the index of its pair in the table. Samples are kept in arrival order, so samples from different threads are interleaved. Thread IDs are 64-bit values that can be large (memory addresses on some platforms) and vary unpredictably. Using a fixed 8-byte encoding avoids the overhead of varint encoding for large values and simplifies parsing since the reader knows exactly where each field begins. The interpreter ID identifies which Python sub-interpreter the thread belongs to, allowing analysis tools to separate activity across interpreters in processes using multiple sub-interpreters. ### Status Byte The status byte is a bitfield encoding thread state at sample time: | Bit | Flag | Meaning | |-----|-----------------------|--------------------------------------------| | 0 | THREAD_STATUS_HAS_GIL | Thread holds the GIL (Global Interpreter Lock) | | 1 | THREAD_STATUS_ON_CPU | Thread is actively running on a CPU core | | 2 | THREAD_STATUS_UNKNOWN | Thread state could not be determined | | 3 | THREAD_STATUS_GIL_REQUESTED | Thread is waiting to acquire the GIL | | 4 | THREAD_STATUS_HAS_EXCEPTION | Thread has a pending exception | | 5 | THREAD_STATUS_MAIN_THREAD | Thread is the interpreter's main thread | Multiple flags can be set simultaneously (e.g., a thread can hold the GIL while also running on CPU). Analysis tools use these to filter samples or visualize thread states over time. Most flags rarely change within a block, so the status bytes are stored as bit planes. Bits set in every sample are recorded once in `common_status_flags`. Each bit in `status_bitmap_flags` has a bitmap of `ceil(sample_count / 8)` bytes in `STATUS_BITMAPS`, in increasing bit order, with sample `i` in bit `i % 8` of byte `i / 8`. Bits that never occur take no space. The reader rebuilds each status byte from `common_status_flags` and the sample's bit in each bitmap. ### Timestamp Delta Encoding Timestamps use delta encoding rather than absolute values. Absolute timestamps in microseconds require 8 bytes each, but consecutive samples from the same thread are typically separated by the sampling interval (e.g., 1000 microseconds), so each timestamp is expected to be the thread's previous timestamp plus the interval, and only the difference from that expected timestamp is stored. The writer tracks the previous timestamp for each thread separately. The expected timestamp for a thread's first sample in a block is the block's `base_timestamp`. This per-thread tracking is necessary because samples are interleaved across threads in arrival order, not grouped by thread. Each residual is a signed 64-bit zigzag varint. A sample taken exactly on schedule has a residual of 0 and takes one byte, while timestamps are still preserved exactly. ### Frame Indexing Each frame in a call stack is represented by an index into the frame table rather than inline data. This provides massive space savings because call stacks are highly repetitive: the same function appears in many samples (hot functions), call stacks often share common prefixes (main -> app -> handler -> ...), and recursive functions create repeated frame sequences. A frame index is typically 1-2 bytes. Inline frame data would be 20-200+ bytes (two strings plus a line number). For a profile with 100,000 samples averaging 30 frames each, this reduces frame data from potentially gigabytes to tens of megabytes. `FRAME_IDS` stores the indices read by `REPLACE_SUFFIX`, using the smallest byte width that fits the largest index in the block. The bytes are split into planes, low bytes first, so similar bytes sit together for compression: ``` indices: 0x1234 0x5678 0x9abc low-byte plane: 0x34 0x78 0xbc high-byte plane: 0x12 0x56 0x9a ``` Frame indices are written outermost-first (the outermost frame has index 0 in the array). This ordering works well with delta compression: function calls typically change frames at the end of the array, while shared frames remain at the start. ## String Table The string table stores deduplicated UTF-8 strings (filenames and function names). It begins at `string_table_offset` and contains entries in order of their assignment during writing: ``` +----------------+ | length: varint | | data: bytes | +----------------+ (repeated for each string) ``` Strings are stored in the order they were first encountered during writing. The first unique filename gets index 0, the second gets index 1, and so on. Length-prefixing (rather than null-termination) allows strings containing null bytes and enables readers to allocate exact-sized buffers. The varint length encoding means short strings (under 128 bytes) need only one length byte. ## Frame Table The frame table stores deduplicated frame entries with full source position information and bytecode opcode: ``` +----------------------------+ | filename_idx: varint | | funcname_idx: varint | | lineno: svarint | | end_lineno_delta: svarint | | column: svarint | | end_column_delta: svarint | | opcode: u8 | +----------------------------+ (repeated for each frame) ``` ### Field Definitions | Field | Type | Description | |------------------|---------------|----------------------------------------------------------| | filename_idx | varint | Index into string table for file name | | funcname_idx | varint | Index into string table for function name | | lineno | zigzag varint | Start line number (-1 for synthetic frames) | | end_lineno_delta | zigzag varint | Delta from lineno (end_lineno = lineno + delta) | | column | zigzag varint | Start column offset in UTF-8 bytes (-1 if not available) | | end_column_delta | zigzag varint | Delta from column (end_column = column + delta) | | opcode | u8 | Python bytecode opcode (0-254) or 255 for None | ### Delta Encoding Position end values use delta encoding for efficiency: - `end_lineno = lineno + end_lineno_delta` - `end_column = column + end_column_delta` Typical values: - `end_lineno_delta`: Usually 0 (single-line expressions) → encodes to 1 byte - `end_column_delta`: Usually 5-20 (expression width) → encodes to 1 byte This saves ~1-2 bytes per frame compared to absolute encoding. When the base value (lineno or column) is -1 (not available), the delta is stored as 0 and the reconstructed value is -1. ### Sentinel Values - `opcode = 255`: No opcode captured - `lineno = -1`: Synthetic frame (no source location) - `column = -1`: Column offset not available ### Deduplication Each unique (filename, funcname, lineno, end_lineno, column, end_column, opcode) combination gets one entry. This enables instruction-level profiling where multiple bytecode instructions on the same line can be distinguished. Strings and frames are deduplicated separately because they have different cardinalities and reference patterns. A codebase might have hundreds of unique source files but thousands of unique functions. Many functions share the same filename, so storing the filename index in each frame entry (rather than the full string) provides an additional layer of deduplication. A frame entry is typically 7-9 bytes rather than two full strings plus location data. ### Size Analysis Typical frame size with delta encoding: - file_idx: 1-2 bytes - func_idx: 1-2 bytes - lineno: 1-2 bytes - end_lineno_delta: 1 byte (usually 0) - column: 1 byte (usually < 64) - end_column_delta: 1 byte (usually < 64) - opcode: 1 byte **Total: ~7-9 bytes per frame** Line numbers and columns use signed varint (zigzag encoding) to handle sentinel values efficiently. Synthetic frames—generated frames that don't correspond directly to Python source code, such as C extension boundaries or internal interpreter frames—use -1 to indicate the absence of a source location. Zigzag encoding ensures these small negative values encode efficiently (−1 becomes 1, which is one byte) rather than requiring the maximum varint length. ## Profile Statistics New files can store measured duration, sampling rate, error rate, and missed sample percentage in an optional 56-byte extension immediately before the footer. Older readers ignore these bytes after parsing the declared number of frame-table entries, and newer readers treat a missing extension as unavailable statistics. ``` Offset Size Type Description +--------+------+---------+----------------------------------------+ | 0 | 8 | double | Measured duration (seconds) | | 8 | 8 | double | Measured sample rate (samples/second) | | 16 | 8 | double | Failed sample percentage | | 24 | 8 | double | Missed sample percentage | | 32 | 4 | uint32 | Optional field presence flags | | 36 | 4 | uint32 | Reserved | | 40 | 8 | bytes | Signature ("TACHSTAT") | | 48 | 4 | uint32 | Extension version (1) | | 52 | 4 | uint32 | Extension size (56) | +--------+------+---------+----------------------------------------+ ``` Putting the signature, version, and size at the end lets readers discover the extension from its fixed position relative to the footer while allowing future versions to add fields before that trailer. Multi-byte values use the same native byte order as the rest of the file and are byte-swapped by cross-endian readers. Readers also accept the original 32-byte extension, which only contains the duration and sampling rate. ## Footer ``` Offset Size Type Description +--------+------+---------+----------------------------------------+ | 0 | 4 | uint32 | String count | | 4 | 4 | uint32 | Frame count | | 8 | 8 | uint64 | Total file size | | 16 | 16 | bytes | Checksum (reserved, currently zeros) | +--------+------+---------+----------------------------------------+ ``` The string and frame counts allow readers to pre-allocate arrays of the correct size before parsing the tables. Without these counts, readers would need to either scan the tables twice (once to count, once to parse) or use dynamically-growing arrays. The file size field provides a consistency check: if the actual file size does not match, the file may be truncated or corrupted. The checksum field is reserved for future use. A checksum would allow detection of corruption but adds complexity and computation cost. The current implementation leaves this as zeros. ## Variable-Length Integer Encoding The format uses LEB128 (Little Endian Base 128) for unsigned integers and zigzag + LEB128 for signed integers. These encodings are widely used (Protocol Buffers, DWARF debug info, WebAssembly) and well-understood. ### Unsigned Varint (LEB128) Each byte stores 7 bits of data. The high bit indicates whether more bytes follow: ``` Value Encoded bytes 0-127 [0xxxxxxx] (1 byte) 128-16383 [1xxxxxxx] [0xxxxxxx] (2 bytes) 16384+ [1xxxxxxx] [1xxxxxxx] ... (3+ bytes) ``` Most indices in profiling data are small. A profile with 1000 unique frames needs at most 2 bytes per frame index. The common case (indices under 128) needs only 1 byte. ### Signed Varint (Zigzag) Standard LEB128 encodes −1 as a very large unsigned value, requiring many bytes. Zigzag encoding interleaves positive and negative values: ``` 0 -> 0 -1 -> 1 1 -> 2 -2 -> 3 2 -> 4 ``` This ensures small-magnitude values (whether positive or negative) encode in few bytes. Frame table deltas use 32-bit zigzag values and timestamp residuals use 64-bit zigzag values. ## Compression The header's compression type controls whether zstd may be used. Each column of each block is encoded separately, using whichever representation is smallest: - **constant**, when every byte of the column is the same, such as the stream IDs of a single-threaded block or residuals of on-time samples; - **zstd**, when compression is enabled and the compressed column plus its stored-size varint is smaller than the raw column; - **raw** otherwise. The string table, frame table, and footer remain uncompressed so readers can access metadata without decompressing the entire file. A tool that only needs to report "this file contains 50,000 samples of 3 threads" can read the header and footer without touching the compressed sample data. Zstd provides an excellent balance of compression ratio and speed. Profiling data compresses very well (often 5-10x) due to repetitive patterns: the same small set of frame indices appears repeatedly, and timestamp residuals cluster around zero. Level 3 compression is used as a default. It gives good compression without slowing down each block flush. Compressing columns separately keeps blocks independent, at the cost of restarting compression history at every block. ## Reading and Writing ### Writing 1. Open the output file and write 64 zero bytes as a placeholder header 2. Initialize empty string and frame dictionaries for deduplication 3. For each sample: - Intern any new strings, assigning sequential indices - Intern any new frames, assigning sequential indices - Append the sample to the current block's columns - Flush the block when it reaches its size or stream limit 4. Flush the last block 5. Write the string table (length-prefixed strings in index order) 6. Write the frame table (varint-encoded entries in index order) 7. Write measured profile statistics, when available 8. Write the footer with final counts 9. Seek to offset 0 and write the header with actual values The writer maintains two dictionaries: one mapping strings to indices, one mapping (filename_idx, funcname_idx, lineno) tuples to frame indices. These enable O(1) lookup during interning. To find `REUSE` candidates, it also keeps a small per-block cache of complete stacks, confirmed by a full comparison. When a block is flushed, the status and frame ID columns are transposed into bit and byte planes, the `REPEAT` counts are inserted into the instruction column, and each column is encoded as described in [Compression](#compression). ### Reading 1. Read the header magic number to detect endianness (set `needs_swap` flag if the magic appears byte-swapped) 2. Validate version and read remaining header fields (byte-swapping if needed) 3. Seek to end − 32 and read the footer (byte-swapping counts if needed) 4. Read measured profile statistics when the optional extension is present 5. Allocate string array of `string_count` elements 6. Parse the string table, populating the array 7. Allocate the frame array 8. Parse the frame table, populating the array 9. For each block, decode its columns into a reusable buffer and replay its samples in order, resolving indices to strings/frames (byte-swapping the stream table if needed) The reader builds lookup arrays rather than dictionaries since it only needs index-to-value mapping, not value-to-index. Consecutive samples of one stream with the same stack and status are passed to the collector in a single call, with up to 8,192 timestamps. The block header and columns are validated before any sample of the block is replayed. Stack instructions are checked as they are replayed, so a block with a bad instruction may already have passed earlier samples to the collector before the error is raised. The reader rejects a block when: - a size, encoding, or stream or frame reference is out of range, or a column has trailing data; - the stream table lists a thread twice or has a stream with no samples; - a stream's first instruction in the block is `REPEAT`, or a `REPEAT` count is zero or runs past the block; - a stack would be deeper than 256 frames, or `REUSE` refers to frame IDs that have not been read yet; - not every frame ID is used; - status bitmap padding bits are not zero; - it uses zstd in an uncompressed file. ## Platform Considerations ### Byte Ordering and Cross-Platform Portability The binary format uses **native byte order** for all multi-byte integer fields when writing. However, the reader supports **cross-endian reading**: files written on a little-endian system (x86, ARM) can be read on a big-endian system (s390x, PowerPC), and vice versa. The magic number doubles as an endianness marker. When read on a system with different byte order, it appears byte-swapped (`0x48434154` instead of `0x54414348`). The reader detects this and automatically byte-swaps all fixed-width integer fields during parsing. Writers must use `memcpy()` from properly-sized integer types when writing fixed-width integer fields. When the source variable's type differs from the field width (e.g., `size_t` written as 4 bytes), explicit casting to the correct type (e.g., `uint32_t`) is required before `memcpy()`. On big-endian systems, copying from an oversized type would copy the wrong bytes—high-order zeros instead of the actual value. The reader tracks whether byte-swapping is needed via a `needs_swap` flag set during header parsing. All fixed-width fields in the header, footer, and stream tables are conditionally byte-swapped using Python's internal byte-swap functions (`_Py_bswap32`, `_Py_bswap64` from `pycore_bitutils.h`). Variable-length integers (varints) are byte-order independent since they encode values one byte at a time using the LEB128 scheme, so they require no special handling for cross-endian reading. ### Memory-Mapped I/O On Unix systems (Linux, macOS), the reader uses `mmap()` to map the file into the process address space. The kernel handles paging data in and out as needed, no explicit read() calls or buffer management are required, multiple readers can share the same physical pages, and sequential access patterns benefit from kernel read-ahead. The implementation uses `madvise()` to hint the access pattern to the kernel: `MADV_SEQUENTIAL` indicates the file will be read linearly, enabling aggressive read-ahead. `MADV_WILLNEED` requests pre-faulting of pages. On Linux, `MAP_POPULATE` pre-faults all pages at mmap time rather than on first access, moving page fault overhead from the parsing loop to the initial mapping for more predictable performance. For large files (over 32 MB), `MADV_HUGEPAGE` requests transparent huge pages (2 MB instead of 4 KB) to reduce TLB pressure when accessing large amounts of data. On Windows, the implementation falls back to standard file I/O with full file buffering. Profiling data files are typically small enough (tens to hundreds of megabytes) that this is acceptable. The writer buffers samples in the current block and writes each completed block with a single call, so individual samples never cause a write. ## Future Considerations The optional profile-statistics block provides an extensible metadata area. The Python-version field retains one reserved byte. The 16-byte checksum field in the footer is currently unused. The version field allows incompatible changes with graceful rejection. New compression types could be added (compression_type > 1). Any changes that alter the meaning of existing fields or the parsing logic should increment the version number to prevent older readers from misinterpreting new files.