Why another columnar file format — and what makes F3 different?

Apache Parquet and ORC have served analytical systems remarkably well. But a file format has an awkward lifespan: the data may need to remain readable for decades, while the hardware, compression algorithms and execution engines around it keep changing. Improvements in encoding often require readers to be upgraded before the new files can be widely exchanged.

F3 — the Future-proof File Format — takes a different approach. Its designers focus on three things: efficient data layout, interoperability, and extensibility. The intriguing idea is not simply better compression. It is that a file can describe how its contents are encoded and, when necessary, carry the code needed to decode them.

1. Two sizes, two jobs

Traditional columnar formats already break data into groups and pages. F3 makes an explicit distinction between the units in which data is read from storage and the units in which it is encoded.

                   ONE COLUMN IN AN F3 FILE

         ┌───────────────────────────────────┐
         │         Row group / column        │
         ├───────────────────────────────────┤
         │ IOUnit A  — the I/O boundary      │
         │ ┌──────────┬──────────┬─────────┐ │
         │ │ EncUnit 1│ EncUnit 2│EncUnit 3│ │
         │ │ encoding │ encoding │encoding │ │
         │ │    X     │    Y     │   X     │ │
         │ └──────────┴──────────┴─────────┘ │
         ├───────────────────────────────────┤
         │ IOUnit B  — another I/O boundary  │
         │ ┌──────────────┬────────────────┐ │
         │ │  EncUnit 4   │   EncUnit 5    │ │
         │ └──────────────┴────────────────┘ │
         └───────────────────────────────────┘

             IOUnit  = unit of physical I/O
             EncUnit = unit of encoding choice

An IOUnit groups data for I/O. An EncUnit is the smaller, contiguous region to which a particular encoding applies. An IOUnit may contain several EncUnits; the file’s metadata records where they are and how to interpret them. These choices need not be tied together.

Why care? Because optimal I/O size depends on storage and access patterns, while optimal encoding depends on the values being encoded. There is no reason for the two decisions to have exactly the same boundary.

2. The decoder travels with the data

F3’s more radical idea concerns longevity. Imagine a column compressed using an encoding that your reader has never encountered. Traditionally that means obtaining a newer reader or adding a codec. F3 offers another path: an encoding unit may reference a WebAssembly (Wasm) decoder embedded in the file.

                   READING AN F3 FILE

                 ┌─────────────────┐
                 │ F3 file         │
                 │ Data + metadata │
                 │ + Wasm decoders │
                 └────────┬────────┘
                          │
                          v
                 ┌─────────────────┐
                 │ Inspect encoding│
                 └────────┬────────┘
                          │
                 ┌────────┴─────────┐
                 v                  v
        ┌────────────────┐  ┌────────────────┐
        │ Known encoding │  │ Unknown native │
        │ Native decoder │  │ encoding       │
        └────────┬───────┘  └────────┬───────┘
                 │                   │
                 │                   v
                 │          ┌────────────────┐
                 │          │ Embedded Wasm  │
                 │          │ decoder, if    │
                 │          │ provided       │
                 │          └────────┬───────┘
                 └──────────┬────────┘
                            v
                      Decoded values

Native decoding remains the desirable fast path. The Wasm decoder is a portability fallback, not a requirement to run a virtual machine for every value. It also isn’t permission to execute arbitrary code without controls: embedded decoders need sandboxing, resource limits and an explicit trust policy.

The principle is elegant: a file should not become unreadable merely because its encoding outlives the original software that wrote it.

3. What F3 keeps familiar

F3 is not proposing to abandon the columnar ecosystem. Its metadata uses FlatBuffers, and its footer includes a serialized Apache Arrow schema. Data is organized into row groups, columns, IOUnits and EncUnits, with metadata describing the physical layout and optional decoder references.

┌────────────────────────────────────────────┐
│              F3 FILE                       │
├────────────────────────────────────────────┤
│ Encoded column data (row groups / IOUnits) │
├────────────────────────────────────────────┤
│ Optional embedded Wasm decoder binaries    │
├────────────────────────────────────────────┤
│ Row-group and column metadata              │
├────────────────────────────────────────────┤
│ Footer: Arrow schema, offsets, versions    │
├────────────────────────────────────────────┤
│ Postscript: footer location and validation │
└────────────────────────────────────────────┘
        Schematic; not a byte-level specification

That gives F3 a plausible evolutionary path: preserve the idea of portable, self-describing analytical files while giving encoding and I/O layout more independence.

4. Where it might matter — and where it does not yet

F3 is worth watching for large analytical datasets, long-lived archives, and data interchange between systems that may not share the same native codecs. The separation of I/O and encoding boundaries also makes it interesting for workloads with different storage characteristics or heterogeneous columns.

It is not a replacement for every data format. It is not designed for transactional record updates, nor does carrying Wasm eliminate the cost and security considerations of running a decoder. And importantly, the upstream authors call F3 a research prototype and warn against production use. Its specification and implementation are still evolving. Research results are promising, but they are not a substitute for workload-specific benchmarks or operational maturity.

5. Bringing F3 to Elixir

I’ve published an experimental Elixir library, f3, to start exploring this format from the BEAM ecosystem. The initial release is 0.1.0.

Add it to mix.exs:

{:f3, "~> 0.1.0"}

The package integrates native code through precompiled NIFs for macOS Apple Silicon and Linux x86-64 when installed from Hex. Building from a Git checkout requires explicitly opting into native compilation with F3_BUILD_NATIVE=1.

At this stage, treat it as an experimental library for exploring F3, not a production data-storage dependency. Its purpose is also to bring an interesting question to Elixir: how far can we take efficient, portable columnar data processing while retaining the BEAM’s strengths in concurrency, streaming and fault isolation?

F3 may or may not become the next widely adopted columnar format. But the idea behind it deserves attention: data portability should include the ability to understand the data, not just to move its bytes.

Further reading: F3 upstream repository · Research paper (ACM DOI: 10.1145/3749163) · F3 file-format schema · Elixir f3 on Hex