This is part 7 of F3. Upstream pin bd92506447dc1333820cfe9a34ebbf497fe0219e.

This article compares jobs. It does not compare speeds. The recorded benchmark did not run a Parquet tool. See part 9 for that run.

Three jobs

Apache Arrow is an in-memory columnar representation. Engines pass arrays to each other without first agreeing on a file. The F3 footer stores an Arrow IPC schema so a reader can name fields and types. That schema is metadata. It is not an Arrow C Data Interface, and this library does not return Arrow arrays across the NIF. A decoded batch is an Elixir term.

Apache Parquet is a file format for exchanging columnar data. Readers in many languages already agree on it. A Parquet file does not carry a new decoder for an encoding the reader has never seen. That is the constraint the F3 paper is arguing with. This library does not read or write Parquet.

F3 is a research file format. The pinned writer puts Vortex-encoded values in the file, or a Wasm decoder when the writer was asked for one. A reader can refuse that decoder. This library does: wasm: :off is the default, and :sandbox is the only other accepted value.

flowchart LR
  arrow[Arrow in memory]
  parquet[Parquet file]
  f3[F3 file]
  arrow --- schema[Arrow IPC schema in the footer]
  f3 --- schema
  parquet --- interchange[Interchange without an embedded decoder]
  f3 --- decoder[Optional Wasm decoder]

What this library actually opens

small.f3 is three rows and an Arrow IPC schema in the footer. F3.schema/1 returns that schema. F3.read/2 returns columnar batches when the native decoder is compiled, and :native_unavailable when it is not. Neither call produces a Parquet file or an Arrow array.

From the f3 repository root:

mix run examples/tutorial_07.exs

The script prints the three jobs and the schema line for small.f3. It does not print a timing.

What is experimental?

Upstream F3 at this pin is a research prototype. The footer schema is Arrow IPC. The payloads are not Parquet row groups and they are not Arrow arrays. Shared dictionaries are still :unsupported_encoding. Nothing here says F3 is faster or slower than Parquet.

Next

Previous: F3: Taking the Decoder with the Data

Read next: F3: Building a Safe Binary-File Reader on the BEAM

Sources