Apache Arrow Columnar Format (IPC)
Arrow is the interchange layer between Python/R/JVM analytics, DuckDB, and Parquet readers. IPC framing is how columnar data crosses process boundaries without JSON overhead.
What It Defines
In-memory columnar format with a serialized IPC/streaming representation: schema message, record batch messages, optional compression, and `ARROW1` magic for file format. Feather v2 is Arrow IPC with LZ4/Zstd compression. Zero-copy sharing between processes/languages.
Canonical (Normative)
Convenient (Practical)
Open-source foundation publishing columnar and analytics file formats widely used in data pipelines: Apache Parquet (columnar storage) and Apache Arrow (in-memory columnar + IPC serialization). Specs are maintained on parquet.apache.org and arrow.apache.org.
Related Specs
Parquet is the standard lakehouse/warehouse on-disk format (Spark, DuckDB, BigQuery export, Snowflake). Column chunk layout explains why analytics workloads beat row-oriented JSON/CSV.
Protobuf is the wire format for gRPC and many high-performance internal APIs. Schemas are also used for event/message contracts in Kafka/pubsub systems.
CBOR is the binary envelope behind COSE cryptographic messages and many constrained-device protocols. Typed length-prefixed encoding without a schema — unlike Protobuf.