Serialization formats

The problem

A profiler runs on a microservice handling 50 MB/sec of data. The report shows 40% of CPU time inside JSON.parse() and JSON.stringify(). The service serializes a simple event object on every message. The JSON is human-readable and easy to debug. It is also destroying the latency profile.

Engineers on the team switch to Avro. Payload sizes drop by 70%. CPU usage falls immediately. The system performs well for months. Then the Avro schema registry goes down during a deploy. New consumer instances spin up, attempt to fetch the writer schema, and fail. The Kafka consumers that were already running continue to work. Any new pod that starts cannot deserialize a single message until the registry recovers.

These are the two failure modes that serialization format choice forces you to navigate: JSON is too slow at scale, and schema-dependent binary formats introduce operational dependencies that can take down consumers silently.

What serialization is

Serialization converts an in-memory object into a flat sequence of bytes suitable for network transmission or disk storage. Deserialization reverses it: the bytes come in and an object comes out. Every time your service sends a request, writes to a queue, or persists a record, serialization happens.

Analogy: Think of a moving company packing the contents of an apartment. One approach is to write the name of every item on the box. Another is to number each item in a catalogue and only write the number on the box. The catalogue approach produces smaller boxes and faster packing, but the recipient needs the catalogue to unpack. That catalogue is the schema.

How it works

The same object encoded in three formats shows the tradeoff immediately:

Input object: { userId: 12345, name: "alice", active: true }

JSON (text, ~42 bytes):
{"userId":12345,"name":"alice","active":true}
  Field names are spelled out in full as strings in every message.

MessagePack (binary, ~25 bytes):
83 a6 75 73 65 72 49 64 cd 30 39 a4 6e 61 6d 65 a5 61 6c 69 63 65 a6 61 63 74 69 76 65 c3
  Field names are still included but binary-encoded. No schema needed.

Protobuf (binary, ~15 bytes):
08 b9 60 12 05 61 6c 69 63 65 18 01
  Field names are gone. Field 1 (varint 12345), field 2 (string "alice"), field 3 (bool true).
  A .proto schema file maps field numbers back to names.

Protobuf encodes the integer 12345 as a variable-length integer (b9 60 in little-endian base-128). The boolean true is 01. The entire key-value structure is encoded as field-number tags followed by values, with zero bytes spent on field names.

The format you choose determines where the schema information lives: encoded inside every message (JSON, MessagePack), in a shared .proto file compiled into the binary (Protobuf), or in an external registry looked up at runtime (Avro).

The wire-format size difference is not abstract. Encoding the same object in each format produces measurably different byte counts, and that difference multiplies across every message the system sends.

spawnSync d2 ENOENT

Protobuf's 3x size advantage over JSON comes almost entirely from replacing string field names with 1-2 byte integers. For a message with 5 fields averaging 10 characters each, that is 50 bytes of field-name overhead eliminated from every single message.

Schema evolution

Schema evolution is how the format handles the fact that your data model will change. Adding fields. Removing fields. Renaming fields. These happen in every long-lived system.

JSON: No schema is required. By convention, receivers ignore fields they do not recognize. You can add new fields today and old clients will silently skip them. You can remove fields and hope all consumers have been updated. There is no enforcement, no versioning, and no protection against an engineer who renames a field and breaks every existing consumer without knowing it.

Protobuf: Field numbers are permanent. When you define a field, you assign it a number. That number is what gets encoded in the binary. You can add new fields with new numbers, and old clients will skip any field number they do not recognize. You can remove fields, but you must then reserve their numbers so no future field reuses the slot. Reusing a number for a different type causes silent data corruption on old readers.

message UserEvent {
  int32 user_id = 1;
  string name = 2;
  bool active = 3;
  // Field 4 was "email", now removed
  reserved 4;
  reserved "email";
  // New field adds with a fresh number
  string region = 5;
}

Avro: The schema is required on both the write side and the read side. A schema registry stores every schema version. When a consumer deserializes a message, it fetches the writer's schema from the registry and compares it to its own reader schema. The registry performs compatibility checks: can the reader schema safely read data written by the writer schema? This is the most rigorous evolution model, and it makes the registry a hard dependency.

Performance comparison

The size and speed differences between formats are not marginal. At 50 MB/sec, a 3x size reduction means 33 MB/sec instead of 100 MB/sec out of your network interface. At 5,000 messages/sec, a 10x parse speedup means 100 microseconds of CPU instead of 1 millisecond.