When JSON Stops Being Good Enough
For the first two years of our payment platform, every event — authorizations, captures, refunds, chargebacks — was serialized as JSON and pushed to Kafka. It worked fine. JSON is human-readable, every language has a parser, and debugging is as simple as piping a topic to jq.
Then our transaction volume crossed 50K events per second, and we started seeing problems. Our Go consumer services were spending 15-20% of their CPU time just on json.Unmarshal. The average event payload was 2.3 KB in JSON. Multiply that by 50K/s and you're pushing 115 MB/s of raw event data through Kafka — and that's before replication.
We evaluated three alternatives: MessagePack, Avro, and Protocol Buffers. Protobuf won for two reasons: Go has first-class support via the official google.golang.org/protobuf module, and the schema evolution story is exactly what financial systems need — you can add fields without breaking existing consumers.