Home > Positioning > Persons > Kreps

Jay Kreps

Jay Kreps is an American software engineer, co-creator of Apache Kafka and co-founder and CEO of Confluent, the company built around it.

Life and career

Kreps studied computer science at the University of California, Santa Cruz (bachelor’s and master’s). As an engineer and later lead architect for data infrastructure at LinkedIn, he and his colleagues Neha Narkhede and Jun Rao built Kafka around 2010 to carry the company’s event and operational data as a single pipeline; it was open-sourced in 2011.

In 2013 he wrote the influential essay “The Log: what every software engineer should know about real-time data’s unifying abstraction,” arguing that an ordered, append-only log is the structure common to database replication, messaging, and data integration; he expanded it into the short O’Reilly book I ❤ Logs (2014). In 2014 he co-founded Confluent with Narkhede and Rao and has led it as CEO, including through its 2021 public listing. He joined the board of directors of Anthropic in 2024.

The log

Kreps’s best-known idea is the argument of “The Log”: that a single structure — an ordered, append-only sequence of records — underlies database replication, state-machine replication, and data integration, and that making it a shared piece of infrastructure simplifies how systems are built and kept consistent. Apache Kafka is the realisation of that idea; the subject pages carry the full treatment of the system.

Where his work sits

Kreps’s contribution is engineering and articulation together — building a piece of infrastructure and naming the abstraction (the log) that made its significance legible. The pairing is what gave the work its reach: Kafka spread as software, and the log essay gave a generation of event-driven and streaming systems a shared way to talk about what they were doing.

Key works

Apache Kafka (co-creator, c. 2010) — distributed event-streaming platform
“The Log” (2013) / I ❤ Logs (O’Reilly, 2014)
Confluent (co-founder, 2014)