Arena: Market Conditions Before Confluent
Engineering teams at internet companies and large enterprises · 2010 · United States first, then worldwide
Disconnected workflowsChanging technical requirements
A growing company ran dozens of applications and data stores, and each one needed changes made in the others. There were a hundred ways to store data and no standard way to move it CF1. Teams wired each source to each user with a custom connection, so 5 sources and 8 applications meant 40 integrations, each with its own formats to reconcile CF2. Databases were updated in batches, so a copy was out of date as soon as the next event happened. Message queues passed data along in real time, but reading a message removed it, so two applications could not share the same stream without extra routing CF2. Engineers who looked for a product that kept every system current with every other found none CF1.
How each step happened
Step 1 of 5 · 2010–14
Log-based streaming, built by Kafka's creators
Kafka's innovation was to treat a company's data as a log: an ordered record of every event, kept after it is read. Jay Kreps, Neha Narkhede and Jun Rao built it at LinkedIn to combine messaging systems with the log that databases use internally, as a distributed system that replicated and partitioned the data across machines CFX-1 CF2. Any number of applications could read the same log, and each source or application needed one connection to Kafka instead of one to every other system CF2. It scaled better than queues at large volumes and ran on any infrastructure, while AWS's later answer fed only AWS's own stores CF2 CFX-5.
The design came from the founders' own problem: LinkedIn needed, Kreps writes, "to unite all the different applications and data stores that made up a global social network into one coherent system", and years of testing products and reading research turned up nothing that did it CF1. They ran Kafka on billions of messages at LinkedIn, open-sourced it through the Apache Software Foundation in 2011 and founded Confluent in September 2014 CF1 CF2. Confluent's engineers remained Kafka's main contributors CF2.
Rivals AWS answered with Kinesis in November 2013, a managed stream of its own that fed data into Amazon S3, DynamoDB and Redshift CFX-5. By 2021 analysts found Kinesis slower than Kafka on message throughput and usable only on AWS CF2.
Novel ArchitectureFounder Domain Expertise
Step 2 of 5 · 2015–21
A practitioner community makes Kafka the standard
Confluent organized Kafka's users into a community. Kafka Summit, announced in September 2015 and first held in April 2016, brought Kafka's core maintainers together with developers building streaming systems, when thousands of companies, among them Netflix, Uber and Goldman Sachs, already used Kafka CFX-2. By 2021 Kafka had more than 60,000 members in over 200 meetup groups and was estimated to be used by over 70% of the Fortune 500 CF1.
The community made Kafka the default for moving data. The 2021 filing states that "modern applications are expected to integrate with Apache Kafka, and the technical skill set for Kafka has become a critical requirement in the industry" CF1. Each new user made Kafka skills and connectors built by outside developers more valuable CF1. Developers could download Confluent's free edition and start on their own, then pay for support and enterprise features, priced by the number of machines, as their deployment grew CF2.
The standard was open, so its pull worked for any supplier that ran Kafka. Confluent had to win on running Kafka, not on owning it.
Rivals AWS adopted the standard its customers had chosen. Amazon MSK, generally available in May 2019, ran open-source Kafka versions 1.1.1 and 2.1.0 CFX-3, and analysts read it as proof that developers preferred Kafka to Kinesis CF2.
Practitioner CommunityProtocol network effectsFreemium Model
Step 3 of 5 · 2017–20
Fully managed Kafka replaces the in-house Kafka team
Running Kafka in production took a specialist team. Confluent Cloud, generally available in November 2017, took that work over: Confluent handled patches, fixes and scaling, customers started on a free trial and paid for what they used, and Kafka's committers answered severe problems within 60 minutes CF1.
Customers bought it to avoid hiring. In 2019 Nuuly, Urban Outfitters' clothing-rental service, chose Confluent over options including major cloud providers because it could build warehouse and order-management systems in six months without hiring "hard-to-find Kafka engineers" CF1. SmartThings, Samsung's smart-home unit, had run Kafka itself for more than five years; after moving to Confluent in September 2020 it cut its dedicated Kafka headcount by 80% CF1. Cloud revenue grew from $14.4 million in 2019 to $31.4 million in 2020 CF1.
Rivals MSK arrived 18 months after Confluent Cloud. AWS replaced failed servers and applied patches, but customers still configured brokers, spread partitions across them, created topics and ran their own Kafka tools, and paid per broker-hour CFX-3 CFX-4.
Managed ServiceFree TrialUsage-Based Pricing
Step 4 of 5 · 2018–24
Kafka rebuilt for the cloud, sold on every cloud
A platform clouds could not resell
Confluent built the tools around Kafka as its own products: Schema Registry to keep data formats consistent, ksqlDB to process streams with SQL, and more than 120 pre-built connectors by March 2021 CF1. In December 2018 Kreps moved these components to a license that bars offering them as a competing cloud service, while Kafka itself stayed under the Apache license; he wrote that some cloud providers take open-source code into their own offerings and invest only in their proprietary products CFX-6 CF1. Any cloud could host Kafka; only Confluent could run the complete platform.
Built for the cloud, on every cloud
Confluent sold Cloud as "a completely different experience than what would result from taking on premise software and simply offering it on cloud virtual machines" CF1. It ran on AWS, Google Cloud and Microsoft Azure from November 2019, and from August 2020 customers could buy it through all three clouds' marketplaces CF1. Buyers with data on several clouds or in their own data centers got one supplier CF2. The Kora engine, published in 2023, rebuilt Kafka for this service: cells isolate customers, older data moves to cloud object storage, and software rebalances clusters and handles hundreds of faults a week without an operator CFX-7. By 2024 Kora ran tens of thousands of clusters in more than 70 regions and scaled 30 times faster than self-managed Kafka CFX-7.
Rivals MSK ran only on AWS and lacked Confluent's proprietary connectors, leaving developers on open-source ones, and offered no counterpart to ksqlDB CF2. Its pay-as-you-go storage and 20 times faster scaling came in November 2024 CFX-9.
Novel ArchitectureMulti-ProductDistribution PartnershipsCompetitive Alternatives
Step 5 of 5 · 2020–26
Catch-up barrier in Kafka expertise
Kafka's open protocol let a customer move its streams to MSK. What kept customers with Confluent was the lead built in steps 1 to 4: the founders' knowledge of Kafka, committer-led support, the proprietary platform and an engine AWS matched only in November 2024, when MSK Express brokers added pay-as-you-go storage, 20 times faster scaling and 90% faster recovery CF1 CFX-7 CFX-9.
The lead paid off inside accounts. Customers started small, often on the free trial, and sales teams took them from a pilot to a first production project and then to a company-wide platform CF1 CF2. At Domino's, one team's real-time store analytics grew into an enterprise system as other teams saw the value CF1. Existing customers spent 25% more in 2020 than a year earlier, and customers paying at least $100,000 a year rose from 374 in March 2020 to 561 in March 2021 CF1. That growth took heavy selling: sales and marketing absorbed 71% of revenue in the year before the IPO, and 2020 brought a $233.2 million operating loss on $236.6 million of revenue CF3 CF1.
By 2025 revenue was $1,166.7 million, $624 million of it from Confluent Cloud, and 1,521 customers paid $100,000 or more a year CFX-8. IBM completed its purchase in March 2026 at an enterprise value of about $11 billion CF4. The price goes back to step 1: the people who built Kafka at LinkedIn became the company that ran it best.
Rivals MSK stayed in the market and spent 2019 to 2024 closing operational gaps; Express brokers in November 2024 brought scaling and storage that Confluent's engine already had CFX-9 CFX-7.
Accumulated assets & catch-up barriersLand and ExpandProduct-Led Sales