Enterprise Data Infrastructure
Real-Time Data Pipelines: Build vs. Buy in 2026
IBM just closed an $11 billion acquisition of Confluent, the company behind the Kafka platform running inside 40% of the Fortune 500. If you’re the person who has to decide whether your team spends the next two years operating a Kafka cluster or signing a vendor contract instead, that deal just changed your negotiating position and your risk profile at the same time.
This isn’t another explainer on what real-time data pipeline architecture looks like. It’s the cost and procurement question underneath it: should your organization build this infrastructure in-house, or buy it? The answer depends less on technology and more on talent, timeline, and what problem you’re actually solving.
In This Article
The $11 Billion Signal: Why IBM Bought Confluent
On March 17, 2026, IBM completed its acquisition of Confluent for $31 a share in cash, a deal worth roughly $11 billion that was first announced back in December 2025. Confluent’s Kafka-based streaming platform sits inside more than 6,500 enterprises, and Confluent itself claims over 40% of the Fortune 500 run its commercial platform, up from 27% a few years ago (that figure is vendor-reported, worth noting, but the trend direction lines up with everything else happening in this market). Confluent delisted from Nasdaq. CEO Jay Kreps stayed on to run the business; the board did not survive the transition.
Full details are in the IBM/Confluent 8-K filing with the SEC.
Why does this matter to you if you’re not a Confluent customer? Because it confirms real-time data infrastructure has graduated from “specialized add-on” to core enterprise plumbing, the kind large vendors pay double-digit billions to own outright. IBM has done this playbook before, with Red Hat and with HashiCorp. Pricing and packaging tend to shift toward IBM’s enterprise contract structure within 12 to 18 months of close. If you’re renewing a Confluent agreement this year, read the fine print now, not at renewal time.
“Real-time data is the fuel for AI.” Jay Kreps, Co-founder & CEO, Confluent (now an IBM company)
Two days before the deal closed on the calendar of relevant 2026 news, Databricks launched LTAP, its lake transactional and analytical processing platform, at its Data + AI Summit on June 16. We covered that architecture in depth in our Databricks LTAP breakdown. This piece picks up where that one leaves off: not how the stack works, but whether you should build one yourself or hand the problem to a vendor.
The Real Question Isn’t “Real-Time or Not”
Here’s the framing most vendor content skips. The decision in front of you isn’t whether real-time data matters. Confluent’s fifth annual Data Streaming Report, which surveyed 4,625 IT leaders across 14 countries, found that 72% say a lack of real-time infrastructure is stalling their AI scaling efforts. That number is now close to consensus.
The decision that actually determines your budget, your headcount, and your risk exposure for the next three years is whether you build that infrastructure yourself or buy it from someone who already operates it at scale. Those are very different bets, and the research doesn’t point toward one obvious winner.
The number to actually use
Skip the round, dramatic “$19 million lost” figures floating around this topic. They don’t trace back to a named company or a disclosed methodology. Gartner’s substantiated estimate puts the annual cost of poor data quality and flawed decisions at $9.7 million to $15 million per organization, a real, citable figure worth anchoring your internal business case to instead.
The True Cost of Building In-House
Building your own real-time pipeline on open-source Kafka and Flink looks cheap on a licensing spreadsheet. It rarely looks cheap on a headcount spreadsheet.
Operating Kafka and Flink reliably in production, not just standing up a proof of concept, requires engineers who understand distributed systems, state management, and cluster operations under load. That talent is genuinely scarce, and the market has been telling you so. Decodable, a streaming startup, got acquired rather than scaled independently. Google retired its managed BigQuery Flink engine. Several Pulsar-focused startups exited the space entirely in the last two years. None of that happens in a market where in-house streaming is easy to staff and operate.
Adoption data backs this up from a different angle. Integrate.io’s 2026 stats roundup found that 72% of organizations now use event-driven architecture in some form, but only 13% report reaching org-wide maturity with it. That gap, adoption without maturity, is exactly where in-house builds tend to stall: teams get Kafka running, then spend eighteen months fighting operational debt instead of shipping features.
What “building” actually costs
- Specialized headcount: platform engineers who understand Kafka/Flink ops don’t come cheap, and they’re in short supply.
- On-call burden: streaming infrastructure that breaks at 2 a.m. is now your problem, not a vendor’s SLA.
- Opportunity cost: every sprint spent on cluster management is a sprint not spent on the product your customers actually see.
- Ramp time: reaching production-grade maturity typically takes longer than teams budget for, based on that 72%-adopted-but-13%-mature gap.
The Case for Buying, and Its Fine Print
The market case for buying is straightforward. Next Move Strategy Consulting projects the global data pipeline market growing from $14.5 billion in 2025 to $58.6 billion by 2035, a 16.8% compound annual growth rate, with real-time streaming as the fastest-growing segment. Separately, Integrate.io compiles market data showing data pipeline tools growing at a 26.8% CAGR toward $48.33 billion by 2030, well ahead of traditional ETL’s 17.1% growth rate. Capital is flowing toward managed platforms, not toward custom builds.
Steven Karan, VP of AI Transformation at Capgemini Australia and New Zealand, made the underlying point plainly to CIO.com in June: the lakehouse has become foundational infrastructure, not a niche analytics tool.
“The lakehouse isn’t just for analytics anymore.” Steven Karan, VP of AI Transformation, Capgemini Australia and New Zealand, via CIO.com
Read the full context in CIO.com’s June 2026 feature on enterprise lakehouse adoption.
For most enterprises, a managed platform, whether that’s Confluent Cloud, Databricks, or a cloud-native streaming service, wins on total cost of ownership once you factor in engineering time and talent scarcity. Building in-house tends to only pay off at very large, sustained data volumes with a platform team you already have in place. If that’s not your situation, buying is the less risky bet.
Why the Vendor ROI Numbers Deserve Skepticism
Here’s where you need to slow down before you take a vendor’s ROI slide into a budget meeting. Confluent’s own 2026 survey reports that half of organizations achieve 5x or greater ROI on data streaming, and 88% achieve at least 2x. Those numbers are real, in the sense that Confluent really did survey 4,625 IT leaders and really did get those responses. What they’re not is independent.
This is a vendor-commissioned survey of self-selected respondents who had already invested in streaming technology before answering the survey. People who bought the platform and regret it don’t tend to fill out vendor satisfaction surveys. Use these numbers as a directional signal that streaming can pay off, not as a guarantee that it will pay off for your organization specifically.
Gartner’s own research offers a more sobering counterweight. Gartner projects that 60% of AI projects lacking AI-ready data will be abandoned through 2026, and separately that 70% of agentic AI use cases will fail to deliver expected value. Rita Sallam, Distinguished VP Analyst at Gartner, has pointed to mismatched cost models as a primary cause, meaning organizations are overspending on top-tier real-time infrastructure to solve problems that didn’t need it. Read Gartner’s original predictions in the 2026 Data & Analytics predictions release.
Our read: this signals that the failure mode in 2026 isn’t “real-time infrastructure doesn’t work.” It’s “we bought infrastructure sized for a problem we hadn’t actually defined yet.” Sequencing, not tooling, is where most build vs. buy decisions go wrong.
That sequencing point shows up elsewhere too. Precisely’s 2025 Data Integrity Trends Report found that 64% of organizations cite data quality as their top data-integrity challenge, and organizations lose roughly 25% of annual revenue to quality-related inefficiencies. Buying a faster pipeline doesn’t fix bad data. It just delivers bad data to your AI agents faster than before.
The Compliance Gotcha Nobody Mentions
What vendors won’t lead with
Popular real-time serving engines including Apache Pinot and Apache Druid don’t natively support UPDATE or DELETE operations on ingested records. If you operate under GDPR or CCPA and need to honor a right-to-erasure request, that’s not a minor technical footnote. It’s an architectural constraint that can force a redesign after you’ve already committed budget and headcount to a platform choice.
This is exactly the kind of detail that gets skipped in an architecture pitch deck and shows up eighteen months later as an unplanned engineering sprint. If your organization operates in the EU, the UK, or California, put this question in front of any vendor or open-source stack before you sign anything: how does erasure actually work at the storage layer, not just at the application layer?
A Practical Build vs. Buy Framework
Only about 22% of enterprises say they’re confident their current IT infrastructure can actually support new AI applications, according to survey data cited in a joint Confluent and Databricks announcement. That confidence gap is where the build vs. buy decision actually gets made, usually under time pressure. Here’s a simplified way to think about it.
| Factor | Lean Build | Lean Buy |
|---|---|---|
| Data volume | Very large, sustained, predictable | Variable or growing unpredictably |
| Platform engineering talent | Already in-house and retained | Scarce, expensive to hire, or nonexistent |
| Latency requirement | True sub-second, mission-critical | 5-15 minute near-real-time is acceptable |
| Compliance complexity | Deep in-house legal/eng coordination | Vendor handles erasure and audit tooling |
| Time to value | 12-24 months acceptable | Need production in under 6 months |
Most enterprises land closer to the “buy” column than they expect, mainly because true sub-second streaming is only justified for a narrow set of use cases: fraud detection, dynamic pricing, and AI-agent workflows that can’t tolerate stale inputs. An estimated 80% of business analytics needs are served just fine by a five to fifteen minute refresh cycle, which is dramatically cheaper to operate than full streaming, whether built or bought.
One more data point worth sitting with: our own reporting on the Databricks LTAP rollout found DoorDash measuring a 35.7% feature mismatch between its batch and streaming ML pipelines, the root cause being two systems computing the same metric two different ways. We’d flag that figure as sourced through our own LTAP coverage rather than independently re-verified from DoorDash directly, but the underlying lesson holds regardless of the exact number: running parallel batch and streaming systems creates definitional drift that neither a build nor a buy decision fixes on its own. It has to be solved with a shared semantic layer.
Amit Kinha, Field CTO at DoiT International and a FinOps Foundation board member, made this point to CIO.com: without a semantic layer, an AI agent won’t reliably know where to look for the data it needs. That’s a governance problem, not an infrastructure problem, and it sits underneath whichever build vs. buy path you choose.
Frequently Asked Questions
What is a real-time data pipeline?
A system that ingests, processes, and delivers data continuously as it’s generated, rather than in scheduled batches. Most are built on Apache Kafka for ingestion, Apache Flink for stream processing, and a serving layer like ClickHouse, Pinot, or a lakehouse platform.
Is it cheaper to build or buy a real-time data pipeline?
For most enterprises, buying a managed platform is cheaper on a total cost of ownership basis once engineering time, on-call burden, and talent scarcity are factored in. Building in-house typically only wins at very large, sustained data volumes with a dedicated platform team already in place.
What is the ROI of real-time data streaming?
Confluent’s 2026 vendor-commissioned survey reports 88% of organizations achieving 2x or greater ROI and half achieving 5x or greater. These figures come from self-selected adopters already invested in the technology, not an independent audit, so treat them as directional rather than universal.
Does every enterprise need real-time data?
No. Most business analytics needs are well served by a five to fifteen minute near-real-time refresh cycle. True sub-second streaming earns its cost mainly for fraud detection, dynamic pricing, and AI-agent-driven automation that can’t tolerate stale inputs.
What This Means Going Forward
Here’s what’s different by the end of reading this versus the start. The build vs. buy decision on real-time data pipelines isn’t really about Kafka versus a managed platform anymore. It’s about whether your organization has the talent to operate streaming infrastructure at 2 a.m. when it breaks, and whether your data is clean enough that faster delivery actually helps instead of just breaking things faster.
Watch three things over the next 6 to 18 months. First, how IBM repositions Confluent’s pricing for its installed base, since that will set a template other vendors follow. Second, whether Databricks’ LTAP approach, unifying transactional and analytical processing, pulls more build-it-yourself shops toward a single managed platform instead of stitching together Kafka, Flink, and a separate serving layer. Third, whether Gartner’s abandonment predictions for AI-ready data projects actually materialize, which would be the clearest signal yet that the market overbought infrastructure relative to the data quality work it needed to do first.
If you’re making this call for your organization right now, the sequencing matters more than the tooling. Fix your data quality and semantic layer first. Then decide, with clear eyes about your own talent bench, whether building or buying gets you to production faster and cheaper. For most teams, the honest answer in 2026 is buy, with data quality work done before, not after, the contract gets signed.
Related reading on NeuralWired: Databricks LTAP Real-Time Analytics Stack 2026, AI Agent Implementation Failures, and Enterprise AI Implementation Roadmap.
Want this kind of analysis in your inbox weekly? Subscribe to The Neural Loop at neuralwired.com/newsletter.
More posts
-
Denmark CPR Data Breach: How a Company’s Legitimate Access Exposed 8.8 Million Records
Nobody picked the lock in the Denmark CPR data breach. According to the ministry, a company’s lawful access to the Central Person Register was misused, exposing the details of about 8.8 million people. Here is what happened, why a CPR number cannot simply be changed, and what to watch next.
-
Pennsylvania’s Measles Outbreak Nears 1,000 Cases as the State and CDC Disagree on the Death Toll
Pennsylvania says five residents have died of measles this year, while the CDC’s national count lists two. This look at the Pennsylvania measles outbreak explains why the two tallies differ and what could change them next.
-
SEC Clears the Way for 3x Bitcoin and Ether ETPs, but None Can Be Traded Yet
The SEC has approved a Cboe rule that would let triple-leveraged bitcoin and ether funds list in the US, but you cannot buy one yet. Here is what the approval covers, what the sponsor’s own filing says about the risks, and what has to happen before the first 3x bitcoin ETF-style product appears on a…
-
Weak September Jobs Report Puts a Fed Rate Hike on the Back Foot as Treasury Yields Hover Near 19-Year Highs
US employers added only 29,000 jobs in September, far below forecasts and just weeks after the Federal Reserve raised rates. The September jobs report has traders doubting an October hike, even as Treasury yields stay near 19-year highs. Here is what the numbers show and what to watch before the Fed’s next meeting.
-
OpenAI Parts Ways With Three Safety Staff Over Alleged Information Sharing, Days After FTC Opens AI Safety Probe
OpenAI says three safety staff mishandled sensitive information, but it hasn’t said what was shared or with whom. The dismissals landed days after a canceled model launch and a new FTC probe. Here is what is confirmed, what is disputed, and what to watch next.
-
Can Britain Rejoin the EU? What Andy Burnham Actually Said, and What Happens Next
Andy Burnham never called for Britain to rejoin the EU in his conference speech, but a radio interview the next day put “all the way” on the table. Here is what he actually said, how Europe responded, and what rejoining would take.
-
OpenAI’s AI Agents Reached Government Websites in Two Countries. Here Is What Is Known So Far
OpenAI’s AI agents have reached beyond a single company breach and into government systems in the US and Australia, touching SEC, Census Bureau and Medicare-linked data. As Congress and the UN Security Council scrutinize the fallout, here is what has been confirmed so far, and what is likely to happen next.