StreamSets is an enterprise data integration and DataOps platform for resilient hybrid-cloud pipelines, now part of IBM. See features and pricing.
StreamSets is a DataOps platform that lets data engineering teams build, run, and monitor data pipelines that move information between databases, cloud data warehouses, message queues, and applications. It was founded in 2014 by Arvind Prabhakar and Girish Pancha, both veterans of Informatica and Yahoo, with the goal of making data pipelines resilient to constant change rather than brittle scripts that break every time an upstream schema shifts.
The platform's signature idea is the 'smart data pipeline,' which senses and automatically adapts to data drift, such as new columns, renamed fields, or type changes, instead of failing outright. In 2024, StreamSets became part of IBM after IBM acquired it and webMethods from Software AG in a deal worth about 2.3 billion dollars, and the product is now marketed and sold as IBM StreamSets.
StreamSets provides a single visual design experience for batch, change data capture, ETL, ELT, and streaming pipeline patterns, along with a centralized control hub for monitoring and governing pipelines running across hybrid and multicloud infrastructure. A Python SDK allows engineers to templatize pipelines and generate hundreds of them programmatically, which is useful for organizations replicating similar pipeline patterns across many source systems, such as multi-tenant SaaS data or franchise/branch reporting.
StreamSets does not publish self-serve pricing. As an IBM enterprise product, cost is negotiated with IBM sales based on the number of connectors, data volume, number of pipelines, and whether the deployment is cloud-managed or self-hosted. Prospective buyers should expect an enterprise sales cycle rather than a credit-card checkout.
StreamSets is used to build, run, and monitor data pipelines that move and transform data between databases, cloud warehouses, applications, and streaming systems, with built-in resilience to schema changes.
StreamSets began with open-source roots but is now a commercial enterprise product sold by IBM; it is not a free, community-maintained open-source tool today.
IBM completed its acquisition of StreamSets, alongside webMethods, from Software AG in July 2024, and the product is now sold as IBM StreamSets.
StreamSets does not publish pricing. Cost is quoted by IBM sales based on data volume, connectors, number of pipelines, and deployment model.
A smart data pipeline is StreamSets' term for a pipeline that automatically detects and adapts to data drift, such as new or renamed fields, instead of failing when source data changes.
Yes, StreamSets supports streaming and change data capture patterns in addition to traditional batch ETL and ELT processing.
Yes, StreamSets execution engines can be deployed on-premises, in private cloud, or across major public clouds in hybrid configurations.
StreamSets is aimed at data engineers and platform teams; it offers a low-code visual designer but still assumes familiarity with data integration concepts.