Apache Kafka is an increasingly foundational component of enterprise Big Data architectures. Data streaming and event-driven systems have rapidly supplanted batch-driven processes since Kafka was first invented by engineers at LinkedIn in 2011. It was then brought into the Apache Software Foundation (ASF) as open-source Apache Kafka as well as made into a commercial product by the good folks over at Confluent. Over the ensuing decade, the drive for real-time stream processing has seen Kafka spread to tens of thousands of companies, including the majority of the Fortune 100.
Apache Kafka is capable of delivering reliable, scalable, high-throughput data streams across a myriad of data sources and sinks. A great number of our open source users and enterprise customers — like IBM, General Electric, Nauto, Grab, and Lookout — use Scylla and Kafka together. For those looking for more information, you can read about using Kafka and Scylla together to, for example, create a scalable backend for an IoT service.
New Shard-Aware Kafka Connector for Scylla
Because Scylla is an API-compatible implementation of Apache Cassandra, to date users who wished to connect Scylla to Kafka have been able to use the Kafka Cassandra Connector (also known as the Confluent Cassandra Sink Connector). Users could simply swap out their Cassandra database with Scylla transparently with no code changes.
However the disadvantage of that approach was that Kafka performance was not as speedy as could be, since it was not fully taking advantage of Scylla’s underlying shard-per-core, shared-nothing architecture.
To remedy this, we are now introducing a new shard-aware Kafka sink connector for Scylla. This will allow you to use Apache Kafka and the Confluent platform to their fullest advantages.
Using Scylla and Apache Kafka Together
Scylla’s high performance NoSQL database is a natural fit with Apache Kafka. Both support the massive scalability and high throughput required in modern event streaming data architectures.
Users are guaranteed flexibility, either via open source NoSQL database, enterprise-grade or cloud-hosted fully managed solutions for both technologies. For example, consider the following combinations:
- Scylla Open Source and open source Apache Kafka
- Scylla Enterprise and the Confluent Platform
- Scylla Cloud (NoSQL DBaaS) and Confluent Cloud managed cloud services
With the new Kafka Scylla Connector you now have the final missing piece with all the flexibility and performance you need to build the next generation of Big Data applications.
Learn More in Our Webinar
Learn more about the advantages of using Scylla together with the Confluent Cloud platform in our webinar Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL.