The Scylla team is pleased to announce the release of Scylla Enterprise 2018.1.0, a production-ready Scylla Enterprise major release.

The Scylla Enterprise 2018.1 release is based on Scylla Open Source 2.1 and includes new features which are only offered to our Enterprise customers, Enterprise version bug fixes, and backported bug fixes from upstream releases 1.7, 2.0, 2.1 and master.

Read more about Scylla Enterprise here.

Related Links

• Get Scylla 2018.1. (customers only, or 30-day evaluation)
• Upgrade from 2017.1.x to 2018.1
• Upgrade from Scylla Open Source 2.1 to Scylla 2018.1

Scylla Enterprise customers are encouraged to upgrade to Scylla Enterprise 2018.1 and are welcome to contact our Support Team with questions.

New Features in Scylla Enterprise 2018.1

Enterprise only features in 2018.1

• Auditing
  Scylla Enterprise includes an auditing capability, which allows administrators to track system and users activity for deployments with multiple users.
  Scylla auditing can work in Keyspace, Table or Category granularity. Categories are:
  AUTH Logs login events
  DML Logs insert, update, delete, and other data manipulation language (DML) events
  DDL Logs object and role create, alter, drop, and other data definition language (DDL) events
  DCL Logs grant, revoke, create a role, drop a role, and list roles events
  QUERY Logs all queriesThe audit log can be output the system log, or to a Scylla table. More on Auditing in Scylla Enterprise: http://docs.scylladb.com/operating-scylla/auditing.
• IBM Power support
  Scylla Enterprise 2018.1 is also available on the IBM POWER8 architecture, running on CentOS 7 and RHEL.

New Features in 2018.1
(Where mentioned, issue numbers are from the upstream open source project)

• Counters. Available in Scylla open source from 1.7, Counters are now part of Scylla Enterprise
  More on Counters: http://docs.scylladb.com/using-scylla/counters/
• Time Window Compaction Strategy (TWCS). An improvement and simplification to Date Tiered Compaction Strategy (DTCS) as the go-to strategy for time series data. TWCS uses Size-Tiered Compaction Strategy (STCS) on “time windowed“ groups of SSTables, using the maximum timestamp of the SSTable to decide which time window each SSTable will be part of. #1432
• Row Cache. In Pre 2.0 releases, Scylla cached full partitions only and we chose not to cache partitions larger than 10MB. With the introduction of row cache, Scylla can cache a single CQL row out of a large partition, removing this limitation. This change will significantly improve Scylla performance for use cases with large partitions such as Time Series.
• Heat Weighted Load Balancing. When a Scylla node restarts for any reason (upgrade, config update, error), it starts with an empty cache. This means the node read path will always hit the disk until it’s cache is repopulated. During this time, this node will be slower, and as a result, can slow down the entire cluster. Heat Weighted Load Balancing solves this issue by optimizing the node selection in the coordinator based on the heat of each node. A hot node, with a good cache-hit-ratio, will get more reads requests than a cold node with the same data and a lower cache hit ratio. After some time, the cold node will become hot, and the load balancing will become uniform again. #1455. More on Heat Weighted Load Balancing
• Improved Performance for CPU-bound workloads. More here.
• DNS resolution. You can now use DNS names (FQDN), not only IPs, in scylla.yaml for seed nodes, listen_address, broadcast_address and broadcast_rpc_address #2357.
• CQL additions:
  • Cast functions between types, for example: SELECT avg(cast(count as double)) FROM myTable.
  • Support Duration data type #2240.
  • CQL prepared statements are now properly evicted so that you’re not able to OOM a server just by preparing many unique statements. See #2474
  • Support non-reserved keywords as columns (without quotes) #2507, for example, a column named “frozen”.
  • Update to CQL 3.3.1, to align with Cassandra 2.2 release
  • New data types: tinyint, smallint, date, and time #1284
    Binary protocol v4 (required by the new data types).
  • Support UNSET in prepared statements #2039
  • CQL aggregate functions now support floating point data types #2168
• Transitional Auth – To enable access control without downtime, use transitional modes to temporarily support applications and users that do not have accounts without interrupting services. More here
• New REST API /storage_service/force_terminate_repair allows aborting of a running repair and all the data streams related to it. #2105.
• Ext4 support. Scylla now supports the ext4 file system in production, not just in dev mode. Using XFS is still recommended for best performance #2381.
• CPU isolation. Improve latency using isolation between background tasks like memtable flushes and compaction and foreground tasks like CQL read/write. More here
• Java tools. Java tools provided with Scylla are upgraded to match Cassandra 3.0. In particular:
  • sstableloader supports Cassandra 3.0 sstable format, allowing easy migration from Cassandra 2.2 and 3.x clusters to Scylla
  • sstable2json is renamed to sstabledump
  • sstablesplit, sstableupgrade, sstablescrub, json2sstable, sstablekeys, and token-generator are temporarily removed (all break compatibility to current SSTable format)
• CompressionInfo.db files, part of the SSTable format, are stored more efficiently in memory, allowing higher disk: RAM ratios #1946.

System Impacts

• Installation updates
  Starting from Scylla Enterprise 2018.1, Scylla packages for Ubuntu 16.04 and Debian 8 are signed. You can find the keys and instructions per Linux distribution here.
• Upgrade procedure. You can only upgrade to Scylla Enterprise 2018.1 from Scylla Enterprise 2017.1.6 or later, or from Scylla Open Source 2.1 or later. Upgrade from earlier Scylla releases is not supported.
• Kernel support
  Scylla Enterprise 2018.1 requires Linux kernel 3.10.0-514 or later.
• Driver support
  Relevant for upgrading from Scylla Enterprise 2017.1 (ignore if you are upgrading from Scylla Open Source 2.1).Certain older versions of Cassandra drivers are no longer supported. With the introduction of MV, Scylla has moved to Cassandra 3.0 compatible system tables. This means that certain old Cassandra drivers, which do not work with Cassandra 3.0, will not work with Scylla.To ensure your application will work properly after a Scylla upgrade, we highly recommend upgrading your Cassandra driver before upgrading to Scylla 2018.1. The latest drivers are backward compatible and will work with all Scylla 2017.1, 1.x and 2.x releases, past, and future.At the very least, your Cassandra driver should be in the list below, using the version listed or later.
  – C++ : 2.3+
  – C#: 3.0+
  – Java: 3.0+
  – Node.js: 3.0+
  – PHP: 1.2+
  – Python: 3.5+
  – Ruby: 3.0+
  – GoCQL commit 8d27cdeae224463cadc1ee973386862baaf6662e (Mar 1, 2016) or later.
• Counter Migration. Scylla does not support Counter SSTables created by Apache Cassandra 2.0 or older. Note that if you upgraded from Cassandra 2.0 to 2.1, you may still have old Counters in the SSTables.The proper way to migrate Counters data from Cassandra to Scylla is to run a major compaction on the Cassandra node, and use sstableloader to load the SSTable to Scylla.
• Deprecated partitioners. Byte-ordered and random partitioners were deprecated in Scylla Open Source 2.0, and Scylla Enterprise 2018.1 and will be removed in a future version. Please open a support ticket if you use or are planning to use these partitioners.
• New intra-node sharding algorithmThe intra-node sharding algorithm determines how data is divided among cores in a node. The change results in a better distribution of data among cores, especially in large clusters. This improves performance across the cluster. When upgrading from 2017.1, please check scylla.yaml for murmur3_partitioner_ignore_msb_bits. If you do *not* have this line, please consult Scylla support on how to safely upgrade.

Metrics updates from 2017.1

Scylla Monitoring stack now includes Scylla 2018.1 dashboard. For a full list of metrics, update see here.

Known Issues

When installing Scylla Enterprise on Power Architecture, scylla_setup fail to install node_exporter. Without node_exporter OS level metric is not reported to Scylla Monitoring Stack. Bypass: install node_exporter packages using the public repo.

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Scylla Enterprise Release 2018.1.0 appeared first on ScyllaDB.

The Intel Memory Group is behind the revolutionary Optane SSD drive that provides breakthrough performance and is 5-8x faster at Low Queue Depths than traditional SSD’s. Intel began working with ScyllaDB staff last year to build a big memory system at high-volume scale. They chose Scylla because they needed a solution that can fully leverage the hardware to derive the best possible performance.

The Intel team likes that Scylla is drop-in compatible with Apache Cassandra and didn’t require any changes to their applications. They also appreciate that Scylla optimizes workloads without manual user intervention or downtime.

“With Scylla you’ve got open source development methodology providing that very fast delivery so you’ve got the best of today’s worlds.”  Frank Ober, Solutions Architect, Intel

Watch the video below to see what Frank Ober of Intel has to share about his experience with Scylla.

Next Steps

• • Learn more about Scylla from our product page.
  • See what our users are saying about Scylla.
  • Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
  • Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Intel’s Memory Group on the Advantages of Scylla appeared first on ScyllaDB.

This is part 10 of a series of blog posts that provides a story arc for Scylla Training.

In the previous post, we learned how developers at Division 3 can create applications that can interact with the Mutant Monitoring System by showing how to use the Cassandra libraries available in programming languages. Due to recent violent events in the mutant community, Division 3 implemented a new policy for Scylla Administrators to learn how to backup and restore the mutant data in the cluster. In this post, we will learn how to backup and restore data from the Mutant Monitoring System.

Starting the Scylla Cluster

The Scylla Cluster should be up and running with the data imported from the previous blog posts.

The MMS Git repository has been updated to provide the ability to automatically import the keyspaces and data. If you have the Git repository cloned already, you can simply do a “git pull” in the scylla-code-samples directory.

git clone https://github.com/scylladb/scylla-code-samples.git
cd scylla-code-samples/mms

Modify docker-compose.yml and add the following line under the environment: section of scylla-node1:

- IMPORT=IMPORT

Now we can build and run the containers:

docker-compose build
docker-compose up -d

After roughly 60 seconds, the existing MMS data will be automatically imported. When the cluster is up and running, we can run our application code.

Backing Up the Data

First, we need to backup our schema with the following commands:

docker exec -it mms_scylla-node1_1 bash
cqlsh -e "DESC SCHEMA" > /schema.cql

With the schema backed up, we can create a snapshot of the two keyspaces used for the Mutant Monitoring System: tracking and catalog. Snapshots are taken using nodetool snapshot. The command first flushes the MemTables from memory to SSTables on disk, then creates a hard link for each SSTable in each keyspace. With time, SSTables are compacted, but the hard link keeps a copy of each file. This takes up and increasing amount of disk space.

nodetool snapshot tracking

Requested creating snapshot(s) for [tracking] with snapshot name [1524840557127] Snapshot directory: 1524840557127

nodetool snapshot catalog

Requested creating snapshot(s) for [catalog] with snapshot name [1524840566260] Snapshot directory: 1524840566260

The snapshot is created in the Scylla data directory /var/lib/scylla/data and It will have the following structure: keyspace_name/table_name-UUID/snapshots/snapshot_name.

Simulating Data Loss

All of the data is backed up on node1 only. Division 3 must now prepare to handle cyber attacks from the mutants and other external organizations. In this scenario, we will simulate such an attack by using cqlsh to delete the Mutant Monitoring Keyspaces.

cqlsh
drop keyspace tracking;
drop keyspace catalog;

To verify that the keyspaces are gone, run the following command:

describe keyspaces;

The output should be:

system_traces system_schema system

Great, we only have the default keyspaces now. We can now learn how to restore the data.

Restoring the Data

To restore the data, we first need to re-create the keyspaces from the backup done previously:

docker exec -it mms_scylla-node1_1 bash
cqlsh -e "SOURCE '/schema.cql'"

Run the nodetool drain command to ensure the data is flushed to the SSTables:

nodetool drain

Next, we will need to kill the Scylla processes in each container and delete the contents of the commit log. In each container, run the following commands:

pkill scylla
rm -rf /var/lib/scylla/commitlog/*

Repeat for containers mms_scylla-node2_1 and mms_scylla-node3_1.

Return back to mms_scylla-node1_1:

docker exec -it mms_scylla-node1_1 bash

The first snapshot restore will be for the catalog keyspace. We first need to find the original snapshot folder which is the oldest directory in /var/lib/scylla/data/catalog:

cd /var/lib/scylla/data/catalog/
ls -al

total 16
drwx------ 4 root root 4096 Apr 27 15:03 .
drwxr-xr-x 1 scylla scylla 4096 Apr 27 14:40 ..
drwx------ 3 root root 4096 Apr 27 15:03 mutant_data-32afab304a2c11e8b776000000000002
drwx------ 4 root root 4096 Apr 27 14:58 mutant_data-dcffd0a04a2811e89412000000000000

Luckily the directories are numbered sequentially. The original data directory with the snapshot is mutant_data-dcffd0a04a2811e89412000000000000 and the current data directory created after importing the schema is mutant_data-32afab304a2c11e8b776000000000002. We can now copy the contents from the snapshot directory to the new data directory:

cp -rf mutant_data-dcffd0a04a2811e89412000000000000/snapshots/1524840566260/* mutant_data-32afab304a2c11e8b776000000000002/

Repeat the process for restoring the tracking keyspace in /var/lib/scylla/data/tracking. When that is complete, we can start Scylla in each container with the following commands:

export SCYLLA_CONF='/etc/scylla'
/usr/bin/scylla --developer-mode 1 --options-file /etc/scylla/scylla.yaml&
/usr/lib/scylla/jmx/scylla-jmx -l /usr/lib/scylla/jmx&

Repeat for mms_scylla-node2_1 and mms_scylla-node3_1. It will take a few minutes for the nodes to form a cluster. You should see messages like this:

INFO 2018-04-27 17:00:35,574 [shard 0] gossip - Connect seeds again ... (11 seconds passed)
INFO 2018-04-27 17:00:36,574 [shard 0] gossip - Connect seeds again ... (12 seconds passed)
INFO 2018-04-27 17:00:37,575 [shard 0] gossip - Connect seeds again ... (13 seconds passed)

When the nodes form a cluster, you should see similar messages as shown below:

INFO 2018-04-27 17:05:27,927 [shard 0] storage_service - Node 172.18.0.3 state jump to normal
INFO 2018-04-27 17:05:28,024 [shard 0] gossip - Favor newly added node 172.18.0.3
INFO 2018-04-27 17:05:28,058 [shard 0] storage_service - Node 172.18.0.2 state jump to normal
INFO 2018-04-27 17:05:29,034 [shard 0] gossip - Favor newly added node 172.18.0.2

After the nodes are online, we can run a repair and then verify that our data is restored properly with the following commands:

nodetool repair

You should see many messages like this:

INFO 2018-04-27 17:31:28,744 [shard 1] repair - Repair 302 out of 498 ranges, id=1, shard=1, keyspace=system_traces, table={sessions_time_idx, node_slow_log_time_idx, events, sessions, node_slow_log}, range=(1506223645386967449, 1537077568234766066]

This means that repair is syncing the data across the cluster. The entire process may take a few minutes. When the repair process is complete, we can run queries to ensure that the data is restored to both keyspaces:

cqlsh -e 'select * from tracking.tracking_data;'
cqlsh -e 'select * from catalog.mutant_data;'

Conclusion

In conclusion, we followed the directive from Division 3 to teach each Scylla Administrator to learn how to backup and restore a cluster. The process began with backing up the schema followed by creating a snapshot of the data. We simulated an attack by deleting the keyspaces and then restoring and repairing the cluster. Stay safe out there and backup as much as you can!

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Mutant Monitoring System Day 10 – Backup and Restore appeared first on ScyllaDB.

The Scylla team is pleased to announce the release of Scylla 2.1.3, a bugfix release of the Scylla 2.1 stable branch. Release 2.1.3, like all past and future 2.x.y releases, is backward compatible and supports rolling upgrades.

Related Links:

• Get Scylla 2.1.3 – Docker, binary packages, and EC2 AMI
• Get started with Scylla 2.1
• Upgrade from 2.1.x to 2.1.y
• Please let us know if you encounter any problems

Bugs Fixed in this Release:

• • Dropping a keyspace with a User Defined Type (UDT) results in an error: “InvalidRequest: Error from server: code=2200 [Invalid query] message=”Unknown type…”. #3068
  • sstableloader: the SSTable files to Scylla upload tool does not support User Defined Type(UDT). scylla-tools-java #57

The post Scylla Release 2.1.3 appeared first on ScyllaDB.

The Scylla team is pleased to announce the release of Scylla 2.0.4, a bugfix release of the Scylla 2.0 stable branch. Release 2.0.4, like all past and future 2.x.y releases, is backward compatible and supports rolling upgrades.  Please note the latest stable open source release is Scylla 2.1.3, and you are encouraged to upgrade to it.

Related Links:

• Get Scylla 2.0.4 – Docker, binary packages, and EC2 AMI
• Get started with Scylla 2.0
• Upgrade from 2.0.x to 2.0.y
• Please let us know if you encounter any problems

Bugs Fixed in this Release:

• • Streaming sender-side failures were not propagated to the receiving side. #1743
  • Streaming many column families at the same time generated many small sstables on the receiving side. #3065
  • Streaming may overload a new node which fails to boot as a result. #3258, 3311
  • CQL prepared statements are now properly evicted, thus preventing server OOM when preparing many unique statements. #2747
  • Authentication requests might return expired permission from cache. #2590
  • Dropping a keyspace with a User Defined Type(UDT) resulted in an error: “InvalidRequest: Error from server: code=2200 [Invalid query] message=”Unknown type…”. #3068
  • Scylla aborted with an “Assertion `end >= _stream_position’ failed” exception. This occurred when querying a partition with no clustering ranges (happened on a counter table with no live rows) which also didn’t have static columns. #3304
  • In some rare cases when one or more nodes were down, a Scylla gossip bug caused Scylla to exit with a segmentation fault. #3299
  • In some cases, a removed node (removed using nodetool) failed to bootstrap. #3331
  • A TLS session can be closed while an IO continuation is still in flight, causing a “system:32, Broken pipe” error #3358
  • In a case where gossip information is not available yet, for example during a cluster init, nodetool gossipinfo may fail with NullPointerException. #3330
  • sstableloader does not support User Defined Type(UDT). scylla-tools-java #57
  • sstableloader cannot bind prepared statement for a table using “date” type. scylla-tools-java #59

The post Scylla Release 2.0.4 appeared first on ScyllaDB.

This is part 11 of a series of blog posts that provides a story arc for Scylla Training.

In the previous post, we explained how a Scylla Administrator can backup and restore a cluster. As the number of mutants is on the rise, Division 3 decided that we must use more applications to connect to the mutant catalog and decided to hire Java developers to create powerful applications that can monitor the mutants. In this post, we will explore how to connect to a Scylla cluster using the Cassandra driver for Java.

When creating applications that communicate with a database such as Scylla, it is crucial that the programming language being used has support for database connectivity. Since Scylla is compatible with Cassandra, we can use any of the available Cassandra libraries. For example in Go, there is Gocql and Gocqlx. In Node.js, there is cassandra-driver. For Java, we have the driver available from Datastax. Since Division 3 wants to start investing in Java, let’s begin by writing a sample Java application.

Creating a Sample Java Application

The sample application that we will go over will connect to a Scylla cluster, display the contents of the Mutant Catalog table, insert and delete data, and show the contents of the table after each action. We will first go through each section of the code and then explain how to run the code in a Docker container that will access the Scylla Mutant Monitoring cluster.

In the application, we first need to import the Datastax Cassandra driver for Java:

The Cluster library provides the ability to connect to the cluster. The ResultSet library will allow us to run queries such as SELECT statements and store the output. The Row library allows us to get the value of a CQL Row returned in a ResultSet. The session library provides the ability to connect to a specific keyspace.

For this application, the main class is called App and the code should be stored in a file called App.java. After the class is defined, we can begin defining which cluster and keyspace to use:

To display the data, we will need to create a function that will run a select statement from the Scylla cluster. The function below will use the ResultSet, Session, and Row libraries to gather the data and print it on the screen:

The next function is used to insert a new Mutant into the Mutant Catalog table. Since this is a simple query, only the Session library is used. After the insert takes place, the previous function, selectQuery(), is called to display the contents of the table after the new data is added.

After the data is added and displayed in the terminal, we will delete it and then display the contents of the table again:

This is the main method of the application and is the starting point for any Java application. In this function, we will add the functions that were created above to be executed in the order below. When all of the functions are completed, the application will exit.

With the coding part done, let’s bring up the Scylla Cluster and then run the sample application in Docker.

Starting the Scylla Cluster

The Scylla Cluster should be up and running with the data imported from the previous blog posts.

The MMS Git repository has been updated to provide the ability to automatically import the keyspaces and data. If you have the Git repository already cloned, you can simply do a “git pull” in the scylla-code-samples directory.

git clone https://github.com/scylladb/scylla-code-samples.git
cd scylla-code-samples/mms

Modify docker-compose.yml and add the following line under the environment: section of scylla-node1:

- IMPORT=IMPORT

Now we can build and run the containers:

docker-compose build
docker-compose up -d

After roughly 60 seconds, the existing MMS data will be automatically imported. When the cluster is up and running, we can run our application code.

Building and Running the Java Example

To build the application in Docker, change into the java subdirectory in scylla-code-samples:

cd scylla-code-samples/mms/java


Now we can build and run the container:

docker build -t java .
docker run -d --net=mms_web --name java java


To connect to the shell of the container, run the following command:

docker exec -it java sh

Finally, the sample Java application can be run:

java -jar App.jar

The output of the application will be:

Conclusion

In this post, we explained how to create a sample Java application that executes a few basic CQL statements with a Scylla cluster using the Datastax Cassandra driver. This is only the basics and there are more interesting topics that Division 3 wants developers to explore. In the next post, we will go over prepared statements using the Java driver.

In the meantime, please be safe out there and continue to monitor the mutants!

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Mutant Monitoring System (MMS) Day 11 – Coding with Java Part 1 appeared first on ScyllaDB.

This is part 12 of a series of blog posts that provides a story arc for Scylla Training.

In the previous post, we explained how to create a sample Java application that executes a few basic CQL statements with a Scylla cluster using the Datastax Cassandra driver. After the code was deployed, we found that several citizens were murdered by mutants because the code was too static and not scalable. Changes must be made in order for Division 3 to protect people better by building highly-scalable and performing applications to monitor mutants. In this post, we will explore how we can optimize the existing Java code with prepared statements.

What Are Prepared Statements?

Prepared statements will enable developers at Division 3 to optimize our applications more efficiently. Most or all of the cassandra-compatible drivers support prepared statements. With that in mind, what you learn here can benefit you regardless of the programming language used. A prepared statement is basically a query that is parsed by Scylla and then saved for later use. One of the useful benefits is that you can continue to reuse that query and modify variables in the query to match variables such as names, addresses, and locations. Let’s dive a little deeper to see how it works.

When asked to prepare a CQL statement, a client library will send a CQL statement to Scylla. Scylla will then create a unique fingerprint for that CQL statement by MD5 hashing the CQL statement. Scylla will use this hash to check its query cache to see if it has already cached that CQL statement. If Scylla had seen that CQL statement, it will send back a reference to that cached CQL statement. If Scylla does not have that unique query hash in its cache, it will then proceed to parse the query and insert the parsed output into its cache.

INSERT INTO tb (key, val) VALUES (?,?)

The client will then be able to send an execute request specifying the statement id and providing the (bound) variables.

For more information on prepared statements, please click here. Now let’s go over how to change our existing Java application to support prepared statements.

Changing the Existing Java Application

To get started, we will first need to add a few libraries to our application:

import com.datastax.driver.core.PreparedStatement;
import com.datastax.driver.core.BoundStatement;

The PreparedStatement and BoundStatement libraries provide the functions to create prepared statements. Moving on, we can add two prepared statements to our application:

The first prepared statement is named insert. This statement will add data programmatically for first_name, last_name, address, and picture_location based on input from the application. The second prepared statement is named delete and will delete entries in the table based input gathered for first_name and last_name. We will reuse these statements later to add and delete data in the mutant_data table.

The first section of the application to replace is the insertQuery function as follows:

This function will take input for first_name, last_name, address, and picture_location and then bind to our prepared statement named insert and execute the query. By using prepared statements, we can reuse these functions over and over to add data to the catalog table.

The second section of the application to replace is the deleteQuery function as follows:

In this function, we will take first_name and last_name inputs and then bind and execute the delete prepared statement. Using this prepared statement, we can reuse these functions over and over to delete data from the catalog table.

Finally, we need to modify the main function as follows to pass input to the functions when the application starts:

First, the contents of the catalog table will be displayed followed by calling the insertQuery functions to add two additional mutants. After each insert is done, the contents of the table will be displayed. Finally, each user that was added is deleted and the contents of the table is shown after each delete.

With the coding part done, let’s bring up the Scylla Cluster and then run the sample application in Docker.

Starting the Scylla Cluster

The Scylla Cluster should be up and running with the data imported from the previous blog posts.

The MMS Git repository has been updated to provide the ability to automatically import the keyspaces and data. If you have the Git repository already cloned, you can simply do a “git pull” in the scylla-code-samples directory.

git clone https://github.com/scylladb/scylla-code-samples.git
cd scylla-code-samples/mms

Modify docker-compose.yml and add the following line under the environment: section of scylla-node1:

- IMPORT=IMPORT

Now we can build and run the containers:

docker-compose build
docker-compose up -d

After roughly 60 seconds, the existing MMS data will be automatically imported. When the cluster is up and running, we can run our application code.

Building and Running the Java Example

To build the application in Docker, change into the java subdirectory in scylla-code-samples:

cd scylla-code-samples/mms/java


Now we can build and run the container:

docker build -t java .
docker run -d --net=mms_web --name java java


To connect to the shell of the container, run the following command:

docker exec -it java sh

Finally, the sample Java application can be run:

java -jar App.jar

The output of the application will be:

Conclusion

In this post we explained what prepared statements are and how they can enable developers at Division 3 to optimize their applications more efficiently. We also learned how to modify our existing Java application to take advantage of prepared statements. Division 3 recommends that you keep experimenting with prepared statements and continue to make your applications more efficient.

Stay safe out there!

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Mutant Monitoring System (MMS) Day 12 – Coding with Java Part 2 appeared first on ScyllaDB.

This is a guest blog post by Soumya Simanta, an architect at Ola Cabs, the leading ride-hailing service in India.

On-demand ride-hailing is a real-time business where responding to quick spikes in demand patterns is a critical need. These spikes are more profound during unusual circumstances such as bad weather conditions, special events, holidays, etc. It is critical that our software systems are architected to support this real-time nature of our business. However, creating a distributed software system that can satisfy this requirement is a challenge. An important component of any web-scale system is the database. Given the rapid growth of our organization, we wanted our choice of database to support some critical quality attributes. Our primary concerns were support for high throughput, low latency, and high availability (multi-zone and multi-datacenter support). Other key requirements were that the product is open source, required minimal maintenance and administration, had a rich ecosystem of tools and, finally, a database that is battle-tested in production.

Like many web-scale companies, we quickly realized that we do not need an ACID-compliant database for all our use cases. With proper design, we could model many of our systems around “eventually consistency,” thereby trading-off consistency while gaining all the other goodness that comes with an AP-compliant database such as Cassandra. Although we were already using Cassandra for some use cases, we were not sure if it was the best long-term fit for us. Primarily because it leverages the JVM runtime and therefore has the well-known latency issues and tuning overhead that comes with the JVM’s garbage collector.

Around early 2016 a new database, Scylla, caught our attention. The primary reason for our interest was that Scylla was advertised as a drop-in replacement for Cassandra, was written in native language (C++) and was developed by the creators of battle-tested software products such as the KVM hypervisor, OSv, and Seastar. While we recognized that Scylla did not yet support all the features of Cassandra and that it was not yet battle-tested in production, we were intrigued by the close-to-the-hardware approach they had taken in building their database. Very early on, we had discussions with Scylla’s core team and they assured us that they would soon add many important features. With this assurance, we were confident that a long-term bet on Scylla would yield a positive outcome for us. We created a roadmap where we would gradually adopt Scylla into our software ecosystem based on application criticality.

For our first use case in March of 2016, we deployed Scylla 1.0 in passive mode along with another database as our primary datastore. Only writes were performed to Scylla. As expected, Scylla being write-optimized by design performed reasonably well in terms of latency and throughput. However, what was surprising for us was the stability of the database. There were no errors or crashes and it ran without any maintenance for more than 3 months. This was very promising for us. Next, we started using Scylla 1.4 for both reads and writes. We had to modify our data model, which is a limitation of the Cassandra architecture itself and not specific to Scylla. For the initial few cases, we performed our installation of Scylla from scratch. However, we quickly realized that we were not getting the performance that was advertised. So we moved to an official AMI image that has tuned OS and database configurations provided by the ScyllaDB folks.

Our initial configurations were based on using EBS volumes. The advantage of using EBS was the protection against data loss in case of node failures. However, for certain use cases, we decided to move to ephemeral (NVMe) disks. In another system, we used Scylla as a persistent cache where we could tolerate the risk of data loss due to ephemeral node failures. We mitigated the risk of disk failures by setting the correct replication factor, storing data across multiple Availability Zones (AZs) and ensuring that we could regenerate data quickly. This gave us a significant performance boost in terms of latencies because Scylla is optimized for NVMe and an extra network hop is avoided as compared to using EBS.

We have deployed Scylla across multiple microservices and platforms in our organization. We currently run two types of workloads. First are real-time reads at high throughput along with batch writes from our machine learning pipelines and Spark jobs at daily or hourly intervals. The second type of workload is cases with a uniform mix of reads and writes. We initially used the standard Cassandra driver from Java. For the last year, we have been using the Gocql connector for Golang for most of our new use cases. In our experience, our overall application latencies provided by the Go-based apps are better with less jitter and significantly less application memory footprint when compared with Java-based applications. In most cases, we are using Scylla without a caching layer because by using the proper hardware for the server nodes we were able to get our target latency profiles (< 5ms 99-tile) with just Scylla. This not only resulted in a simplified architecture but also helped us save on the hardware cost for the caching layer.

Figure 1. Application Latency Profile. Application Go code running on Docker using Gocql driver for Scylla 2.1 on a 5 node i3.8xlarge cluster)

Figure 2. Scylla Prometheus Dashboard (for the application shown in Figure 1)

Figure 3: Application Latency Profile

Another valuable feature that was introduced in Scylla was native metrics monitoring support for Prometheus. This was extremely helpful in debugging and identifying bottlenecks in our data model and sharing our results for debugging with the Scylla community. Sharing all the metrics with the Scylla community and core developers reduced the performance debugging cycle. Additionally, the metrics enabled us to make cost-performance tradeoffs by selecting the right hardware specs for our latency/throughput requirements. Another takeaway was that by tweaking the schema (e.g., reduction in partition sizes) we could get significant improvements in performance. We highly recommend that all Scylla users configure the out-of-box monitoring provided by Scylla from day one.

In our two-year journey of using it in production, Scylla has lived up to our expectations. We have graduated from using Scylla for very simple and non-critical use cases to deploying it for some of our mission-critical flows. Scylla’s development team has delivered on their core promise of doing continuous releases to push out new features but has never compromised on the quality or performance of their product. The team is also very receptive to its community’s needs and is always ready to help on Scylla’s Slack channel and mailing lists. Some features we are looking forward to are the secondary indexes (currently in experimental mode), incremental repair, support for Apache Cassandra 3.x storage (finally patches arrived in the dev mailing list) and user-defined types (UDFs).

The post Ola Cabs on Their First Two Years of Using Scylla in Production appeared first on ScyllaDB.

The Scylla team is pleased to announce the release of Scylla Enterprise 2018.1.0, a production-ready Scylla Enterprise minor release. Scylla Enterprise 2018.1.1 is a bug fix release for the 2018.1 branch, the latest stable branch of Scylla Enterprise.

The 2018.1 branch is based on Scylla open source 2.1 and includes backported bug fixes from upstream releases (1.7, 2.0, 2.1) as well as enterprise-only bug fixes. Read more about Scylla Enterprise here.

Related Links

• Get Scylla 2018.1.1 (customers only, or 30-day evaluation)
• Upgrade from 2017.1.x to 2018.1.1
• Upgrade from Scylla Open Source 2.1 to Scylla 2018.1
• Upgrade from Scylla 1.6 to Scylla 2018.1
• Submit a ticket

Scylla Enterprise customers are encouraged to upgrade to Scylla Enterprise 2018.1.1 in coordination with the Scylla support team.

Bug Fixes in this Release (With Open Source Issue Number Where Applicable):

• Upgrading to latest version of the RHEL kernel causes Scylla to lose access to the RAID 0 data directory. #3437 A detailed notice has been sent to all relevant customers.
• An upgrade from Scylla 2017.1 to 2018.1 may cause frequent and redundant schema updates.  #3394
• Multi-DC writes may fail during schema changes. #3393
• Installing Scylla 2018.1 on an old CentOS kernel results in the following error “systemd[5370]: Failed at step CAPABILITIES spawning /usr/bin/scylla: Invalid argument”. In Scylla 2018.1.1 the dependency on kernels later than kernel-3.10.0-514.el7 is explicitly added to Scylla Enterprise package spec, making Scylla installations on older kernels impossible. #3176

Known Issues

Scylla Enterprise 2018.1.1 was not validated on IBM POWER8 architecture. If you are using 2018.1. on Power, please wait for future notices on upgrade availability. In the meantime, you should explicitly install Scylla Enterprise 2018.1.0, not 2018.1.

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Scylla Enterprise Release 2018.1.1 appeared first on ScyllaDB.

A web-based photo-sharing and printing service, Snapfish is dedicated to helping people transform their photos into beautiful personalized objects, such as photo books, mugs, prints and gifts that can be shared and treasured. On a typical day, Snapfish handles more than 1 million uploads. During the holiday season, that number can easily quadruple. These real-time numbers are compounded by backups, which have to move 40 billion files a year via 80,000 simultaneous reads and writes per minute, without impacting production workloads.

The NoSQL database platform in use at Snapfish wasn’t up to the task, so the team evaluated other options. Scylla hit Snapfish’s production throughput requirement of 3,000 read operations per second. The team saw even better results after moving to a 3-node Scylla cluster running on AWS. In the end, performance improved by 5X using Scylla as compared to MongoDB.

“What initially interested us is the number of concurrent operations that Scylla supports. In the end, Scylla turned out to be the fastest NoSQL out there.” Brent Williams, Principal Engineer, Snapfish

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Snapfish Moves from MongoDB to Scylla for Global Scale and High Performance appeared first on ScyllaDB.

The Scylla team is pleased to announce the release of Scylla Enterprise 2018.1.2, a production-ready Scylla Enterprise minor release. Scylla Enterprise 2018.1.2 is a bug fix release for the 2018.1 branch, the latest stable branch of Scylla Enterprise.

The 2018.1 branch is based on Scylla open source 2.1 and includes backported bug fixes from upstream releases (1.7, 2.0, 2.1) as well as enterprise-only bug fixes. Read more about Scylla Enterprise here.

Related Links

• Get Scylla 2018.1.2 (customers only, or 30-day evaluation)
• Upgrade from 2017.1.x to 2018.1.2
• Upgrade from Scylla Open Source 2.1 to Scylla 2018.1
• Upgrade from Scylla 1.6 to Scylla 2018.1
• Submit a ticket

Scylla Enterprise customers are encouraged to upgrade to Scylla Enterprise 2018.1.2 in coordination with the Scylla support team.

Scylla Enterprise 2018.1.2 solved the IBM POWER8 architecture issue reported for 2018.1.0. You can now download and install Scylla Enterprise 2018.1 for RHEL on IBM POWER8.

Additional Issues Solved in This Release (with Open Source Issue Reference When Applicable)

• Closing a secure connection (TLS) may cause a core dump.
• Wrong Commit log error handling may cause a core dump. #3440

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Scylla Enterprise Release 2018.1.2 appeared first on ScyllaDB.

This is part 13 of a series of blog posts that provides a story arc for Scylla Training.

In the previous post, we went over prepared statements and how developers at Division 3 can optimize their applications more efficiently by using them. The Mutant Monitoring System had been receiving a lot of data and Division 3 wanted to find better ways to sort and store data so it can be quickly analyzed with applications. Luckily, having found an exciting feature in Scylla called Materialized Views, they provided us with directives to learn how to use it to help our application developers prevent further acts of terror. In this post, we will learn what Materialized Views are and how to use it with the Mutant Monitoring System.

What are Materialized Views?

Material Views automate the tedious and inefficient work that must be done when an application maintains several tables with the same data that’s organized differently. In the past, every time the application wanted to write data, it needed to write to both tables, and reads were done directly (and efficiently) from the desired table. However, ensuring any level of consistency between the data in the two or more views required complex and slow application logic.

Scylla’s Materialized Views feature moves this complexity out of the application and into the servers. This approach makes it much easier for applications to have multiple views of their data. The application just declares the additional views, new view tables are created, and on every update to the original table (known as the “base table”), the additional view tables are automatically updated as well.

Please note that Materialized Views are not production ready and are still considered experimental in Scylla 2.1, 2.2 and 2.3.

To get started with Materialized Views, we will first need to bring up the Scylla Cluster.

Starting the Scylla Cluster

The Scylla Cluster should be up and running with the data imported from the previous blog posts.

The MMS Git repository has been updated to provide the ability to automatically import the keyspaces and data. If you have the Git repository already cloned, you can simply do a “git pull” in the scylla-code-samples directory.

git clone https://github.com/scylladb/scylla-code-samples.git
cd scylla-code-samples/mms

Modify docker-compose.yml and add the following line to the environment: section of scylla-node1:

- IMPORT=IMPORT

Now we can build and run the containers:

docker-compose build
docker-compose up -d

After roughly 60 seconds, the existing MMS data will be automatically imported. When the cluster is up and running, we can learn how to create a few Materialized Views.

Starting the Load Generator

Before we create a Materialized View, we will need to use a load generator tool that will start adding tracking data for each mutant in the Scylla cluster. To get started, we will first need to build the Docker container:

cd scylla-code-samples/mms/loadgen
docker build -t loadgen .

To start the load generator, run the following command:

docker run -d --net=mms_web --name loadgen loadgen

After 60 seconds, the load generator will begin populating the keyspace with data. You will be able to see when it starts by viewing the container logs:

docker logs -f loadgen

Creating the Materialized Views

Now let’s connect to a container running Scylla so we can use cqlsh to interact with the cluster:

docker exec -it mms_scylla-node1_1 cqlsh

use tracking;

Now it is time to create our first Materialized View. According to our original table schema for tracking, tracking_data, our primary keys are first_name, last_name, and timestamp. When creating a Materialized View, we will need to reference all of the primary keys that were used when creating the tracking_data table:

Let’s assume that Division 3 wants to retrieve only the mutant’s timestamp at a certain location. The following Materialized View will show only the timestamp and location for each mutant without the other data columns:

Now we will be able to see the data from the Materialized View with the following command:

To narrow the results down by city, we can add a location to the select query. Let’s see who is in Cincinnati:

Conclusion

In this post, we went over Materialized Views and showed a few examples of how to create views from the Mutant Monitoring System data. With Materialized Views, developers will be able to create intensive applications that can store and sort through data faster and more efficiently with less coding because those tasks can be handled by the Scylla cluster rather than in the applications themselves.

Please be safe out there and continue to monitor the mutants!

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Mutant Monitoring System (MMS) Day 13 – Materialized Views appeared first on ScyllaDB.

Numberly is a leading programmatic advertising firm that helps brands collect and activate consumer data across digital channels. The company’s multi-platform, multi-device, multi-format solutions include real-time bidding, targeted display advertising, CRM program and an advanced data management platform. A big part of Numberly’s service offering is helping clients make sense of multiple sources of data to build and improve their relationships with customers and prospects.

Alexys Jacob-Monier, Numberly’s Chief Technology Officer, wrote a blog post about evaluating and choosing the right NoSQL data store, in which he explains, “Our line of business is tied to our capacity at mixing and correlating a massive amount of different types of events coming from various sources that all have their own identifiers.”

Challenges with MongoDB and Hadoop

Numberly had previously handled real-time ID matching using MongoDB and batch ID matching using Apache Hive. This required them to maintain two copies of every ID matching table. Neither MongoDB nor Hive was able to sustain both the read/write lookup/update ratio while performing within the low latencies that Numberly’s SLAs required, so the company was saddled with the operational burden of ensuring data consistency between the two data stores.

“ScyllaDB is the kind of Open Source project we love at Numberly: a core of talented and benevolent people powering a smart and lightning fast piece of software!” –Alexys Jacob-Monier, Chief Technology Officer, Numberly

They found that MongoDB’s primary/secondary architecture hindered performance because of a loss of write throughput. Alexys summarized his 7-year MongoDB experience with, “To say the least, it is inefficient and cumbersome to operate and maintain.”

Numberly looked at using Apache Cassandra but was apprehensive about using a Java solution because of the heap sizes and tuning, and garbage collection that can temporarily stop the service.

Numberly was looking for a database that could sustain their ID matching tables’ workloads while maintaining consistently low upsert/write and lookup/read latencies. They realized that the right database would not only reduce costs but also lead to better data consistency and result in greater operational simplicity.

Scylla Lowers Cost and Complexity

Numberly started by running Scylla through rigorous load testing. Scylla passed with flying colors, leading Numberly to replace their 15-node Mongo cluster with a 3-node Scylla cluster.

“ScyllaDB is the kind of Open Source project we love at Numberly: a core of talented and benevolent people powering a smart and lightning fast piece of software!”

With its production Scylla Enterprise deployment, Numberly has realized a number of benefits, including:

• Cost savings
• Ease of operations–reducing the footprint by 80%
• Production reliability
• Data consistency
• Datacenter awareness
• Performance boosts

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Numberly Moves from MongoDB to Scylla to Simplify Operations and Reduce Costs appeared first on ScyllaDB.

Got a Real-Time Big Data Story to Share?

Scylla Summit 2018 is coming to the San Francisco Bay Area November 6-7. Our sessions will cover real-world Scylla use cases, roadmap plans, technical product demonstrations, integrations with the NoSQL ecosystem, training sessions, the latest Scylla news, and much more.

Everyone has a story about their journey to find the best database solution for their specific use case. What’s yours? Are you a developer, engineer, designer, or product manager leveraging the close-to-the-hardware design of Scylla or Seastar? Are you an executive, entrepreneur, or innovator faced with navigating the impact of NoSQL databases on your organization? If so, come and share your experiences to help build community and your own profile within it.

We’re looking for compelling case studies, technical sessions, technical and organizational best practices, and more. See below for a list of suggested topics, and feel free to recommend others. The deadline for submissions is Friday, August 17th, 2018.

Speakers receive complimentary admission to the 2-day summit and pre-summit training.

Topics

Scylla Use Cases

• Innovative Scylla use cases are fascinating to our audience, especially if you can speak to the value your company gained from your Scylla deployment, latency improvements, scale, reduced costs and such.
• Vertical, horizontal, and diagonal use case showcases: FinTech, AdTech, IoT, Cyber, Recommendation Engines, etc
• Database migration—share with our community how you migrated from another database solution (Cassandra, Mongo, MySQL, etc.) to Scylla—whether it’s a repeatable process or an edge case, these presentations are always popular with our audience.

Big Data Ecosystem and Scylla/Cassandra

• Share your stack, code, and how you’ve made a difference with Spark, Graph, Search, and Kafka. Share fantastic challenges and how you solved them.
  Running a time-series database on top of Scylla/Cassandra? Hack GoCQL or even develop the new Seastar based C++ driver? Akka fan? Share your story and contributions!
• Containers, Kubernetes and Helm Charts, Prometheus monitoring and additional production tooling

Seastar and Scylla Development

• Have you developed an interesting database enhancement? New gossip suggestions? Another compaction strategy?
• Scylla on new, emerging hardware platforms
• Seastar-based developments—DPDK and TCP/IP stacks, Java/legacy rewrites, etc.

Evals and Tests

• Running interesting QA tests? Have a chaos monkey? If so, join our zoo!
• Performance tests are always interesting to our technical audience.

Required Information

You’ll be asked to include the following information for your proposal:

• Proposed title
• Description
• Suggested main topic
• Audience information
• Who is this presentation for?
• What will the audience take away?
• What prerequisite knowledge would they need?
• Videos or live demos included in presentation?
• Length of the presentation (30-45 minutes recommended)

8 Tips for Submitting a Scylla Summit Proposal

Help us understand why your presentation is right for Scylla Summit 2018. Please keep in mind that this event is made by and for technical professionals. All presentations and supporting materials must be respectful and inclusive.

1. Be authentic. Your peers need original ideas with real-world scenarios, relevant examples, and knowledge transfer
2. Give your proposal a simple and straightforward title
3. Include as much detail about the presentation as possible
4. Keep proposals free of marketing, sales and vendor pitches
5. We tend to ignore proposals submitted by PR agencies and require that we can reach the suggested speaker directly. Improve the proposal’s chances of being accepted by working closely with the presenter(s) to write a jargon-free proposal that will be interesting and provide value for attendees
6. The best sessions are concise and focused—overviews aren’t great in this format, the narrower your topic is, the more you give your audience to take home
7. Explain why people will want to attend and what they’ll take away from it
8. Live demos sometimes don’t go as planned, so we don’t recommend them

SUBMIT A SPEAKER PROPOSAL

Super Early Bird Registration for Scylla Summit 2018 opens soon – follow us on Twitter or subscribe to our blog to stay up to date!

The post Scylla Summit 2018 – Be a Speaker appeared first on ScyllaDB.

This is part 14 of a series of blog posts that provides a story arc for Scylla Training.

In the previous post, we went over Materialized Views and showed a few examples of how to create views from the Mutant Monitoring System (MMS) data. Times are getting tough now. Attacks from the mutants are on the rise all over the country and Division 3 has not handled this problem effectively with the monitoring applications. Since the applications were not properly preventing or stopping attacks in a timely manner, Division 3 now wants to dive back into data analytics to learn how to prevent the attacks. In this post, we will go over how to use Apache Spark, Hive, and Superset to analyze and visualize the data from the Mutant Monitoring system.

As our mutant data increases in size over time, we need to be able to quickly analyze it using a cluster of machines and that is where Apache Spark comes in. Apache Spark is a distributed system and unified analytics engine for large-scale data processing. Using Hive, we can query data from a Scylla cluster using SQL syntax. Superset provides a graphical web interface that is used to visualize data from Hive.

To get started with using Spark, Hive, and Superset, we will need to bring up the containers.

Running the Scylla, Spark, and Superset Containers

The first step is to clone the scylla-code-samples repository and change to the Spark directory from the terminal.

git clone https://github.com/scylladb/scylla-code-samples.git
cd scylla-code-samples/mms/spark

Now we can build and run the containers:

docker-compose build
docker-compose up -d

After roughly 60 seconds, the existing MMS data will be automatically imported into Scylla. To make the data and graphs more interesting, we also included our load generator tool which is a simple Node.js application that populates the tracking keyspace with data. By this time, the Spark and Superset containers are running as well. We now need to learn how to set up Spark and Hive so they can communicate with Scylla.

Running Spark and Hive

After Spark is running, we will run the Thrift server inside the container. Thrift is an RPC framework that allows applications to interact with Scylla. In the command below, we tell Thrift to connect to Hive inside the container and use scylla-node1 as the Cassandra connection host:

With the Thrift server started, we will now need to create the Hive table with the following commands:

The output of the command should be:

After this command is run, Hive will know that we are using the tracking keyspace and tracking_data table from the Scylla cluster. With Spark and Hive set up, we can now move on to Superset.

When Superset is run for the first time, you will not be able to log into the web interface because an admin password is not configured. The command below sets the password for the Superset web interface.

Now we can begin to import the Mutant Monitoring data set for Superset to use. Navigate to the scylla-code-samples/mms/spark directory in your terminal and run the following commands:

Using Superset

The next step is to log into the Superset web interface located at http://127.0.0.1:8088/ with the username and password of “admin” and import the dashboard.

Import the dashboard.json located in scylla-code-samples/mms/spark:

Click Upload.

Click on Dashboards on the top of the page followed by clicking on the “Main” dashboard.

In this dashboard, you will be able to see a heat pie chart that displays the heat signature of each mutant, a table view of the tracking_data keyspace, and a line chart that shows heat, telepathy, and speed over time.

To modify or create your own charts, click on Charts on the top of the screen. From here you can click on the green plus icon to create a new chart or modify an existing one by clicking on the chart. Click on the Graph chart and let’s examine the contents.

In the image below, you can see how the chart was made and we make changes as needed. In the query box, we are taking data from the tracking keyspace and running queries based on speed, heat, and telepathy powers and grouping the results by the mutants first name. After a change is made, you can click on “Run Query” to see the results and then click save to make the changes permanent.

Conclusion

Division 3 was not taking the proper actions to analyze the Mutant Monitoring data effectively and took action to do better in the future. In this post, we explained how to do data analytics on a Scylla cluster using Apache Spark, Hive, and Superset. Apache Spark is a powerful distributed data analytics system and Hive provides an SQL interface for interacting with data from Scylla. Combined with Superset, Division 3 has a easy and powerful solution to visualize the data from the Mutant Monitoring System. Please be safe out there and continue to monitor the mutants so we can protect as many lives as possible.

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Mutant Monitoring System (MMS) Day 14 – Analyzing Data with Spark, Hive, and Superset appeared first on ScyllaDB.

Come Talk Real-Time Big Data With Us…

We’ve got a busy schedule in the weeks ahead. Come join us at one of these exciting events.

Spark+AI Summit – San Francisco, CA – June 5-6
First up, we are sponsors at Spark+AI Summit in San Francisco June 5-6. Visit us at booth #318 to see a live demo of Scylla serving 1/2 million write requests per second with consistent millisecond latency. Plus play our fun iPad game to get a Scylla t-shirt and a chance to win more cool swag.

Montréal Big Data Meetup – Montréal, QC – June 5
Next stop is Montreal, where we will sponsor the Montréal Big Data Meetup. Come hear the session, ‘Real-time big data at scale: 1 million queries per second with single-digit millisecond latencies’, by Mina Naguib from AdGear and Glauber Costa from ScyllaDB. The talk will present the architecture and design decisions which allow Scylla users to achieve high throughput and predictably low latencies while loweing the overall TCO of the system. You will learn about a real example of how that played out in practice by AdGear. Join us June 5th from 6:15-8:15. Food, drinks, and swag provided!

BDM58 (NEW LOCATION!): 1M queries/s with single-digit millisecond latency

Tuesday, Jun 5, 2018, 6:15 PM

Maison Notman
51 Sherbrooke Ouest & St-Laurent (Clark) Montréal, QC

113 Big Data Montrealers Attending

Big Data Montreal would like to invite you to its 58th meeting! ================================================= NEW LOCATION: Notman House – see below ================================================= Join us on Tuesday June 5th 2018 at 6:00PM to attend a conference, as well as to network with other Big Data enthusiasts from Montreal! All are wel…

Check out this Meetup →

Fullstack LX – Lisbon, Portugal – June 5
June 5th is a busy night! ScyllaDB engineer Duarte Nunes is presenting ‘NoSQL at Ludicrous Speed’ at this month’s Fullstack LX in Lisbon. Attend the meetup to get the low-level details of how Scylla employs a totally asynchronous, shared nothing programming model, relies on its own memory allocators, and meticulously schedules all its IO requests, plus learn how Scylla fully utilizes the underlying hardware resources.

Fullstack LX June edition

Tuesday, Jun 5, 2018, 6:30 PM

Talkdesk
Rua Tierno Galvan, Torre 3 das Amoreiras, 15º Piso 1070-274 Lisboa, PT

39 Membros Attending

DETAILS ScyllaDB: NoSQL at Ludicrous Speed ScyllaDB is a NoSQL database compatible with Apache Cassandra, distinguishing itself by supporting millions of operations per second, per node, with predictably low latency, on similar hardware. Achieving such speed requires a great deal of mechanical sympathy: ScyllaDB employs a totally asynchronous, shar…

Check out this Meetup →

SQL NYC, The NoSQL & NewSQL Database Meetup – New York, NY – June 19th
Later this month we’ll be in New York for the SQL NYC NoSQL Database Meetup. Join this informative session to hear Oracle Data Cloud’s Principal Software Engineer discuss his team’s use cases for a redesigned streaming system. He will cover the challenges they encountered while prototyping various data stores to meet their needs and the client-side changes that ultimately made the project a success. Plus snacks and swag!

Choosing the Right NoSQL Datastore: Journeying From Apache Cassandra to Scylla

Tuesday, Jun 19, 2018, 6:30 PM

Galvanize, 2nd Floor
303 Spring St. New York, NY

87 Data Enthusiasts Attending

• Free fun-food, beer, swag and more! • Aaron Stockton, Principal Software Engineer, Oracle Data Cloud • A Database Month event http://www.DBMonth.com/database/oracle-data-cloud Join this informative session to hear from Oracle Data Cloud’s Principal Software Engineer discuss his team’s use cases for a redesigned streaming system, along with the ch…

Check out this Meetup →

The post ScyllaDB Coming to a City Near You! appeared first on ScyllaDB.

From a bird’s eye view, the task of a database is simple: The user inserts some data and later fetches it. But as we look closer things get much more complex. For example, the data needs to go to a commit log for durability, it needs to get indexed, and it’s rewritten many times so it can easily be fetched.

All of those tasks are internal processes of the database that will compete for finite resources like CPU, disk, and network bandwidth. However, the payoffs for privileging one or the other aren’t always clear. One example of such internal processes are compactions, which are a fact of life in any database, like Scylla, with a storage layer based on Log Structured Merge (LSM) trees.

LSM trees consist of append-only immutable files originating from the database writes. As writes keep happening, the system is potentially left with data for a key present in many different files, which makes reads very expensive. Those files are then compacted in the background by the compaction process according to a user-selected compaction strategy. If we spend fewer resources compacting existing files, we will likely be able to achieve faster write rates. However, reads are going to suffer as they will now need to touch more files.

What’s the best way to set the amount of resources to use for compactions? A less-than-ideal option is to push the decision to the user in the form of tunables. The user can then select in the configuration file the bandwidth that will be dedicated to compactions. The user is then responsible for a trial-and-error tuning cycle to try to find the right number for the matching workload.

At ScyllaDB, we believe that this approach is fragile. Manual tuning is not resilient to changes in the workload, many of which are unforeseen. The best rate for your peak load when resources are scarce may not be the best rate when the cluster is out of business hours–when resources are plentiful. But even if the tuning cycle can somehow indeed find a good rate, that process significantly increases the cost of operating the database.

In this article, we’ll discuss the approach ScyllaDB prescribes for solving this problem. We borrow from the mathematical framework of industrial controllers to make sure that compaction bandwidth is automatically set to a fair value while maintaining a predictable system response.

A Control Systems Primer

While we can’t magically determine the optimal compaction bandwidth by looking at the system, we can set user-visible behaviors that we would like the database to abide by. Once we do that we can then employ Control Theory to make sure that all pieces work in tandem at specified rates so that the desired behavior is achieved. One example of such a system is your car’s cruise control. While it is impossible to guess the individual settings for each part of the that would combine to cause the car to move at the desired speed, we can instead simply set the car’s cruising speed then expect the individual parts to adjust to make that happen.

In particular, we will focus in this article on closed-loop control systems—although we use open-loop control systems in Scylla as well. For closed-loop control systems we have a process being controlled, and an actuator, which is responsible for moving the output to a specific state. The difference between the desired state and the current state is called the error, which is fed back to the inputs. For this reason, closed-loop control systems are also called feedback control systems.

Let’s look at another example of a real-world closed-loop control system: we want the water in a water tank to be at or around a certain level, we will have a valve working as an actuator, and the difference between the current level and the desired level is the error. The controller will open or close the valve so that more or less water flows from the tank. How much the valve should open is determined by the transfer function of the controller. Figure 1 shows a simple diagram of this general idea.

Figure 1: An industrial controller, controlling the water level in a tank. The current level is measured and fed to a feedback-loop controller. Based on that an actuator will adjust the rate of water flow in the tank. Systems like this are commonly used in industrial plants. We can leverage that knowledge to control database processes as well.

The Pieces of the Puzzle: Schedulers and the Backlog Controller

Inside Scylla, the foundation of our control processes are the Schedulers embedded in the database. We discussed the earliest of them, the I/O Scheduler, extensively in a three-part article (part 1, part 2, and part 3). The Schedulers work as the actuators in a control system. By increasing the shares of a certain component we increase the rate at which that process is executed—akin to the valve in Figure 1 allowing more (or less) fluid to pass through. Scylla also embeds a CPU Scheduler, which plays a similar role for the amount of CPU used by each of the database’s internal processes.

To design our controller it’s important to start by reminding ourselves of the goal of compactions. Having a large amount of uncompacted data will lead to a read to both read amplification and space amplification. We have read amplification because each read operation will have to read from many SSTables, and space amplification because overlapping data will be duplicated many times. Our goal is to get rid of this amplification.

We can then define a measurement of how much work remains to be done to bring the system to a state of zero amplification. We call that the backlog. Whenever new bytes are written into the system, we generate bytes of backlog into the future. Note that there isn’t a one-to-one relationship between and , since we may have to rewrite the data many times to achieve a state of zero amplification. When the backlog is zero, everything is fully compacted and there is no read or write amplification.

We are still within our goal if we allow the system to accumulate backlog, as long as the backlog doesn’t grow out of bounds. After all, there isn’t a need to compact all work generated by incoming writes now. The amount of bandwidth used by all components will always add up to a fixed quantity—since the bandwidth of CPU and disk are fixed and constant for a particular system. If the compaction backlog accumulates, the controller will increase the bandwidth allocated to compactions, which will cause the bandwidth allocated to foreground writes to be reduced. At some point, a system in a steady state should reach equilibrium.

Different workloads have a different steady-state bandwidth. Our control law will settle on different backlog measures for them; a high-bandwidth workload will have a higher backlog than a low-bandwidth workload. This is a desirable property: higher backlog leaves more opportunity for overwrites (which reduce the overall amount of write amplification), while a low-bandwidth write workload will have a lower backlog, and therefore fewer SSTables and smaller read amplification.

With this in mind, we can write a transfer function that is proportional to the backlog.

Determining Compaction Backlog

Compaction of existing SSTables occurs in accordance with a specific compaction strategy, which selects which SSTables have to be compacted together, and which and how many should be produced. Each strategy will do different amounts of work for the same data. That means there isn’t a single backlog controller—each compaction strategy has to define its own. Scylla supports the Size Tiered Compaction Strategy (STCS), Leveled Compaction Strategy (LCS), Time Window Compaction Strategy (TWCS) and Date Tiered Compaction Strategy (DTCS).

In this article, we will examine the backlog controller used to control the default compaction strategy, the Size Tiered Compaction Strategy. STCS is described in detail in a previous post. As a quick recap, STCS will try to compact together SSTables of similar size. If at all possible, we try to wait until 4 SSTables of similar sizes are created and compact that. As we compact SSTables of similar sizes, we may create much larger SSTables that will belong to the next tier.

In order to design the backlog controller for STCS, we start with a few observations. The first is that when all SSTables are compacted into a single SSTable, the backlog is zero as there is no more work to do. As the backlog is a measure of work still to be done for this process, this follows from the definition of the backlog.

The second observation is that the larger the SSTables, the larger the backlog. This is also easy to see intuitively. Compacting 2 SSTables of 1MB each should be much easier on the system than compacting 2 SSTables of 1TB each.

A third and more interesting observation is that the backlog has to be proportional to the number of tiers that a particular SSTable still have to move through until it is fully compacted. If there is a single tier, we’ll have to compact the incoming data being currently written with all the SSTables in that tier. If there are two tiers, we’ll compact the first tier but then will have to compact every byte present in that tier – even the ones that were already sealed in former SSTables into another tier.

Figure 2 shows that in practice. As we write a new SSTable, we are creating a future backlog as that new SSTable will have to be compacted with the ones in its tier. But if there is a second tier, then there will be a second compaction eventually, and the backlog has to take that into account.

Figure 2: SSTables shown in blue are present in the system already when a new SSTable is being written. Because those existing SSTables live in two different Size Tiers, the new SSTable creates a future backlog roughly twice as big as if we had a single tier—since we will have to do two compactions instead of one.

Note that this is valid not only for the SSTable that is being written at the moment as a result of data being flushed from memory—it is also valid for SSTables being written as a result of other compactions moving data from earlier tiers. The backlog for the table resulting from a previous compaction will be proportional to the number of levels it still has to climb to the final tier.

It’s hard to know how many tiers we have. That depends on a lot of factors including the shape of the data. But because the tiers in STCS are determined by the SSTable sizes, we can estimate an upper bound for the number of tiers ahead of a particular SSTable based on a logarithmic relationship between the total size in the table for which the backlog is calculated and the size of a particular SSTable— since the tiers are of exponentially larger sizes.

Consider, for example, a constant insert workload with no updates that continually generates SSTables sized 1GB each. They get compacted into SSTables of size 4GB, which will then be compacted into SSTables of size 16GB, etc.

When the first two tiers are full, we will have four SSTables of size 1GB and four more SSTables of size 4GB. The total table size is 4 * 1 + 4 * 4 = 20GB and the table-to-SSTable ratios are 20/1 and 20/4 respectively. We will use the base 4 logarithm, since there are 4 SSTables being compacted together, to yield for the 4 small SSTables:

and for the large ones

So we know that the first SSTable, sized 1GB, belongs to the first tier while the SSTable sized 4GB belongs in the second.

Once that is understood, we can write the backlog of any existing SSTable that belongs to a particular table as:

where  is the backlog for SSTable , is the size of SSTable . The total backlog for a table is then

To derive the formula above, we used SSTables that are already present in the system. But it is easy to see that it is valid for SSTables that are undergoing writes as well. All we need to do is to notice that is, in fact, the partial size of the SSTable—the number of bytes written so far.

The backlog increases as new data is written. But how does it decrease? It decreases when bytes are read from existing SSTables by the compaction process. We will then adjust the formula above to read:

where is the number of bytes already read by compaction from sstable . It will be 0 in SSTables that are not undergoing compactions.

Note that when there is a single SSTable, , and since , there is no backlog left—which agrees with our initial observations.

The Compaction Backlog Controller in Practice

To see this in practice, let’s take a look at how the system responds to an ingestion-only workload, where we are writing 1kB values into a fixed number of random keys so that the system eventually reaches steady state.

We will ingest data at maximum throughput, making sure that even before any compaction starts, the system is already using 100% of its resources (in this case, it is bottlenecked by the CPU), as shown in Figure 3. As compactions start, the internal compaction process uses very little CPU time in proportion to its shares. Over time, the CPU time used by compactions increases until it reaches steady state at around 15%. The proportion of time spent compacting is fixed, and the system doesn’t experience fluctuations.

Figure 4 shows the progression of shares over time for the same period. Shares are proportional to the backlog. As new data is flushed and compacted, the total disk space fluctuates around a specific point. At steady state, the backlog sits at a constant place where we are compacting data at the same pace as new work is generated by the incoming writes.

A very nice side effect of this is shown in Figure 5. Scylla CPU and I/O Schedulers enforce the amount of shares assigned to its internal processes, making sure that each internal process consumes resources in the exact proportion of their shares. Since the shares are constant in steady state, latencies, as seen by the server, are predictable and stable in every percentile.

Figure 3: Throughput of a CPU in the system (green), versus percentage of CPU time used by compactions (yellow). In the beginning, there are no compactions. As time progresses the system reaches steady state as the throughput steadily drops.

Figure 4: Disk space assigned to a particular CPU in the system (yellow) versus shares assigned to compaction (green). Shares are proportional to the backlog, which at some point will reach steady state

Figure 5: 95th, 99th, and 99.9th percentile latencies. Even under 100% resource utilization, latencies are still low and bounded.

Once the system is in steady state for some time, we will suddenly increase the payload of each request, leading the system to ingest data faster. As the rate of data ingestion increases, the backlog should also increase. Compactions will now have to move more data around.

We can see the effects of that in Figure 6. With the new ingestion rate, the system is disturbed as the backlog grows faster than before. However, the compaction controller will automatically increase shares of the internal compaction process and the system will reach a new equilibrium.

In Figure 7 we revisit what happens to the percentage of CPU time assigned to compactions now that the workload changes. As requests get more expensive the throughput in request/second naturally drops. But, aside from that, the larger payloads will lead to compaction backlog accumulating quicker. The percentage of CPU used by compactions increases until a new equilibrium is reached. The total drop in throughput is a combination of both of those effects.

Having more shares, compactions are now using more of the system resources, disk bandwidth, and CPU. But the amount of shares is stable and doesn’t have wide fluctuations leading to a predictable outcome. This can be observed through the behavior of the latencies in Figure 8. The workload is still CPU-bound. There is now less CPU available for request processing, as more CPU is devoted to compactions. But since the change in shares is smooth, so is the change in latencies.

Figure 6: The ingestion rate (yellow line) suddenly increases from 55MB/s to 110MB/s, as the payload of each request increases in size. The system is disturbed from its steady state position but will find a new equilibrium for the backlog (green line).

Figure 7: Throughput of a CPU in the system (green), versus percentage of CPU time used by compactions (yellow) as the workload changes. As requests get more expensive the throughput in request/second naturally drops. Aside from that, the larger payloads will lead to compaction backlog accumulating quicker. The percentage of CPU used by compactions increases.

Figure 8: 95th, 99th and 99.9th percentile latencies after the payload is increased. Latencies are still bounded and move in a predictable way. This is a nice side effect of all internal processes in the system operating at steady rates.

Conclusions

At any given moment, a database like ScyllaDB has to juggle the admission of foreground requests with background processes like compactions, making sure that the incoming workload is not severely disrupted by compactions, nor that the compaction backlog is so big that reads are later penalized.

In this article, we showed that isolation among incoming writes and compactions can be achieved by the Schedulers, yet the database is still left with the task of determining the amount of shares of the resources incoming writes and compactions will use.

Scylla steers away from user-defined tunables in this task, as they shift the burden of operation to the user, complicating operations and being fragile against changing workloads. By borrowing from the strong theoretical background of industrial controllers, we can provide an Autonomous Database that adapts to changing workloads without operator intervention.

As a reminder, Scylla 2.2 is right around the corner and will ship with the Memtable Flush controller and the Compaction Controller for the Size Tiered Compaction Strategy. Controllers for all compaction strategies will soon follow.

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Taming the Beast: How Scylla Leverages Control Theory to Keep Compactions Under Control appeared first on ScyllaDB.

We recently hosted the webinar, Steering the Sea Monster: Integrating Scylla with Kubernetes. We received a lot of great questions during the live webinar, so here they are along with our answers. Miss the live webinar? You can access the on-demand version here.

What’s the difference between Helm and the YAML manifest files?

Kubernetes manifest files (written in YAML) describe the specification of Kubernetes API objects like StatefulSets and Services. Helm is a packing system that can combine and automate the deployment of many manifest files and help an administrator to manage them effectively. Helm calls groups of manifests (and their configuration) “Helm Charts”.

If I lose a Scylla pod of a StatefulSet, and rather than trying to bring that Node back, I create a new Node, does the StatefulSet replace the old Scylla pod from the old node by rescheduling it on a new node? Or does it generate a new pod replica in the sequence?

The new Scylla pod will replace the downed pod. For example, let’s image that we initially had scylla-0, scylla-1, and scylla-2 pods. If pod scylla-1 goes down, the StatefulSet will create an additional pod and it will still be called scylla-1.

Are multi-datacenters supported when Scylla is deployed with Kubernetes?

Yes, multi-datacenters are supported with Scylla on Kubernetes. To deploy a Scylla cluster in each region, you are required to create a StatefulSet per region and each region can be scaled separately. The StatefulSet will include the definition of the end-point-snitch, seed nodes information, and prefix information per datacenter.

How does Kubernetes know that a pod is ready?

A pod is ready when Kubernetes has completed a status probe, in our case nodetool status. Once nodetool reports that the pod is up and running, Kubernetes will mark it as ready.

The StatefulSet is responsible for creating DNS records for nodes?

Yes, the StatefulSet is responsible for setting an internal DNS server to register the different instances that the StatefulSet consists of. For example, if the StatefulSet includes four instances and the pod moves to a different host, the DNS information will be updated accordingly.

Is there a performance penalty when running Scylla on Kubernetes + containers?

At the time of writing this blog post, there is a performance degradation using Scylla in Docker containers. For Kubernetes, we are using the Scylla default Docker container as the base image. From recent testing conducted, we have seen 25% to 40% degradation of performance. Scylla is actively working to improve its performance in Docker containers.

Why use Docker for Scylla if performance is not as good as using the AMI?

Some users prefer to have a single deployment platform that can be used across multiple infrastructure systems, e.g., cloud, on-prem. Kubernetes provides the users with a single view of the platform. AMIs are specific for AWS deployments. While It is possible to create preconfigured images for many on-prem and cloud solutions, Kubernetes and Docker offer a single image to be used over any infrastructure.

How would you upgrade Scylla in this deployment?

Users are required to take a snapshot of the current data and configuration of the image. Once the persistent storage directory is backed up, we can remove the current container and attach the upgraded version to the pre-existing persistent storage used in the previous container.

Will the Helm Chart be available?

Yes, the Helm Charts are currently available from Scylla code examples, and we are working to submit the charts into the main Helm project repositories.

How is the deployment/decommission affected by the amount of data?

In the case of decommissioning or adding pods, Scylla will redistribute the data. Streaming of data between nodes implies less CPU, I/O , and network bandwidth to the normal operations. The more data, the more throughput is required from the I/O and network system to maintain SLA.

Our next webinar, ‘Analytics Showtime: Powering Spark with Scylla’ is on June 27th where we will cover best practices, use cases, and performance tuning. We hope you can join us! REGISTER NOW

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Kubernetes and Scylla: 10 Questions and Answers appeared first on ScyllaDB.

The Scylla team is pleased to announce the release of Scylla Enterprise 2018.1.3, a production-ready Scylla Enterprise minor release. Scylla Enterprise 2018.1.3 is a bug fix release for the 2018.1 branch, the latest stable branch of Scylla Enterprise.

More about Scylla Enterprise here.

Critical Patch

Scylla Enterprise 2018.1.3 fixes possible data loss when using Leveled Compaction Strategy.  The issue causes Scylla to miss a small fraction of data in a full table scan. This was originally observed in decommission (which performs a full table scan internally), where some data (<1% in a test) was not streamed.

In addition to full scan query, scans are used internally as part of compaction and streaming, including decommissioning, adding a node, and repairs. Our investigation into the matter concluded that Scylla can cause data loss while running any of these actions. The issue is limited to tables using LCS and does not affect tables using other compaction strategies.

If you are using LCS, you should upgrade to Scylla Enterprise 2018.1.3 ASAP.

We will share more details on the issue soon.

Related Links

• Get Scylla 2018.1.3 (customers only, or 30-day evaluation)
• Upgrade from 2018.1.0 to 2018.1.2
• Upgrade from 2017.1.x to 2018.1
• Upgrade from Scylla Open Source 2.1 to Scylla 2018.1
• Upgrade from Scylla 1.6 to Scylla 2018.1
• Submit a ticket

Scylla Enterprise customers are encouraged to upgrade to Scylla Enterprise 2018.1.3 in coordination with the Scylla support team.

Additional Issues Solved in This Release (with Open Source Issue Reference When Applicable)

• Additional issue solved in this release (with open source issue reference when one exist)
• Ec2MultiRegionSnitch does not (always) honor prefer the local DC, which result with redundant requests to remote DC #3454
• When using TLS for interconnect connections, shutting down a node generates errors on system_error (error system:32, Broken pipe) other nodes #3461

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Scylla Enterprise Release 2018.1.3 appeared first on ScyllaDB.

Today we released Scylla Open Source 2.1.4, a bugfix release of the Scylla 2.1 stable branch. Release 2.1.4, like all past and future 2.x.y releases, is backward compatible and supports rolling upgrades.

Critical Patch

Scylla 2.1.4 fixes a possible data loss when using Leveled Compaction Strategy #3513. The issue causes Scylla to miss a small fraction of data in a full table scan. This was originally observed in decommission (which performs a full table scan internally), where some data (<1% in a test) was not streamed.

In addition to a full scan query, scans are used internally as part of compaction and streaming, including decommission, adding a node, and repairs. Our investigation into the matter concluded that Scylla can cause data loss while running any of these actions.

The issue is limited to tables using LCS and does not affect tables using other compaction strategies. If you are using LCS, you should upgrade to Scylla 2.1.4 ASAP. 

Action to Take

The problem may be mitigated by restoring backups of the relevant table. If you are using LCS and have relevant backups, please contact our support team for additional information on how to run the restore procedure.

How This Happened

We take data integrity very seriously and are investigating why this issue was not identified earlier. Our initial findings are that a low-level optimization around disjoint SSTable merging introduced the bug in the 2.1 release. It surfaced only in our 2.2 testing since it happened very rarely with 2.1 based code. The Scylla cluster test suite did detect the issue, however, meeting quorum persistence papered over it together with the test suite itself – one of the roles of this suite is to run disruptors (corruption emulation, node and data center failures) against the cluster and to trigger corruptions and repairs. The bug was not identified since the test suite incorrectly concluded that it is part of the disruptor activity of the suite. We are now working to improve the cluster test suite’s ability to detect errors.

Please contact us with any questions or concerns. We will publish a full root cause analysis report as soon as possible and disclose enhancements to prevent such a case in the future.

Related Links

Get Scylla 2.1.4 – Docker, binary packages. AMI will be published soon.
Get started with Scylla 2.1
Upgrade from 2.1.x to 2.1.y
Please let us know if you encounter any problems.

Additional bugs fixed in this release

• Scylla AMI error: “systemd: Unknown lvalue ‘Ambient / Unknown lvalue ‘AmbientCapabilities’ “ Issue is solved by moving to a new CentOS 7.4.1708 base image #3184
• Upgrading to latest version of RHEL kernel causes Scylla to lose access to the RAID 0 data directory #3437 (detailed notice has been sent to all relevant customers)
• Wrong Commit log error handling may cause a core dump #3440
  Closing a secure connection (TLS) may cause a core dump #3459
• When using TLS for interconnect connections, shutting down a node generates errors:”on system_error (error system:32, Broken pipe) other nodes” #3461
• Ec2MultiRegionSnitch does not (always) honor or prefer the local DC, which results with redundant requests to remote DC #3454

Next Steps

• Learn more about Scylla from our product page.
• See what our users are saying about Scylla.
• Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
• Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Scylla Open Source Release 2.1.4 appeared first on ScyllaDB.