<\/a><\/noindex><\/p>\nSo, here are the insights we've gained from experience.<\/h2>\n <\/p>\n
Yes, a downed node is not a tragedy. That\u2019s the essence of Cassandra's fault tolerance. But a node can be alive and still start to degrade in performance.<\/b>As it turned out, this immediately affects the performance of the entire cluster.<\/p>\n
<\/p>\n
Cassandra won\u2019t safeguard where Oracle provided safety with its constraints.<\/b>And if the application author didn\u2019t realize this in advance, a duplicate arriving for Cassandra is no worse than the original. If it's there, it will be inserted.<\/p>\n
<\/p>\n
Out-of-the-box Cassandra was sharply disapproved of by security: there's no user action logging, nor permission segregation.<\/b>Call information relates to personal data, which means that all attempts to request or modify it in any way must be logged with the possibility of subsequent auditing. It is also necessary to recognize the need to separate rights at different levels for different users. A simple operations engineer and a super admin who can freely delete an entire keyspace are different roles, with different responsibilities and competencies. Without such a delineation of access rights, the value and integrity of the data will quickly come into question, faster than with a consistency level of ANY. <\/p>\n
<\/p>\n
We did not consider that call data requires both serious analytics and periodic sampling under various conditions. Since the selected records are expected to be deleted and rewritten (we need to maintain data actualization processes even when initial data inputs are incorrect), Cassandra is not suitable for this task. Cassandra is like a piggy bank \u2014 it is convenient for storage, but it doesn't allow for calculations.<\/b><\/p>\n
<\/p>\n
We faced a problem with transferring data to test zones.<\/b> (5 nodes in testing versus 20 in production). In this case, using a dump will not be possible.<\/p>\n
<\/p>\n
The problem of updating the application data schema that writes to Cassandra. Rollback will create a multitude of tombstones, which can unpredictably degrade performance.<\/b>Cassandra is optimized for writing, and before writing, it does not think much. Any operation with existing data in it is also a write operation. That is, by deleting unnecessary entries, we simply create even more records, and only some of them will be marked as tombstones.<\/p>\n
<\/p>\n
Timeouts during insertion. Cassandra is excellent for writing, but sometimes the incoming stream can be significantly perplexing for it.<\/b>This happens when the application starts cycling through several records that cannot be inserted for some reason. We will need a real DBA to monitor gc.log, system and debug logs for slow queries and metrics on compaction pending.\n<\/p>\n
<\/p>\n
Several data centers in the cluster. Where to read from and where to write to?<\/b>
<\/p>\n
How we approached this<\/h2>\n <\/p>\n
To prevent node performance issues, we disabled SWAP<\/b>. And now, when memory is insufficient, the node should go down instead of generating long GC pauses.<\/p>\n
<\/p>\n
So, we no longer rely on the database logic. Application developers are retraining and starting to actively build safety nets in their own code.<\/b> An ideal clear separation of data storage and processing.<\/p>\n
<\/p>\n
We purchased support from DataStax.<\/b> The boxed version of Cassandra is no longer being developed (the last commit was in February 2018). At the same time, Datastax offers excellent service and a wide range of customized and adapted solutions for existing information systems.<\/p>\n
<\/p>\n
I also want to note that Cassandra is not very convenient for query selections. Of course, CQL is a significant step toward user-friendliness (compared to Thrift). But if you have entire departments accustomed to such convenient joins, free filtering by any field, and query optimization capabilities, and these departments are focused on addressing issues and crises, then a solution based on Cassandra appears hostile and foolish to them. We began to tackle how to help our colleagues make selections. <\/p>\n
<\/p>\n
We considered two options. In the first option, we write calls not only to C*, but also to the archival Oracle database. However, unlike C*, this database only stores calls for the current month (which is sufficient for the re-tarification cases). Here, I immediately saw the following problem: if we write synchronously, we lose all the advantages of C*, related to fast insertion; if asynchronously, there is no guarantee that all necessary calls actually made it to Oracle. There was one advantage, but a significant one: for operation, we still have the familiar PL\/SQL Developer, which means we can practically implement the 'Facade' pattern. The alternative option is to implement a mechanism that extracts calls from C*, pulls some enrichment data from the corresponding tables in Oracle, joins the obtained datasets, and provides us with the result, which we can then somehow use (rollback, repeat, analyze, marvel). The downsides: the process becomes quite multi-step, and in addition, there is no interface for the operations staff.<\/p>\n
<\/p>\n
In the end, we settled on the second option. For selections from different databases, we used Apache Spark.<\/b> The essence of the mechanism came down to Java code that extracts data from C* based on specified keys (subscriber, call time \u2013 keys of the section), as well as the required enrichment data from any other database. It then joins them in memory and outputs the result to the result table. We drew a web interface over Spark, and it turned out to be quite suitable for operation.<\/p>\n
\n
In tackling the issue of updating the data, the production test again considered several solutions. Both the transfer via Sstloader and the option of splitting the cluster in the test zone into two parts, each of which alternately connects to the production cluster, powering itself from it were discussed. During the update, it was planned to swap them: the part that worked in the test would be cleared and introduced into production, while the other would start working with the data separately. However, upon further reflection, we evaluated more rationally which data should be transferred and understood that the calls themselves are an inconsistent entity for tests, generated quickly if needed, and that the production dataset has no value for transferring to the test. There are a few warehouse objects worth transferring, but this is literally a couple of tables, and they are not very heavy. Therefore, we sought the solution through Spark again, which allowed us to write and actively use a data transfer script between the production and test tables.<\/b><\/p>\n
<\/p>\n
Our current deployment policy allows us to operate without rollbacks.<\/b> Before production, there is a mandatory rollout to the test environment, where errors are not as costly. In case of failure, you can always drop the keyspace and roll out the entire schema from the beginning.<\/p>\n
<\/p>\n
To ensure high availability of Cassandra, a DBA and more than just that are needed. Everyone working with the application must understand where and how to check the current situation and how to diagnose problems promptly.<\/b> To this end, we actively use DataStax OpsCenter (Administration and monitoring of workloads), the system metrics of the Cassandra Driver (the number of write timeouts in C*, the number of read timeouts in C*, maximum latency, etc.), and monitor the operation of the application itself that interacts with Cassandra.\n<\/p>\n
<\/p>\n
When we reflected on the previous question, we realized where our main risk might lie. It concerns the data representation forms that pull information from several independent queries to the storage. As a result, we could end up with quite inconsistent information. However, this issue would also be relevant if we were working with just one data center. So, the most sensible approach here is, of course, to create a batch function for reading data in an external application, which will ensure data retrieval in a unified timeframe. Regarding the separation of reading and writing in terms of performance, we were halted by the risk that during some connectivity loss between data centers, we could end up with two completely inconsistent clusters.<\/p>\n
<\/p>\n
As a result, at this point we settled on a write consistency level of EACH_QUORUM and a read level of LOCAL_QUORUM.<\/b><\/p>\n
<\/p>\n
Brief impressions and conclusions<\/h2>\n <\/p>\n
To evaluate the solution in terms of operational support and future development prospects, we decided to think about where else such a development could be applied.<\/p>\n
<\/p>\n
If we consider it quickly, then data scoring for programs like 'Pay when convenient' (loading information into S* and calculations on Spark scripts), handling claims with aggregation by directions, storing roles, and computing based on the role matrix for user access rights. <\/p>\n
<\/p>\n
As we can see, the repertoire is broad and diverse. And if we were to choose sides among supporters\/opponents of NoSQL, we would align ourselves with the supporters, as we have gained our advantages, particularly where we expected them.<\/p>\n
<\/p>\n
Even the out-of-the-box Cassandra variant allows for horizontal scaling in real-time, effortlessly addressing the issue of data increase in the system. We managed to isolate a very high-load mechanism for calculating call aggregates into a separate contour, as well as separate the schema and application logic, eliminating the detrimental practice of writing custom jobs and objects directly in the database. We gained the ability to select and configure, for the sake of acceleration, which data centers we will perform calculations in, and which ones will handle data writing, thus securing ourselves against failures of individual nodes as well as the entire data center.<\/p>\n
<\/p>\n
When applying our architecture to new projects, especially with some prior experience, I would like to take into account the nuances described above right from the start, avoiding certain mistakes and smoothing out some rough edges that were initially unavoidable.<\/p>\n
<\/p>\n
For example, to track updates of Cassandra in a timely manner<\/b>, because many issues we encountered were already known and had been addressed.<\/p>\n
<\/p>\n
Do not place both the database and Spark on the same nodes<\/b> (or strictly separate based on allowable resource usage), as Spark can consume more memory than allowed, and we will quickly face issue number 1 from our list.<\/p>\n
<\/p>\n
Enhance monitoring and operational expertise even during the project testing phase. <\/b>Initially consider all potential consumers of our solution as much as possible<\/b>, because this will ultimately determine the structure of the database.<\/p>\n
<\/p>\n
Revisit the resulting schema several times for possible optimization. Identify which fields can be serialized. Determine what additional tables we need to create to accommodate the required information most accurately and optimally (for instance, recognizing that the same data can be stored in different tables, based on various criteria, can significantly save processing time during read queries).<\/p>\n
<\/p>\n
It\u2019s a good idea to immediately allow for the application of TTL and clean up outdated data.<\/b><\/p>\n
<\/p>\n
When exporting data from Cassandra the application logic should operate on the principle of FETCH, so that not all rows are loaded into memory at once, but are selected in batches.<\/b><\/p>\n
<\/p>\n
It is recommended to check the system's fault tolerance before transitioning the project to the described solution by conducting a series of crash tests<\/b>, such as simulating data loss in one data center, restoring corrupted data over a period, or experiencing network drops between data centers. Such tests will not only allow us to evaluate the pros and cons of the proposed architecture but will also provide valuable practice for the engineers conducting them, and the skills gained will be quite valuable in case system failures occur in production.<\/p>\n
<\/p>\n
When dealing with critical information (such as billing data and subscriber debt calculations), it is essential to pay attention to tools that can reduce risks arising from the nuances of the DBMS. For instance, using the nodesync utility (Datastax) and developing an optimal strategy for its use so that we do not create excessive load on Cassandra for the sake of consistency<\/b> and use it only for specific tables during certain periods.<\/p>\n
<\/p>\n
So, what about Cassandra after six months of use? Overall, there are no unresolved issues. We also avoided serious crashes and data loss. Yes, we had to consider compensating for some previously unforeseen problems, but ultimately it did not significantly tarnish our architectural decision. If you want to try something new and are not afraid of potential disappointments, be prepared to learn that nothing comes for free. You will need to delve into the documentation and gather your individual pains more than with the old legacy solution, and no theory will tell you in advance what specific challenges await you.<\/p>\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"\u0413\u043e\u0432\u043e\u0440\u044f\u0442, \u0432 \u0436\u0438\u0437\u043d\u0438 \u0432\u0441\u0435 \u0441\u0442\u043e\u0438\u0442 \u043f\u043e\u043f\u0440\u043e\u0431\u043e\u0432\u0430\u0442\u044c \u0445\u043e\u0442\u044f \u0431\u044b \u0440\u0430\u0437. \u0418 \u0435\u0441\u043b\u0438 \u0432\u044b \u043f\u0440\u0438\u0432\u044b\u043a\u043b\u0438 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0441 \u0440\u0435\u043b\u044f\u0446\u0438\u043e\u043d\u043d\u044b\u043c\u0438 \u0421\u0423\u0411\u0414, \u0442\u043e \u043f\u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u043d\u0430 \u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0435 \u0441 NoSQL \u0441\u0442\u043e\u0438\u0442 \u0432 \u043f\u0435\u0440\u0432\u0443\u044e \u043e\u0447\u0435\u0440\u0435\u0434\u044c \u0445\u043e\u0442\u044f \u0431\u044b \u0434\u043b\u044f \u043e\u0431\u0449\u0435\u0433\u043e \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u044f. \u0421\u0435\u0439\u0447\u0430\u0441 \u0432 \u0441\u0438\u043b\u0443 \u0431\u0443\u0440\u043d\u043e\u0433\u043e \u0440\u0430\u0437\u0432\u0438\u0442\u0438\u044f \u044d\u0442\u043e\u0439 \u0442\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0438 \u043e\u0447\u0435\u043d\u044c \u043c\u043d\u043e\u0433\u043e \u043f\u0440\u043e\u0442\u0438\u0432\u043e\u0440\u0435\u0447\u0438\u0432\u044b\u0445 \u043c\u043d\u0435\u043d\u0438\u0439 \u0438 \u0433\u043e\u0440\u044f\u0447\u0438\u0445 \u0441\u043f\u043e\u0440\u043e\u0432 \u043d\u0430 \u044d\u0442\u0443 \u0442\u0435\u043c\u0443, \u0447\u0442\u043e \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e \u043f\u043e\u0434\u043e\u0433\u0440\u0435\u0432\u0430\u0435\u0442 \u0438\u043d\u0442\u0435\u0440\u0435\u0441. \u0415\u0441\u043b\u0438 \u0432\u043d\u0438\u043a\u043d\u0443\u0442\u044c […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":28210,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-37585","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t
![\"How](\"\/wp-content\/uploads\/2019\/08\/4845d37a9928f888639c4ebeb7807787.jpg\")
![\"How](\"\/wp-content\/uploads\/2019\/08\/c7b095dc8879011adb751c96427af512.jpg\")
<\/p>\n