{"id":69568,"date":"2020-02-20T15:24:23","date_gmt":"2020-02-20T12:24:23","guid":{"rendered":"https:\/\/prohoster.info\/blog\/google-cloud-spanner-horoshij-plohoj-zloj"},"modified":"2020-03-03T16:14:50","modified_gmt":"2020-03-03T13:14:50","slug":"google-cloud-spanner-horoshij-plohoj-zloj","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/google-cloud-spanner-horoshij-plohoj-zloj","title":{"rendered":"Google Cloud Spanner: The Good, The Bad, and The Ugly","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><b><i>Hello, Hubbers. As always, we continue to share interesting material in anticipation of our new course launches. Today, especially for you, we have translated an article about Google Cloud Spanner, timed to coincide with the launch of the course. <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/eoP2\/\">AWS for Developers<\/a><\/noindex>.<\/i><\/b><\/p>\n<p><img decoding=\"async\" alt=\"Google Cloud Spanner: The Good, The Bad, and The Ugly\" src=\"\/wp-content\/uploads\/2020\/02\/b27b8a09543d10b078480c317553ca46.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<i>Originally published in <noindex><a rel=\"nofollow\" href=\"https:\/\/www.lightspeedhq.com\/blog\/\">the Lightspeed HQ blog<\/a><\/noindex>.<\/i><\/p>\n<p>As a company offering a wide range of cloud POS solutions for retailers, restaurateurs, and online sellers worldwide, Lightspeed employs several different types of database platforms for various transactional, analytical, and search use cases. Each of these database platforms has its own strengths and weaknesses. Therefore, when Google introduced Cloud Spanner to the market\u2014with promising features unseen in the world of relational databases, such as virtually unlimited horizontal scalability and a 99.999% service level agreement (SLA)\u2014we couldn't miss the chance to get our hands on it!<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nTo provide a comprehensive overview of our experience with Cloud Spanner as well as the evaluation criteria we used, we will cover the following topics:<\/p>\n<ol>\n<li>Our Evaluation Criteria<\/li>\n<li>Cloud Spanner in a Nutshell<\/li>\n<li>Our Assessment<\/li>\n<li>Our Conclusions<\/li>\n<\/ol>\n<p>\n<img decoding=\"async\" alt=\"Google Cloud Spanner: The Good, The Bad, and The Ugly\" src=\"\/wp-content\/uploads\/2020\/02\/29ed599362fb189283faa9215c18297a.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>1. Our Evaluation Criteria<\/h2>\n<p>\nBefore delving into the features of Cloud Spanner, its similarities and differences with other solutions on the market, let's first discuss the key use cases we had in mind when considering where to deploy Cloud Spanner in our infrastructure:<\/p>\n<ul>\n<li>As a replacement for (the predominant) traditional SQL database solutions<\/li>\n<li>As an OLTP solution with OLAP support<\/li>\n<\/ul>\n<p><\/p>\n<blockquote><p><i><b>Note:<\/b> For simplicity and ease of comparison, this article compares Cloud Spanner with MySQL variants from the GCP Cloud SQL and Amazon AWS RDS families.<\/i><\/p><\/blockquote>\n<p><\/p>\n<h4>Using Cloud Spanner as a replacement for traditional SQL database solutions<\/h4>\n<p>\nIn an environment of <i>traditional <\/i>databases, when the response time for a database query approaches or even exceeds predefined application thresholds (primarily due to an increasing number of users and\/or requests), there are several ways to reduce response times to acceptable levels. However, most of these solutions involve manual intervention.<\/p>\n<p>For example, the first step to take is to look at various database performance-related parameters and adjust them to best fit application usage patterns. If that proves insufficient, one can choose vertical or horizontal scaling of the database.<\/p>\n<p>Vertical scaling of an application involves upgrading the server instance, usually by adding more processors\/cores, more RAM, faster storage, etc. Adding more hardware resources leads to increased database performance, primarily measured in transactions per second and transaction latency for OLTP systems. Relational database systems (which use a multithreaded approach), such as MySQL, scale well vertically.<\/p>\n<p>This approach has several drawbacks, but the most obvious is the maximum server size available on the market. Once the largest server instance limit is reached, the only way left is horizontal scaling.<\/p>\n<p>Horizontal scaling is an approach where more servers are added to a cluster to ideally increase performance linearly with the addition of more servers. Most <i>traditional <\/i>database systems poorly scale horizontally or do not scale at all. For example, MySQL can horizontally scale for read operations by adding slave readers but cannot horizontally scale for write operations.<\/p>\n<p>On the other hand, due to its nature, Cloud Spanner can easily scale horizontally with minimal intervention.<\/p>\n<p>Fully-featured<i> Database as a Service<\/i> should be evaluated from different perspectives. As a basis, we took the most popular cloud DBs \u2014 for Google, GCP Cloud SQL, and for Amazon, AWS RDS. In our evaluation, we focused on the following categories:<\/p>\n<ul>\n<li>Feature comparison: SQL extent, DDL, DML; connection libraries\/connectors, transaction support, and so on.<\/li>\n<li>Development support: ease of development and testing.<\/li>\n<li>Administration support: instance management \u2014 for example, scaling up\/down and upgrading instances; SLA, backup and recovery; security\/access control.<\/li>\n<\/ul>\n<p><\/p>\n<h4>Using Cloud Spanner as an OLTP solution with OLAP support<\/h4>\n<p>\nAlthough Google does not explicitly state that Cloud Spanner is intended for analytical processing, it shares some attributes with other engines such as Apache Impala &amp; Kudu and YugaByte, which are designed for OLAP workloads.<\/p>\n<p>Even if there were only a slight chance that Cloud Spanner included a horizontally scalable HTAP (hybrid transactional\/analytical processing) engine with (more or less) a usable set of OLAP features, we believe it would deserve our attention.<\/p>\n<p>With that in mind, we considered the following categories:<\/p>\n<ul>\n<li>Data loading, indexing, and partitioning support<\/li>\n<li>Query performance and DML<\/li>\n<\/ul>\n<p><\/p>\n<h2>2. Cloud Spanner in Brief<\/h2>\n<p>\nGoogle Spanner is a clustered relational database management system (RDBMS) that Google uses for several of its own services. Google made it publicly available to Google Cloud Platform users in early 2017.<\/p>\n<p>Here are some attributes of Cloud Spanner:<\/p>\n<ul>\n<li>Highly consistent scalable RDBMS cluster: uses hardware time synchronization to ensure data consistency.<\/li>\n<li>Support for cross-table transactions: transactions can span multiple tables\u2014not limited to a single table (unlike Apache HBase or Apache Kudu).<\/li>\n<li>Tables based on primary key: all tables must have a declared primary key (PK), which can consist of multiple columns of the table. Tabular data is stored in PK order, making it very efficient and fast for lookups by PK. Like other PK-based systems, the implementation should be modeled with pre-thought-out use cases for achieving <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/spanner\/docs\/schema-and-data-model\">optimal performance.<\/a><\/noindex>.<\/li>\n<li>Interleaved tables: tables can have physical dependencies on each other. Rows in a child table can be matched to rows in a parent table. This approach speeds up the search for relationships that can be defined at the data modeling stage, such as co-locating customers and their invoices.<\/li>\n<li>Indexes: Cloud Spanner supports secondary indexes. An index consists of indexed columns and all columns of the primary key. Additionally, the index can also include other non-indexed columns if desired. The index can interleave with the parent table to speed up queries. Several constraints apply to indexes, such as the maximum number of additional columns that can be stored in the index. Additionally, queries through indexes may not be as straightforward as in other RDBMS.<\/li>\n<\/ul>\n<p>\n<i>Cloud Spanner automatically selects an index only in rare cases. In particular, Cloud Spanner does not automatically select a secondary index if the query requests any columns that are not stored in <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/spanner\/docs\/secondary-indexes\">the index <\/a><\/noindex>\u00bb.<\/i><\/p>\n<ul>\n<li>Service Level Agreement (SLA): deployment in a single region with an SLA of 99.99%; multi-region deployments with a 99.999% SLA. Although the SLA itself is just an agreement and not any guarantee, I believe Google employees indeed have some accurate data to make such a serious claim. (For reference, 99.999% means 26.3 seconds of service unavailability per month.)<\/li>\n<li>More: <noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/spanner\/\">https:\/\/cloud.google.com\/spanner\/<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<blockquote><p><i><b>Note:<\/b> The Apache Tephra project adds enhanced transaction support to Apache HBase (now also implemented in Apache Phoenix as a beta version).<\/i><\/p><\/blockquote>\n<p><\/p>\n<h2>3. Our assessment<\/h2>\n<p>\nSo, we've all read Google's statements about the advantages of Cloud Spanner \u2014 virtually unlimited horizontal scaling while maintaining strong consistency and a very high SLA. While these requirements are extremely difficult to achieve, our goal was not to refute them. Instead, let's focus on other aspects that concern most database users: clarity and ease of use.<\/p>\n<h4>We evaluated Cloud Spanner as a replacement for Sharded MySQL.<\/h4>\n<p>\nGoogle Cloud SQL and Amazon AWS RDS are two of the most popular OLTP databases on the cloud market, offering a very large feature set. However, to scale these databases beyond the size of a single node, you need to perform application sharding. This approach adds complexity to both applications and administration. We examined how Spanner fits into the scenario of merging multiple segments into a single instance and what features (if any) might need to be sacrificed.<\/p>\n<h4>Support for SQL, DML, and DDL, as well as connectors and libraries?<\/h4>\n<p>\nFirstly, when starting with any database, you need to build a data model. If you think you can connect JDBC Spanner to your favorite SQL tool, you\u2019ll find that you can query your data using it, but cannot use it to create tables or modify (DDL) or perform any insert\/update\/delete (DML) operations. Google\u2019s official JDBC does not support either.<\/p>\n<blockquote><p><i>\"Currently, drivers do not support DML or DDL statements.\"<\/i><br \/>\nSpanner Documentation<\/p><\/blockquote>\n<p>\nThe situation with the GCP console is not better \u2014 you can only send SELECT queries. Fortunately, there is a community-supported JDBC driver with DML and DDL support, including transactions. <noindex><a rel=\"nofollow\" href=\"http:\/\/github.com\/olavloite\/spanner-jdbc\">github.com\/olavloite\/spanner-jdbc<\/a><\/noindex>. While this driver is extremely useful, the lack of a native JDBC driver from Google is surprising. Fortunately, Google provides quite an extensive range of client libraries (based on gRPC): C#, Go, Java, node.js, PHP, Python, and Ruby.<\/p>\n<p>The virtually mandatory use of custom Cloud Spanner APIs (due to the lack of DDL and DML in JDBC) leads to some limitations for related areas of code, such as connection pools or database binding frameworks (e.g., Spring MVC). Generally, when using JDBC, you can freely choose your favorite connection pool (e.g., HikariCP, DBCP, C3PO, etc.) that is tested and works well. In the case of custom Spanner APIs, we have to rely on the frameworks\/pools\/bindings\/sessions that we built ourselves.<\/p>\n<p>Key-oriented design allows Cloud Spanner to be very fast when accessing data via the key, but also leads to some query issues.<\/p>\n<ul>\n<li>You cannot update the primary key value; you must first delete the record with the original primary key and reinsert it with the new value. (This is similar to other PK-oriented databases\/storage engines.)<\/li>\n<li>Any UPDATE and DELETE operators must specify the primary key in the WHERE clause, thus there cannot be empty DELETE all operators \u2014 there must always be a subquery, for example: UPDATE xxx WHERE id IN (SELECT id FROM table1)<\/li>\n<li>The absence of an auto-increment option or anything similar that sets a sequence for the primary key field. For this to work, the appropriate value must be generated on the application side.<\/li>\n<\/ul>\n<p><\/p>\n<h4>Secondary indexes?<\/h4>\n<p>\nGoogle Cloud Spanner has built-in support for secondary indexes. This is a very nice feature that is not always available in other technologies. Apache Kudu currently does not support secondary indexes at all, while Apache HBase does not support indexes directly but can add them through Apache Phoenix.<\/p>\n<p>Indexes in Kudu and HBase can be modeled as a separate table with a different composition of primary keys, but the atomicity of operations performed with the parent table and associated index tables must be managed at the application level and is not trivial in a proper implementation.<\/p>\n<p>As mentioned in the Cloud Spanner review, its indexes may differ from MySQL indexes. Therefore, care should be taken when constructing queries and profiling to ensure the proper index is used where necessary.<\/p>\n<h4>Views?<\/h4>\n<p>\nA very popular and useful object in a database is views. They can be beneficial for a wide range of use cases; two of my favorites are the level of logical abstraction and the level of security. Unfortunately, Cloud Spanner does NOT support views. However, this only partially limits us, as there is no column-level granularity for access permissions, where views could be an acceptable solution.<\/p>\n<p>In the Cloud Spanner documentation, in the section that details quotas and limitations (<noindex><a rel=\"nofollow\" href=\"https:\/\/cloud.google.com\/spanner\/quotas\">spanner\/quotas<\/a><\/noindex>), there is, in particular, one that may be problematic for some applications: Cloud Spanner out of the box has a limit of a maximum of 100 databases per instance. Obviously, this can become a serious obstacle for a database intended to scale beyond 100 databases. Fortunately, after speaking with our Google technical representative, we found out that this limit can be increased to virtually any value through Google support.<\/p>\n<h4>Development support?<\/h4>\n<p>\nCloud Spanner offers a fairly decent support for programming languages to work with its API. The officially supported libraries include C#, Go, Java, node.js, PHP, Python, and Ruby. The documentation is quite detailed, but, as with other advanced technologies, the community is relatively small compared to the most popular database technologies, which can lead to increased time spent resolving less common use cases or issues.<\/p>\n<h4>So, what about support for local development?<\/h4>\n<p>\nWe did not find a way to create a Cloud Spanner instance in a local environment. The closest we got was a Docker image, <noindex><a rel=\"nofollow\" href=\"https:\/\/hub.docker.com\/r\/cockroachdb\/cockroach\">CockroachDB<\/a><\/noindex>, which is generally similar, but practically very different. For example, CockroachDB can use PostgreSQL JDBC. Since the development environment should closely resemble the production environment, Cloud Spanner is not ideal because it requires relying on a full Spanner instance. To save costs, you can choose a single-region instance.<\/p>\n<h4>Administration support?<\/h4>\n<p>\nCreating a Cloud Spanner instance is very simple. You just need to choose between creating a multi-region or a single-region instance, specify the region(s), and the number of nodes. Within less than a minute, the instance will be up and ready to work.<\/p>\n<p>Several basic metrics are directly available on the Spanner page in the Google console. More detailed views are available through Stackdriver, where you can also set metric thresholds and alert policies.<\/p>\n<h4>Access to resources?<\/h4>\n<p>\nMySQL offers extensive and very detailed settings for user permissions\/roles. Access to a specific table or even just to a subset of its columns can be easily configured. Cloud Spanner uses the Google Identity &amp; Access Management (IAM) tool, which allows policies and permissions to be set only at a very high level. The most detailed option is a database-level permission, which does not fit most production cases. This limitation forces you to add additional security measures in your code, infrastructure, or both to prevent unauthorized use of Spanner resources.<\/p>\n<h4>Backups?<\/h4>\n<p>\nSimply put, there are no backups in Cloud Spanner. While Google's high SLA requirements may guarantee that you won't lose any data due to hardware or database failures, human errors, application defects, and so on. We all know the rule: high availability does not replace a sensible backup strategy. Currently, the only way to back up data is to programmatically stream it out of the database to a separate storage environment.<\/p>\n<h4>Query Performance?<\/h4>\n<p>\nWe used Yahoo! Cloud Serving Benchmark for data loading and query testing. The table below presents the YCSB workload B with a read ratio of 95% and write ratio of 5%.<\/p>\n<p><img decoding=\"async\" alt=\"Google Cloud Spanner: The Good, The Bad, and The Ugly\" src=\"\/wp-content\/uploads\/2020\/02\/8fca14875af33c8ed3cd04b602c52b78.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<i>* The load test was performed on a compute engine (CE) n1-standard-32 (32 vCPU, 120 GB memory), and the test instance was never a bottleneck in the tests.<\/i><br \/>\n<i>** The maximum number of threads in one YCSB instance is 400. A total of six parallel instances of YCSB tests needed to be run to achieve a total of 2400 threads.<\/i><\/p>\n<p>Looking at the test results, particularly the combination of CPU load and TPS, we clearly see that Cloud Spanner scales quite well. The high load created by a large number of threads is offset by the increased number of nodes in the Cloud Spanner cluster. Although the latency appears quite high, especially when working with 2400 threads, more accurate figures may require retesting with 6 smaller computational engine instances. Each instance would run one YCSB test instead of one large CE instance with 6 parallel tests. This way, it will be easier to distinguish between Cloud Spanner request latencies and the latencies added by the network connection between Cloud Spanner and the CE instance where the test is executed.<\/p>\n<h3>How does Cloud Spanner perform as an OLAP?<\/h3>\n<p><\/p>\n<h4>Partitioning?<\/h4>\n<p>\nDividing data into physically and\/or logically independent segments, known as partitions, is a very popular concept inherent to most OLAP engines. Partitions can significantly improve query performance and database maintainability. A deeper dive into partitions would warrant a separate article(s), so let's just mention the importance of having a partitioning and sub-partitioning scheme. The ability to break down data into partitions and even further into sub-partitions is key to the performance of analytical queries.<\/p>\n<p>Cloud Spanner does not support partitions as such. It divides data internally into so-called <i>splits<\/i>based on primary key ranges. The splitting is done automatically to balance load in the Cloud Spanner cluster. A very convenient feature of Cloud Spanner is the splitting of the core load of the parent table (a table that isn\u2019t interleaved with others). Spanner automatically determines if <i>splits <\/i>data is read more frequently than data in other <i>splits<\/i>splits, and can decide on further splitting. This way, more nodes can be involved in the request, which effectively increases throughput.<\/p>\n<h4>Data loading?<\/h4>\n<p>\nThe Cloud Spanner method for large data is the same as with regular loading. To achieve maximum performance, you need to follow certain recommendations, including:<\/p>\n<ul>\n<li>Sort your data by the primary key.<\/li>\n<li>Split them into 10*<i>nodes<\/i> separate sections.<\/li>\n<li>Create a set of workloads that load data in parallel.<\/li>\n<\/ul>\n<p>\nAll Cloud Spanner nodes are utilized during such data loading.<\/p>\n<p>We used workload A YCSB to generate a dataset of 10M rows.<\/p>\n<p><img decoding=\"async\" alt=\"Google Cloud Spanner: The Good, The Bad, and The Ugly\" src=\"\/wp-content\/uploads\/2020\/02\/89b9c0353716be2d389b3e8da82781ad.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<i>* The load test was conducted on a n1-standard-32 compute engine (32 vCPUs, 120 GB memory), and the test instance was never a bottleneck in the tests.<br \/>\n** A configuration of 1 node is not recommended for any production load.<\/i><\/p>\n<p>As mentioned earlier, Cloud Spanner automatically handles splits based on their load, so results improve after several consecutive test runs. The results presented here are the best outcomes we achieved. Looking at the numbers above, we can see how Cloud Spanner scales (well) with an increasing number of nodes in the cluster. The figures that stand out represent extremely low average latencies that contrast with the mixed workload results (95% for reads and 5% for writes), as described in the section above.<\/p>\n<h4>Scaling?<\/h4>\n<p>\nIncreasing and decreasing the number of Cloud Spanner nodes is a one-click task. If you want to load data quickly, you might consider boosting the instance to the maximum (in our case, it was 25 nodes in the US-EAST region), and then reduce the number of nodes to suit your regular load after all data is in the database, bearing in mind the limit of 2 TB\/node.<\/p>\n<p>We were reminded of this limit even with a much smaller database. After several iterations of load tests, our database was around 155 GB, and when scaling down to a 1-node instance, we encountered the following error:<\/p>\n<p><img decoding=\"async\" alt=\"Google Cloud Spanner: The Good, The Bad, and The Ugly\" src=\"\/wp-content\/uploads\/2020\/02\/4a46c97e7943b5105bf178316dc67433.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nWe managed to scale down from 25 to 2 instances, but got stuck at two nodes.<\/p>\n<p>The increase and decrease of nodes in a Cloud Spanner cluster can be automated using the REST API. This can be particularly useful for alleviating system load during peak hours.<\/p>\n<h4>Performance of OLAP queries?<\/h4>\n<p>\nInitially, we planned to dedicate significant time to our Spanner evaluation of this part. After several SELECT COUNT queries, we quickly realized that testing would be brief and that Spanner would NOT be suitable as an OLAP engine. Regardless of the number of nodes in the cluster, a simple count of rows in a table of 10M rows took 55 to 60 seconds. Additionally, any query requiring more memory to store intermediate results resulted in an OOM error.<\/p>\n<p><code>SELECT COUNT(DISTINCT(field0)) FROM usertable; \u2014 (10M distinct values) -&gt; SpoolingHashAggregateIterator ran out of memory during new row.<\/code><\/p>\n<p>Some figures for TPC-H queries can be found in Todd Lipcon's article. <noindex><a rel=\"nofollow\" href=\"https:\/\/kudu.apache.org\/2017\/10\/23\/nosql-kudu-spanner-slides.html\">Nosql-kudu-spanner-slides.html<\/a><\/noindex>, slides 42 and 43. These figures align with our own results (unfortunately).<\/p>\n<p><img decoding=\"async\" alt=\"Google Cloud Spanner: The Good, The Bad, and The Ugly\" src=\"\/wp-content\/uploads\/2020\/02\/d070c839bbc62b1ba1bebbaf64127e93.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h4>4. Our findings<\/h4>\n<p>\nGiven the current state of Cloud Spanner's features, it is hard to envision it as a straightforward replacement for existing OLTP solutions, especially when your needs outgrow its capabilities. Significant time would need to be spent building a solution that addresses the shortcomings of Cloud Spanner.<\/p>\n<p>When we began evaluating Cloud Spanner, we expected its management features to be on par with, or at least not too far from, other Google SQL solutions. However, we were surprised by the complete lack of backups and very limited access control to resources. Not to mention the absence of views, lack of local development environment, unsupported sequences, JDBC without DML and DDL support, and so on.<\/p>\n<p>So, where does someone go if they need to scale their transactional database? It seems there isn't a one-size-fits-all solution available in the market. There are many closed and open-source solutions (some mentioned in this article), each with their strengths and weaknesses, but none offer a SaaS with a 99.999% SLA and a high degree of consistency. If a high SLA level is your main goal and you're not inclined to build your own solution for multiple cloud environments, Cloud Spanner might be the solution you're looking for. But you should be aware of all its limitations.<\/p>\n<p>To be fair, Cloud Spanner was only made publicly available in the spring of 2017, so it's reasonable to expect that some of its current shortcomings may eventually disappear (hopefully), and when that happens, it could be a game changer. After all, Cloud Spanner isn't just a side project for Google. Google uses it as a foundation for other Google products. And when Google recently replaced Megastore in Google Cloud Storage with Cloud Spanner, it allowed Google Cloud Storage to become strictly consistent for object listings worldwide (which still doesn't apply to <noindex><a rel=\"nofollow\" href=\"https:\/\/cloudplatform.googleblog.com\/2018\/02\/how-Google-Cloud-Storage-offers-strongly-consistent-object-listing-thanks-to-Spanner.html\">Amazon\u2019s<\/a><\/noindex> <noindex><a rel=\"nofollow\" href=\"https:\/\/cloudplatform.googleblog.com\/2018\/02\/how-Google-Cloud-Storage-offers-strongly-consistent-object-listing-thanks-to-Spanner.html\">S3<\/a><\/noindex>).<\/p>\n<p>So there\u2019s still hope\u2026 we remain hopeful.<\/p>\n<p>That's all for now. Like the author of this article, we too continue to hope, but what do you think about it? Share your thoughts in the comments.<\/p>\n<p><b>We invite everyone interested to attend our <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/eoP2\/\">free webinar<\/a><\/noindex> where we will explain the course in detail<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/eoP2\/\"> AWS for Developers<\/a><\/noindex> from OTUS.<\/b><br \/>\n<br \/>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/489012\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0445\u0430\u0431\u0440\u043e\u0432\u0447\u0430\u043d\u0435. \u0422\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043d\u043e\u0432\u044b\u0445 \u043a\u0443\u0440\u0441\u043e\u0432. \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u044c\u043d\u043e \u0434\u043b\u044f \u0432\u0430\u0441 \u043c\u044b \u043f\u0440\u0435\u0432\u0435\u043b\u0438 \u0441\u0442\u0430\u0442\u044c\u044e \u043e Google Cloud Spanner, \u043f\u0440\u0438\u0443\u0440\u043e\u0447\u0438\u0432 \u0435\u0435 \u043a \u0437\u0430\u043f\u0443\u0441\u043a\u0443 \u043a\u0443\u0440\u0441\u0430 \u00abAWS \u0434\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a\u043e\u0432\u00bb. \u041f\u0435\u0440\u0432\u043e\u043d\u0430\u0447\u0430\u043b\u044c\u043d\u043e \u043e\u043f\u0443\u0431\u043b\u0438\u043a\u043e\u0432\u0430\u043d\u043e \u0432 \u0431\u043b\u043e\u0433\u0435 Lightspeed HQ. \u041a\u0430\u043a \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044f, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u043e\u0431\u043b\u0430\u0447\u043d\u044b\u0445 POS-\u0440\u0435\u0448\u0435\u043d\u0438\u0439 \u0434\u043b\u044f \u0440\u043e\u0437\u043d\u0438\u0447\u043d\u044b\u0445 \u0442\u043e\u0440\u0433\u043e\u0432\u0446\u0435\u0432, \u0440\u0435\u0441\u0442\u043e\u0440\u0430\u0442\u043e\u0440\u043e\u0432 \u0438 \u043e\u043d\u043b\u0430\u0439\u043d-\u043f\u0440\u043e\u0434\u0430\u0432\u0446\u043e\u0432 \u043f\u043e \u0432\u0441\u0435\u043c\u0443 \u043c\u0438\u0440\u0443, Lightspeed \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":69569,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-69568","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0445\u0430\u0431\u0440\u043e\u0432\u0447\u0430\u043d\u0435. \u0422\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043d\u043e\u0432\u044b\u0445 \u043a\u0443\u0440\u0441\u043e\u0432.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/google-cloud-spanner-horoshij-plohoj-zloj\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Google Cloud Spanner: \u0445\u043e\u0440\u043e\u0448\u0438\u0439, \u043f\u043b\u043e\u0445\u043e\u0439, \u0437\u043b\u043e\u0439 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0445\u0430\u0431\u0440\u043e\u0432\u0447\u0430\u043d\u0435. \u0422\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043d\u043e\u0432\u044b\u0445 \u043a\u0443\u0440\u0441\u043e\u0432.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/google-cloud-spanner-horoshij-plohoj-zloj\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-20T12:24:23+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-03T13:14:50+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Google Cloud Spanner: The Good, The Bad, The Ugly | ProHoster","description":"Hello, Habr readers. Traditionally, we continue to share interesting material in anticipation of the start of new courses.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/google-cloud-spanner-horoshij-plohoj-zloj","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Google Cloud Spanner: \u0445\u043e\u0440\u043e\u0448\u0438\u0439, \u043f\u043b\u043e\u0445\u043e\u0439, \u0437\u043b\u043e\u0439 | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0445\u0430\u0431\u0440\u043e\u0432\u0447\u0430\u043d\u0435. \u0422\u0440\u0430\u0434\u0438\u0446\u0438\u043e\u043d\u043d\u043e \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u044b\u043c \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u043e\u043c \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043d\u043e\u0432\u044b\u0445 \u043a\u0443\u0440\u0441\u043e\u0432.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/google-cloud-spanner-horoshij-plohoj-zloj","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-20T12:24:23+00:00","article:modified_time":"2020-03-03T13:14:50+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"69568","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:20:25","updated":"2022-09-30 15:10:07","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/69568","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=69568"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/69568\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/69569"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=69568"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=69568"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=69568"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}