S3 object storage command translated an article about what criteria are important when choosing an object storage. Below is the text from the author.
When it comes to object storage, people tend to think of only one characteristic - price per TB/GB. Of course, this metric is important, but it makes the approach one-sided and equates object storage with an archive storage tool. Plus, this approach reduces the importance of object storage for the enterprise technology stack.
When choosing object storage, you should pay attention to five characteristics:
- performance;
- scalability
- compatible with S3;
- failure response;
- integrity.
These five characteristics are new metrics for object storage, along with cost. Let's consider them all.
Performance
Traditional object stores don't perform well. Service providers constantly donated to her in pursuit of low prices. However, things are different with modern object storage.
The speed of various storage approaches or even exceeds Hadoop. Modern requirements for read and write speed: from 10 GB / s for hard drives, up to 35 GB / s for NVMe.
This bandwidth is enough for Spark, Presto, Tensorflow, Teradata, Vertica, Splunk and other modern computing frameworks in the analytics stack. The fact that MPP databases are configured for object storage means that it is increasingly being used as the primary storage.
If your storage system does not provide the required speed, you cannot use the data and extract values ββfrom it. Even if you are retrieving data from object storage into an in-memory processing structure, you still need bandwidth to transfer data to and from memory. For obsolete object stores, it is not enough.
This is the key point: the new performance metric is throughput, not latency. It is required for scalable data and is the norm in today's data infrastructure.
And while benchmarks are a good way to measure performance, it can't be accurately measured before running the application in the environment. Only after it can you tell where exactly the bottleneck is: in software, disks, network, or at the level of computing.
Scalability
Scalability refers to the number of petabytes that fit into one namespace. Vendors claim easy scalability, but remain silent about how massive monolithic systems become fragile, complex, unstable, and expensive as they scale.
A new measure of scalability is the number of namespaces or clients you can serve. The metric is taken directly from hyperscalers, where storage building blocks are small but scale to billions of units. In general, this is a cloud metric.
When building blocks are small, they are easier to optimize, that is, provide security, access control, policy management, lifecycle, and updates without interruption. And ultimately ensure performance. The size of the building block is a function of the controllability of the failure area, and this is how highly resilient systems are built.
Multi-client has many characteristics. While the setting refers to how organizations provide access to data and applications, it also refers to the applications themselves and the logic behind their isolation from each other.
Characteristics of the modern approach to multi-client:
- In a short time, the number of customers can grow from several hundred to several million.
- Clients are completely isolated from each other. This allows them to run different versions of the same software and store objects with different configurations, permissions, features, security levels, and service levels. This is necessary when scaling new servers, upgrades, and geographies.
- The storage is elastically scalable, resources are provided on demand.
- Each operation is API driven and automated without human intervention.
- Software can be hosted in containers and use standard orchestration systems such as Kubernetes.
S3 compatible
Amazon S3 API is the de facto standard for object storage. Each vendor of object storage software claims to be compatible with it. Compatibility with S3 is binary: either it is fully implemented, or it is not.
In practice, there are hundreds and thousands of edge scenarios where something goes wrong when using object storage. Especially from vendors of proprietary software and services. Its main use cases are direct archiving or backup, so there are few reasons to call the API, the use cases are uniform.
Significant advantages of open source software. It covers most edge scenarios given the size and variety of applications, operating systems, and hardware architecture.
All this is important for application developers, so it is worth testing the application with storage providers. Open source simplifies the process - it's easier to understand which platform is right for your application. The provider can be used as a single point of entry into the vaults, which means it will suit your needs.
Open source means applications are vendor-neutral and more transparent. This ensures a long application life cycle.
And a few more notes about open source and S3.
If you're running a big data application, S3 SELECT improves performance and efficiency by an order of magnitude. It does this by using SQL to retrieve only the objects you need from the store.
The key point is support for bucket notifications. Bucket notifications facilitate serverless computing, an essential component of any microservice architecture that is delivered as a service. Considering that object storage is actually cloud storage, this capability becomes critical when cloud applications use object storage.
Finally, the S3 implementation must support the Amazon S3 server-side encryption APIs: SSE-C, SSE-S3, SSE-KMS. It's even better if S3 supports tamper protection, which is really secure.
Failure response
A metric that is probably often overlooked is how the system handles failures. Failures happen for a variety of reasons, and the object store must handle them all.
For example, there is a single point of failure, the metric of this is zero.
Unfortunately, many object storage systems use special nodes that must be enabled for the cluster to function properly. These include name nodes or metadata servers - this creates a single point of failure.
Even where there are multiple points of failure, the ability to withstand catastrophic failures is paramount. Drives fail, servers fail. The key is to create software designed to handle failures as normal. If a disk or node fails, such software will continue to work without changes.
Built-in erasure and data degradation protection ensures that you can lose as many disks or nodes as you have parity blocks - usually half the disks. And only then the software will not be able to return data.
Failure is rarely tested under load, but such testing is required. Simulation of a failure under load will show the cumulative costs incurred after a failure.
Consistency
A consistency score of 100% is also referred to as strong consistency. Consistency is a key component of any storage system, but strong consistency is rare. For example, Amazon S3 ListObject is not strictly consistent, it is only at the end consistent.
What is meant by strict consistency? For all operations after a confirmed PUT operation, the following must be true:
- The updated value is visible when reading from any node.
- The update is protected by node failure redundancy.
This means that if you unplug the plug in the middle of a recording, nothing will be lost. The system never returns corrupted or outdated data. This is a high bar that matters for many scenarios, from transactional applications to backup and recovery.
Conclusion
These are new object storage metrics that reflect usage patterns in today's organizations where performance, consistency, scalability, fault domains, and S3 compatibility are the building blocks for cloud applications and big data analytics. I recommend using this list in addition to price when building modern data stacks.
About Mail.ru Cloud Solutions object storage: .
What else to read:
- .
- .
- .
Source: habr.com