Once they were at the top of the food chain. Millennia. And then the unthinkable happened: clouds covered the sky, and they ceased to exist. At the other end of the world, events occurred that changed the climate: cloudiness increased. Dinosaurs became too big and slow: their attempts to survive were doomed to failure. The apex predators ruled the Earth for 100 million years, growing bigger and stronger. They evolved into what seemed to be the perfect creature at the top of the food chain, but the Universe changed the face of our planet in an instant.
Ironically, it was the clouds that wiped out the dinosaurs 66 million years ago. In the same way, clouds are now destroying classic data storage systems that are βat the top of the food chainβ. In both cases, the problem was not the clouds themselves, but the ability to adapt to a changing world. In the case of the dinosaurs, everything happened quickly: the destructive effect of the clouds came within days or weeks of the meteorite impact (or volcanic eruption - the choice of theory is up to you). In the case of classical data warehouses, the process takes years, but it is, of course, irreversible.
The Triassic: The Big Iron Age and the Emergence of Migratory Applications
So what happened? The established ecosystem included entry-level and mid-range storage, enterprise-level systems, and direct-attached storage (DAS). These categories were defined by analysts, had their own market volumes, indicators of cost, reliability, performance, scalability. And then something strange happened.
The advent of virtual machines meant that many applications could run on the same server at the same time, probably from several owners - such changes immediately called into question the future of direct-attached storage. Then the owners of the largest hyperscale infrastructures (hyperscalers): Facebook, Google, eBay, etc., tired of paying a lot of money for storage systems, developed their own applications that ensured data availability on regular servers instead of large "iron" storage systems. Then Amazon introduced something strange to the market called the Simple Storage Service, or S3. Not a block, not a file, but something fundamentally new: it became impossible to buy a system, it became possible to buy only a service. Wait, what's that bright light in the sky? Another asteroid?
Jurassic: the era of "good enoughsaurs"
We entered the storage development phase with the βgood enoughβ ideology. Customers using storage, noticing what hyperscalers had done, began to question the validity of the ten or even hundred times the extra cost over hardware that they paid for their enterprise storage systems. Mid-range arrays began to win market share from high-end systems. Products such as showed rapid growth. EMC Symmetrix, the once-dominant array (from the word "massive") of the enterprise class, still held some territory, but it was rapidly decreasing. Many users have begun migrating their data to AWS.
On the other hand, storage innovators began to borrow ideas from hyperscalers, using distributed horizontally scalable systems technologies, an ideology that is the opposite of vertical scaling. It is expected that the new storage software will be able to run on conventional servers, just like hyperscalers. No more 10-100 times the cost of the equipment itself. In theory, you can use any servers - the choice depends on your preferences. The era of software-defined storage (SDS) has begun: clouds have covered the sky, temperatures have dropped, and top predator populations have begun to decline.
Cretaceous: the beginning of the evolution of software-defined storage
The early days of software-defined storage were tumultuous. Much was promised, but little was delivered. At the same time, an important technological shift took place: flash memory became a modern alternative to "rotating rust" (HDD). It was a period of many storage startups and easy-to-give venture capital money. Everything would be fine if not for one problem: data storage requires a serious attitude. It turned out that customers like their data. If they lose access to them, or a couple of wrong bits are found in terabytes of data, they worry and worry very much. Most startups didn't survive. Customers received cool functionality, but not everything was good with basic tools. Bad recipe.
Cenozoic period: storage arrays dominate
Few people talk about what happened after, because it's not very interesting - customers continue to buy all the same classic storage arrays. Of course, those who have moved their applications to the clouds have also moved their data there. But for the vast majority of customers who do not want to move to the cloud completely, or do not want to move at all, the same Hewlett Packard Enterprise continued to offer classic arrays.
We live in 2019, so why is there still a multi-billion dollar storage business based on technologies from the Y2K era? Because they work! Simply put, the requirements of mission-critical applications were not met by products built on the hype wave. Products such as HPE 3PAR remained the best choice for enterprise customers, and the next evolution of the HPE 3PAR architecture is - it only confirms.
In turn, the capabilities of software-defined storage were excellent: horizontal scalability, the use of standard servers ... But the price for this was: unstable availability, unpredictable performance and specific scalability rules.
The complexity of customer requirements is that they never get simpler. No one will say that loss of data integrity or increased downtime is acceptable. That is why an architecture is so important for storage that simultaneously meets the requirements of today's rapidly evolving data centers and at the same time, in the search for a compromise, is not devoid of key characteristics of enterprise-class storage.
Tertiary period: emergence of new forms of life
Let's try to figure out how one of the newcomers to the storage market, Datera, managed to cope with such a difficult mixture of historically established and new storage requirements. First of all, due to the implementation of an architecture focused on solving the dilemma described above. It is impossible to modify an old architecture to meet the challenges of a modern data center, just as it is impossible to modify the architecture of an average software-defined storage system to meet the requirements of enterprise-class systems: dinosaurs did not become mammals because the temperature dropped.
Building a solution that meets the demands of enterprise storage while capturing the full value of today's data center agility is no easy task, but it was exactly what Datera set out to do. Datera specialists have been working on this for five years and have found a recipe for "cooking" enterprise-class software-defined storage.
The main difficulty faced by Datera was that it had to use the logical operator "AND" instead of the much simpler "OR". Consistent availability, AND predictable performance, AND architectural scalability, AND orchestration-as-code, AND standardized hardware, AND policy enforcement, AND flexibility, AND analytics-driven governance, "AND" security, "AND" integration with open ecosystems. The logical operator "AND" is one character longer than "OR" - this is the main difference.
Quaternary: Modern Data Centers and Rapid Climate Change Drive Software-Defined Storage
So how did Datera create an architecture that meets the demands of traditional enterprise-class storage and meets the demands of today's data center at the same time? It all comes down to that pesky AND operator again.
There was no point in solving one task at a time to satisfy individual requirements. The sum of such elements will not become a single whole. As in any complex system, careful study of the whole complex of balanced trade-offs was important here. When developing, Datera specialists focused on three main principles:
- application-specific management;
- a single mechanism for ensuring data flexibility;
- high performance due to reduced overhead costs.
The common property of these principles is simplicity. Easily manage your system, easily manage your data with a single, elegant engine, and deliver predictable (and high) performance at the cost of reduced costs. Why is simplicity so important? Experienced storage wizards know that it's impossible to meet the storage requirements of today's dynamic data center with just granular management, multiple data management tools, and hyper-optimization to drive performance. A complex of such techniques is already familiar to us as a storage dinosaur.
Familiarity with these principles has served Datera well. The architecture they have developed has, on the one hand, the availability, performance, and scalability of modern enterprise-class storage, and, on the other hand, the flexibility and speed required for a modern software-defined data center.
Availability of Datera in Russia
Datera is a global technology partner of Hewlett Packard Enterprise. Datera products are tested for compatibility and performance with various server models .
You can learn more about the Datera architecture at 31 October.
Source: habr.com