Recently, I learned that (reportedly due to licensing changes). This made me think, as Iāve seen a lot of articles over the past few years claiming how terrible MongoDB is and that no one should ever use it. But during this time, MongoDB has evolved into a much more mature product. What happened? Is all the hate really due to marketing missteps with the new DBMS? Or are people just using MongoDB inappropriately?
If you think Iām defending MongoDB, please read the at the end of the article.
New Trend
Iāve been in the software industry longer than I care to admit, but I have still only experienced a fraction of the trends that have struck our field. Iāve witnessed the rise of 4GL, AOP, Agile, SOA, Web 2.0, AJAX, blockchainā¦ the list goes on. Every year, new trends emerge. Some fade quickly, while others fundamentally change software development methods.
Around each new trend, a certain public excitement is generated: people either jump on the bandwagon themselves or see the noise created by others and follow the crowd. This process was codified by Gartner in the . Although controversial, this chart roughly describes what happens with technologies before they ultimately become useful.
Yet every once in a while, a new innovation appears (or experiences a resurgence, as is the case here), driven solely by one specific implementation. In the case of NoSQL, the hype was heavily influenced by the emergence and rapid ascent of MongoDB. MongoDB didnāt start this trend; in fact, major internet companies faced challenges in handling large volumes of data that led to the revival of non-relational databases. The movement began with projects like Googleās Bigtable and Facebookās Cassandra, but MongoDB became the most well-known and accessible NoSQL database implementation available to most developers.
Note: You might think Iām conflating document databases with columnar databases, key/value stores, or any of the numerous other types of data storage that fall under the general definition of NoSQL. And youād be right. But at the time, chaos ruled. Everyone was obsessed with NoSQL; it became absolutely necessary, although many did not see differences in different technologies. For many, MongoDB has become synonymous with NoSQL.
And developers jumped at the opportunity. The idea of a schema-less database that magically scales to solve any problem was quite enticing. Around 2014, it seemed like everywhere that a relational database like MySQL, Postgres, or SQL Server was used a year ago, MongoDB databases were being deployed instead. When asked why, you could get responses ranging from the mundane "it's the scalability of the web" to the more thoughtful "my data is very loosely structured and fits nicely into a schema-less database."
It's important to remember that MongoDB and document databases, in general, address a number of problems with traditional relational databases:
- Strict schema: with a relational database, if you have dynamically generated data, you are forced to either create a bunch of random "different" data columns, cram in data blobs, or use a configuration ā¦ which has significant downsides.
- Scaling difficulties: if the data is so much that it doesnāt fit on one server, MongoDB offered mechanisms to scale it across multiple machines.
- Complex schema modifications: no migrations! In a relational database, changing the database structure can become a huge problem (especially when thereās a lot of data). MongoDB greatly simplified this process, making it so easy that you can simply update the schema on the fly and move forward quickly.
- Write performance: MongoDB's performance was good, especially with proper tuning. Even the out-of-the-box MongoDB configuration, often criticized, demonstrated some impressive performance metrics.
All the risks are on you.
The potential benefits of MongoDB were enormous, especially for certain classes of problems. Reading the list above without understanding the context and lacking experience may give the impression that MongoDB is indeed a revolutionary DBMS. The only issue was that the aforementioned advantages came with a number of caveats, some of which are listed below.
Fairness to say, no one at 10gen/MongoDB Inc. will claim that what follows is untrue; these are simply compromises.
- Loss of Transactions: Transactions are a core feature of many relational databases (not all, but most). Transactionality means you can perform multiple operations atomically and ensure that data remains consistent. Of course, with a NoSQL database, transactionality might be within a single document, or you can use two-phase commits to achieve transactional semantics. However, you will have to implement this functionality yourself... which can be a complex and labor-intensive task. Often, you don't realize the issues until you see data in the database entering invalid states because it's impossible to guarantee the atomicity of operations. Note: Many have informed me that transactions were introduced in MongoDB 4.0 last year, but with several limitations. The takeaway from the article remains the same: Evaluate how well the technology meets your needs.
- Loss of Relational Integrity (Foreign Keys): If your data has relationships, you will need to enforce them in the application. Having a database that respects these relationships will offload significant work from the application and, consequently, from your programmers.
- Inability to Enforce Data Structure: Strict schemas can sometimes be a huge issue, but they are also a powerful mechanism for good data structuring when used correctly. Document databases like MongoDB provide incredible schema flexibility, but this flexibility removes the responsibility for keeping the data clean. If you don't take care of it, you will ultimately have to write a lot of code in the application to account for the data stored in a form that you do not expect. As often said in our company, Simple Threadā¦ applications will eventually be rewritten, but data will live forever. Note: MongoDB supports schema validation; it is useful but does not provide the same guarantees as a relational database. First of all, adding or modifying schema validation does not affect existing data in the collection. You must ensure that you update the data according to the new schema yourselves. Decide if this is sufficient for your needs.
- Proprietary query language / loss of tool ecosystem: the emergence of SQL was an absolute revolution, and since then nothing has changed. It is an incredibly powerful language, but also quite complex. The need to construct database queries in a new language, consisting of JSON fragments, is seen as a significant step back by those with SQL experience. There is a whole universe of tools that interact with SQL databases: from IDEs to reporting tools. Transitioning to a database that does not support SQL means that you cannot use most of these tools or you need to translate the data into SQL to use it, which can be more complicated than you think.
Many developers who turned to MongoDB did not fully understand the trade-offs and often dove in headfirst, setting it up as their primary data store. After such a commitment, it was often incredibly difficult to revert.
What could have been done differently?
Not everyone jumped in headfirst and hit rock bottom. But quite a few projects installed MongoDB in places where it simply did not fitāand they will have to live with it for many years. If these organizations had taken some time and methodically considered their technology choices, many would have made a different decision.
How to choose the right technology? There were several attempts to create a systematic framework for assessing technologies, such as and , but I think that is overly complicated.
Many technologies can be reasonably assessed by asking just two fundamental questions. The problem lies in finding people who can responsibly answer them, spending time to find answers and without prejudice.
If you are not facing a particular problem, you do not need a new tool. Point.
Question 1: What problems am I trying to solve?
If you are not facing any problem, you donāt need a new tool. Period. Thereās no need to look for a solution and then invent a problem. If you haven't encountered an issue that the new technology solves significantly better than your existing technology, then there's nothing to discuss. If youāre considering this technology just because youāve seen others use it, think about the problems they face and ask yourself if you have similar issues. Itās easy to adopt technology simply because others are using it; the challenge lies in understanding whether youāre facing the same problems.
Question 2: What am I giving up?
This is definitely a tougher question because it requires digging deep and gaining a good understanding of both the old and new technologies. Sometimes, you canāt truly understand the new one until youāve built something with it, or until you have someone on your team who has that experience.
If you lack both, it makes sense to think about the minimal possible investments required to ascertain the value of this tool. And if you make the investments, how difficult will it be to reverse the decision?
People always mess things up
As you try to answer these questions as objectively as possible, remember one thing: youāll have to contend with human nature. There are a number of cognitive biases that need to be overcome to effectively evaluate technology. Here are just a few:
- ā everyone knows about it, yet itās still hard to fight. Just make sure the technology truly meets your actual needs.
- ā many developers tend to underestimate the technologies they've worked with for a long time and overestimate the benefits of new technology. This cognitive bias affects not just programmers; everyone is susceptible.
- ā we tend to focus on what is present and overlook what is missing. This can lead to chaos in combination with the novelty effect, as you not only inherently overvalue the new technology but also ignore its shortcomings..
An objective assessment is not easy to achieve, but understanding the fundamental cognitive biases will help make more rational decisions.
Summary
When an innovation arises, it is important to cautiously address two questions:
- Does this tool solve a real problem?
- Do we understand the trade-offs well?
If you cannot confidently answer these two questions, take a few steps back and think.
So was MongoDB actually the right choice? Of course, yes; like most engineering technologies, it depends on many factors. Among those who answered these two questions, many have benefited from MongoDB and continue to do so. Those who did not, I hope, learned a valuable and not too painful lesson about navigating the hype cycle.
Disclaimer
I want to clarify that I feel neither love nor hatred towards MongoDB. It's just that we did not have any issues for which MongoDB was the best solution. I know that 10gen/MongoDB Inc. initially acted quite boldly by setting unsafe defaults and promoting MongoDB everywhere (especially at hackathons) as a universal solution for any type of data. That was likely a poor decision. But it supports the approach described here: these issues could be identified very quickly even with a superficial evaluation of the technology.
Source: habr.com
