The first API of MoegoSklada appeared 10 years ago. Throughout this time, we have been working on existing versions of the API and developing new ones. Meanwhile, several versions of the API have already been buried.
This article will cover a lot: how the API was created, why it is needed for cloud services, what it offers to users, the pitfalls we've encountered, and what we plan to do next.
My name is Oleg Alekseev , I am the Technical Director and co-founder of MoegoSklada.
Why create an API for a service
Our clients, numbering in the tens of thousands of entrepreneurs, actively use cloud solutions: banking, online stores, inventory management, CRM. Once they connect to one, it becomes hard to stop. Soon, the fifth, eighth, or tenth service makes an entrepreneur's work easier, but users manually transfer data between these cloud services. The work turns into a nightmare.
An obvious solution is to give users the ability to transfer data between cloud services. For instance, importing and exporting data as files that can then be uploaded to the required service. The files usually need to be modified to fit each service's format. This is more or less simple manual work, but as the number of these services grows, it becomes increasingly difficult to perform.
Therefore, the next step is an API. With it, a cloud service benefits by connecting multiple services at a single point. The emergence of such an ecosystem attracts new clients through additional capabilities. A product with new features becomes more profitable and useful.
Creating your own application programming interfaces attracts third-party vendors in the form of developers who learn about your product through the API. They start building solutions based on the provided API and earn money by automating their clients' tasks.
MoegoSklada's accounting system is built on simple processes. The main focus is on working with primary documents, the ability to receive and ship products, and generating reports for businesses based on primary data. There is also data transfer, for example, to cloud accounting, and receiving data from banking systems or retail outlets. Additionally, we work with online stores: we obtain information about products and send stock data.
The First API of MoegoSklada
Over the 10 years of MoegoSklada's operation with the API, we have developed various integrations that allow data exchange, work with banks, process payments, and utilize external telephony.
In the first year, we enabled the export of any data in XML format. At that time, users found it much more understandable and familiar to keep their data offline rather than in some cloud, and we provided them with that option. The export was initiated manually from the interface, so it couldn't yet be called an API.
At that time, we began collaborating with the company Rusagro — they were already using a 'mature' ERP for production planning and sales, while loading wagons at factories was automated in MoegoSklada. That’s how we got our first seeds of a real API: the exchange between our service and the ERP happened by sending a large file with data for all types of documents.
This is a decent option for batch data exchange, but along with the documents, their dependencies had to be transmitted: information about goods, counterparties, and warehouses. Such a heap can be easily generated during export, but it's quite difficult to unpack during import, as all the information arrives in one package: both about new documents and about existing ones.
The first XML API didn't last long — after two years, we began restructuring it. Even at the start of its operation, we made several mistakes in building the interface.
How the XML API was made: an illustration from one of our architects. By the way, expect his articles.
Here are our main mistakes:
- JAXB markup was done directly on entity beans. To communicate with the database, we use Hibernate, and JAXB markup was also applied to these same beans. This mistake became apparent almost immediately: any update to the data structure necessitated urgent notifications for all users of the API or the construction of workarounds to ensure compatibility with the previous data structure.
- The API developed as an addition, and initially, we didn’t define what part of the product it constituted. We didn't consider whether the API was something important and whether backward compatibility needed to be supported for its early clients. At one point, the number of API users made up about 5% of the small total, and they were not given much attention. The universal filtering that was created back then led to our use as a backend. This filtering was not quite GraphQL but something similar — it worked through many query string parameters. With such a powerful tool, users found it hard to resist, and requests were routed to us directly from the UI of their online stores. The situation became an unpleasant surprise because providing such a service should require a different pricing model and a fundamentally different understanding of the API as a product.
- Because the API evolved not as a primary product, the API documentation was produced and published as an afterthought — through reverse engineering. This approach seems fairly simple and convenient but contradicts contract work. This is when there's a component with a pre-established working scheme. The developer implements it according to that scheme and task, the component undergoes testing, and the client receives a product that matches the analyst's design. Reverse engineering, on the other hand, throws a product onto the market that simply exists: with workarounds, odd solutions, and cobbled-together parts instead of the needed functionality.
- The entire stream of requests coming through the API could only be analyzed as Nginx or application server logs. This didn’t allow us to isolate subject areas, except perhaps by splitting them by users and subscribers. If there’s no way to regulate the registration of applications or clients, analyzing the situation becomes impossible. This issue had the least impact on the API's development; it was more about understanding its demand and functionality.
Attempt number two: REST API
In 2010, we attempted to build a data exchange system with an online accounting software called BuhSoft. It didn't take off. However, during the integration process, a full-fledged API emerged: a REST exchange service where there were no indulgences like making RPC calls. All communication with the API was reduced to the standard REST mode: the request string contains the entity name, and the operation performed on it is defined using the HTTP method. We added filtering based on the last update of entities, allowing users to build replication with their systems.
That same year, an API for exporting warehouse and inventory balances became available. Users gained access to the most valuable parts of the system through the API — the exchange of primary documents and calculated data on balances and product costs.
In December 2015, RetailCRM published the first third-party library for accessing our API. It was actively used, and at the same time, the service's popularity grew, with the API load increasing faster than that of the web interface. Eventually, the growth turned into a surge in load.
And this surge, indicated by the arrow to the left, completely astonished the server that manages our API. We spent a week figuring out what exactly generated this load. It turned out that it was those requests being routed to our API from clients' front ends. About 50 clients were responsible for the entire load. That’s when we realized one of our mistakes — the complete lack of limits.
As a result, we introduced a limit on the number of simultaneous requests. From one account, it became possible to open no more than two requests at once. This is sufficient for operating in replication mode for data exchange in batch mode. Those who wanted to use us as a backend were now required to better align with the tariffs, as they needed to work with multiple accounts in their software tools.
Bringing it in line
Since 2014, the demand for the existing API has become an important part of the business, generating the largest volume of data exchange with clients. In 2015, we launched a project to tidy up the API. We chose JSON as the format instead of XML and began building it based on the features we identified during the implementation of the previous version:
- The ability to manage versions. Versioning allows for the development of a new version without affecting the existing application or disrupting user operations.
- The ability for a user to see metadata in the response they receive.
- The ability to exchange large documents. When processing a document with more than 4-5 thousand items, it becomes a problem for the server: a long transaction, a long HTTP request. We built a special mechanism that allows updating the document in parts and managing individual items of that document, sending them to the server.
- Replication tools — were also present in the previous version.
- Load limits — as a legacy from the mistakes made in the previous version. We introduced limits on the number of requests per time interval, the number of parallel requests, and requests from a single IP address.
Since then, we have released two minor versions of the API and launched several specialized APIs, but the overall approach has remained unchanged. The updated exchange format and new architecture have allowed us to fix shortcomings in the API much faster.
MyWarehouse API today
Today, MyWarehouse API solves many tasks:
- data exchange with online stores, accounting systems, banks;
- retrieving calculation data, reports;
- serving as a backend for client applications — our mobile applications and desktop cashier work via the API
- sending notifications about data changes in MyWarehouse — webhooks;
- telephony;
- loyalty systems.
Based on the API, our CEO Askar Rakhimberdiev wrote a telegram bot in four hours that pulls stock data through the API:
Now for the dry numbers.
Here are our statistics from the old REST API:
- 400 companies;
- 600 users;
- 2 million requests per day;
- 200 GB per day of outgoing traffic.
And here is what we have achieved with all MyWarehouse APIs:
- over 70 integrations (some of which can be viewed here );
- 8500 companies;
- 12,000 users;
- 46 million requests per day;
- 2 TB per day of outgoing traffic.
What's Next
Plans for API development are actively being discussed. We strive to incorporate user feedback based on their operational experiences. While we cannot implement everything immediately, a new version of the API with more user-friendly metadata and a less convoluted structure, along with OAuth for authentication and an API for integrated applications, is on the horizon.
Stay updated with news on the dedicated site for developers integrating with MyWarehouse: .
Source: habr.com
