In the first part, we discussed why we decided to replace the old BMS system in our data centers with a new one. And not just replace it, but develop it from scratch to meet our requirements. In the second part, we explain how we did it.
Market analysis
Taking into account the wishes mentioned in and the decision to abandon the upgrade of the existing system, we wrote a technical specification to search for solutions on the market and made inquiries to several major companies specializing only in the creation of industrial SCADA systems.
The first responses from them showed that the market leaders in monitoring systems predominantly continue to operate on physical servers, although the migration to the cloud in this segment has already begun. As for the reservation of virtual machines, no one supported this option. Moreover, it seemed that none of the prominent developers in the market even demonstrated an understanding of the necessity for reservation: "the cloud never goes down" was the most common response. In fact, we were offered to place the monitoring of the data center in the cloud, physically located within the same data center.
Here, I need to make a brief digression about the contractor selection process. Price does matter, of course, but during any tender for the implementation of a complex project, at the dialogue stage with suppliers, you start to feel who among the candidates is more motivated and capable of realizing it.
This is especially noticeable with complex projects.
By the nature of the clarifying questions regarding the technical specification, contractors can be divided into those interested simply in selling (the typical push from a sales manager is felt) and those interested in developing a product, listening to and understanding the client, making constructive adjustments to the technical specification even before the final selection (despite the real risk of improving someone else's technical specification and losing the tender), ultimately simply those ready to take on a professional challenge and create a good product.
All of this made us pay attention to a relatively small local developer – the Sunline Group, which responded to most of our requirements immediately and was ready to meet all our needs regarding the new BMS.
Risks
While major players were trying to understand what we wanted and engaged in a slow email exchange involving pre-sales specialists, a local developer arranged a meeting at our office with their technical team. At this meeting, the contractor expressed their willingness to participate in the project once again and, most importantly, explained how the required system would be implemented.
Before the meeting, we identified two risks of working with a team that lacks backing from a large national or international company:
- The specialists might overestimate their capabilities and consequently fail to deliver. For example, they may use complex software or design unfeasible reservation algorithms.
- After the project is implemented, the project team could disband, putting product support at risk.
To minimize these risks, we invited our in-house development specialists to the meeting. The employees of the potential contractor were thoroughly questioned about the foundation of the system, how they plan to implement the reservation, and other matters in which we, as an operations service, are not sufficiently competent.
The verdict was positive: the architecture of the existing BMS platform is modern, simple, and reliable, can be further developed, and the proposed reservation and synchronization scheme is logical and functional.
We addressed the first risk. We eliminated the second risk after receiving confirmation from the contractor of their readiness to provide us with the source code and documentation of the system, as well as selecting the Python programming language, which is well known to our specialists. This guaranteed us the ability to support the system on our own without any difficulties or a lengthy training period for employees in case the developing company exits the market.
An additional advantage of the platform was that it was implemented in Docker containers: in this environment, the core, web interface, and product database operate. This approach offers many benefits, including pre-configured settings for the highest deployment speed compared to 'the classic method' and effortless addition of new devices to the system. The 'all together' principle greatly simplifies the system's implementation: it is enough to unpack the system, and it can be put into operation immediately.
With this solution, it is easier to back up the system, and upgrades can be implemented in a separate environment without stopping the overall operation of the solution.
After both risks were minimized, the contractor provided a proposal. It detailed all the most important parameters of the BMS system for us.
Failover
The new BMS system was to be hosted in the cloud on a virtual machine.
No hardware, no servers, and all the inconveniences and risks associated with this deployment model – the cloud solution allowed us to eliminate them forever. It was decided that the system would operate in our cloud across two data center locations in Saint Petersburg and Moscow. These are two fully functional systems working in active standby mode with access for all authorized specialists.
The two systems back each other up, ensuring full redundancy for both computing power and data transmission channels. Additional security measures have also been implemented, including data and channel backups, system backups, complete virtual machine backups, and separate monthly backups of the database (the most valuable resource in terms of management and analysis).
It is worth noting that the backup option as part of the BMS solution was developed specifically at our request. The backup scheme looked like this:
Support
A crucial aspect for the effective operation of the BMS solution is technical support.
Here it is simple: the new system would cost us 35,000 rubles per month for an SLA of 'response within 8 hours,' which means 35,000 x 12 / 80 = $5,250 per year. The first year is free.
For comparison: the support for the old BMS from the vendor cost $18,000 per year, increasing the amount for each new device added! Moreover, the company did not provide a dedicated manager; all interaction occurred through the sales manager, who was interested in us as a potential buyer, affecting the handling of requests accordingly.
For less money, we received full product support, with an account manager involved in product development, a single point of entry, etc. The support became significantly more flexible – thanks to direct access to developers for timely adjustments on any aspects of system operation, integration through the API, etc.
Updates
According to the proposed offer for the new BMS, all updates are included in the support cost, meaning no additional payments are required. The only exception is for the development of additional functionality beyond what is specified in the technical specifications.
The old system required payment for both updates of the built-in free software (like Java) and for bug fixes. There was no way to opt out of this, as the system as a whole ‘lagged’ without updates due to old versions of internal components.
And, of course, it was impossible to update the software without purchasing a support package.
Flexible approach
Another fundamental requirement concerned the interface. We wanted to ensure access through a web browser from anywhere, without the mandatory presence of an engineer on the data center premises. Additionally, we aimed to create an animated interface so that the dynamics of infrastructure functioning would be more visually apparent to the on-duty engineers.
The new system also needed to support formulas for calculating the performance of virtual sensors in engineering systems – for example, for optimal distribution of electrical power across equipment racks. For this, it is necessary to have access to all familiar mathematical operations applicable to the sensor readings.
Next, access to the SQL database was required to retrieve the necessary data about the equipment's operation – namely, all records of monitoring two thousand devices and two thousand virtual sensors, generating approximately 20 thousand variables.
We also needed a rack equipment accounting module that provides a graphical representation of the layout of devices in each unit, counting the total weight of the hardware, maintaining a device library, and providing detailed information about each element.
Agreement on technical specifications and contract signing
At the time when it was necessary to start working on the new system, correspondence with 'big' companies was still far from discussing the costs of their proposals, so we compared the received commercial proposal with the costs of upgrading the old BMS (see ), and as a result, it turned out to be more attractive in price and suitable for our requirements.
The choice was made.
After selecting the contractor, lawyers began drafting the contract, while the technical teams on both sides refined the technical specifications. As is known, detailed and competent technical specifications are the foundation of success for any work. The more specifics in the technical specifications, the fewer disappointments like 'but we wanted it differently.'
I will provide two examples of the level of detail in the technical specifications:
- The duty data centers are authorized to add new devices to the BMS, most often PDUs. In the old BMS, this was an 'administrator' level, allowing for changing the variable settings of all devices, and it was impossible to separate functions. We were not satisfied with this. In the existing basic version of the new platform, the scheme was similar. We immediately stated in the technical specifications that we wanted to separate these roles: only an authorized employee should change the settings, but the duty personnel should still have the ability to add devices. This scheme was approved for implementation.
- In any standard BMS, there are three typical categories of notifications: RED – immediate response required, YELLOW – can be observed, BLUE – 'Informational'. We traditionally used 'blue' notifications to monitor excesses in commercial parameters, such as exceeding the power limit of a client's rack. This type of notification was intended for managers and was not of interest to the operations service, but in the old BMS, it regularly cluttered the list of active incidents and interfered with operational work. We considered the logic and color differentiation of the notification banners successful and retained it, however, in the specifications, we specifically noted that 'blue' notifications should quietly 'pile up' in a separate section, where they would be addressed by commercial specialists without distracting the on-duty staff.
Formats for building charts and generating reports, outlines of interfaces, a list of devices to be monitored, and many other details were prescribed with the same level of detail.
This was truly a creative endeavor for three working groups – the client service, which dictated its requirements and conditions; technical specialists from both sides, whose task was to transform these conditions into technical documentation; and the contractor's programming team, which implemented the client's requirements based on the developed technical documentation… Ultimately, we adapted some of our non-critical requirements to fit the functionality of the existing platform, and some the contractor committed to expand for us.
Parallel operation of two systems
The time for implementation has come. In practice, this meant that we were giving the contractor the opportunity to deploy a BMS prototype in our virtual cloud and provide network access to all devices requiring monitoring.
At the same time, the new system was not yet ready for operation. At this stage, it was crucial for us to maintain monitoring in the old system while simultaneously providing access to devices for the new system. It is impossible to build a system properly without seeing the devices in it, which in turn cannot be disconnected from monitoring by the old system.
Whether the devices can withstand simultaneous polling by two systems was not evident without real testing. There was a possibility that dual simultaneous polling would lead to frequent response failures from the devices, resulting in numerous errors regarding device unavailability, which in turn would block the operation of the old monitoring system.
The network department set up virtual routes from the prototype of the new BMS deployed in the cloud to the devices, and we received the results:
- devices connected via SNMP protocol virtually never disconnected due to simultaneous queries,
- devices connected through gateways using modbus-TCP protocols had issues that were resolved by reasonably reducing their polling frequency.
And then we began to observe how a new system was being built before our eyes, featuring familiar devices, but in a different interface – convenient, fast, and accessible even from a phone.
We will discuss what we achieved in the end in the third part of our article.
Source: habr.com
