В , I talked about the background of the creation of Veliam and the decision to distribute it through the SaaS system. In this article, I will talk about what I had to do to make the product not local, but public. About how they started distribution and what problems they encountered.
Planning
The current backend for users was on Linux. Almost every organization has Windows servers, which cannot be said about Linux. The main strength of Veliam is remote connections to servers and network equipment behind NAT. But this functionality was very tightly tied to the fact that the router had to be a Mikrotik. And this obviously would not satisfy many. I first started thinking about adding support for most common vendor routers. But I understood that this is an endless race with the expansion of the list of supported firms. Moreover, those that are already supported may have a different set of commands for changing NAT rules from model to model. VPN seemed to be the only way out.
Since we decided to distribute the product, but not as open source, it became impossible to include various libraries with open licenses such as GPL. This is generally a separate issue, after deciding to sell the product, I had to sort out half of the libraries due to the fact that they were GPL. When they wrote for themselves, it was normal. But not suitable for distribution. The first VPN that comes to mind is OpenVPN. But it's GPL. Another option was to use the Japanese SoftEther VPN. His license allowed him to be included in his product. After a couple of days of various tests, how to integrate it in such a way that the user did not need to configure anything at all and know about SoftEther VPN, a prototype was obtained. Everything was as it should be. But, for some reason, this scheme still confused us, and we eventually abandoned it. But of course they refused after they came up with another option. As a result, everything was done on normal TCP connections. Some of the connections work through the coordinator, some directly through the Nat Hole Punching (NHP) technology, which was also implemented in Free Pascal. I must say that I had never even heard of NHP before. And it could not occur to me that it is possible to connect 2 network devices, both of which are behind NAT directly. I studied the topic, understood the principle of work and sat down to write. The plan is implemented, the user connects with one click to the desired device behind NAT via RDP, SSH or Winbox without entering passwords and setting up a VPN. Moreover, most of these connections go past our coordinator, which has a good effect on ping and the cost of servicing these connections.
Transfer of the server part from Linux to Windows
There were several problems during the transition to Windows. The first is that the built-in wmic in windows does not allow you to make WQL queries. And in our system, everything was already built on them. And there was something else, but now I forgot why they finally abandoned its use. Possibly differences between versions of Windows. And the second problem is multithreading. Having not found a good third-party utility under a “permissible” license for us, I launched the Lazarus IDE again. And wrote the necessary utility. The required list of objects is given as an input and what requests should be made, and in response I receive data. And all this in multi-threaded mode. Great.
After I set up pthreads for PHP Windows, I thought that everything would start right up, but that was not the case. After some debugging time, I realized that pthreads seems to work, but it didn’t work on our system. It became clear that there is some peculiarity of working with pthreads in Windows. So it was. I read the documentation, and it was written there that the number of threads is limited for Windows, and, as far as I remember, implicitly. This has become a problem. Because when I started to reduce the number of threads on which the application was running, it was doing the work very slowly. I opened the IDE again and the functionality for multi-threaded object pinging was added to the same utility. Well, to the heap already and port scanning there too. Actually, after that, the need for pthreads for PHP disappeared, and it is no longer used. Further, several more functionalities were added to this utility and it works to this day. After that, an installer for Windows was assembled, which included Apache, PHP, MariaDB, the PHP application itself and a set of utilities for interacting with the system, written in Free Pascal. As for the installer, I thought that I would quickly solve this issue, because. this is a super common and necessary thing for almost every software. Whether I not so searched, whether still something. But I constantly came across products that were either not flexible enough, or expensive and also inflexible. And yet, I found a free installer, in which it will be possible to provide for any Wishlist. This is InnoSetup. I'm writing about this here because I had to look, maybe I'll save someone time.
Cancellation of the plugin in favor of your client
I wrote earlier that the client side was a browser with a "plugin". So there were times when Chrome was updated and the layout was a little crooked, then Windows was updated and the custom uri scheme flies off. I really did not want to have such surprises in the public version of the product. Moreover, custom uri began to fly after each Windows update. Microsoft simply deleted all non-her branches in the right section. Also, Google Chrome now does not allow remembering the choice to open or not an application from a custom uri, and asks this question every time you click on the monitored object. Well, in general, normal interaction with the user's local system was necessary, which the browser does not provide. The simplest option in such a scheme seems to be to simply make your own browser, as many people now do through Electron. But already many things were written in Free Pascal, including in the server part, so we decided to make the client in the same language, and not produce a zoo. This is how the client was written with Chromium on board. After that, he began to acquire various harnesses.
Release
Finally chose a name for the system. We were constantly iterating over different options as we went through the process of converting from on-premises to SaaS. Since we initially planned to enter not only the domestic market, the main criterion for choosing a name was the presence of an unoccupied or not very expensive domain in the “.com” zone. Some functions/modules have not yet been ported from the local version to Veliam, but we decided that we would release with the current functionality, and finish the rest as updates. In the very first version, there was no HelpDesk, Veliam Connector, it was impossible to change notification trigger thresholds, and much more. We bought a Code Sign Certificate, signed the client and server parts. We wrote a website for the product, started the procedures for registering software, trademarks, etc. Basically ready to start. Light euphoria from the work done and from the fact that someone will probably use your product, although we had no doubts about this. And then stop. The partner said that it is impossible to enter the market without notifications in messengers. It is possible without many other things, but not without this. After a short debate, integration with Telegram was added, which suited us. Of all the current messengers, this is the only one that gives access to its APIs for free and without any complicated approval procedures. The same WhatsApp offers to contact providers who charge good money for using their services, all letters asking for access without spacers were ignored. Well, viber ... I don’t know who uses it now, because. spam and ads are rampant. At the end of December, after a series of internal testing and testing among friends, we opened registration for everyone and posted the software for download.
Start of distribution
From the very beginning, we understood that we needed a small stream of system users so that they would test the product in combat mode and give some kind of first feedback. Several purchased posts in VK have borne fruit. The first registrations have gone.
It must be said here that entering the market when your company does not have a famous name, and at the same time providing agentless monitoring functionality in which you need to enter accounts from your servers and workstations, is very difficult. A lot of people are scared by this. From the very beginning we understood that there would be problems with this and were ready for this both technically and morally. All remote connections, despite the fact that RDP and SSH are already encrypted by default, are additionally encrypted by our software according to the AES standard. All data from local servers is transferred to the cloud via HTTPS. Accounts are stored in encrypted form. Encryption keys for all subsystems are individual for all clients. For remote connections, session encryption keys are generally used.
All we can do in this situation to make people calmer is to be as open as possible, work on security and never get tired of answering people's questions.
For many, the convenience and functionality of the software outweigh the fear, and they register. Some personalities in published posts in VK wrote that this software cannot be used. this is a collection of their passwords and a no-name company in general. I must say that such an opinion was not one person. Many simply do not understand that when they install other proprietary software on a server that runs as a service, it also has full rights in the system and they do not need accounts in order to do something illegal (it is clear that you can change the user from which the service is started, but here, you can enter any account). In fact, people's fears are understandable. Installing software on a server is a common thing, but entering an account is already a little scary and intimate, since a good half of people have one password for all services, and making a separate account even for a test is lazy. But at the moment there are a huge number of services that people trust with their credentials and more. And we aspire to be one of them.
A lot of comments were of such a plan that we stole it somewhere. This surprised us a little. Well, okay, the opinion of one person, but such comments were found in various publications from different people. They didn't know how to react at first. Either be sad that some people have the opinion that in Russia no one can do anything on their own, but can only steal, or rejoice that they think that this can only be stolen.
We have now completed the procedure for obtaining the EV Code Sign Certificate. To get it, you need to go through a series of checks and send a bunch of documents about the company, some of which must be certified by a lawyer. Obtaining an EV Code Sign certificate in a pandemic is a separate topic for an article. The procedure dragged on for a month. And it was not a month of waiting, but of constant requests for additional documents. Maybe the pandemic has nothing to do with it, and the procedure took so long for everyone? Share.
Some say that we will not use it because there is no FSTEC certificate. We have to explain that we can’t get it and won’t because to get this certificate, encryption must be in accordance with GOST, and we plan to distribute software not only in Russia and use AES.
All these comments evoked some uncertainty that it is possible to promote a product in which you need to enter accounts without being at the same time heard. Even though we knew there would be those who were very negative about it. After the number of registrations exceeded a thousand, we stopped thinking about it. Especially after, in addition to the negativity of those who have not even tried the product, very pleasant reviews began to appear. I must say that these positive reviews are the biggest motivator for product development.
Adding remote access functionality for employees
One of the frequent tasks from clients is “make Vanya access to his computer from home”. We raised VPN on Mikrotik and made accounts for users. But this is a real problem. Users are unable to watch the manual and follow the steps to connect via VPN. Different versions of Windows. In one Windows, everything is connected well, in another, a different protocol is needed. And in general, this has always been associated with the reconfiguration of network equipment, which acted as a VPN server, and not all employees have access to it, and this was inconvenient.
But we already have remote connections to servers and network equipment. Why not use a ready-made transport and make a separate small utility that you can simply give the user to connect. I just wanted to make sure that the user did not enter anything abstruse there. Just one button "connect". But how will this utility understand where to connect if it has only one button. There was an idea of online assembly of the necessary application on our servers. The system administrator presses the “download shortcut” button, and a command is sent to the cloud to build an individual binary with hardwired information on connecting to the desired server / computer via RDP. In general, this could be done. But this is a long time, the administrator would have to wait first until the binary is compiled, and then downloaded. Of course, it would be possible to add just a second file with a config, but these are already 2 files, and for simplicity, the user needs one. One file, one button and no installers. After reading a little Google, I came to the conclusion that if some information is added to the end of the compiled “.exe”, then it does not deteriorate (well, almost). You can even add war and peace there, and it will work as before. It's a sin not to use it. Now you can simply unpack the application right in the client itself on the go, by the way it is called the Veliam Connector, and simply add the information you need to connect to it at the end. And the application itself knows what to do with it. Why did I write “well, almost” a little higher in brackets? Because you have to pay for this convenience in that the application loses its EDS signature. But we, at this stage, believe that this is a small price to pay for such convenience.
Third Party Module Licenses
I already wrote above that after it was decided to make the product publicly available, and not just for our own use, I had to work hard and look for replacements for some modules that did not allow us to include them in our product. But after the release, a very unpleasant thing was accidentally discovered. The Veliam Server, which was on the client side, included the MariaDB DBMS. And it's licensed under the GPL. The GPL license implies that the software must be open source, and if MariaDB is included in our product, which has this license, then our product must be under this license. But fortunately, this license has the goal of open source, and not the punishment in court of those who accidentally made a mistake. If the copyright holder has a claim, he notifies the violator in writing and he must eliminate the violation within 30 days. We discovered our mistake ourselves and did not receive letters, and immediately began to consider options for how to solve the problem. The solution turned out to be obvious - the transition to SQLite. This database has no licensing restrictions. Most modern browsers use SQLite, and a bunch of other programs as well. I found information on the Internet that SQLite is considered the most common DBMS in the world, just because of browsers, but I did not look for proofs, so this is inaccurate information. I began to study what the transition to SQLite threatens.
This becomes already a non-trivial task when there are several hundred servers installed on customers with MariaDB and data in it. Some MariaDB features are not available in SQLite. Well, for example, in the code we used queries like
Select * FROM `table` WHERE `id`>1000 FOR UPDATE
This construction not only makes a selection from the table, but also blocks the row data. And a few more designs also had to be rewritten. But besides the fact that we had to rewrite a lot of queries, we also had to come up with a mechanism that, when updating the Veliam Server at the client, will port all the data to the new DBMS and delete the old one. Also, transactions in SQLite didn't work and that was a real problem. But after reading the expanses of the World Wide Web, I found without any problems that transactions in SQLite can be enabled by passing a simple command when connecting
PRAGMA journal_mode=WAL;As a result, the task is completed and now the server part of the clients works on SQLite. We did not notice any changes in the operation of the system.
New Help Desk
It was necessary to port the HelpDesk system from the internal version to the SaaS version, but with some changes. The first thing I wanted to do was integration with the client's domain in terms of transparent authorization of users in the system. Now the user, in order to enter HelpDesk and leave a request in the system, simply clicks on the shortcut on the desktop and the browser opens. The user does not enter any credentials. The module for Apache SSPI, which is part of the Veliam Server, automatically authorizes the user under a domain account. To leave a request in the system, when the user is outside the corporate network, he clicks on the button, and he receives a link in the mail, by which he logs in to the HelpDesk system without passwords. If the user is disabled or deleted in the domain, then the account in HelpDesk will also stop working. Thus, the system administrator does not need to keep track of accounts both in the domain and in HelpDesk. The employee quit - he disabled the account in the domain and that's it, he will not log into the system not from the corporate network, not via a link. For this integration to work, the system administrator needs to create one GPO, which и .
The second thing we consider essential for HelpDesk systems, at least for ourselves, is a one-click connection to the applicant directly from the application. Moreover, connections must go through if the system administrator is on a different network. For outsourcing, this is a must; for full-time system administrators, it is also often very necessary. There are already several products that do a great job with remote connections. And we decided to make integrations for them. Now we have done integration for VNC, and in the future we plan to add Radmin and TeamViewer. Using our network transport for remote connections to the infrastructure, we made it so that VNC connects to remote workstations behind NAT. The same will happen with Radmin. Now, in order to connect to the user, you just need to click the “connect to the applicant” button in the application itself. The VNC client opens and connects to the applicant, regardless of whether you are on the same network with him or sitting at home in slippers. Previously, the system administrator, using GPO, must install VNC Server on everyone's workstations.
Now we ourselves are moving to the new HelpDesk and using integration with the domain and VNC. This is very convenient for us. Now we can not pay for TeamViewer, which we have been using for more than three years for our support team.
What we plan to do next
When we released the product, we did not make any paid plans, but simply limited the free plan to 50 monitoring objects. Five dozen network devices and servers should be enough for everyone, we thought. And then began to receive requests to increase the limit. To say we were a little shocked is an understatement. Are companies that have so many servers really interested in our software? We expanded the limit for free for those who made such requests. In response to their request, we asked some of them why they need so many, do they really have such a large number of servers and network equipment. And it turned out that the system administrators began to use the system in a way that we had not planned at all. Everything turned out to be simple - not only servers, but also workstations began to monitor with our software. Hence the many requests to expand the limits. Now we have already introduced paid tariffs and limits can be expanded independently.
Servers almost always work with either storage or local drives in a RAID array. And we initially made a product for them. And SMART monitoring was uninteresting for this task. But given the fact that people have adapted software for monitoring workstations, there have been requests for the implementation of SMART monitoring. We will implement it soon.
With the advent of Veliam Connector, it became unnecessary to deploy a VPN server on a corporate network, or do RDGW, or simply forward ports to the necessary machines to connect via RDP. A lot of people use our system only for these remote connections. Veliam Connector is available only under Windows, and some company users connect from home laptops running MacOS to workstations or terminals on the corporate network. And it turns out that the system administrator is forced, because of several users, to return to the issue of forwarding or VPN anyway. Therefore, now we are already finishing making a version of Veliam Connector for MacOS. Users of their favorite Apple technology will also be able to connect to corporate infrastructure with one click.
I really like the fact that, having a large number of system users, you don’t have to rack your brains about what people need and what will be more convenient. they write their wishes themselves, so there are a lot of plans for development in the near future.
In parallel, we are now planning to translate the system into English and distribute it abroad. We do not yet know how we will distribute the product outside our country, we are looking for options. Perhaps there will be a separate article about this later. Perhaps one of those who read this article will be able to suggest the desired vector, well, or he knows and knows how to do it and will offer his services. We will be grateful for your help.
Source: habr.com