In , I talked about the background of Veliam's creation and the decision to distribute it through the SaaS system. In this article, I will discuss what had to be done for the product to become public rather than local. I will cover how we started distribution and the challenges we faced.
Planning
The current server part for users was on Linux. Nearly every organization has Windows servers, which cannot be said about Linux. The main strength of Veliam is remote connections to servers and network equipment behind NAT. However, this functionality was tightly bound to the requirement that the router must be a Mikrotik. This clearly would not satisfy many users. I initially began thinking about adding support for routers from the most common vendors. But I realized that this would lead to an endless race to expand the list of supported companies. Moreover, even those that are already supported can vary from model to model in terms of command sets for changing NAT rules. The only solution to this situation seemed to be VPN.
Since we decided to distribute the product, but not as open source, we can no longer include various libraries with open licenses like GPL. This is a separate topic altogether; after deciding to sell the product, we had to sift through half of the libraries due to them being GPL. When we were developing for our own needs, it was fine. But it's not suitable for distribution. The first VPN that comes to mind is OpenVPN. But that's GPL. There was also the option of using the Japanese SoftEther VPN. Its license allowed it to be included in our product. After a couple of days of testing how to integrate it in a way that users wouldnât need to configure anything or know about SoftEther VPN, we ended up with a prototype. Everything was as it should be. But for some reason, we were still uneasy about this scheme, and ultimately, we abandoned it. Obviously, we only decided to do so after coming up with another option. In the end, we implemented everything over standard TCP connections. Some connections work through a coordinator, while others go directly via the Nat Hole Punching (NHP) technology, which was also implemented in Free Pascal. I must say, I had never even heard of NHP before. It hadn't occurred to me that two network devices behind NAT could be linked directly. I studied the topic, understood the principles, and started coding. The plan was realized; the user connects with one click to the required NAT device via RDP, SSH, or Winbox without entering passwords or configuring a VPN. Moreover, most of these connections bypass our coordinator, which positively affects the ping and the cost of servicing these connections.
Transitioning the server side from Linux to Windows
There were several issues during the switch to Windows. The first was that the built-in wmic in Windows does not allow WQL queries. Our system was completely built on them. There was also something else, but Iâve forgotten why we ultimately abandoned its use. Possibly, it was due to differences between Windows versions. The second issue was multithreading. Unable to find a good third-party utility under a 'permissible' license for us, I started the Lazarus IDE again. I wrote the necessary utility. It takes a list of objects and the specific queries needed as input, and in return, I get the data. All of this runs in a multithreaded mode. Excellent.
After setting up pthreads for PHP on Windows, I thought everything would just run smoothly, but that wasnât the case. After some debugging time, I realized that pthreads seemed to work, but specifically in our system, it didn't. It became clear that there was some peculiarity in how pthreads worked on Windows. And indeed, it was true. I read the documentation, and it stated that the number of threads is limited on Windows, and as far as I remember, it's implicit. This became a problem. Because when I started reducing the number of threads at which the application operated, it ran very slowly. I opened the IDE again, and the utility was enhanced with multithreaded pinging of objects. Additionally, port scanning was also added. After that, the need for pthreads for PHP vanished, and it was no longer used. Subsequently, this utility was further enhanced with several functionalities, and it continues to operate to this day. After that, an installer was created for Windows, which included Apache, PHP, MariaDB, the PHP application itself, and a set of utilities for system interaction written in Free Pascal. Regarding the installer, I thought this would be a quick fix, as itâs an extremely common requirement for nearly every piece of software. Perhaps I wasnât searching correctly or something else. However, I constantly encountered products that were either not flexible enough or were expensive and also inflexible. Nonetheless, I found a free installer that could accommodate any customization I wanted. Itâs InnoSetup. I'm mentioning this here because I had to search for it, and maybe I can save someone else some time.
Abandoning the plugin in favor of my client
I previously mentioned that the client side was a browser with a "plugin." There were times when Chrome would update, causing the layout to break a bit, or when Windows would update, leading to the custom URI scheme getting lost. I really wanted to avoid such surprises in the public version of the product. Moreover, the custom URI started failing after each Windows update. Microsoft simply removed all non-Microsoft branches in the necessary section. Also, Google Chrome now doesn't allow remembering the choice to open or not open the application from the custom URI, and asks this question every time you click on a monitoring object. Overall, there needed to be proper interaction with the user's local system, which the browser does not provide. The simplest solution in this scenario seemed to be creating our own browser, as many are currently doing using Electron. However, many components had already been written in Free Pascal, including the server side, so we decided to make the client in the same language rather than creating a chaotic mix. Thus, a client with Chromium on board was developed. After that, it started to gain various bindings.
Release
We finally settled on a name for the system. We constantly considered various options while converting from the local version to SaaS. Since we initially planned to enter not only the domestic market, the main criterion for selecting a name was the availability of an unused, or not very expensive, domain in the â.comâ zone. Some features/modules had not yet been ported from the local version to Veliam, but we decided to launch with the current functionality and work on the rest as updates. The very first version did not have HelpDesk, Veliam Connector, the ability to change the notification trigger thresholds, and much more. We purchased a Code Sign Certificate and signed both the client and server parts. We created a website for the product and began the processes of registering software, trademarks, etc. In general, we are ready to start. There is a slight euphoria from the work done and from the possibility that someone might use your product, although we had no doubts about that. And then, hold on. A partner said that entering the market without notifications in messengers is not possible. You can do without many other things, but not this. After some discussions, integration with Telegram was added, which suited us. Among all current messengers, it is the only one that provides free access to its API without any complicated approval procedures. WhatsApp, on the other hand, suggests contacting providers who charge quite a bit for their services, and all requests for direct access have been ignored. As for Viber... I donât know who uses it now, as spam and advertising are overwhelming there. At the end of December, after several rounds of internal testing and testing among friends, we opened registration for everyone and released the software for download.
Beginning of Distribution
From the very beginning, we understood that we needed a small flow of users of the system so that they could test the product in live mode and provide some initial feedback. A few purchased posts on VK produced results. The first registrations started coming in.
It must be said that entering the market when your company doesnât have a well-known name, while offering agentless monitoring functionality that requires users to input their server and workstation credentials, is quite challenging. This intimidates a lot of people. From the very beginning, we understood that this would be a problem and we were technically and morally prepared for it. All remote connections, despite RDP and SSH being encrypted by default, are additionally encrypted by our software using AES standards. All data from local servers is transmitted to the cloud via HTTPS. Credentials are stored in an encrypted form. The encryption keys for all subsystems are unique for each client. Session encryption keys are used for remote connections.
All we can do in this situation to reassure people is to be as transparent as possible, work on security, and continually address their concerns.
For many, the convenience and functionality of the software outweigh their fears, and they register. Some individuals in posted comments on VK stated that this software is unusable as it collects their passwords and is from an unknown company. Itâs worth mentioning that this view was not held by just one person. Many simply do not realize that when they install another proprietary software on their servers, which operates as a service, it also has full system rights and doesnât need credentials to do anything illegal (itâs clear that the user under which the service runs can be changed, but the same applies here; any account can be entered). In reality, peopleâs fears are understandable. Installing software on a server is a common task, but providing credentials is a bit scary and intimate, as a good half of people use the same password for all services, and creating a separate account even for testing is tedious. However, at the moment, there are a huge number of services that people trust with their credentials and more. We strive to become one of them.
There were many comments suggesting that we stole this from somewhere. This surprised us a bit. Well, it's one person's opinion, but such comments appeared in various publications from different people. At first, we didn't know how to react to this. Should we be sad that some people think that in Russia no one can do anything themselves and can only steal, or should we be glad that they believe such things can only be stolen?
We have now completed the process of obtaining an EV Code Sign Certificate. To obtain it, a series of checks must be passed, and a bunch of company documents must be submitted, some of which need to be notarized. Getting the EV Code Sign certificate during the pandemic is a whole separate topic for an article. The process took a month, and it was not a month of waiting, but a month full of requests for additional documents. Maybe the pandemic isn't to blame, and everyone else experienced a similarly lengthy process? Please share.
Some say they won't use it because there is no FSTEC certificate. We have to explain that we cannot obtain it, and we won't, because to get that certificate, encryption must be according to GOST standards, and we plan to distribute software not only in Russia and use AES.
All these comments sowed some uncertainty about whether it is possible to promote a product that requires user accounts while not being in the public eye. Even considering that we knew there would be those who had a very negative attitude toward this. After the number of registrations exceeded a thousand, we stopped worrying about it. Especially after, in addition to the negativity from those who hadn't even tried the product, very pleasant reviews started to emerge. It should be noted that these positive reviews are the biggest motivator for product development.
Adding remote access functionality for employees
One common request from clients is, "Enable Ivan to access his computer from home." We set up VPN on MikroTik and created accounts for users. However, this is truly a challenge. Users struggle to follow the instructions and connect via VPN step by step. There are different versions of Windows. In one version, everything connects smoothly, whereas another version requires a different protocol. Overall, this has always been tied to reconfiguring the network equipment serving as the VPN server, and not all employees have access to it, making it inconvenient.
But we already have remote connections to servers and network equipment. Why not utilize existing transport and create a small utility that we can simply provide to users for connections? I wanted to ensure that users do not have to input anything complex, just a single "connect" button. But how will this utility know where to connect if it only has one button? There was an idea for an online assembly of the necessary application on our servers. The system administrator presses the "download shortcut" button, and a command is sent to our cloud for building a custom binary with the embedded connection information for the required server/computer via RDP. Overall, this could be done. However, it takes time, and the administrator would have to wait for the binary to compile and then download it. Of course, a second config file could be added, but that makes two files, and for simplicity, the user needs just one. One file, one button, and no installers. After exploring a bit online, I concluded that if you add some information to the end of the compiled " .exe", it doesn't break (well, almost). You could add the entire War and Peace, and it would still work as before. It would be a shame not to take advantage of this. Now we can simply unpack the application on the fly right within the client; by the way, it's called Veliam Connector, and just append the necessary connection information to it. The application knows what to do with it. Why did I write "well, almost" in parentheses above? Because this convenience comes at the cost of the application losing its digital signature. But at this stage, we consider this a small price for such convenience.
Third-party module licenses
I mentioned earlier that after deciding to make the product publicly available rather than just for internal use, we had to put in quite a bit of effort and look for replacements for certain modules that prevented us from including them in our product. However, after the release, we unintentionally discovered a rather unpleasant issue. The Veliam Server on the client side included the MariaDB database. And it is under the GPL license. The GPL license stipulates that software must be open source, and if our product includes MariaDB, which has this license, then our product must also be under this license. Fortunately, the purpose of this license is to promote open source, not to punish those who accidentally make mistakes. If the rights holder has a claim, they must notify the violator in writing, and the violator is given 30 days to correct the violation. We discovered our mistake on our own, received no letters, and immediately began to consider options for resolving the issue. The solution turned out to be obvious â switching to SQLite. This database has no licensing restrictions. Most modern browsers use SQLite, as do many other applications. I found information online that SQLite is considered the most widely used database in the world, primarily because of browsers, but I didn't look for evidence, so that information may not be accurate. I started exploring what the implications of switching to SQLite would be.
This becomes quite a non-trivial task when there are several hundred servers with MariaDB installed at client sites and data within them. Some features of MariaDB are not available in SQLite. For example, in the code, we used queries like
Select * FROM `table` WHERE `id`>1000 FOR UPDATE
This construct not only selects from the table but also locks the data rows. A few other constructs also had to be rewritten. However, in addition to having to rewrite many queries, we also needed to devise a mechanism that would port all data to the new database and remove the old one during an update of Veliam Server at the client's site. Additionally, transactions did not work in SQLite, which was a real problem. But after reading through the vastness of the World Wide Web, I easily found that transactions in SQLite can be enabled by passing a simple command during the connection.
PRAGMA journal_mode=WAL;As a result, the task is completed, and now the server-side for clients runs on SQLite. We haven't noticed any changes in the system's performance.
New HelpDesk
The HelpDesk system needed to be ported from the internal version to the SaaS version, but with some modifications. The first thing we wanted to do was integrate with the client's domain for seamless user authentication in the system. Now, a user can simply click on a shortcut on their desktop to open the browser and access HelpDesk without entering any credentials. The Apache SSPI module included with Veliam Server automatically authenticates the user using their domain account. If a user is outside the corporate network and wants to submit a request, they can click a button, and a link will be sent to their email, allowing them to log into HelpDesk without passwords. If a user's account is disabled or removed from the domain, their account in HelpDesk will also cease to function. This way, the system administrator doesn't need to monitor accounts in both the domain and HelpDesk separately. If an employee leaves, they just disable the account in the domain, and that's it; they wonât be able to access the system, neither from the corporate network nor through the link. For this integration to work, the system administrator needs to set up a GPO that and .
Secondly, we consider it extremely necessary for HelpDesk systemsâat least for ourselvesâto have a one-click connection to the requester directly from the ticket. Moreover, connections should be possible even if the system administrator is on a different network. This is essential for outsourcing and often very necessary for in-house system administrators as well. There are already several products that handle remote connections wonderfully. We decided to create integrations for them. Currently, we have integrated with VNC, and we plan to add Radmin and TeamViewer in the future. Using our network transport for remote connections to the infrastructure, we have made it so that VNC connects to remote workstations behind NAT. The same will apply to Radmin. Now, to connect to a user, you just need to click the "connect to requester" button in the ticket. The VNC client opens and connects to the requester, regardless of whether you are on the same network or sitting at home in slippers. Previously, the system administrator must install the VNC Server on all workstations using Group Policy.
We are currently transitioning to a new HelpDesk and using the integration with the domain and VNC. This is very convenient for us. Now we no longer need to pay for TeamViewer, which we used for more than three years to support our help desk.
What we plan to do next
When we launched the product, we did not introduce any paid plans; we simply limited the free plan to 50 monitoring objects. We thought fifty network devices and servers should be enough for everyone. Then requests to increase the limit started coming in. To say we were somewhat shocked would be an understatement. Could it be that companies with such a number of servers were interested in our software? We expanded the limit for those who requested it for free. In some cases, we asked why they needed so many; did they really have such a large number of servers and network equipment? It turned out that system administrators began using the system in ways we never anticipated. It was simple â they started monitoring not just servers but also workstations with our software. Hence, the many requests for limit increases. We have now introduced paid plans, and users can extend their limits on their own.
Servers almost always work either with SANs or with local disks in RAID arrays. We originally designed the product for them. Monitoring SMART wasnât a priority for this task. However, given that people adapted the software to monitor workstations, requests for SMART monitoring implementation started coming in. We will be rolling it out soon.
With the arrival of Veliam Connector, there is no longer a need to deploy a VPN server in the corporate network, set up RDGW, or simply forward ports to the necessary machines for RDP access. Many users utilize our system solely for these remote connections. Veliam Connector is currently available only for Windows, while some company users connect from their MacOS laptops to workstations or terminals in the corporate network. This means system administrators have to revisit port forwarding or VPN setup due to a few users. Therefore, we are finalizing the Veliam Connector version for MacOS. Users of their beloved Apple devices will also get the opportunity to connect to the corporate infrastructure with a single click.
I really like that, with a large number of system users, there's no need to rack my brain about what people want and what would be more convenient. They write their own requests, so there are plenty of development plans in the near future.
At the same time, we are currently planning to translate the system into English and distribute it abroad. We are still unsure how we will distribute the product outside our country and are exploring options. Perhaps later there will be a separate article about this. Maybe someone who read this article can suggest the right direction, or knows how to do it and will offer their services. We would appreciate any help.
Source: habr.com
