Since buildbot is a distributed system, it would be logical to make a separate build host for each architecture and operating system. In our case, these will be LXC containers (in the case of linux) and qemu (in the case of windows):
vm-srv-build1 - centos 7, there will be a buildbot master (master) and one of the workers (worker)
vm-srv-build2 - debian 10, for building DEB packages
vm-srv-build3 - windows 10, for assembly, you yourself understand what
We will collect Rac GUI - graphic face to 1C rac for managing a cluster of servers. Under Linux, standard tools for each OS will be used; to build an exe file under windows from a tcl script, use freewrap.
Installation
GNU / Linux
For installation, there is enough documentation on the network 1,2. Yes, and it does not cause any special problems:
For master:
Of course, it would be more correct to build packages for each OS, but this is not within the scope of the article. We will also omit the description of setting up containers for work, I will only note that I use ProxMox VE. And you will also need to install packages for each axis required for building (centos: rpmdevtools, etc; debian: build-essential, dh-make, pbuilder, etc.)
Project builds and buildbot services will run as an unprivileged user, so you need to create it on all hosts involved in the process:
adduser buildbot
Next, we will configure the automatic start of services, respectively, on each of the hosts (containers):
After that, you can create a directory infrastructure for "workers" (on all hosts), for this you register under the buildbot user and execute the following commands:
On the first host vm-srv-build1:
su - buildbot
mkdir /home/buildbot/worker
cd ~
buildbot-worker create-worker --umask=0o22 --keepalive=60 worker vm-srv-build1:4000 CentOS 123456
On the second host vm-srv-build2:
su - buildbot
mkdir /home/buildbot/worker
cd ~
buildbot-worker create-worker --umask=0o22 --keepalive=60 worker vm-srv-build1:4000 Debian-10 123456
On worker hosts, the buildbot-worker service can be started
systemctl start buildbot-worker
MS Windows
As a βworkerβ for building under windows, a virtual machine with the latest release of Win10 will be used.
For work you need:
After all of the above is installed, you can install the buildbot itself:
pip install buildbot-worker
Create a working directory
md c:worker
And let's run
buildbot-worker start c:worker
If everything works (see the c: workertwistd.log log), then you can register our βworkerβ as a service by adding an item with the working directory to the registry (commands are executed in powershell running as administrator):
With the "workers" that's all, you can not touch them further, all control comes from the master.
Wizard setup
To begin with, let's create the infrastructure for the master (on the main host), for this we register under the buildbot user and execute the following commands:
su - buildbot
mkdir /home/buildbot/master
cd ~
buildbot create-master master
For ready-made packages, create a builds directory
mkdir /home/buildbot/builds
A master.cfg file has been created in the /home/buildbot/master/ directory. This file is a python code and contains a description of all the mechanisms of the system, and we will work with it in the future.
To automate the assembly of packages of different versions, so as not to have to climb into the code of the master.cfg file, in the main script of the rac_gui.tcl program in the header, lines with the current version and release have been added:
And based on these lines, buildbot will number the packages. For pulling out the data, a call to the console grep is used. In buildbot, you just canβt define variables for βworkersβ (in any case, I didnβt find how). This is what properties are for. Those. to the build process, add the steps to determine the version and release and, accordingly, set the version and release properties. Properties can be set in various ways, in this case it is a call to the console command:
To set the correct release and version numbers, a standard sed call is used, i.e. the command replaces the values ββinside the spec file with the necessary ones
We copy the finished assembled package and the archive with the source codes to the master. But you can immediately copy files from the working one to your repository or to the site.
This is the end of RPM. Now let's start describing the algorithm for building a DEB package. Since the processes of building packages for different systems are independent of each other, many steps will be repeated.
For an RPM package, some of the following procedures are done by rpm itself when building and are described inside the spec, for debian you have to do it here:
We save the file and you can try to start the wizard service:
systemctl restart buildbot-master
In the log, we will check that everything is in order with the config and everything is working normally. All our workers should now connect, which will be happily reported in the »»'/home/buildbot/master/twistd.log»»' log:
2019-07-24 16:50:35+0300 [-] Loading buildbot.tac...
2019-07-24 16:50:35+0300 [-] Loaded.
2019-07-24 16:50:35+0300 [-] twistd 19.2.1 (/usr/bin/python3.6 3.6.8) starting up.
2019-07-24 16:50:35+0300 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2019-07-24 16:50:35+0300 [-] Starting BuildMaster -- buildbot.version: 2.3.1
2019-07-24 16:50:35+0300 [-] Loading configuration from '/home/buildbot/master/master.cfg'
2019-07-24 16:50:36+0300 [-] /usr/local/lib/python3.6/site-packages/buildbot/config.py:90: buildbot.config.ConfigWarning: [0.9.0 and later] `buildbotNetUsageData` is not configured and defaults to basic.
This parameter helps the buildbot development team to understand the installation base.
No personal information is collected.
Only installation software version info and plugin usage is sent.
You can `opt-out` by setting this variable to None.
Or `opt-in` for more information by setting it to "full".
2019-07-24 16:50:36+0300 [-] Setting up database with URL 'sqlite:/state.sqlite'
2019-07-24 16:50:36+0300 [-] setting database journal mode to 'wal'
2019-07-24 16:50:36+0300 [-] adding 1 new services, removing 0
2019-07-24 16:50:36+0300 [-] adding 1 new change_sources, removing 0
2019-07-24 16:50:36+0300 [-] gitpoller: using workdir '/home/buildbot/master/gitpoller-work'
2019-07-24 16:50:36+0300 [-] adding 3 new builders, removing 0
2019-07-24 16:50:36+0300 [-] adding 1 new schedulers, removing 0
2019-07-24 16:50:36+0300 [-] initializing www plugin 'waterfall_view'
2019-07-24 16:50:36+0300 [-] initializing www plugin 'console_view'
2019-07-24 16:50:36+0300 [-] initializing www plugin 'grid_view'
2019-07-24 16:50:36+0300 [-] NOTE: www plugin 'sitenav' is installed but not configured
2019-07-24 16:50:36+0300 [-] initializing www plugin 'waterfall_view'
2019-07-24 16:50:36+0300 [-] initializing www plugin 'console_view'
2019-07-24 16:50:36+0300 [-] initializing www plugin 'grid_view'
2019-07-24 16:50:36+0300 [-] NOTE: www plugin 'sitenav' is installed but not configured
2019-07-24 16:50:36+0300 [-] BuildbotSite starting on 80
2019-07-24 16:50:36+0300 [-] Starting factory <buildbot.www.service.BuildbotSite object at 0x7fe31c2657b8>
2019-07-24 16:50:36+0300 [-] adding 3 new workers, removing 0
2019-07-24 16:50:36+0300 [-] PBServerFactory starting on 4000
2019-07-24 16:50:36+0300 [-] Starting factory <twisted.spread.pb.PBServerFactory object at 0x7fe31c147470>
2019-07-24 16:50:37+0300 [-] BuildMaster is running
2019-07-24 16:50:37+0300 [-] buildbotNetUsageData: sending {'installid': 'b6193b126b96689351d2fe95787c5a03fc0879f9', 'versions': {'Python': '3.6.8', 'Buildbot': '2.3.1', 'Twisted': '19.2.1'}, 'platform': {'platform': 'Linux-4.15.18-10- pve-x86_64-with-centos-7.6.1810-Core', 'system': 'Linux', 'machine': 'x86_64', 'processor': 'x86_64', 'python_implementation': 'CPython', 'version': '#1 SMP PVE 4.15.18-32', 'distro': 'centos:7'}, 'plugins': {'buildbot/worker/base/Worker': 3, 'buildbot/config/BuilderConfig': 3, 'buildbot/schedulers/basic/SingleBranchScheduler': 1, 'buildbot/reporters/mail/MailNotifier': 1, 'buildbot/changes/gitpoller/GitPoller': 1, 'buildbot/steps/worker/MakeDirectory': 1, 'buildbot/steps/source/git/Git': 3, 'buildbot/steps/shell/ShellCommand': 9, 'buildbot/steps/package/rpm/rpmbuild/RpmBuild': 1}, 'db': 'sqlite', 'mq': 'simple', 'www_plugins': ['waterfall_view', 'console_view', 'grid_view']}
2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] worker 'CentOS' attaching from IPv4Address(type='TCP', host='127.0.0.1', port=37332)
2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] Got workerinfo from 'CentOS'
2019-07-24 16:50:37+0300 [-] bot attached
2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] Worker CentOS attached to Rac-GUI-RPM-builder
2019-07-24 16:50:37+0300 [-] buildbotNetUsageData: buildbot.net said: ok
2019-07-24 16:50:39+0300 [Broker,1,192.168.55.15] worker 'Windows-10' attaching from IPv4Address(type='TCP', host='192.168.5.145', port=49831)
2019-07-24 16:50:39+0300 [Broker,1,192.168.55.15] Got workerinfo from 'Windows-10'
2019-07-24 16:50:40+0300 [-] bot attached
2019-07-24 16:50:40+0300 [Broker,1,192.168.55.15] Worker Windows-10 attached to Rac-GUI-WIN-builder
2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] worker 'Debian-10' attaching from IPv4Address(type='TCP', host='192.168.5.9', port=44430)
2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] Got workerinfo from 'Debian-10'
2019-07-24 16:50:41+0300 [-] bot attached
2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] Worker Debian-10 attached to Rac-GUI-DEB-builder
This completes the setup process. You can view the current state through the web-morde. Where you can also see build errors, kick a frozen process if something went wrong, etc.
Immediately after the launch of our hard workers, you can see through the menu "Builds" -> "Workers"
After the first build process is done (i.e. changes in the Git repository), the status of the processes will appear on the first page.
If you click on the desired line with the mouse, a page will open with the current state of this process, where you can see what is happening, what errors, etc.