Buildbot v příkladech

Potreboval sem nastaviti postopek zbiranja in dostave paketa programske opreme iz Git repozitorija na spletno stran. In ko sem nedavno videl članek o buildbotu na Habré (povezava na koncu), sem se odločil, da ga preizkusim in uporabim.

Ker je buildbot distribuiran sistem, bo logično, da bomo za vsako arhitekturo in operacijski sistem ustvarili ločen zbiralni gost. V našem primeru bodo to LXC kontejnerji (v primeru linuxa) in qemu (v primeru windows):

• vm-srv-build1 — centos 7, tukaj bo buildbot master in eden od delavnikov
• vm-srv-build2 — debian 10, za zbiranje DEB paketov
• vm-srv-build3 — windows 10, za zbiranje, sami veste, za kaj

Zbirali bomo Rac GUI — grafični vmesnik za 1C rac za upravljanje s grozdom strežnikov. Pod linuxom bomo uporabili sistemska sredstva za vsak OS, za zbiranje exe datoteke pod windows iz tcl skripte pa se uporablja freewrap.

Namestitev

GNU/Linux

Glede namestitve je na spletu dovolj dokumentacije. 1,2In ne povzroča posebnih težav:
Za master:

pip3 install buildbot
pip3 install twisted
pip3 install autobahn
pip3 install pysqlite3
pip3 install sqlalchemy sqlalchemy-migrate
pip3 install buildbot-www buildbot-grid-view buildbot-console-view buildbot-waterfall-view
pip3 install python-dateutil

Za 'delavce' bo dovolj tega:

pip3 install buildbot-worker

Seveda, bolj pravilno bi bilo zbirati pakete za vsak OS, vendar to ne sodi v okvir članka. Prav tako bomo preskočili opis nastavitev kontejnerjev za delovanje, omenjeno pa le, da uporabljam ProxMox VE. In še namestiti bo treba pakete za vsak operacijski sistem, potrebne za zbiranje (centos: rpmdevtools itd; debian: build-essential, dh-make, pbuilder itd.)

Zbiranje projektov in storitve buildbot se bodo izvajale v imenu neprivilegiranega uporabnika, zato ga je treba ustvariti na vseh gostiteljih, ki sodelujejo v procesu:

adduser buildbot

Nato bomo nastavili samodejni zagon storitev, ustrezno na vsakem od gostiteljev (kontejnerjev):

Systemd enota za zagon masterja:

touch /etc/systemd/buildbot-master.service 

[Unit]
Description=BuildBot master service
After=network.target

[Service]
User=buildbot
Group=buildbot
WorkingDirectory=/home/buildbot/master
ExecStart=/usr/local/bin/buildbot start --nodaemon
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

in 'delavca'

touch /etc/systemd/buildbot-worker.service 

[Unit]
Description=BuildBot worker service
After=network.target

[Service]
User=buildbot
Group=buildbot
WorkingDirectory=/home/buildbot/worker
ExecStart=/usr/local/bin/buildbot-worker start --nodaemon

[Install]
WantedBy=buildbot-master.service

Ker so vsi skripti (v našem primeru) shranjeni v /usr/local/, je treba pot do njih zapisati v okoljske spremenljivke:

nano /root/.bash_profile

PATH=$PATH:$HOME/.local/bin:$HOME/bin:/usr/local/bin

Nato, lahko ustvarite infrastrukturo imenikov za »delavce« (na vseh gostiteljih), registrirate se pod uporabnikom buildbot in izvedete naslednje ukaze:

Na prvem gostitelju vm-srv-build1:

su - buildbot
mkdir /home/buildbot/worker
cd ~
buildbot-worker create-worker --umask=0o22 --keepalive=60 worker vm-srv-build1:4000 CentOS 123456

Na drugem gostitelju vm-srv-build2:

su - buildbot
mkdir /home/buildbot/worker
cd ~
buildbot-worker create-worker --umask=0o22 --keepalive=60 worker vm-srv-build1:4000 Debian-10 123456

Na delovnih gostiteljih lahko zaženete storitev buildbot-worker

systemctl start buildbot-worker

MS Windows

Kot »delavec« za gradnjo pod Windows bo uporabljena virtualna naprava z najnovejšo različico Win10.
Za delovanje boste potrebovali:

• Python
• PyWIN
• Git
• Za namestitev različnega v Pythonu je bilo potrebno prenesti Build Tools for Visual Studio 2019
• 7Zip

Ko je vse zgoraj navedeno nameščeno, lahko namestite tudi buildbot:

pip install buildbot-worker

Ustvarimo delovni imenik

md c:worker

In zaženimo

buildbot-worker start c:worker

Če vse deluje (glejte log c:workertwistd.log), lahko našega »delavca« registrirate kot storitev, tako da dodate točko v register z delovnim imenikom (ukazi se izvajajo v powershellu, zagnanem kot skrbnik):

buildbot_worker_windows_service.exe --user VM-SRV-BUILD3buildbot --password 123456 --startup auto install
New-ItemProperty -path Registry::HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesBuildBotParameters -Name directories -PropertyType String -Value c:worker

In lahko zaženete storitev

Start-Service buildbot

S »delavci« je to vse, naprej jih ni treba motiti, vsa upravljanja potekajo iz glavnega strežnika.

Nastavitev glavnega strežnika

Najprej ustvarimo infrastrukturo za glavni strežnik (na glavnem gostitelju), za to se registriramo pod uporabnikom buildbot in izvedemo naslednje ukaze:

su - buildbot
mkdir /home/buildbot/master
cd ~
buildbot create-master master

Za pripravljene pakete ustvarimo imenik builds

mkdir /home/buildbot/builds

V imeniku /home/buildbot/master/ je bila ustvarjena datoteka master.cfg. Ta datoteka predstavlja kodo v pythonu in vsebuje opis vseh mehanizmov delovanja sistema, s katerim bomo sodelovali v nadaljevanju.

nano /home/buildbot/master/master.cfg

import os, re
from buildbot.plugins import steps, util, schedulers, worker, changes, reporters

c= BuildmasterConfig ={}

# Opis našeg radnika.
c['workers'] = [ worker.Worker('CentOS', '123456'), worker.Worker('Debian-10', '123456'), worker.Worker('Windows-10', '123456')]
c['protocols'] = {'pb': {'port': 4000}} 

# Pomoćniku se govori koji repozitorij pratiti
c['change_source'] = []
c['change_source'].append(changes.GitPoller(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       project = 'Rac-GUI',
       branches = True,
       pollInterval = 60
      )
)

# Usluga za pokretanje gradnje
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
       name="Rac-GUI-schedulers",
       change_filter=util.ChangeFilter(branch='master'),
       builderNames=["Rac-GUI-RPM-builder", "Rac-GUI-DEB-builder", "Rac-GUI-WIN-builder"],
       properties = {'owner': 'admin'}
       )
)
@util.renderer

######################################3
# Gradnja RPM-paketa
 rac_gui_build_RPM = util.BuildFactory()

rac_gui_build_RPM.addStep(steps.Git(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       workdir = 'rac-gui',
       haltOnFailure = True,
       submodules = True,
       mode='full',
       progress = True)
)

Za automatizaciju gradnje paketa različitih verzija, kako ne bi trebalo ulaziti u kod datoteke master.cfg, u glavnom skriptnom programu rac_gui.tcl dodane su linije sa trenutnom verzijom i izdanjem:

######################################################
#        Rac GUI
...
# version: 1.0.3
# release: 1

I na osnovu tih linija buildbot će numerisati pakete. Za izvlačenje podataka koristi se poziv konzolnog grep-a. U buildbot-u ne može se jednostavno odrediti varijable za "radnike" (barem, nisam pronašao kako). Za to služe svojstva (property). To jest, u proces gradnje dodajemo korake za određivanje verzije i izdanja i prema tome postavljamo svojstva version i release. Svojstva se mogu postavljati na različite načine, u ovom slučaju to je poziv konzolne komande:

# Добавим определение версии из основного файла
rac_gui_build_RPM.addStep(
       steps.SetPropertyFromCommand(
       command="grep version ../rac-gui/rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
       )
)
# Добавим определение релиза из основного файла
rac_gui_build_RPM.addStep(
       steps.SetPropertyFromCommand(
       command="grep release ../rac-gui/rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
       )
)

Dobijene vrednosti možemo umetnuti putem poziva util.Interpolate().

# Запакуем исходники
rac_gui_build_RPM.addStep(
       steps.ShellCommand(
          command=["tar", "czf", util.Interpolate("/home/buildbot/rpmbuild/SOURCES/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"), "../rac-gui"]
       )
)

Treba napomenuti da, pošto se host koristi i za ručnu gradnju paketa, gradnja će se odvijati prema standardnim putevima.

# Копируем spec
rac_gui_build_RPM.addStep(steps.ShellCommand(
       command=["cp", "../rac-gui/rac_gui.spec", "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))

Za postavljanje ispravnih brojeva izdanja i verzija koristi se poziv standardnog sed-a, tj. komanda menja vrednosti unutar spec-datoteke na potrebne

# меняем версию
rac_gui_build_RPM.addStep(steps.ShellCommand(
       command=["sed", "-i", util.Interpolate("s/.*Version:.*/Version:t%(prop:version)s/"), "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))
# меняем релиз
rac_gui_build_RPM.addStep(steps.ShellCommand(
       command=["sed", "-i", util.Interpolate("s/.*Release:.*/Release:t%(prop:release)s/"), "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))

# запускаем процесс сборки
rac_gui_build_RPM.addStep(steps.RpmBuild(
       specfile="/home/buildbot/rpmbuild/SPECS/rac_gui.spec",
       dist='.el5',
       topdir='/home/buildbot/rpmbuild',
       builddir='/home/buildbot/rpmbuild/build',
       rpmdir='/home/buildbot/rpmbuild/RPMS',
       sourcedir='/home/buildbot/rpmbuild/SOURCES'
      )
)

Gotov sastavljeni paket i arhiva sa izvorima se kopiraju na master. Ali se može odmah sa radnog računara kopirati u svoj repozitorij ili na sajt.

# Скопируем файл на мастер
rac_gui_build_RPM.addStep(
       steps.FileUpload(
          workersrc=util.Interpolate("/home/buildbot/rpmbuild/RPMS/noarch/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
          masterdest=util.Interpolate("/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")
      )
)

rac_gui_build_RPM.addStep(
       steps.FileUpload(
       workersrc=util.Interpolate("/home/buildbot/rpmbuild/SOURCES/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
          masterdest=util.Interpolate("/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")
       )
)

Pokrenimo na masteru proces kopiranja sastavljenih paketa na hosting putem FTP-a. Za to se koristi skript na tcl.

rac_gui_build_RPM.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
                util.Interpolate("--local-file=/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
                util.Interpolate("--remote-file=uploads/rac-gui/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")]
        )
)

rac_gui_build_RPM.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
                util.Interpolate("--local-file=/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
                util.Interpolate("--remote-file=uploads/rac-gui/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")]
        )
)

Завершив работу с RPM, мы переходим к описанию алгоритма сборки DEB-пакета. Процессы сборки пакетов для различных систем независимы друг от друга, поэтому многие шаги будут повторяться.

rac_gui_build_DEB = util.BuildFactory()

rac_gui_build_DEB.addStep(steps.Git(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       haltOnFailure = True,
       submodules = True,
       mode='full',
       workdir='build',
       progress = True)
)

# Добавим определение версии из основного файла
rac_gui_build_DEB.addStep(
       steps.SetPropertyFromCommand(
       command="grep version rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
       )
)

# Добавим определение релиза из основного файла
rac_gui_build_DEB.addStep(
       steps.SetPropertyFromCommand(
       command="grep release rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
       )
)

# Переименуем запускаемый файл
    rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["mv", "rac_gui.tcl", "racgui"]))

Для RPM-пакетов часть нижеследующих процедур выполняется самим rpm при сборке и описана внутри спецификации, в то время как для Debian это необходимо делать здесь.

# Поменяем пути к библиотекам
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["sed", "-i", "s+^set dir(lib)+set dir(lib) /usr/share/rac-gui/lib ;#+g", "racgui"]))

# Поменяем пути к файлам
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["sed", "-i", "s+[pwd]+/usr/share/rac-gui+g", "racgui"]))

# заархивируем исходники
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["tar", "czf", util.Interpolate("../rac-gui_%(prop:version)s.orig.tar.gz"), "."]))

# Соберём пакет
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["dpkg-buildpackage"]))

# Скопируем файл на мастер
rac_gui_build_DEB.addStep(
       steps.FileUpload(
       workersrc=util.Interpolate("../rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
           masterdest=util.Interpolate("/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")
       )
)
rac_gui_build_DEB.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
             util.Interpolate("--local-file=/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
             util.Interpolate("--remote-file=uploads/rac-gui/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")]
        )
)

С DEB покончено, теперь переходим к Windows!

rac_gui_build_WIN = util.BuildFactory()

rac_gui_build_WIN.addStep(steps.Git(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       haltOnFailure = True,
       submodules = True,
       mode='full',
       workdir='build',
       progress = True)
)

Поскольку в Windows по умолчанию нет grep и sed (или они есть?), мы будем использовать PowerShell.

# Добавим определение версии из основного файла
rac_gui_build_WIN.addStep(
       steps.SetPropertyFromCommand(
       command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'version:') -split 's')[2]",
       property="version"
       )
    )

# Добавим определение релиза из основного файла
rac_gui_build_WIN.addStep(
       steps.SetPropertyFromCommand(
       command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'release:') -split 's')[2]",
       property="release"
       )
)

# Создадим запускаемый файл
rac_gui_build_WIN.addStep(steps.ShellCommand(
       command=["c:binfreewrap.exe", "rac_gui.tcl"]))

# запакуем то, что получилось
rac_gui_build_WIN.addStep(steps.ShellCommand(
       command=["c:Program Files7-zip7z.exe", "a", "-r", util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"), "..build"]))

# скопируем на мастер
rac_gui_build_WIN.addStep(
       steps.FileUpload(
       workersrc=util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
           masterdest=util.Interpolate("/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")
       )
)
# Скопируем файл на хостинг
rac_gui_build_WIN.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
            util.Interpolate("--local-file=/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
            util.Interpolate("--remote-file=uploads/rac-gui/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")]
       )
)

# Тут определяем какие сборщики у нас есть
c['builders'] = [
        util.BuilderConfig(name="Rac-GUI-RPM-builder", workername='CentOS', factory=rac_gui_build_RPM),
        util.BuilderConfig(name="Rac-GUI-DEB-builder", workername='Debian-10', factory=rac_gui_build_DEB),
        util.BuilderConfig(name="Rac-GUI-WIN-builder", workername='Windows-10', factory=rac_gui_build_WIN),
]

Для уведомления о состоянии процесса сборки мы будем использовать электронную почту.

c['services'] = []

template=u'''
    <h4>Status di costruzione: {{ summary }}</h4>
    <p> Lavoratore utilizzato: {{ workername }}</p>
    {% for step in build['steps'] %}
    <p> {{ step['name'] }}: {{ step['result'] }}</p>
    {% endfor %}
    <p><b> -- Il Buildbot</b></p>
    '''

mailNotifier = reporters.MailNotifier(fromaddr="builder@domain.ru",
             sendToInterestedUsers=False,
             mode=('all'),
             extraRecipients=["admin@domain.ru"],
             relayhost="mail.domain.ru",
             smtpPort=587,
             smtpUser="builder@domain.ru",
             smtpPassword="******",
             messageFormatter=reporters.MessageFormatter(
                                 template=template, template_type='html',
                                 wantProperties=True, wantSteps=True))

c['services'].append(mailNotifier)

# Impostazioni principali del master
c['title'] = "Il processo di costruzione"
c['titleURL'] = "http://vm-srv-build1:80/"
c['buildbotURL'] = "http://vm-srv-build1/"
c['www'] = dict(port=80,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
c['db'] = {
       'db_url' : "sqlite:////state.sqlite"
}

Сохраняя файл, можно попробовать запустить службу мастера:

systemctl restart buildbot-master

В логе мы убедимся, что с конфигурацией все в порядке и все работает в нормальном режиме. Все наши работники должны теперь подключиться, как будет радостно сообщено в логе »»’/home/buildbot/master/twistd.log»»’:

 2019-07-24 16:50:35+0300 [-] Nalagam buildbot.tac...
 2019-07-24 16:50:35+0300 [-] Naloženo.
 2019-07-24 16:50:35+0300 [-] twistd 19.2.1 (\/usr\/bin\/python3.6 3.6.8) se začenja.
 2019-07-24 16:50:35+0300 [-] razred reaktorja: twisted.internet.epollreactor.EPollReactor.
 2019-07-24 16:50:35+0300 [-] Zagon BuildMaster -- buildbot.version: 2.3.1
 2019-07-24 16:50:35+0300 [-] Nalagam konfiguracijo iz '\/home\/buildbot\/master\/master.cfg'
 2019-07-24 16:50:36+0300 [-] \/usr\/local\/lib\/python3.6\/site-packages\/buildbot\/config.py:90: buildbot.config.ConfigWarning: [0.9.0 in novejše] `buildbotNetUsageData` ni konfigurirano in privzeto je osnovno.
        Ta parameter pomaga ekipi za razvoj buildbota razumeti bazo namestitev.
        Osebni podatki se ne zbirajo.
        Samo informacije o različici namestitvenega programa in uporabi vtičnikov se pošljejo.
        Lahko se `opt-out` nastavite tako, da to spremenite v None.
        Ali `opt-in` za več informacij tako, da to nastavite na "full".

 2019-07-24 16:50:36+0300 [-] Nastavljanje baze podatkov z URL-jem 'sqlite:\/state.sqlite'
 2019-07-24 16:50:36+0300 [-] nastavljanje načina dnevnika baze podatkov na 'wal'
 2019-07-24 16:50:36+0300 [-] dodajanje 1 nove storitve, odstranjevanje 0
 2019-07-24 16:50:36+0300 [-] dodajanje 1 novega vir sprememb, odstranjevanje 0
 2019-07-24 16:50:36+0300 [-] gitpoller: uporablja delovno mapo '\/home\/buildbot\/master\/gitpoller-work'
 2019-07-24 16:50:36+0300 [-] dodajanje 3 novih graditeljev, odstranjevanje 0
 2019-07-24 16:50:36+0300 [-] dodajanje 1 novega razporejevalca, odstranjevanje 0
 2019-07-24 16:50:36+0300 [-] inicializacija www vtičnika 'waterfall_view'
 2019-07-24 16:50:36+0300 [-] inicializacija www vtičnika 'console_view'
 2019-07-24 16:50:36+0300 [-] inicializacija www vtičnika 'grid_view'
 2019-07-24 16:50:36+0300 [-] OPOZORILO: www vtičnik 'sitenav' je nameščen, vendar ni konfiguriran
 2019-07-24 16:50:36+0300 [-] inicializacija www vtičnika 'waterfall_view'
 2019-07-24 16:50:36+0300 [-] inicializacija www vtičnika 'console_view'
 2019-07-24 16:50:36+0300 [-] inicializacija www vtičnika 'grid_view'
 2019-07-24 16:50:36+0300 [-] OPOZORILO: www vtičnik 'sitenav' je nameščen, vendar ni konfiguriran
 2019-07-24 16:50:36+0300 [-] BuildbotSite se začenja na 80
 2019-07-24 16:50:36+0300 [-] Zagon tovarne <buildbot.www.service.BuildbotSite object at 0x7fe31c2657b8>
 2019-07-24 16:50:36+0300 [-] dodajanje 3 novih delavcev, odstranjevanje 0
 2019-07-24 16:50:36+0300 [-] PBServerFactory se začenja na 4000
 2019-07-24 16:50:36+0300 [-] Zagon tovarne <twisted.spread.pb.PBServerFactory object at 0x7fe31c147470>
 2019-07-24 16:50:37+0300 [-] BuildMaster je v teku
 2019-07-24 16:50:37+0300 [-] buildbotNetUsageData: pošiljanje {'installid': 'b6193b126b96689351d2fe95787c5a03fc0879f9', 'versions': {'Python': '3.6.8', 'Buildbot': '2.3.1', 'Twisted': '19.2.1'}, 'platform': {'platform': 'Linux-4.15.18-10- pve-x86_64-with-centos-7.6.1810-Core', 'system': 'Linux', 'machine': 'x86_64', 'processor': 'x86_64', 'python_implementation': 'CPython', 'version': '#1 SMP PVE 4.15.18-32', 'distro': 'centos:7'}, 'plugins': {'buildbot\/worker\/base\/Worker': 3, 'buildbot\/config\/BuilderConfig': 3, 'buildbot\/schedulers\/basic\/SingleBranchScheduler': 1, 'buildbot\/reporters\/mail\/MailNotifier': 1, 'buildbot\/changes\/gitpoller\/GitPoller': 1, 'buildbot\/steps\/worker\/MakeDirectory': 1, 'buildbot\/steps\/source\/git\/Git': 3, 'buildbot\/steps\/shell\/ShellCommand': 9, 'buildbot\/steps\/package\/rpm\/rpmbuild\/RpmBuild': 1}, 'db': 'sqlite', 'mq': 'simple', 'www_plugins': ['waterfall_view', 'console_view', 'grid_view']}
 2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] delavec 'CentOS' se priključuje iz IPv4Address(type='TCP', host='127.0.0.1', port=37332)
 2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] Prejel delavske informacije od 'CentOS'
 2019-07-24 16:50:37+0300 [-] bot je priključen
 2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] Delavec CentOS priključen na Rac-GUI-RPM-builder
 2019-07-24 16:50:37+0300 [-] buildbotNetUsageData: buildbot.net je dejal: ok
 2019-07-24 16:50:39+0300 [Broker,1,192.168.55.15] delavec 'Windows-10' se priključuje iz IPv4Address(type='TCP', host='192.168.5.145', port=49831)
 2019-07-24 16:50:39+0300 [Broker,1,192.168.55.15] Prejel delavske informacije od 'Windows-10'
 2019-07-24 16:50:40+0300 [-] bot je priključen
 2019-07-24 16:50:40+0300 [Broker,1,192.168.55.15] Delavec Windows-10 priključen na Rac-GUI-WIN-builder
 2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] delavec 'Debian-10' se priključuje iz IPv4Address(type='TCP', host='192.168.5.9', port=44430)
 2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] Prejel delavske informacije od 'Debian-10'
 2019-07-24 16:50:41+0300 [-] bot je priključen
 2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] Delavec Debian-10 priključen na Rac-GUI-DEB-builder

Na tej točki je postopek nastavitve zaključen. Trenutno stanje si lahko ogledate prek spletnega vmesnika, kjer lahko vidite tudi napake pri gradnji, prekinete visoko proces, če je kaj šlo narobe itd.

Takoj po zagonu naših delavcev lahko stanje preverite prek menija „Builds“ -> „Workers“

Ko bo prvi postopek gradnje (tj. spremembe v Git repozitoriju) izveden, se bo na prvi strani prikazalo stanje procesov.

Če kliknete na ustrezno vrstico, se bo odprla stran s trenutnim stanjem tega postopka, kjer lahko vidite, kaj se dogaja, kakšne so napake itd.

Celoten konfigurator mojstra lahko dobite tukaj

import os, re
from buildbot.plugins import steps, util, schedulers, worker, changes, reporters

c= BuildmasterConfig ={}

c['workers'] = [ worker.Worker('CentOS', '123456'), worker.Worker('Debian-10', '123456'), worker.Worker('Windows-10', '123456')]
c['protocols'] = {'pb': {'port': 4000}} 

c['change_source'] = []
c['change_source'].append(changes.GitPoller(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    project = 'Rac-GUI',
    branches = True,
    pollInterval = 600
    ))

# служба запуска сборки
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
    name="Rac-GUI-schedulers",
    change_filter=util.ChangeFilter(branch='master'),
    builderNames=["Rac-GUI-RPM-builder", "Rac-GUI-DEB-builder", "Rac-GUI-WIN-builder"],
    properties = {'owner': 'admin'}
    ))
@util.renderer
def get_name_version_release(props):
    prog_name = "rac-gui"
    prog_version = "1.0.3"
    prog_release = "3"

    return {
    "prog_name": prog_name
    #"prog_version": prog_version,
    #"prog_release": prog_release
}

rac_gui_build_RPM = util.BuildFactory()

rac_gui_build_RPM.addStep(steps.Git(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    workdir = 'rac-gui',
    haltOnFailure = True,
    submodules = True,
    mode='full',
    progress = True)
    )

# Добавим определение версии из основного файла
rac_gui_build_RPM.addStep(
    steps.SetPropertyFromCommand(
    command="grep version ..\/rac-gui\/rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
    )
)

# Добавим определение релиза из основного файла
rac_gui_build_RPM.addStep(
    steps.SetPropertyFromCommand(
    command="grep release ..\/rac-gui\/rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
    )
)

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["tar", "czf", util.Interpolate("\/home\/buildbot\/rpmbuild\/SOURCES\/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"), "..\/rac-gui"]))

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["cp", "..\/rac-gui\/rac_gui.spec", "\/home\/buildbot\/rpmbuild\/SPECS\/rac_gui.spec"]))

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["sed", "-i", util.Interpolate("s\/.*Version:.*\/Version:t%(prop:version)s\/"), "\/home\/buildbot\/rpmbuild\/SPECS\/rac_gui.spec"]))

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["sed", "-i", util.Interpolate("s\/.*Release:.*\/Release:t%(prop:release)s\/"), "\/home\/buildbot\/rpmbuild\/SPECS\/rac_gui.spec"]))

rac_gui_build_RPM.addStep(steps.RpmBuild(
    specfile="\/home\/buildbot\/rpmbuild\/SPECS\/rac_gui.spec",
    dist='.el5',
    topdir='\/home\/buildbot\/rpmbuild',
    builddir='\/home\/buildbot\/rpmbuild\/build',
    rpmdir='\/home\/buildbot\/rpmbuild\/RPMS',
    sourcedir='\/home\/buildbot\/rpmbuild\/SOURCES'
    ))

# Скопируем файл на мастер
rac_gui_build_RPM.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("\/home\/buildbot\/rpmbuild\/RPMS\/noarch\/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
        masterdest=util.Interpolate("\/home\/buildbot\/builds\/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")
    )
)

rac_gui_build_RPM.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("\/home\/buildbot\/rpmbuild\/SOURCES\/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
        masterdest=util.Interpolate("\/home\/buildbot\/builds\/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")
    )
)

rac_gui_build_RPM.addStep(
    steps.MasterShellCommand(
    command=["\/usr\/local\/bin\/deploy-ftp.tcl",
     util.Interpolate("--local-file=\/home\/buildbot\/builds\/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
     util.Interpolate("--remote-file=uploads\/rac-gui\/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")]
    )
)

rac_gui_build_RPM.addStep(
    steps.MasterShellCommand(
    command=["\/usr\/local\/bin\/deploy-ftp.tcl",
     util.Interpolate("--local-file=\/home\/buildbot\/builds\/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
     util.Interpolate("--remote-file=uploads\/rac-gui\/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")]
    )
)
####################################
##      DEB 
####################################

rac_gui_build_DEB = util.BuildFactory()

rac_gui_build_DEB.addStep(steps.Git(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    haltOnFailure = True,
    submodules = True,
    mode='full',
    workdir='build',
    progress = True)
    )

# Добавим определение версии из основного файла
rac_gui_build_DEB.addStep(
    steps.SetPropertyFromCommand(
    command="grep version rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
    )
)

# Добавим определение релиза из основного файла
rac_gui_build_DEB.addStep(
    steps.SetPropertyFromCommand(
    command="grep release rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
    )
)

# Переименуем запускаемый файл
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["mv", "rac_gui.tcl", "racgui"]))

# Поменяем пути к библиотекам
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["sed", "-i", "s+^set dir(lib)+set dir(lib) \/usr\/share\/rac-gui\/lib ;#+g", "racgui"]))

# Поменяем пути к файлам
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["sed", "-i", "s+[pwd]+\/usr\/share\/rac-gui+g", "racgui"]))

# заархивируем исходники
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["tar", "czf", util.Interpolate("..\/rac-gui_%(prop:version)s.orig.tar.gz"), "."]))

# Соберём пакет
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["dpkg-buildpackage"]))

# Скопируем файл на мастер
rac_gui_build_DEB.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("..\/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
        masterdest=util.Interpolate("\/home\/buildbot\/builds\/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")
    )
)

rac_gui_build_DEB.addStep(
    steps.MasterShellCommand(
    command=["\/usr\/local\/bin\/deploy-ftp.tcl",
     util.Interpolate("--local-file=\/home\/buildbot\/builds\/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
     util.Interpolate("--remote-file=uploads\/rac-gui\/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")]
    )
)
############################################
#       WIN
############################################

rac_gui_build_WIN = util.BuildFactory()

rac_gui_build_WIN.addStep(steps.Git(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    haltOnFailure = True,
    submodules = True,
    mode='full',
    workdir='build',
    progress = True)
    )

# Добавим определение версии из основного файла
rac_gui_build_WIN.addStep(
    steps.SetPropertyFromCommand(
    command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'version:') -split 's')[2]",
    property="version"
    )
)

# Добавим определение релиза из основного файла
rac_gui_build_WIN.addStep(
    steps.SetPropertyFromCommand(
    command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'release:') -split 's')[2]",
    property="release"
    )
)

# Создадим запускаемый файл
rac_gui_build_WIN.addStep(steps.ShellCommand(
    command=["c:binfreewrap.exe", "rac_gui.tcl"]))

# запакуем то, что получилось
rac_gui_build_WIN.addStep(steps.ShellCommand(
    command=["c:Program Files7-zip7z.exe", "a", "-r", util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"), "..build"]))

# скопируем на мастер
rac_gui_build_WIN.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
        masterdest=util.Interpolate("\/home\/buildbot\/builds\/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")
    )
)

# Скопируем файл на хостинг
rac_gui_build_WIN.addStep(
    steps.MasterShellCommand(
    command=["\/usr\/local\/bin\/deploy-ftp.tcl",
     util.Interpolate("--local-file=\/home\/buildbot\/builds\/rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
     util.Interpolate("--remote-file=uploads\/rac-gui\/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")]
    )
)

c['builders'] = [
        util.BuilderConfig(name="Rac-GUI-RPM-builder", workername='CentOS', factory=rac_gui_build_RPM),
        util.BuilderConfig(name="Rac-GUI-DEB-builder", workername='Debian-10', factory=rac_gui_build_DEB),
        util.BuilderConfig(name="Rac-GUI-WIN-builder", workername='Windows-10', factory=rac_gui_build_WIN),
]

c['services'] = []

template=u'''
<h4>Status di costruzione: {{ summary }}</h4>
<p> Lavoratore utilizzato: {{ workername }}</p>
{% for step in build['steps'] %}
<p> {{ step['name'] }}: {{ step['result'] }}</p>
{% endfor %}
<p><b> -- Il Buildbot</b></p>
'''

mailNotifier = reporters.MailNotifier(fromaddr="112@icvibor.ru",
          sendToInterestedUsers=False,
          mode=('all'),
          extraRecipients=["my@domain.local"],
          relayhost="mail.domain.local",
          smtpPort=587,
          smtpUser="buildbot@domain.local",
          smtpPassword="**********",
          messageFormatter=reporters.MessageFormatter(
                                 template=template, template_type='html',
                                 wantProperties=True, wantSteps=True))

c['services'].append(mailNotifier)

c['title'] = "Proces izgradnje"
c['titleURL'] = "http://vm-srv-build1:80/"

c['buildbotURL'] = "http://vm-srv-build1/"

c['www'] = dict(port=80,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
c['db'] = {
    'db_url' : "sqlite://state.sqlite"
}

Материалы

Pri pripravi članka so bili uporabljeni naslednji materiali:

• Primer izvedbe Continuous Integration s pomočjo BuildBot
• Buildbot: zgodba z primeri še o enem sistemu neprekinjene integracije
• Nastavitev neprekinjene integracije v Buildbotu na Ubuntu 16.04
• In seveda dokumentacija o izdelku Buildbot Tutorial

Izvor: habr.com