ProHoster > Blogi > Haldamine > Buildboti näited

Buildboti näited

Mul oli vaja seadistada ehitamise ja tarnimise protsess programmi paketidelt Git-repositooriumisse. Nägin siin hiljuti Habr's artiklit buildboti kohta (link allpool) ja otsustasin seda proovida.

Kuna buildbot on hajutatud süsteem, on loogiline luua iga arhitektuuri ja opsüsteemi jaoks eraldi ehitusserver. Meie puhul on need LXC-konteinerid (Linux'i puhul) ja qemu (Windows'i puhul):

  • vm-srv-build1 — CentOS 7, siin on buildboti master ja üks töötaja (worker)
  • vm-srv-build2 — Debian 10, DEB-pakettide kogumiseks
  • vm-srv-build3 — Windows 10, eks, teate ise, mille jaoks

Kogume Rac GUI — 1C rac graafiline liides serveriklastri haldamiseks. Linux'i puhul kasutatakse igale opsüsteemile sobivaid standardvahendeid, Windows'i exe-faili kogumiseks tcl-skriptist kasutatakse freewrap.

Installeerimine

GNU/Linux

Internetis on piisavalt teavet paigaldamise ja dokumentatsiooni kohta. 1,2Ja erilisi probleeme see ei tekita:
Masteri jaoks:

pip3 install buildbot
pip3 install twisted
pip3 install autobahn
pip3 install pysqlite3
pip3 install sqlalchemy sqlalchemy-migrate
pip3 install buildbot-www buildbot-grid-view buildbot-console-view buildbot-waterfall-view
pip3 install python-dateutil

Töötajatele piisab sellest:

pip3 install buildbot-worker

Muidugi, oleks õigesti koguda pakette iga OS-i jaoks, kuid see ei kuulu selle artikli raamidesse. Samuti jätame välja konteinerite seadistamise kirjelduse, mainin ainult, et kasutan ProxMox VE. Ja veel on vaja installida igale operatsioonisüsteemile vajalikud paketid (centos: rpmdevtools jne; debian: build-essential, dh-make, pbuilder jne).

Projektide koostamine ja buildboti teenused käivitatakse mitterohtlikust kasutajast, seega tuleb see luua kõigil hostidel, mis osalevad protsessis:


adduser buildbot

Edasi seadistame teenuste automaatse käivitamise vastavalt igal hostil (konteineris):

Systemd üksus meistri käivitamiseks:

touch /etc/systemd/buildbot-master.service 

[Unit]
Description=BuildBot master service
After=network.target

[Service]
User=buildbot
Group=buildbot
WorkingDirectory=/home/buildbot/master
ExecStart=/usr/local/bin/buildbot start --nodaemon
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

ja „töötaja“

touch /etc/systemd/buildbot-worker.service 

[Unit]
Description=BuildBot worker service
After=network.target

[Service]
User=buildbot
Group=buildbot
WorkingDirectory=/home/buildbot/worker
ExecStart=/usr/local/bin/buildbot-worker start --nodaemon

[Install]
WantedBy=buildbot-master.service

Kuna kõik skriptid (meie puhul) asuvad /usr/local/, tuleb nende teed keskkonnamuutujatesse kirjutada:

nano /root/.bash_profile

PATH=$PATH:$HOME/.local/bin:$HOME/bin:/usr/local/bin

Seejärel saab luua kataloogide infrastruktuuri "töötajate" jaoks (kõigil hostidel), selleks logitakse sisse kasutajana buildbot ja täidetakse järgmised käsud:

Esimesele hostile vm-srv-build1:

su - buildbot
mkdir /home/buildbot/worker
cd ~
buildbot-worker create-worker --umask=0o22 --keepalive=60 worker vm-srv-build1:4000 CentOS 123456

Teisele hostile vm-srv-build2:

su - buildbot
mkdir /home/buildbot/worker
cd ~
buildbot-worker create-worker --umask=0o22 --keepalive=60 worker vm-srv-build1:4000 Debian-10 123456

Töötajate hostidel saab buildbot-worker teenuse käivitada

systemctl start buildbot-worker

MS Windows

Windowsi buildimiseks kasutatakse virtuaalmasinat, kus on viimane Win10 versioon.
Tööks on vajalik:

Kui kõik ülaltoodud on paigaldatud, saab paigaldada ka buildboti:

pip install buildbot-worker

Loo töökataloog

md c:worker

Ja käivita

buildbot-worker start c:worker

Kui kõik töötab (vt logi c:workertwistd.log), siis saab meie "töötaja" registreerida teenusena, lisades registrisse kirje töökaustaga (käsklused täidetakse powershellis, mis on käivitatud administraatori õiguste all):

buildbot_worker_windows_service.exe --user VM-SRV-BUILD3buildbot --password 123456 --startup auto install
New-ItemProperty -path Registry::HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesBuildBotParameters -Name directories -PropertyType String -Value c:worker

Ja on võimalik teenust käivitada

Alusta teenust buildbot

Sellega 'töötajatega' on kõik, edasi ei pea neid puudutama, kogu haldus toimub meistrist.

Meistri seadistamine

Alustamiseks loome infrastruktuuri meistrile (peamisel hostil), selleks logime sisse kasutajana buildbot ja täidame järgmised käsud:

su - buildbot
mkdir /home/buildbot/master
cd ~
buildbot create-master master

Valmis paketid loome katalooge builds

mkdir /home/buildbot/builds

Kataloogis /home/buildbot/master/ loodi fail master.cfg. See fail on python-kood ja sisaldab süsteemi töösüsteemi kõiki mehhanisme, millega me edaspidi töötame.

nano /home/buildbot/master/master.cfg

import os, re
from buildbot.plugins import steps, util, schedulers, worker, changes, reporters

c = BuildmasterConfig = {}

# Meie töötajate kirjeldus.
c['workers'] = [ worker.Worker('CentOS', '123456'), worker.Worker('Debian-10', '123456'), worker.Worker('Windows-10', '123456')]
c['protocols'] = {'pb': {'port': 4000}} 

# Näitab masterile, millist repo jälgida
c['change_source'] = []
c['change_source'].append(changes.GitPoller(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       project = 'Rac-GUI',
       branches = True,
       pollInterval = 60
      )
)

# Koostamise käivitamise teenus
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
       name="Rac-GUI-schedulers",
       change_filter=util.ChangeFilter(branch='master'),
       builderNames=["Rac-GUI-RPM-builder", "Rac-GUI-DEB-builder", "Rac-GUI-WIN-builder"],
       properties = {'owner': 'admin'}
       )
)
@util.renderer

######################################3
# RPM-paketi koostamine
 rac_gui_build_RPM = util.BuildFactory()

rac_gui_build_RPM.addStep(steps.Git(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       workdir = 'rac-gui',
       haltOnFailure = True,
       submodules = True,
       mode='full',
       progress = True)
)

Pakettide erinevate versioonide automaatseks koostamiseks, et ei peaks koodifaili master.cfg sisse minema, on peamise programmi rac_gui.tcl päisesse lisatud read praeguse versiooni ja väljaandmise kohta:

######################################################
#        Rac GUI
...
# version: 1.0.3
# release: 1

Ja nende ridade põhjal numberib buildbot paketid. Andmete väljavõtmiseks kasutatakse konsolil grep'i käsku. Buildbotis ei saa ''töötajate jaoks'' muutujaid lihtsalt määrata (igal juhul ei leidnud ma, kuidas). Selleks on omadused (property). Seega, lisame ehitusprotsessi samme versiooni ja välja määratlemiseks ning seega määrame omadused version ja release. Omadusi saab määrata erinevatel viisidel, antud juhul on see konsolikäsk:

# Добавим определение версии из основного файла
rac_gui_build_RPM.addStep(
       steps.SetPropertyFromCommand(
       command="grep version ../rac-gui/rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
       )
)
# Добавим определение релиза из основного файла
rac_gui_build_RPM.addStep(
       steps.SetPropertyFromCommand(
       command="grep release ../rac-gui/rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
       )
)

Saadud väärtusi saab asendada util.Interpolate() kutsumise teel.

# Запакуем исходники
rac_gui_build_RPM.addStep(
       steps.ShellCommand(
          command=["tar", "czf", util.Interpolate("/home/buildbot/rpmbuild/SOURCES/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"), "../rac-gui"]
       )
)

Siin tuleks märkida, et kuna hosti kasutatakse ka pakettide käsitsi koostamiseks, toimub koostamine standardsetel teedel.

# Копируем spec
rac_gui_build_RPM.addStep(steps.ShellCommand(
       command=["cp", "../rac-gui/rac_gui.spec", "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))

Korrektselt välja antud versiooni ja versiooni numbrite seadmiseks kasutatakse standardset sed-käsku, st käsk asendab väärtusi spec-failis vajalikuga.

# меняем версию
rac_gui_build_RPM.addStep(steps.ShellCommand(
       command=["sed", "-i", util.Interpolate("s/.*Version:.*/Version:t%(prop:version)s/"), "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))
# меняем релиз
rac_gui_build_RPM.addStep(steps.ShellCommand(
       command=["sed", "-i", util.Interpolate("s/.*Release:.*/Release:t%(prop:release)s/"), "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))

# запускаем процесс сборки
rac_gui_build_RPM.addStep(steps.RpmBuild(
       specfile="/home/buildbot/rpmbuild/SPECS/rac_gui.spec",
       dist='.el5',
       topdir='/home/buildbot/rpmbuild',
       builddir='/home/buildbot/rpmbuild/build',
       rpmdir='/home/buildbot/rpmbuild/RPMS',
       sourcedir='/home/buildbot/rpmbuild/SOURCES'
      )
)

Valmis ehitatud pakett ja arhiiv allikafaile kopeeritakse masterisse. Kuid võite ka otse tööfailidest kopeerida oma reposse või veebilehele.

# Скопируем файл на мастер
rac_gui_build_RPM.addStep(
       steps.FileUpload(
          workersrc=util.Interpolate("/home/buildbot/rpmbuild/RPMS/noarch/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
          masterdest=util.Interpolate("/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")
      )
)

rac_gui_build_RPM.addStep(
       steps.FileUpload(
       workersrc=util.Interpolate("/home/buildbot/rpmbuild/SOURCES/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
          masterdest=util.Interpolate("/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")
       )
)

Käivitame masteris protsessi koostatud pakettide kopeerimiseks hostimisse FTP kaudu. Selleks kasutatakse skripti tcl.

rac_gui_build_RPM.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
                util.Interpolate("--local-file=/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
                util.Interpolate("--remote-file=uploads/rac-gui/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")]
        )
)

rac_gui_build_RPM.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
                util.Interpolate("--local-file=/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
                util.Interpolate("--remote-file=uploads/rac-gui/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")]
        )
)

RPM-iga on lõpetatud. Nüüd hakkame kirjeldama DEB-paketi koostamise algoritmi. Kuna erinevate süsteemide pakettide koostamise protsessid on üksteisest sõltumatud, korduvad paljud sammud.


rac_gui_build_DEB = util.BuildFactory()

rac_gui_build_DEB.addStep(steps.Git(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       haltOnFailure = True,
       submodules = True,
       mode='full',
       workdir='build',
       progress = True)
)

# Lisame versiooni määratlemise põhifailist
rac_gui_build_DEB.addStep(
       steps.SetPropertyFromCommand(
       command="grep version rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
       )
)

# Lisame välja määratlemise põhifailist
rac_gui_build_DEB.addStep(
       steps.SetPropertyFromCommand(
       command="grep release rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
       )
)

# Muudame käivitatava faili nime
    rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["mv", "rac_gui.tcl", "racgui"]))

RPM-paketi jaoks tehakse osa allolevaid protseduure ise rpm-i poolt koostamise ajal ja need on kirjeldatud spetsifikatsioonis; debiani puhul tuleb see siin teha:

# Поменяем пути к библиотекам
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["sed", "-i", "s+^set dir(lib)+set dir(lib) /usr/share/rac-gui/lib ;#+g", "racgui"]))

# Поменяем пути к файлам
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["sed", "-i", "s+[pwd]+/usr/share/rac-gui+g", "racgui"]))

# заархивируем исходники
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["tar", "czf", util.Interpolate("../rac-gui_%(prop:version)s.orig.tar.gz"), "."]))

# Соберём пакет
rac_gui_build_DEB.addStep(steps.ShellCommand(
       command=["dpkg-buildpackage"]))

# Скопируем файл на мастер
rac_gui_build_DEB.addStep(
       steps.FileUpload(
       workersrc=util.Interpolate("../rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
           masterdest=util.Interpolate("/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")
       )
)
rac_gui_build_DEB.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
             util.Interpolate("--local-file=/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
             util.Interpolate("--remote-file=uploads/rac-gui/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")]
        )
)

DEB on nüüd lõpetatud, nüüd Windows!

rac_gui_build_WIN = util.BuildFactory()

rac_gui_build_WIN.addStep(steps.Git(
       repourl = 'https://bitbucket.org/svk28/rac-gui.git',
       haltOnFailure = True,
       submodules = True,
       mode='full',
       workdir='build',
       progress = True)
)

Kuna Windowsis ei ole tavaliselt grep'i ega sed'i (või on?), siis kasutame PowerShell'i.

# Добавим определение версии из основного файла
rac_gui_build_WIN.addStep(
       steps.SetPropertyFromCommand(
       command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'version:') -split 's')[2]",
       property="version"
       )
    )

# Добавим определение релиза из основного файла
rac_gui_build_WIN.addStep(
       steps.SetPropertyFromCommand(
       command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'release:') -split 's')[2]",
       property="release"
       )
)

# Создадим запускаемый файл
rac_gui_build_WIN.addStep(steps.ShellCommand(
       command=["c:binfreewrap.exe", "rac_gui.tcl"]))

# запакуем то, что получилось
rac_gui_build_WIN.addStep(steps.ShellCommand(
       command=["c:Program Files7-zip7z.exe", "a", "-r", util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"), "..build"]))

# скопируем на мастер
rac_gui_build_WIN.addStep(
       steps.FileUpload(
       workersrc=util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
           masterdest=util.Interpolate("/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")
       )
)
# Скопируем файл на хостинг
rac_gui_build_WIN.addStep(
        steps.MasterShellCommand(
            command=["/usr/local/bin/deploy-ftp.tcl",
            util.Interpolate("--local-file=/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
            util.Interpolate("--remote-file=uploads/rac-gui/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")]
       )
)

# Тут определяем какие сборщики у нас есть
c['builders'] = [
        util.BuilderConfig(name="Rac-GUI-RPM-builder", workername='CentOS', factory=rac_gui_build_RPM),
        util.BuilderConfig(name="Rac-GUI-DEB-builder", workername='Debian-10', factory=rac_gui_build_DEB),
        util.BuilderConfig(name="Rac-GUI-WIN-builder", workername='Windows-10', factory=rac_gui_build_WIN),
]

Koostamisprotsessi oleku teavitamiseks kasutame e-posti.


c['services'] = []

template=u'''
    <h4>Ehitus olek: {{ summary }}</h4>
    <p> Kasutatud töötaja: {{ workername }}</p>
    {% for step in build['steps'] %}
    <p> {{ step['name'] }}: {{ step['result'] }}</p>
    {% endfor %}
    <p><b> -- Buildbot</b></p>
    '''

mailNotifier = reporters.MailNotifier(fromaddr="builder@domain.ru",
             sendToInterestedUsers=False,
             mode=('all'),
             extraRecipients=["admin@domain.ru"],
             relayhost="mail.domain.ru",
             smtpPort=587,
             smtpUser="builder@domain.ru",
             smtpPassword="******",
             messageFormatter=reporters.MessageFormatter(
                                 template=template, template_type='html',
                                 wantProperties=True, wantSteps=True))

c['services'].append(mailNotifier)

# Peamised seaded
c['title'] = "Ehitusprotsess"
c['titleURL'] = "http://vm-srv-build1:80/"
c['buildbotURL'] = "http://vm-srv-build1/"
c['www'] = dict(port=80,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
c['db'] = {
       'db_url' : "sqlite:///state.sqlite"
}

Salvestame faili ja saame proovida käivitada meistriteenust:

systemctl restart buildbot-master

Logis kontrollime, et kõik oleks konfigureeritud ja toimiks normaalselt. Kõik meie töötajad peavad nüüd sisse logima, millest logis "»»/home/buildbot/master/twistd.log"»» rõõmsalt teatatakse:

 2019-07-24 16:50:35+0300 [-] Laadimine buildbot.tac...
 2019-07-24 16:50:35+0300 [-] Laaditud.
 2019-07-24 16:50:35+0300 [-] twistd 19.2.1 (/usr/bin/python3.6 3.6.8) käivitub.
 2019-07-24 16:50:35+0300 [-] reaktoriklasse: twisted.internet.epollreactor.EPollReactor.
 2019-07-24 16:50:35+0300 [-] Käivitamine BuildMaster -- buildbot.version: 2.3.1
 2019-07-24 16:50:35+0300 [-] Laadimine konfiguratsioon '/home/buildbot/master/master.cfg'
 2019-07-24 16:50:36+0300 [-] /usr/local/lib/python3.6/site-packages/buildbot/config.py:90: buildbot.config.ConfigWarning: [0.9.0 ja hiljem] `buildbotNetUsageData` ei ole konfigureeritud ja vaikimisi on see põhitasemed.
        See parameeter aitab buildbot arendajatel mõista installatsioonibaasi.
        Isikuandmeid ei koguta.
        Ahnustatakse ainult installitava tarkvara versioonide teavet ja pluginade kasutust.
        Saate `opt-out` seadistades selle muutuja None'iks.
        Või `opt-in` rohkemate üksikasjade jaoks, seadistades selle "täielikuks".

 2019-07-24 16:50:36+0300 [-] Andmebaasi seadistamine URL'ga 'sqlite:/state.sqlite'
 2019-07-24 16:50:36+0300 [-] seadistamine andmebaasi ajakirja režiimi 'wal'
 2019-07-24 16:50:36+0300 [-] lisamine 1 uut teenust, eemaldamine 0
 2019-07-24 16:50:36+0300 [-] lisamine 1 uut change_sources, eemaldamine 0
 2019-07-24 16:50:36+0300 [-] gitpoller: kasutamine töökausta '/home/buildbot/master/gitpoller-work'
 2019-07-24 16:50:36+0300 [-] lisamine 3 uut ehitajat, eemaldamine 0
 2019-07-24 16:50:36+0300 [-] lisamine 1 uut ajastajat, eemaldamine 0
 2019-07-24 16:50:36+0300 [-] www plugin 'waterfall_view' initsialiseerimine
 2019-07-24 16:50:36+0300 [-] www plugin 'console_view' initsialiseerimine
 2019-07-24 16:50:36+0300 [-] www plugin 'grid_view' initsialiseerimine
 2019-07-24 16:50:36+0300 [-] MÄRKUS: www plugin 'sitenav' on installitud, kuid mitte konfigureeritud
 2019-07-24 16:50:36+0300 [-] www plugin 'waterfall_view' initsialiseerimine
 2019-07-24 16:50:36+0300 [-] www plugin 'console_view' initsialiseerimine
 2019-07-24 16:50:36+0300 [-] www plugin 'grid_view' initsialiseerimine
 2019-07-24 16:50:36+0300 [-] MÄRKUS: www plugin 'sitenav' on installitud, kuid mitte konfigureeritud
 2019-07-24 16:50:36+0300 [-] BuildbotSite käivitumine port 80
 2019-07-24 16:50:36+0300 [-] Käivitamine tehas 
 2019-07-24 16:50:36+0300 [-] lisamine 3 uut töötajat, eemaldamine 0
 2019-07-24 16:50:36+0300 [-] PBServerFactory käivitumine port 4000
 2019-07-24 16:50:36+0300 [-] Käivitamine tehas 
 2019-07-24 16:50:37+0300 [-] BuildMaster töötab
 2019-07-24 16:50:37+0300 [-] buildbotNetUsageData: saatmine {'installid': 'b6193b126b96689351d2fe95787c5a03fc0879f9', 'versioonid': {'Python': '3.6.8', 'Buildbot': '2.3.1', 'Twisted': '19.2.1'}, 'platvorm': {'platvorm': 'Linux-4.15.18-10- pve-x86_64-with-centos-7.6.1810-Core', 'süsteem': 'Linux', 'masin': 'x86_64', 'protsessor': 'x86_64', 'python_implementation': 'CPython', 'versioon': '#1 SMP PVE 4.15.18-32', 'distro': 'centos:7'}, 'plugins': {'buildbot/worker/base/Worker': 3, 'buildbot/config/BuilderConfig': 3, 'buildbot/schedulers/basic/SingleBranchScheduler': 1, 'buildbot/reporters/mail/MailNotifier': 1, 'buildbot/changes/gitpoller/GitPoller': 1, 'buildbot/steps/worker/MakeDirectory': 1, 'buildbot/steps/source/git/Git': 3, 'buildbot/steps/shell/ShellCommand': 9, 'buildbot/steps/package/rpm/rpmbuild/RpmBuild': 1}, 'db': 'sqlite', 'mq': 'simple', 'www_plugins': ['waterfall_view', 'console_view', 'grid_view']}
 2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] töötaja 'CentOS' liitub IPv4Address(type='TCP', host='127.0.0.1', port=37332)
 2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] Saadud töötaja info 'CentOS' järgi
 2019-07-24 16:50:37+0300 [-] bot seotud
 2019-07-24 16:50:37+0300 [Broker,0,127.0.0.1] Töötaja CentOS on seotud Rac-GUI-RPM-builderiga
 2019-07-24 16:50:37+0300 [-] buildbotNetUsageData: buildbot.net ütles: ok
 2019-07-24 16:50:39+0300 [Broker,1,192.168.55.15] töötaja 'Windows-10' liitub IPv4Address(type='TCP', host='192.168.5.145', port=49831)
 2019-07-24 16:50:39+0300 [Broker,1,192.168.55.15] Saadud töötaja info 'Windows-10' järgi
 2019-07-24 16:50:40+0300 [-] bot seotud
 2019-07-24 16:50:40+0300 [Broker,1,192.168.55.15] Töötaja Windows-10 on seotud Rac-GUI-WIN-builderiga
 2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] töötaja 'Debian-10' liitub IPv4Address(type='TCP', host='192.168.5.9', port=44430)
 2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] Saadud töötaja info 'Debian-10' järgi
 2019-07-24 16:50:41+0300 [-] bot seotud
 2019-07-24 16:50:41+0300 [Broker,2,192.168.55.99] Töötaja Debian-10 on seotud Rac-GUI-DEB-builderiga

Selle seadistuse protsess on lõppenud. Praegust seisundit saab vaadata veebiliidese kaudu, kus on ka näha kogumise vigu, peatada külmunud protsess, kui miski läks valesti jne.

Pärast meie töötajate käivitamist saab vaadata menüüst „Builds” -> „Workers”.

Buildboti näited

Pärast esimest kogumisprotsessi (st muudatused Git-repos) kuvatakse esimesel lehel protsesside olek.

Buildboti näited

Kui klõpsate soovitud real, avaneb leht, kus on käesoleva protsessi olek, kus on nähtavad toimingud, vead jne.

Buildboti näited

Kogu konfiguratsioonimeistri saab siit.

import os, re
from buildbot.plugins import steps, util, schedulers, worker, changes, reporters

c = BuildmasterConfig = {}

c['workers'] = [ worker.Worker('CentOS', '123456'), worker.Worker('Debian-10', '123456'), worker.Worker('Windows-10', '123456')]
c['protocols'] = {'pb': {'port': 4000}} 

c['change_source'] = []
c['change_source'].append(changes.GitPoller(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    project = 'Rac-GUI',
    branches = True,
    pollInterval = 600
    ))

# Build launch service
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
    name="Rac-GUI-schedulers",
    change_filter=util.ChangeFilter(branch='master'),
    builderNames=["Rac-GUI-RPM-builder", "Rac-GUI-DEB-builder", "Rac-GUI-WIN-builder"],
    properties = {'owner': 'admin'}
    ))
@util.renderer
def get_name_version_release(props):
    prog_name = "rac-gui"
    prog_version = "1.0.3"
    prog_release = "3"

    return {
    "prog_name": prog_name
    #"prog_version": prog_version,
    #"prog_release": prog_release
}

rac_gui_build_RPM = util.BuildFactory()

rac_gui_build_RPM.addStep(steps.Git(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    workdir = 'rac-gui',
    haltOnFailure = True,
    submodules = True,
    mode='full',
    progress = True)
    )

# Add version definition from the main file
rac_gui_build_RPM.addStep(
    steps.SetPropertyFromCommand(
    command="grep version ../rac-gui/rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
    )
)

# Add release definition from the main file
rac_gui_build_RPM.addStep(
    steps.SetPropertyFromCommand(
    command="grep release ../rac-gui/rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
    )
)

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["tar", "czf", util.Interpolate("/home/buildbot/rpmbuild/SOURCES/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"), "../rac-gui"]))

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["cp", "../rac-gui/rac_gui.spec", "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["sed", "-i", util.Interpolate("s/.*Version:.*/Version:t%(prop:version)s/"), "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))

rac_gui_build_RPM.addStep(steps.ShellCommand(
    command=["sed", "-i", util.Interpolate("s/.*Release:.*/Release:t%(prop:release)s/"), "/home/buildbot/rpmbuild/SPECS/rac_gui.spec"]))

rac_gui_build_RPM.addStep(steps.RpmBuild(
    specfile="/home/buildbot/rpmbuild/SPECS/rac_gui.spec",
    dist='.el5',
    topdir='/home/buildbot/rpmbuild',
    builddir='/home/buildbot/rpmbuild/build',
    rpmdir='/home/buildbot/rpmbuild/RPMS',
    sourcedir='/home/buildbot/rpmbuild/SOURCES'
    ))

# Copy file to master
rac_gui_build_RPM.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("/home/buildbot/rpmbuild/RPMS/noarch/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
        masterdest=util.Interpolate("/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")
    )
)

rac_gui_build_RPM.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("/home/buildbot/rpmbuild/SOURCES/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
        masterdest=util.Interpolate("/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")
    )
)

rac_gui_build_RPM.addStep(
    steps.MasterShellCommand(
    command=["/usr/local/bin/deploy-ftp.tcl",
     util.Interpolate("--local-file=/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm"),
     util.Interpolate("--remote-file=uploads/rac-gui/rac-gui-%(prop:version)s-%(prop:release)s.noarch.rpm")]
    )
)

rac_gui_build_RPM.addStep(
    steps.MasterShellCommand(
    command=["/usr/local/bin/deploy-ftp.tcl",
     util.Interpolate("--local-file=/home/buildbot/builds/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz"),
     util.Interpolate("--remote-file=uploads/rac-gui/rac-gui-%(prop:version)s-%(prop:release)s.tar.gz")]
    )
)
####################################
##      DEB 
####################################

rac_gui_build_DEB = util.BuildFactory()

rac_gui_build_DEB.addStep(steps.Git(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    haltOnFailure = True,
    submodules = True,
    mode='full',
    workdir='build',
    progress = True)
    )

# Add version definition from the main file
rac_gui_build_DEB.addStep(
    steps.SetPropertyFromCommand(
    command="grep version rac_gui.tcl | grep -oE 'b[0-9]{1,2}.[0-9]{1,2}.[0-9]{1,2}b'", property="version"
    )
)

# Add release definition from the main file
rac_gui_build_DEB.addStep(
    steps.SetPropertyFromCommand(
    command="grep release rac_gui.tcl | grep -oE 'b[0-9]{1,3}b'", property="release"
    )
)

# Rename the executable file
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["mv", "rac_gui.tcl", "racgui"]))

# Change library paths
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["sed", "-i", "s+^set dir(lib)+set dir(lib) /usr/share/rac-gui/lib ;#+g", "racgui"]))

# Change file paths
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["sed", "-i", "s+[pwd]+/usr/share/rac-gui+g", "racgui"]))

# Archive source files
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["tar", "czf", util.Interpolate("../rac-gui_%(prop:version)s.orig.tar.gz"), "."]))

# Build the package
rac_gui_build_DEB.addStep(steps.ShellCommand(
    command=["dpkg-buildpackage"]))

# Copy file to master
rac_gui_build_DEB.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("../rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
        masterdest=util.Interpolate("/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")
    )
)

rac_gui_build_DEB.addStep(
    steps.MasterShellCommand(
    command=["/usr/local/bin/deploy-ftp.tcl",
     util.Interpolate("--local-file=/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb"),
     util.Interpolate("--remote-file=uploads/rac-gui/rac-gui_%(prop:version)s-%(prop:release)s_amd64.deb")]
    )
)
############################################
#       WIN
############################################

rac_gui_build_WIN = util.BuildFactory()

rac_gui_build_WIN.addStep(steps.Git(
    repourl = 'https://bitbucket.org/svk28/rac-gui.git',
    haltOnFailure = True,
    submodules = True,
    mode='full',
    workdir='build',
    progress = True)
    )

# Add version definition from the main file
rac_gui_build_WIN.addStep(
    steps.SetPropertyFromCommand(
    command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'version:') -split 's')[2]",
    property="version"
    )
)

# Add release definition from the main file
rac_gui_build_WIN.addStep(
    steps.SetPropertyFromCommand(
    command="powershell -command "((Get-Content .rac_gui.tcl | Select-String -Pattern 'release:') -split 's')[2]",
    property="release"
    )
)

# Create executable file
rac_gui_build_WIN.addStep(steps.ShellCommand(
    command=["c:binfreewrap.exe", "rac_gui.tcl"]))

# Archive the result
rac_gui_build_WIN.addStep(steps.ShellCommand(
    command=["c:Program Files7-zip7z.exe", "a", "-r", util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"), "..build"]))

# Copy to master
rac_gui_build_WIN.addStep(
    steps.FileUpload(
    workersrc=util.Interpolate("..rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
        masterdest=util.Interpolate("/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")
    )
)

# Copy file to hosting
rac_gui_build_WIN.addStep(
    steps.MasterShellCommand(
    command=["/usr/local/bin/deploy-ftp.tcl",
     util.Interpolate("--local-file=/home/buildbot/builds/rac-gui_%(prop:version)s-%(prop:release)s.win.zip"),
     util.Interpolate("--remote-file=uploads/rac-gui/rac-gui_%(prop:version)s-%(prop:release)s.win.zip")]
    )
)

c['builders'] = [
        util.BuilderConfig(name="Rac-GUI-RPM-builder", workername='CentOS', factory=rac_gui_build_RPM),
        util.BuilderConfig(name="Rac-GUI-DEB-builder", workername='Debian-10', factory=rac_gui_build_DEB),
        util.BuilderConfig(name="Rac-GUI-WIN-builder", workername='Windows-10', factory=rac_gui_build_WIN),
]

c['services'] = []

template=u'''
<h4>Ehitus olek: {{ summary }}</h4>
<p> Kasutatud töötaja: {{ workername }}</p>
{% for step in build['steps'] %}
<p> {{ step['name'] }}: {{ step['result'] }}</p>
{% endfor %}
<p><b> -- Buildbot</b></p>
'''

mailNotifier = reporters.MailNotifier(fromaddr="112@icvibor.ru",
          sendToInterestedUsers=False,
          mode=('all'),
          extraRecipients=["my@domain.local"],
          relayhost="mail.domain.local",
          smtpPort=587,
          smtpUser="buildbot@domain.local",
          smtpPassword="**********",
          messageFormatter=reporters.MessageFormatter(
                                 template=template, template_type='html',
                                 wantProperties=True, wantSteps=True))

c['services'].append(mailNotifier)

c['title'] = "Ehitamise protsess"
c['titleURL'] = "http://vm-srv-build1:80/"

c['buildbotURL'] = "http://vm-srv-build1/"

c['www'] = dict(port=80,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
c['db'] = {
    'db_url' : "sqlite:///state.sqlite"
}

Materjalid

Artikli ettevalmistamisel kasutati järgmisi materjale:

Allikas: habr.com

Osta usaldusväärne veebihosting DDoS kaitsega, VPS VDS serverid 🔥 Osta usaldusväärne veebihosting DDoS kaitsega, VPS VDS serverid | ProHoster