ProHoster > Π‘Π»ΠΎΠ³ > Kev tswj hwm > Txheej txheem cej luam ntawm Okerr hybrid saib xyuas qhov system

Txheej txheem cej luam ntawm Okerr hybrid saib xyuas qhov system

Ob xyoos dhau los kuv twb tau tshaj tawm Yooj yim poob rau lub vev xaib txog ua oker. Tam sim no muaj qee qhov kev txhim kho ntawm qhov project, thiab kuv kuj tau luam tawm oker server side source code hauv qab qhib daim ntawv tso cai, yog vim li cas kuv thiaj txiav txim siab sau qhov kev tshuaj xyuas luv luv no ntawm Habr.

Txheej txheem cej luam ntawm Okerr hybrid saib xyuas qhov system
[ tag nrho loj ]

Rau leej twg nws yuav txaus siab

Qhov no yuav txaus siab rau koj yog tias koj ua haujlwm hauv pab pawg me lossis ib leeg. Koj tsis muaj kev saib xyuas thiab koj tsis paub tseeb tias koj xav tau tiag tiag. Txawm hais tias koj tau sim qee qhov kev saib xyuas hnyav "rau cov tub loj", tab sis nws "tsis tau tshem tawm" rau koj, lossis nws ua haujlwm hauv qhov kev teeb tsa yuav luag thiab tsis hloov koj lub neej ntau. Thiab tseem - yog tias koj yeej tsis npaj los faib tag nrho cov neeg ua haujlwm (lossis txawm tias ib lub tuam tsev) los saib xyuas lub dashboard yam tsawg kawg ob peb teev hauv ib hnub lossis teeb tsa nws.

Vim li cas oker txawv txawv

Tom ntej no kuv yuav qhia cov yam ntxwv nthuav dav ntawm okerra uas paub qhov txawv ntawm qee qhov kev saib xyuas.

Okerr yog hybrid saib xyuas

Thaum lub sijhawm saib xyuas sab hauv, "tus neeg sawv cev" tab tom khiav ntawm cov tshuab soj ntsuam, uas xa cov ntaub ntawv mus rau cov neeg saib xyuas (piv txwv li, qhov chaw dawb disk). Thaum sab nraud, tus neeg rau zaub mov ua cov tshev hauv lub network (piv txwv li, ping lossis lub vev xaib muaj). Txhua txoj hauv kev muaj nws cov kev txwv. Okerr siv ob qho kev xaiv. Kev kuaj xyuas hauv cov servers yog ua los ntawm tus neeg saib xyuas lub teeb (30Kb) lossis koj tus kheej cov ntawv sau thiab cov ntawv thov, thiab kev kuaj xyuas hauv network tau ua los ntawm okerr sensors hauv ntau lub tebchaws.

okerr tsis yog software xwb, tab sis kuj yog ib qho kev pabcuam

Cov neeg rau zaub mov ib feem ntawm kev saib xyuas yog qhov loj thiab nyuaj, nws nyuaj rau nruab thiab teeb tsa, thiab nws yuav tsum muaj peev txheej. Nrog okerr koj tuaj yeem nruab koj tus kheej saib xyuas server (nws yog dawb thiab opensource), lossis koj tuaj yeem siv tsuas yog cov neeg siv khoom thiab siv cov kev pabcuam ntawm peb lub server. Kuj dawb.

Yog tias kev saib xyuas tso cai rau koj los them rov qab thiab npog qhov tsis muaj kev ntseeg siab ntawm cov servers thiab cov ntawv thov, ces cov lus nug xav tau tshwm sim - leej twg yog tus saib xyuas? Kev soj ntsuam yuav qhia peb li cas txog qhov teeb meem yog tias nws tus kheej "tuag" rau qee qhov laj thawj, cais lossis ua ke nrog koj lwm cov peev txheej (piv txwv li, channel rau cov ntaub ntawv chaw poob)? Thaum siv cov kev pabcuam sab nraud okerr - qhov teeb meem no tau daws - koj yuav tau txais kev ceeb toom txawm tias tag nrho cov ntaub ntawv chaw nrog koj cov servers tsis muaj zog lossis raug tawm tsam los ntawm zombies.

Tau kawg, muaj kev pheej hmoo tias okerr server nws tus kheej yuav tsis muaj, qhov no yog qhov tseeb (raws li koj paub, 90% ntawm kev ntseeg tau ib txwm tau txais yooj yim thiab "dawb", 99% nrog rau qhov tsawg kawg nkaus ntawm kev siv zog, thiab txhua qhov cuaj tom ntej yog exponentially nyuaj dua). Tab sis, ua ntej, qhov muaj feem ntawm qhov tshwm sim no qis dua, thiab qhov thib ob, qhov teeb meem yuav mus tsis tau pom tsuas yog tias nws coincides nrog teeb meem ntawm peb cov servers. Yog tias peb muaj 99.9% kev ntseeg siab, thiab koj muaj 99.9% (tsis yog tus lej siab dhau), ces qhov kev pheej hmoo ntawm kev tsis pom zoo yog 0.1% ntawm 0.1% = 0.0001%. Ntxiv peb cuaj rau koj qhov kev cia siab yuav luag tsis muaj kev siv zog thiab tsis muaj nqi yog qhov zoo heev!

Lwm qhov zoo ntawm kev saib xyuas raws li qhov kev pabcuam yog tias tus kws kho mob lossis lub vev xaib studio tuaj yeem nruab okerr server thiab muab kev nkag mus rau cov neeg siv khoom raws li kev them nyiaj lossis kev pabcuam dawb ntxiv. Koj cov neeg sib tw tsuas yog muaj hosting thiab cov vev xaib, tab sis koj muaj kev ntseeg siab hosting nrog kev saib xyuas.

Okerr yog hais txog qhov ntsuas

Qhov taw qhia yog "lub teeb". Nws muaj ob lub xeev tseem ceeb - ntsuab (OK) lossis liab (ERR). Qhov project muaj ntau pawg (piv txwv li, los ntawm server) ntsuas. Ntawm nplooj ntawv tseem ceeb ntawm qhov project, koj tam sim ntawd pom tias txhua yam yog ntsuab (thiab koj tuaj yeem kaw nws), lossis qee yam teeb liab thiab yuav tsum tau kho. Thaum hloov pauv ntawm cov xeev no, kev ceeb toom raug xa mus. Ib hnub thaum koj tab tom teeb tsa, cov ntsiab lus ntawm qhov project raug xa mus.

Txheej txheem cej luam ntawm Okerr hybrid saib xyuas qhov system

Txhua qhov taw qhia okerr muaj cov xwm txheej uas nws hloov pauv hauv xeev (hauv Zabbix qhov no yog hu ua tus txhais). Piv txwv li, load nruab nrab yuav tsum tsis pub ntau tshaj 2 (ntawm chav kawm, qhov no yog configurable). Thiab rau txhua qhov kev kuaj sab hauv (load nruab nrab, disk dawb, ...) muaj ib tus saib xyuas. Yog tias vim qee yam peb tsis tau txais kev lees paub tiav ntawm lub sijhawm teem tseg, qhov yuam kev raug kaw thiab ceeb toom raug xa mus.

Peb cov qauv kev ua haujlwm ib txwm yog kuaj xyuas email thaum sawv ntxov, thiab saib cov ntsiab lus ntawm lwm cov ntawv (peb teem caij thaum pib ua haujlwm). Yog tias txhua yam nyob hauv nws, peb ua lwm yam tseem ceeb (tab sis kom muaj kev nyab xeeb, peb tuaj yeem saib sai ntawm okerra dashboard thiab xyuas kom meej tias txhua yam ntsuab nyob rau lub sijhawm no). Yog tias muaj kev ceeb toom tuaj txog, peb teb.

Tau kawg, nws yog ib qho ua tau kom yooj yim khaws "cov ntaub ntawv" cov cim (kom pom daim duab ntawm lub network los ntawm kev saib xyuas), tab sis txhua yam ua tiav kom yooj yim, yooj yim thiab sai tsim cov ntsuas tshwj xeeb rau kev saib xyuas thiab xa cov lus ceeb toom.

Lub hom phiaj uas koj tab tom teeb tsa okerr yog nyob rau hauv kev ceeb toom, kom koj tuaj yeem tsim qhov taw qhia hauv ib feeb, nws tuaj yeem "pw tsaug zog" rau ib xyoos, tsuas yog lees txais qhov hloov tshiab, thiab thaum ib xyoos tom qab qee yam tawg, nws teeb thiab xa. kev ceeb toom. Lub feeb koj ib zaug siv los tsim qhov ntsuas tau them; koj tau kawm txog qhov teeb meem tam sim ntawd, ua ntej lwm tus. Nws yog qhov ua tau tias lawv kho nws ua ntej leej twg pom. Ib yam dab tsi uas tau tsa sai sai tsis suav tias yog poob!

Kev ruaj ntseg

Nws yuav yog ib qho kev txaj muag yog tias koj teeb tsa kev saib xyuas kom muaj kev ntseeg siab ntxiv, tab sis vim li ntawd, koj raug tawm tsam hauv lub network los ntawm nws, thiab muaj ntau lub network tsis zoo hauv cov cuab yeej saib xyuas sib txawv (Zabbix, Nagios).

Tus neeg saib xyuas (okerrmod los ntawm pob okerupdate) khiav ntawm qhov system tsis yog network server, tab sis tus neeg siv khoom. Yog li ntawd, tsis muaj cov chaw nres nkoj qhib ntxiv ntawm cov neeg saib xyuas, tus neeg siv tau yooj yim ua haujlwm tom qab lub foob pob hluav taws lossis NAT thiab nyuaj heev (Kuv xav hais tias "ua tsis tau") txhawm rau txhawm rau hla lub network, txij li hauv paus ntsiab lus nws tsis mloog lub network. lub qhov (socket).

Kev saib xyuas tag nrho

Tam sim no peb txoj cai yog tias peb kawm txog txhua yam teeb meem kev lag luam los ntawm okerr. Yog tias tam sim no txoj cai raug ua txhaum cai (okerr tsis tau ceeb toom txog nws qhov tshwm sim yuav tshwm sim (yog tias ua tau) lossis tias nws tau tshwm sim lawm) - peb ntxiv cov tshev rau okerr.

Kev kuaj sab nraud

Zoo ib yam set:

  • ping
  • http status
  • tshawb xyuas qhov siv tau thiab qhov tshiab ntawm SSL daim ntawv pov thawj (yuav ceeb toom yog tias nws yuav tas sijhawm)
  • qhib TCP chaw nres nkoj thiab banner rau nws
  • http grep (cov nplooj ntawv [yuav tsum tsis txhob] muaj cov ntawv tshwj xeeb)
  • sha1 hash txhawm rau ntes nplooj hloov pauv.
  • DNS (DNS cov ntaub ntawv yuav tsum muaj tus nqi tshwj xeeb)
  • WHOIS (yuav ceeb toom yog tias tus sau yuav mus tsis zoo)
  • Antispam DNSBL (tus tswv saib xyuas tawm tsam 50+ antispam blacklists ib zaug)

Kev kuaj sab hauv

Tsis tas li ntawd, cov txheej txheem ncaj ncees (tab sis nthuav dav tau yooj yim).

  • df (qhov chaw dawb disk)
  • load nruab nrab
  • opentcp (qhib TCP mloog sockets - yuav ceeb toom yog tias ib yam dab tsi pib lossis poob)
  • uptime - tsuas yog uptime ntawm lub server. Yuav ceeb toom yog tias nws tau hloov pauv (piv txwv li lub server tau overloaded)
  • client_ip
  • dirsize - peb siv nws los taug qab thaum peb lub tshuab virtual rootfs tshaj qhov tso cai loj, tsis muaj kev qhia txog kev txwv nruj, thiab qhov loj ntawm cov neeg siv cov npe hauv tsev
  • khoob thiab tsis yog - saib xyuas cov ntaub ntawv uas yuav tsum tau khoob (lossis tsis khoob). Piv txwv li, lub cav yuam kev ntawm okerr server nws tus kheej yuav tsum tau khoob, thiab yog tias muaj txawm tias muaj kab hauv nws, kuv yuav tau txais kev ceeb toom thiab xyuas nws. Tab sis mail.log ntawm lub mail server yuav tsum tsis txhob khoob (N feeb tom qab kev sib hloov). Thiab qee zaum nws tau khoob rau peb tom qab hloov kho qhov system, thaum logrotate tsis tuaj yeem rov pib rsyslog kom raug.
  • linecount - tus naj npawb ntawm kab hauv cov ntaub ntawv (xws li wc -l). Peb siv nws li kev hloov pauv rau qhov khoob, thaum lub cav yuam kev tseem tuaj yeem loj hlob, tab sis tsuas yog maj mam (piv txwv li, Googlebot tsoo qee nplooj ntawv kaw). Muaj qhov txwv ntawm 2 kab hauv 20 feeb. Yog tias siab dua, yuav muaj kev ceeb toom

Nthuav kev kuaj xyuas sab hauv

Yog tias koj tau nyeem "diagonally" mus txog qhov no, tam sim no nws yuav txaus siab nyeem kom zoo zoo.

backups

Saib xyuas cov thaub qab hauv phau ntawv qhia. Peb cov ntaub ntawv thaub qab muaj npe xws li "ServerName-20200530.tar.gz". Rau txhua tus neeg rau zaub mov hauv okerr, qhov taw qhia ServerName-DATE.tar.gz yog tsim (hnub hloov pauv rau kab β€œDATE”). Lub xub ntiag ntawm cov thaub qab tshiab thiab nws qhov loj me kuj tau saib xyuas (piv txwv li, nws tsis tuaj yeem tsawg dua 90% ntawm cov thaub qab yav dhau los).

Yuav ua li cas yuav tsum tau ua rau ib tug tshiab thaub qab yuav pib taug qab tom qab peb tau pib tsim nws thiab muab tso rau hauv no directory? Tsis muaj dab tsi! Nov yog txoj hauv kev yooj yim heev thaum koj xav ua "tsis muaj dab tsi" vim tias:

  • Ua "tsis muaj dab tsi" sai sai, nws txuag sijhawm
  • Nws nyuaj rau tsis nco qab ua "tsis muaj dab tsi"
  • Nws yog ib qho nyuaj rau ua "tsis muaj dab tsi", nrog rau qhov yuam kev. Tsis muaj dab tsi yog txoj kev ntseeg siab tshaj plaws

Yog dheev cov ntaub ntawv thaub qab tshiab tsis tshwm sim, yuav muaj kev ceeb toom. Yog tias, piv txwv li, koj tau xiam ib qho ntawm cov servers, thiab yuav tsum tsis muaj kev thaub qab ntxiv, koj yuav tsum tshem tawm qhov taw qhia (ntawm lub vev xaib interface lossis los ntawm lub plhaub ntawm API).

maxfilesz ua

Ua raws li qhov loj ntawm cov ntaub ntawv loj tshaj plaws (feem ntau: /var/log/*). Qhov no tso cai rau koj los ntes cov teeb meem tsis tuaj yeem, piv txwv li, brute quab yuam passwords lossis xa spam los ntawm lub server.

runstatus/cov

Cov no yog ob qho tseem ceeb proxy modules rau khiav lwm cov kev pab cuam ntawm lub server. Runstatus qhia qhov program tawm code rau qhov taw qhia. Piv txwv li, okerr tsis (yuav tsum tau) ib qho module los xyuas tias cov kev pabcuam systemd tau ua haujlwm. Qhov no yog ua los ntawm runstatus (saib hauv qab). Runline - qhia rau tus neeg rau zaub mov cov kab uas qhov kev pab cuam tsim. Piv txwv li, temp_RUN="cat /sys/class/thermal/thermal_zone0/temp" nyob rau hauv lub Runline config ntawm peb cov neeg rau zaub mov tsim ib qho taw qhia servername: temp nrog lub processor kub.

sql

Ua tus lej nug rau MySQL thiab tshaj tawm qhov tshwm sim rau qhov ntsuas. Hauv qhov yooj yim, koj tuaj yeem ua tau, piv txwv li, "SELECT 1" - qhov no yuav kuaj xyuas tias DBMS tag nrho ua haujlwm.

Tab sis ib daim ntawv thov nthuav ntau dua yog, piv txwv li, taug qab cov lej xaj hauv khw hauv online. Yog tias koj paub tias koj muaj 100 lossis ntau qhov kev txiav txim ib teev twg, koj tuaj yeem teem qhov tsawg kawg nkaus rau 100 lossis 80. Yog tias koj qhov kev muag khoom poob qis, koj yuav tau txais kev ceeb toom thiab koj tuaj yeem txiav txim siab.

Nco ntsoov tias nws tsis muaj teeb meem rau qhov laj thawj tsis tuaj yeem tshwm sim:

  • Cov neeg rau zaub mov tsuas yog tsis muaj (de-energized lossis tsis muaj lub network), thiab kev ceeb toom los ntawm qhov tseeb tias qhov ntsuas tau "rotten".
  • Cov neeg rau zaub mov yog overloaded nrog ib yam dab tsi, nws ua hauj lwm maj mam los yog cov pob ntawv poob, nws tsis yooj yim rau cov neeg siv thiab lawv tawm yam tsis tau yuav khoom.
  • Cov neeg rau zaub mov suav nrog cov npe spam thiab xa ntawv los ntawm nws tsis tau txais, cov neeg siv tsis tuaj yeem sau npe
  • Cov phiaj xwm kev tshaj tawm nyiaj txiag tau dhau mus, cov ntawv tshaj tawm tsis tau tig.

Muaj peev xwm muaj ntau yam laj thawj, thiab tag nrho lawv tsis tuaj yeem pom ua ntej, thiab nws nyuaj rau kev taug qab. Tab sis koj tuaj yeem yooj yim saib xyuas qhov ntsuas kawg (kev txiav txim) thiab txiav txim siab los ntawm lawv tias qhov xwm txheej tsis txaus ntseeg thiab tsim nyog yuav tsum tau daws.

Logical ntsuas

Tso cai siv Boolean kab lus (Python syntax) ntawm ib qho module evalidate(tsab xov xwm ntawm Habre). Cov ntaub ntawv los ntawm qhov project thiab nws cov ntsuas yog muaj rau kev qhia. Piv txwv li, nyob rau hauv tshooj hais txog SQL kuaj saum toj no, tej zaum koj yuav tau pom ib tug tsis muaj zog point - thaum nruab hnub peb muaj peev xwm muaj 100 muag ib teev, tab sis thaum hmo ntuj - 20, thiab qhov no yog ib txwm, tsis muaj teeb meem. Kuv yuav ua li cas? Qhov taw qhia yuav tas li ceeb thaum hmo ntuj.

Koj tuaj yeem tsim ob qhov ntsuas, nruab hnub thiab hmo ntuj. Ua ob qho tib si "tsim" (lawv yuav tsis xa ntawv ceeb toom). Thiab tsim ib qho kev taw qhia uas xav kom lub hnub qhia tau OK ua ntej 20:00, thiab tom qab 20:00 nws txaus rau qhov taw qhia hmo ntuj kom OK.

Lwm qhov piv txwv ntawm kev siv qhov ntsuas qhov ntsuas yog kev nce ntxiv. Piv txwv li, tus thawj tswj hwm qhov project unsubscribes los ntawm kev ceeb toom (nws tsis tas yuav ua qhov no, admins yuav tsum teb rau cov teeb meem ib txwm muaj), tab sis sau npe rau qhov ntsuas qhov ntsuas uas tig liab yog tias qhov ntsuas hauv qhov project tsis raug kho nyob rau hauv lub sijhawm teem tseg.

Tsis tas li ntawd, nws muaj peev xwm teem sij hawm tso cai rau kev ua hauj lwm, piv txwv li, los ntawm 3 mus rau 5 teev sawv ntxov. Peb tsis quav ntsej yog tias cov servers thiab qhov chaw sib tsoo thaum lub sijhawm no. Tab sis thaum 5:00 lawv yuav tsum ua haujlwm. Yog tias lawv tsis ua haujlwm rau lwm lub sijhawm - ceeb toom. Qhov taw qhia logical kuj tso cai rau koj coj mus rau hauv tus account server redundancy. Yog tias koj muaj 5 lub web servers, ces cov thawj coj tuaj yeem tua 1-2 servers txhua lub sijhawm. Tab sis yog tias muaj tsawg dua 3 tawm ntawm 5 servers hauv kev sib ntaus sib tua, yuav muaj kev ceeb toom.

Cov piv txwv saum toj no tsis yog kev ua haujlwm zoo, tsis yog qee qhov nta uas yuav tsum tau qhib thiab teeb tsa. Okerra tsis muaj tag nrho cov haujlwm no, tab sis muaj cov qauv kev xav uas tso cai rau koj los siv qhov kev ua haujlwm no (Kwv yees li hauv cov lus programming - yog tias peb muaj tus lej lej, ces peb tsis tas yuav muaj lub luag haujlwm tshwj xeeb rau kev suav 20% ​​VAT. los ntawm cov lus, koj tuaj yeem ua nws tus kheej ua kom haum koj cov kev xav tau).

Logic Indicator yog tej zaum ib qho ntawm ob peb lub ntsiab lus nyuaj hauv okerr, tab sis cov xov xwm zoo yog tias koj tsis tas yuav tsum paub txog nws kom txog thaum koj xav tau. Tab sis tib lub sijhawm, lawv nthuav dav lub peev xwm, thaum ua kom lub kaw lus nws tus kheej yooj yim heev.

Ntxiv koj cov tshev

Kuv xav qhia lub tswv yim tias okerr tsis yog ib txheej ntawm ntau txhiab tus npaj txhij-ua cov tshev rau txhua lub sijhawm, tab sis ntawm qhov tsis sib xws - ua ntej ntawm tag nrho - lub cav yooj yim nrog lub peev xwm yooj yim los tsim koj tus kheej cov tshev. Tsim koj tus kheej cov tshev hauv okerr tsis yog ib txoj haujlwm rau cov neeg nyiag nkas, cov neeg tsim khoom sib koom ua ke, lossis tsawg kawg yog cov neeg siv okerr siab heev, tab sis ib txoj haujlwm ua tau rau txhua tus thawj tswj hwm uas tau teeb tsa Linux thawj zaug ib hlis dhau los.

Kev kuaj xyuas cov nyiaj ua haujlwm yam tsawg kawg nkaus yog ua tiav los ntawm lub module runstatus:

Cov kab no nyob rau hauv config runstatus yuav ceeb toom rau koj yog tias /bin / tseeb dheev tsis pib lossis rov qab ib yam dab tsi uas tsis yog 0.

true_OK=/bin/true

Tsuas yog ib kab - thiab ntawm no peb twb me ntsis lawm nthuav dav functionality oker.

Txawm tias qhov kev kuaj ntawd twb muaj nuj nqis lawm: yog tias tam sim ntawd koj lub server poob, qhov taw qhia sib txuas ntawm okerr server yuav tsis hloov kho raws sijhawm, thiab tom qab lub sijhawm dhau mus, ceeb toom yuav tshwm sim.

Qhov kev kuaj no yuav ceeb toom tias apache2 server tau tsoo (zoo, koj tsis paub ...):

apache_OK="systemctl is-active --quiet apache2"

Yog li, yog tias koj hais ib hom lus programming, thiab tsawg kawg tuaj yeem sau cov ntawv plhaub, ces koj tuaj yeem ntxiv koj cov tshev.

Qhov nyuaj dua - koj tuaj yeem sau (ua ib hom lus) koj tus kheej module rau okerrmod. Hauv qhov yooj yim tshaj plaws nws zoo li no:

#!/usr/bin/python3

print("STATUS: OK")

Tsis yog nws nyuaj heev? Lub module yuav tsum ua qhov kev kuaj nws tus kheej thiab muab cov txiaj ntsig rau STDOUT. Ib tug complex module muab, piv txwv li, qhov no:

$ okerrmod --dump df
NAME: pi:df-/
TAGS: df
METHOD: numerical|maxlim=90
DETAILS: 49.52%, 13.9G/28.2G used, 13.0G free
STATUS: 49.52

NAME: pi:df-/boot
TAGS: df
METHOD: numerical|maxlim=90
DETAILS: 84.32%, 53.1M/62.9M used, 9.9M free
STATUS: 84.32

Nws hloov kho ob peb qhov ntsuas ib zaug (sib cais los ntawm kab khoob), tsim lawv yog tias tsim nyog, qhia cov ntsiab lus pov thawj thiab ib daim ntawv lo uas nws yooj yim los nrhiav cov cim tsim nyog hauv dashboard.

Telegram

Muaj ib tug Telegram bot @OkerrBot. Koj tsis tas yuav clutter koj lub xov tooj nrog nyias daim ntawv thov (Kuv tsis nyiam qhov ntawd rau Pyaterochka koj xav tau ib daim ntawv thov nrog daim ntawv qhia, rau Lenta lwm, rau MTS ib feem peb, thiab lwm yam rau txhua tus, txhua tus, tag nrho). Ib telegram txaus. Los ntawm telegram koj tuaj yeem tau txais kev ceeb toom tam sim ntawd, tshawb xyuas qhov xwm txheej ntawm qhov project thiab muab lus txib kom rov kuaj xyuas txhua qhov teeb meem. Peb tawm hauv kev ua yeeb yam / dav hlau, tsis txhob tuav peb tus ntiv tes ntawm lub mem tes ob teev, qhib lub xov tooj, nias ib lub pob hauv chatbot, thiab xyuas kom meej tias txhua yam zoo.

Nplooj ntawv xwm txheej

Niaj hnub no, cov nplooj ntawv xwm txheej yuav luag yuav tsum muaj rau txhua lub lag luam uas muaj IT, lub luag haujlwm tus cwj pwm ntawm kev ntseeg tau thiab ua rau nws cov neeg siv khoom / cov neeg siv kev hwm.

Xav txog qhov xwm txheej - tus neeg siv xav ua ib yam dab tsi, saib cov ntaub ntawv lossis tso qhov kev txiav txim, thiab qee yam tsis ua haujlwm. Nws tsis paub dab tsi tshwm sim, ntawm nws sab qhov teeb meem yog thiab thaum twg nws yuav raug daws. Tej zaum koj lub tuam txhab tsuas muaj lub vev xaib tsis ua haujlwm? Los yog nws puas tau tawg rau lub hlis dhau los thiab nws yuav raug kho hauv ob xyoos? Tab sis koj yuav tsum tau yuav lub tub yees tam sim no, nws twb nyob rau hauv lub laub ... Thiab nws yog ib qho txawv kiag li thaum ib tug neeg pom tias ib yam dab tsi tsis ncaj ncees lawm rau koj (tsawg kawg yog qhov tseeb tias qhov teeb meem tsis nyob ntawm nws sab), hais tias tus Qhov teeb meem tau pom, uas koj twb tau ua haujlwm rau nws, thiab tej zaum kuj tau sau cia lub sijhawm kwv yees rau kev kho. Tus neeg siv tuaj yeem sau npe thiab tau txais email ceeb toom thaum qhov teeb meem raug kho thiab nws tuaj yeem ua qhov nws xav tau (yuav lub tub yees).

Txheej txheem cej luam ntawm Okerr hybrid saib xyuas qhov system

Teeb meem thiab downtime tshwm sim rau txhua tus. Tab sis cov neeg siv thiab cov neeg koom tes ntseeg ntau dua cov neeg uas muaj pob tshab thiab lub luag haujlwm hauv lawv txoj hauv kev rau qhov no.

no tshuaj xyuas 10 lwm cov haujlwm uas tso cai rau koj los tsim cov nplooj ntawv xwm txheej. Nov yog cov piv txwv ntawm cov nplooj ntawv qhov project no zoo li cas Nab hab sej ΠΈ Dropbox. okerr xwm txheej nplooj.

Kev yuam kev

Txhawm rau kom tsis txhob ua cov kab lus no ntev dua, kuv yuav rov hais dua rau kuv tsab xov xwm dhau los - Yooj yim poob rau lub vev xaib . Yog tias koj tuaj yeem ua tus neeg rau zaub mov duplicate, tom qab ntawd siv qhov tsis ua haujlwm, koj yeej yuav tsis muaj sijhawm poob ntev - sai li sai tau pom qhov teeb meem, cov neeg siv yuav tau xa mus rau lub chaw ua haujlwm thaub qab. Thiab nws zoo li kuv tias qhov no yog qhov nthuav heev, ci ntsa iab uas tsis tshua muaj nyob txhua qhov chaw.

Tsawg qhov yuav tsum tau ua

Rau okerr servers, peb siv cov tshuab nrog RAM ntawm 2Gb. Rau network sensors, txawm tias 512Mb txaus. Cov neeg siv khoom feem feem ntau yuav luag xoom. (Lub hnab yas okerupdate hnyav 26 Kb, tab sis yuav tsum tau Python3 thiab cov tsev qiv ntawv txheem). Tus neeg siv khoom khiav los ntawm cov ntawv cron, yog li nws tsis muaj kev nco tsis tu ncua. Ntawm cov tshuab peb saib xyuas, peb muaj cov sensors (super-cheap VPS nrog 512Mb RAM) thiab Raspberry Pi. Nws tuaj yeem ua tau txawm tias tsis muaj tus neeg siv khoom xa cov tshiab ntawm curl! (saib hauv qab no)

Noj qhov no rau hauv tus account - okerr, tej zaum feem ntau dawb kev saib xyuas los ntawm cov muaj, vim tias txawm siv lwm qhov chaw qhib dawb xws li Zabbix lossis Nagios, koj yuav tsum faib cov peev txheej (server) rau nws, thiab qhov no yog nyiaj txiag. Tsis tas li ntawd, qee qhov kev saib xyuas server tseem yuav tsum tau ua. Nrog okerr, qhov no tuaj yeem raug tshem tawm. Lossis koj tsis tas yuav tshem nws thiab siv koj tus kheej server, nyob ntawm seb koj nyiam qhov zoo tshaj plaws.

API thiab kev koom ua ke rau hauv cov software tsim tshwj xeeb

Yooj yim thiab qhib architecture. okerr muaj qhov yooj yim zoo nkauj API, uas yog ib qho yooj yim mus ua hauj lwm nrog. Xav tau los tsim 1000 qhov ntsuas? Ib daim ntawv plhaub ntawm 3-4 kab yuav ua li no. Yuav tsum tau kho dua 1000 qhov ntsuas? Nws kuj yog ib qho yooj yim heev. Piv txwv li, peb xav muab ob npaug rau-xyuas tag nrho peb HTTPS daim ntawv pov thawj los ntawm Lavxias teb sab sensor:

#!/bin/sh

for indicator in `okerrclient --api-filter sslcert`
do
    echo set location for $indicator
    okerrclient --api-set location=ru retest=1 --name $indicator
done

Koj tuaj yeem hloov kho qhov taw qhia siv peb cov neeg siv khoom module, txawm tias tsis muaj nws, tsuas yog ntawm curl.

# short and nice (using okerrupdate and config file)
$ okerrupdate MyIndicator OK

# only curl is enough!
$ curl -d 'textid=MyProject&name=MyIndicator&secret=MySecret&status=OK' https://bravo.okerr.com/

Koj tuaj yeem hloov kho cov ntsuas ncaj qha los ntawm koj qhov program. Piv txwv li, xa lub plawv dhia cov cim kom okerr paub tias nws khiav thiab tsa lub tswb yog tias nws tsoo lossis khov. Los ntawm txoj kev, okerr Cheebtsam ua li ntawd - okerr saib nws tus kheej, thiab teeb meem hauv yuav luag txhua qhov module yuav raug kuaj pom thiab tsim kev ceeb toom txog qhov teeb meem. (Thiab nyob rau hauv rooj plaub no "yuav luag" - lawv raug kuaj xyuas los ntawm lwm tus neeg rau zaub mov)

Nov yog cov cai (simplified) hauv peb lub xov tooj bot:

from okerrupdate import OkerrProject, OkerrExc

op = OkerrProject()
uptimei = op.indicator("{}:telebot_uptime".format(hostname))
...
uptimei.update('OK', 'pid: {} Uptime: {} cmds: {}'.format(
        os.getpid(), dhms(uptime), commands_cnt))

Muaj ib lub tsev qiv ntawv rau kev hloov kho cov ntsuas los ntawm Python cov kev pab cuam okerupdate, rau lwm yam lus tsis muaj tsev qiv ntawv, tab sis koj tuaj yeem hu rau okerrupdate tsab ntawv lossis ua HTTP thov rau okerr server.

okerr pab peb li cas

Okerr hloov peb lub neej. Tseeb tiag. Tej zaum lwm qhov kev saib xyuas yuav ua tau zoo ib yam, tab sis kev ua haujlwm nrog okerr yog qhov yooj yim thiab yooj yim rau peb thiab nws muaj tag nrho cov haujlwm uas peb xav tau (peb ntxiv qhov nws tsis muaj). Los ntawm txoj kev, yog tias muaj qee yam uas ploj lawm, nug thiab kuv yuav ntxiv rau lawv (Kuv tsis cog lus, tab sis kuv xav kom okerr yog qhov kev soj ntsuam zoo tshaj plaws rau cov haujlwm me me). Los yog zoo dua, ntxiv rau koj tus kheej - nws yooj yim.

Peb tswj kom ua neej nyob raws li txoj cai "kawm txog txhua yam teeb meem los ntawm kerra." Yog tias tam sim ntawd muaj teeb meem tshwm sim uas peb tsis tau kawm txog los ntawm okerr, peb ntxiv ib daim tshev rau okerr. (hauv qhov no, los ntawm "peb" Kuv txhais tau tias peb yog cov neeg siv ntawm lub kaw lus, tsis yog tus tsim tawm). Thaum xub thawj qhov no tshwm sim, tab sis tam sim no nws tau dhau los ua tsawg heev.

Saib xyuas

Los ntawm okerr peb saib xyuas cov log loj ntawm txhua lub servers. Nws yog, ntawm chav kawm, tsis yooj yim sua kom xav nyeem txhua kab ntawm lub cav nrog koj ob lub qhov muag, tab sis tsuas yog saib xyuas tus nqi loj hlob twb muab ntau heev. Los ntawm qhov no, peb nrhiav tau spam xa ntawv thiab brute quab yuam kev tshawb nrhiav lo lus zais, thiab thaum qee cov ntawv thov "mus vwm," ib yam dab tsi tsis ua haujlwm rau lawv thiab lawv rov ua dua (txhua zaus ntxiv ob peb kab rau lub cav. ).

SSL daim ntawv pov thawj. Yuav luag tam sim tom qab tso tawm LetsEncrypt peb cov neeg siv khoom pib muab daim ntawv pov thawj SSL dawb rau nws cov neeg siv khoom (kwv yees li ib txhiab ntawm lawv). Thiab nws tau dhau los ua ntuj txiag teb tsaus rau kev tswj hwm! Qhov tseeb yog tias qhov chaw yog "nyob", cov neeg siv khoom ib ntus hais kom lawv ua ib yam dab tsi, programmers ua nws. Lawv tuaj yeem hloov lub xaib mus rau lwm qhov DocumentRoot, piv txwv li. Los yog ntxiv ib qho kev sau tsis raug cai rau lub virtualhost config. Lawm, tom qab no, tsis siv neeg rov ua dua tshiab ntawm daim ntawv pov thawj tawg. Tam sim no peb muaj tag nrho SSL hosts ntxiv rau okerr tau los ntawm lwm qhov ntawm peb cov khoom siv muaj txiaj ntsig los ntawm pob ib 2conf. Cia li tso tawm a2okerr.py - thiab yog tias ntau qhov chaw tshiab tshwm sim ntawm lub server, lawv yuav cia li tshwm sim hauv okerr. Yog tias dheev rau qee yam vim li cas daim ntawv pov thawj tsis tau rov ua dua tshiab, peb lub lis piam ua ntej daim ntawv pov thawj tas sijhawm, peb nyob hauv kev paub, thiab peb yuav txheeb xyuas seb vim li cas nws tsis hloov kho, xws li tus dev. a2certbot.py los ntawm tib lub pob - nws pab tau ntau yam nrog qhov no (nws tam sim ntawd tshawb xyuas cov teeb meem feem ntau - thiab sau dab tsi tau kuaj xyuas zoo, thiab qhov twg muaj feem yuav muaj teeb meem).

Peb saib xyuas hnub tas sij hawm ntawm tag nrho peb cov domains. Thiab tag nrho peb cov mail servers uas xa ntawv kuj raug kuaj tawm tsam 50+ cov npe dub sib txawv. (Thiab qee zaum lawv poob rau hauv lawv). Los ntawm txoj kev, koj puas paub tias Google mail servers kuj raug teev npe dub? Tsuas yog rau kev ntsuas tus kheej, peb ntxiv mail-wr1-f54.google.com rau cov saib xyuas servers, thiab nws tseem nyob hauv SORBS blacklist! (Qhov no yog hais txog tus nqi ntawm "anti-spammers")

Backups - Kuv twb tau sau saum toj no yooj yim npaum li cas los saib xyuas lawv nrog okerr. Tab sis peb saib xyuas ob qhov kev thaub qab tshiab kawg ntawm peb cov neeg rau zaub mov thiab (siv cov khoom siv sib cais uas siv okerr) cov thaub qab uas peb upload rau Amazon Glacier. Thiab, yog, teeb meem tshwm sim ib ntus. Tsis xav tias lawv tau saib.

Peb siv qhov ntsuas escalation. Nws qhia tau hais tias qee qhov teeb meem tsis tau kho tau ntev. Thiab kuv tus kheej, thaum kuv daws qee qhov teeb meem, qee zaum kuv tuaj yeem hnov ​​​​qab txog lawv. Kev nce qib yog qhov ceeb toom zoo, txawm tias koj tab tom saib xyuas koj tus kheej.

Zuag qhia tag nrho, kuv ntseeg tias qhov ua tau zoo ntawm peb txoj haujlwm tau nce los ntawm qhov kev txiav txim loj. Yuav luag tsis muaj sijhawm poob haujlwm (lossis tus neeg siv khoom tsis muaj sijhawm pom nws. Tsuas yog shh!), thaum lub sijhawm ua haujlwm tau me dua thiab cov haujlwm ua haujlwm tau zoo dua. Peb tau txav los ntawm kev ua haujlwm kub ntxhov nrog patching qhov nrog daim kab xev rau kev ua haujlwm siab thiab ntsuas, thaum muaj ntau yam teeb meem tau kwv yees ua ntej thiab muaj sijhawm los tiv thaiv lawv. Txawm tias cov teeb meem uas tau tshwm sim los kuj tau yooj yim kho: thawj zaug, peb paub txog lawv ua ntej cov neeg siv khoom txaus ntshai, thiab qhov thib ob, nws feem ntau tshwm sim tias qhov teeb meem cuam tshuam nrog kev ua haujlwm tsis ntev los no (thaum kuv ua ib yam, kuv tsoo lwm qhov) - yog li nws kub Nws yooj yim dua rau cov kab mob los daws nws.

Tab sis muaj lwm rooj plaub ...

Koj puas paub tias hauv qhov nrov Debian 9 (stretch) xws li pob nrov li phpmyadmin tseem (rau ntau lub hlis!) nyob rau hauv cov xwm txheej tsis zoo? (CVE-2019-6798). Thaum qhov teeb meem tshwm sim, peb tau ceev nrooj npog nws ntau txoj hauv kev. Tab sis kuv teeb tsa kev saib xyuas ntawm nplooj ntawv ruaj ntseg-tracker hauv okerr kom paub thaum twg "zoo nkauj" kev daws teeb meem yuav tawm (ntawm SHA1 cov ntsiab lus). Qhov taw qhia twitched kuv ob peb zaug, nplooj ntawv hloov pauv, tab sis raws li koj tuaj yeem pom, nws tseem (txij li Lub Ib Hlis 2019!) tsis qhia tias qhov teeb meem tau daws lawm. Tej zaum, los ntawm txoj kev, ib tug neeg paub dab tsi qhov teeb meem yog hais tias xws li ib tug tseem ceeb pob yog tseem muaj kev cuam tshuam rau ntau tshaj ib xyoo?

Lwm lub sijhawm hauv qhov xwm txheej zoo sib xws: tom qab muaj qhov tsis zoo hauv SSH, nws yuav tsum tau hloov kho txhua lub servers. Thiab thaum koj teem ib txoj hauj lwm, koj yuav tsum tau tswj kom tiav. (Cov tub txib yuav to taub yuam kev, tsis nco qab, tsis meej pem, thiab ua yuam kev). Yog li ntawd, ua ntej peb ntxiv SSH version kos rau okerr ntawm txhua lub servers, thiab los ntawm okerr peb tau ua kom paub tseeb tias cov kev hloov tshiab tau dov tawm ntawm txhua lub servers. (Zoo! Kuv xaiv hom kev ntsuas no, thiab koj tuaj yeem pom tam sim ntawd tus neeg rau zaub mov twg muaj qhov version). Thaum peb paub tseeb tias txoj haujlwm tiav ntawm txhua lub servers, peb tshem tawm cov ntsuas.

Ob peb zaug muaj ib qho teeb meem uas muaj teeb meem tshwm sim, thiab tom qab ntawd ploj mus ntawm nws tus kheej. (tej zaum paub txhua tus?). Thaum koj pom, thaum lub sijhawm koj kuaj - thiab tsis muaj dab tsi los kuaj - txhua yam twb ua haujlwm zoo. Tab sis ces nws tawg dua. Peb tau muaj qhov no tshwm sim, piv txwv li, nrog cov khoom uas peb muab tso rau Amazon Marketplace (MWS). Qee lub sij hawm, cov khoom thauj khoom tsis raug (cov khoom tsis raug thiab cov nqi tsis raug). Peb xam nws tawm. Tab sis txhawm rau txheeb xyuas nws, nws yog ib qho tseem ceeb kom paub txog qhov teeb meem tam sim ntawd. Hmoov tsis zoo, MWS, zoo li txhua qhov kev pabcuam Amazon, qeeb me ntsis, yog li muaj ib txwm muaj kev lag luam, tab sis tseem, peb muaj peev xwm tsawg kawg yog kwv yees qhov kev sib txuas ntawm qhov teeb meem thiab cov ntawv uas ua rau nws (peb tau kuaj xyuas, daig. nws mus rau okerr, thiab xyuas nws tam sim ntawd tau txais kev ceeb toom).

Ib rooj plaub nthuav tau tsis ntev los no tau ntxiv rau kev sau los ntawm tus tswv tsev loj thiab kim European, uas peb cov neeg siv khoom siv. Dheev, tag nrho peb cov servers ploj ntawm radar! Ua ntej, tus neeg siv khoom nws tus kheej (ceev dua okerra!) pom tias qhov chaw nws ua haujlwm nrog tsis tau qhib thiab ua daim pib txog nws. Tab sis tsis yog tsuas yog ib qhov chaw poob mus, tab sis txhua tus ntawm lawv! (Natasha, peb poob txhua yam!). Ntawm no Okerr pib xa cov ko taw ntev nrog txhua qhov ntsuas uas teeb rau nws. Panic, ceeb, peb khiav hauv lub voj voog (peb yuav ua li cas ntxiv?). Ces txhua yam sawv. Nws hloov tawm tias muaj kev saib xyuas niaj hnub hauv cov ntaub ntawv chaw (ib zaug txhua xyoo) thiab, tau kawg, peb yuav tsum tau ceeb toom. Tab sis qee qhov teeb meem tshwm sim rau lawv thiab lawv tsis ceeb toom peb. Zoo, ntau lub plawv nres, tsawg lub plawv nres. Tab sis tom qab txhua yam rov qab los, koj yuav tsum tau kuaj xyuas txhua yam! Kuv tsis tuaj yeem xav tias kuv yuav ua li cas nrog kuv txhais tes. Okerr sim txhua yam hauv ob peb feeb. Nws tau pom tias feem ntau ntawm cov servers tsuas yog tsis muaj ib ntus, tab sis lawv ua haujlwm. Ib txhia twb overloaded, tab sis kuj sawv raws li lawv yuav tsum tau. Ntawm tag nrho cov poob, peb poob ob lub thaub qab, uas raws li lub crown yuav tsum tau tsim thiab loaded thaum lub txiv tsawb puv no mus. Kuv tsis txawm thab tsim lawv, tsuas yog ib hnub tom qab ntawv ceeb toom tuaj txog tias txhua yam zoo, cov thaub qab tau tshwm sim. Kuv nyiam qhov piv txwv no tiag tiag vim tias okerr tau dhau los ua qhov muaj txiaj ntsig zoo nyob rau hauv qhov xwm txheej uas peb tsis tau xav txog ua ntej, tab sis qhov ntawd yog lub hom phiaj ntawm kev saib xyuas - tiv thaiv qhov tsis tau pom dua.

Rau Okerr sensors, peb siv qhov pheej yig tshaj plaws hosting (qhov zoo thiab kev ntseeg siab tsis tseem ceeb, lawv muaj kev ruaj ntseg). Yog li, peb tsis ntev los no pom ib qho zoo heev hosting thiab super pheej yig, cov qauv zoo heev. Tab sis ... qee zaum nws hloov tawm tias cov kev sib txuas tawm ntawm lub tshuab virtual yog tsim los ntawm lwm tus (ib puag ncig) IP. Tej txuj ci tseem ceeb. Client_ip module nrog https://diagnostic.opendns.com/myip tau txais tus IP yuam kev. Thiab los ntawm cov neeg rau zaub mov cav ntawm qhov ntsuas nws yog qhov tseeb tias qhov hloov tshiab kuj tuaj ntawm tus IP nyob sib ze. Cia peb nrog kev txhawb nqa tam sim no. Nws yog qhov zoo uas peb pom qhov no nyob rau hauv kev thaj yeeb nyab xeeb. Tab sis, piv txwv li, nws feem ntau tshwm sim tias kev nkag mus tau sau npe raws li daim ntawv teev npe IP dawb - thiab yog tias cov neeg rau zaub mov qee zaum blinks zoo li no rau lub sijhawm luv luv - koj tuaj yeem sim ntes qhov teeb meem no ntev heev.

Zoo, ib qho ntxiv - txij li peb tab tom tham txog VPS hosting - peb ib txwm siv cov pheej yig (hetzner, ovh, scaleway). Kuv yeej nyiam nws ob qho tib si nyob rau hauv cov nqe lus ntawm benchmarks thiab stability. Peb kuj siv Amazon EC2 kim dua rau lwm cov haujlwm. Yog li, ua tsaug rau okerr, peb muaj peb tus kheej cov lus qhia. Nkawd ob leeg poob. Thiab kuv yuav tsis hais tias nyob rau lub sijhawm ntev ntawm peb cov kev soj ntsuam, pheej yig hostings zoo li hetzner tau pom tias tsis tshua muaj zog dua EC2. Yog li ntawd, yog tias koj tsis khi rau lwm yam Amazon nta, vim li cas them nyiaj ntau dua? πŸ™‚

Yuav ua li cas yog tom ntej no?

Yog tias nyob rau theem no kuv tsis tau ntshai koj tawm ntawm Okerr tsis tau, ces sim nws! Koj tuaj yeem mus ncaj qha rau qhov txuas no okerr demo tus account (Nyem tam sim no!) Tab sis nco ntsoov tias tsuas muaj ib tus account demo rau txhua tus, yog li yog tias koj ua qee yam, lwm tus neeg hauv tib tus account yuav cuam tshuam rau koj tib lub sijhawm. Los yog (zoo dua) sau npe ntawm qhov txuas rau offsite oker - Txhua yam yog yooj yim, tsis muaj SMS. Yog tias koj tsis nyiam siv koj tus email tiag, koj tuaj yeem siv cov khoom siv pov tseg, xws li mailinator (Kuv pom zoo getnada.com). Xws li cov nyiaj yuav raug muab tshem tawm thaum lub sijhawm, tab sis lawv yuav zoo rau kev sim.

Tom qab sau npe, koj yuav raug hais kom mus kawm (ua ob peb txoj haujlwm tsis nyuaj heev). Cov kev txwv thawj zaug tsawg heev, tab sis rau kev cob qhia lossis ib tus neeg rau zaub mov lawv txaus. Tom qab ua tiav qhov kev cob qhia, cov kev txwv (piv txwv li, qhov siab tshaj plaws ntawm cov ntsuas) yuav nce ntxiv.

Los ntawm cov ntaub ntawv - ua ntej ntawm tag nrho cov WIKI nyob rau sab server thiab ntawm tus neeg siv khoom (okerupdate wiki). Tab sis yog tias muaj qee yam tsis meej, sau ntawv txhawb nqa (ntawm) okerr.com lossis tawm daim pib - peb yuav sim daws txhua yam sai sai.

Yog tias koj siv nws tiag tiag thiab cov kev txwv ntau ntxiv no tsis txaus, sau ntawv rau kev txhawb nqa thiab peb yuav nce nws (dawb).

Koj puas xav nruab okerr server ntawm koj lub server? Ntawm no okerr-dev repository. Peb pom zoo kom txhim kho ntawm lub tshuab virtual huv si, tom qab ntawd koj tuaj yeem ua nws yooj yim nrog cov ntawv teeb tsa. Ntawm koj lub tshuab virtual - tsis muaj kev txwv :-). Zoo, dua, yog tias muaj dab tsi tshwm sim, peb yuav sim pab.

Peb xav kom qhov project no tawm mus, kom lub ntiaj teb no txhim khu kev qha ua tsaug rau peb. Ua tsaug rau cov software dawb thiab cov kev pabcuam, lub ntiaj teb tau dhau los ua phooj ywg thiab tab tom txhim kho ntau dua. Cov peev txheej tuaj yeem khaws cia hauv github pub dawb, thiab rau kev xa ntawv koj tuaj yeem siv gmail dawb. Peb siv dawb ua haujlwm tshiab rau kev txhawb nqa. Rau ib qho ntawm qhov no, koj tsis tas yuav them nyiaj rau cov servers, koj tsis tas yuav rub tawm thiab teeb tsa, thiab koj tsis tas yuav daws teeb meem ntau yam haujlwm. Txhua qhov project tshiab, txhua pab neeg tam sim ntawd muaj xa ntawv, chaw cia khoom thiab CRM. Thiab tag nrho cov no yog qhov zoo heev thiab pub dawb thiab tam sim ntawd. Peb xav kom nws zoo ib yam rau kev saib xyuas - cov tuam txhab me thiab cov phiaj xwm tuaj yeem siv okerr dawb thiab txawm tias nyob rau theem ntawm kev yug me nyuam thiab kev loj hlob muaj kev ntseeg siab ntawm cov neeg laus cov haujlwm loj.

Tau qhov twg los: www.hab.com