Nyob rau hauv 2010 lub tuam txhab muaj 50 servers thiab tus qauv network yooj yim: backend, frontend thiab firewall. Tus naj npawb ntawm cov servers tau loj hlob, tus qauv tau dhau los ua qhov nyuaj: kev teeb tsa, cais VLANs nrog ACLs, tom qab ntawd VPNs nrog VRFs, VLANs nrog ACLs ntawm L2, VRFs nrog ACLs ntawm L3. Lub taub hau yog spinning? Nws yuav lom zem dua tom qab.
Thaum muaj 16 servers, nws tsis tuaj yeem ua haujlwm yam tsis muaj kua muag nrog ntau qhov sib txawv. Yog li peb tuaj nrog lwm txoj kev daws teeb meem. Peb coj Netfilter pawg, ntxiv Consul rau nws raws li cov ntaub ntawv qhov chaw, thiab peb tau txais lub foob pob hluav taws ceev ceev. Lawv hloov ACLs ntawm routers thiab siv lawv ua ib qho hluav taws xob sab nraud thiab sab hauv. Txhawm rau tswj hwm lub cuab yeej, peb tsim BEFW system, uas tau siv txhua qhov chaw: los ntawm kev tswj cov neeg siv nkag mus rau cov khoom lag luam mus rau kev cais cov ntu ntawm ib leeg.
Nws yuav qhia koj tias txhua yam ua haujlwm li cas thiab vim li cas koj thiaj li yuav tsum ua tib zoo saib xyuas qhov system no. Ivan Agarkov () yog lub taub hau ntawm pab pawg kev ruaj ntseg ntawm Pawg Saib Xyuas Kev Ua Haujlwm ntawm lub tuam txhab Minsk qhov chaw txhim kho. Ivan yog ib tug SELinux kiv cua, hlub Perl, thiab sau code. Raws li lub taub hau ntawm pab pawg kev ruaj ntseg cov ntaub ntawv, nws tsis tu ncua ua haujlwm nrog cov cav, thaub qab thiab R & D los tiv thaiv Wargaming los ntawm cov neeg nyiag nkas thiab xyuas kom muaj kev ua haujlwm ntawm txhua qhov game servers hauv tuam txhab.
Keeb kwm keeb kwm
Ua ntej kuv qhia koj tias peb tau ua li cas, kuv yuav qhia koj tias peb tuaj rau qhov no thawj zaug thiab vim li cas thiaj xav tau. Ua li no, cia peb rov qab mus rau 9 xyoo: 2010, Ntiaj Teb ntawm Tanks nyuam qhuav tshwm sim. Wargaming muaj kwv yees li 50 servers.
Lub tuam txhab server loj hlob daim ntawv qhia.
Peb muaj tus qauv network. Rau lub sijhawm ntawd nws yog qhov zoo tshaj plaws.
Network qauv hauv 2010.
Muaj cov neeg phem nyob rau pem hauv ntej uas xav rhuav peb, tab sis nws muaj lub foob pob hluav taws. Tsis muaj firewall ntawm lub backend, tab sis muaj 50 servers nyob ntawd, peb paub lawv tag nrho. Txhua yam ua haujlwm zoo.
Nyob rau hauv 4 xyoo, lub server fleet loj hlob 100 lub sij hawm, mus rau 5000. Thawj qhov kev sib cais network tshwm sim - staging: lawv tsis tuaj yeem mus rau kev tsim khoom, thiab feem ntau muaj ntau yam khiav mus rau qhov uas yuav ua rau muaj kev phom sij.
Network qauv hauv 2014.
Los ntawm inertia, peb siv tib yam khoom siv kho vajtse, thiab tag nrho cov haujlwm tau ua tiav ntawm VLANs cais: ACLs tau sau rau VLANs, uas tso cai lossis tsis lees paub qee yam kev sib txuas.
Xyoo 2016, tus naj npawb ntawm cov servers mus txog 8000. Wargaming absorbed lwm lub studios, thiab cov koom tes ntxiv tau tshwm sim. Lawv zoo li peb, tab sis tsis heev: VLAN feem ntau tsis ua haujlwm rau cov neeg koom tes, koj yuav tsum siv VPN nrog VRF, kev cais tawm yuav nyuaj dua. ACL rwb thaiv tsev sib tov loj hlob.
Network qauv hauv 2016.
Thaum pib ntawm 2018, lub nkoj ntawm cov tshuab tau loj hlob mus rau 16. Muaj 000 ntu, thiab peb tsis suav cov seem, suav nrog cov kaw uas cov ntaub ntawv nyiaj txiag tau khaws cia. Thawv tes hauj lwm (Kubernetes), DevOps, huab kev sib txuas ntawm VPN, piv txwv li, los ntawm IVS, tau tshwm sim. Muaj ntau txoj cai - nws mob heev.
Network qauv thiab kev cais tawm hauv 2018.
Rau kev sib cais peb siv: VLAN nrog ACL ntawm L2, VRF nrog ACL ntawm L3, VPN thiab ntau ntxiv. Ntau dhau lawm.
Teeb meem
Txhua tus nyob nrog ACL thiab VLAN. Dab tsi tsis yog? Lo lus nug no yuav teb los ntawm Harold, zais qhov mob.
Muaj ntau yam teeb meem, tab sis muaj tsib qhov loj heev.
- Geometric nqi nce rau cov cai tshiab. Txhua txoj cai tshiab tau siv sijhawm ntev dua los ntxiv dua li yav dhau los, vim tias nws yuav tsum xub saib seb puas muaj txoj cai ntawd lawm.
- Tsis muaj firewall hauv ntu. Cov ntu tau sib cais los ntawm ib leeg, thiab twb tsis muaj peev txheej txaus hauv.
- Cov kev cai raug siv rau lub sijhawm ntev. Cov neeg ua haujlwm tuaj yeem sau ib txoj cai hauv zos los ntawm tes hauv ib teev. Lub ntiaj teb no siv ob peb hnub.
- Teeb meem nrog kev txheeb xyuas cov cai. Ntau precisely, nws tsis tau. Thawj txoj cai tau sau rov qab rau xyoo 2010, thiab feem ntau ntawm lawv cov kws sau ntawv tsis ua haujlwm rau lub tuam txhab lawm.
- Qib qis ntawm kev tswj hwm infrastructure. Qhov no yog qhov teeb meem tseem ceeb - peb tsis paub zoo tias muaj dab tsi tshwm sim hauv peb lub tebchaws.
Nov yog qhov uas tus kws tshaj lij network zoo li hauv 2018 thaum nws hnov: "Xav tau qee qhov ACL ntxiv."
Lub ntsiab
Thaum pib ntawm 2018, nws tau txiav txim siab ua ib yam dab tsi txog nws.
Tus nqi ntawm kev sib koom ua ke yog pheej loj zuj zus. Lub ntsiab lus pib yog tias cov chaw zov me nyuam loj tau tso tseg kev sib cais VLANs thiab ACLs vim tias cov khoom siv khiav tawm ntawm lub cim xeeb.
Kev daws: peb tshem tawm tib neeg qhov tseem ceeb thiab automated muab kev nkag mus rau qhov siab tshaj plaws.
Cov cai tshiab siv sijhawm ntev los siv. Kev daws: ua kom nrawm rau kev siv cov cai, ua kom nws faib thiab sib npaug. Qhov no yuav tsum muaj kev faib tawm kom cov kev cai raug xa mus rau lawv tus kheej, tsis muaj rsync lossis SFTP rau ib txhiab lub tshuab.
Tsis muaj firewall hauv ntu. Lub firewall nyob rau hauv ntu pib tuaj rau peb thaum cov kev pabcuam sib txawv tshwm sim hauv tib lub network. Kev daws: siv lub firewall ntawm tus tswv tsev - host-based firewalls. Yuav luag txhua qhov chaw peb muaj Linux, thiab txhua qhov chaw peb muaj iptables, qhov no tsis yog teeb meem.
Teeb meem nrog kev txheeb xyuas cov cai. Kev daws: Khaws tag nrho cov cai hauv ib qho chaw rau kev tshuaj xyuas thiab kev tswj xyuas, yog li peb tuaj yeem tshawb xyuas txhua yam.
Qib qis ntawm kev tswj hwm infrastructure. Kev daws: nqa ib daim ntawv teev tag nrho cov kev pabcuam thiab kev nkag ntawm lawv.
Qhov no yog ntau ntawm cov txheej txheem kev tswj hwm ntau dua li kev ua haujlwm. Qee lub sij hawm peb muaj 200-300 qhov kev tshaj tawm tshiab hauv ib lub lis piam, tshwj xeeb tshaj yog thaum muaj kev tshaj tawm thiab hnub so. Ntxiv mus, qhov no tsuas yog rau ib pab pawg ntawm peb DevOps. Nrog ntau qhov kev tshaj tawm, nws tsis tuaj yeem pom qhov chaw nres nkoj, IPs, thiab kev sib koom ua ke xav tau. Yog li ntawd, peb xav tau kev cob qhia tshwj xeeb rau cov thawj coj saib xyuas kev pabcuam uas nug cov pab pawg: "Yuav ua li cas nyob qhov twg thiab vim li cas koj thiaj li coj nws?"
Tom qab txhua yam peb tau pib, tus kws tshaj lij network hauv 2019 pib zoo li no.
Consul
Peb txiav txim siab tias peb yuav muab txhua yam uas peb pom nrog kev pab los ntawm cov thawj tswj kev pabcuam rau hauv Consul thiab los ntawm qhov ntawd peb yuav sau iptables cov cai.
Peb txiav txim siab ua li no li cas?
- Peb yuav sau tag nrho cov kev pabcuam, network thiab cov neeg siv.
- Cia peb tsim iptables cov cai raws li lawv.
- Peb automate tswj.
- ....
- NYIAJ.
Consul tsis yog tej thaj chaw deb API, nws tuaj yeem khiav ntawm txhua qhov ntawm thiab sau rau iptables. Txhua yam uas tseem tshuav yog los nrog kev tswj tsis siv neeg uas yuav ntxuav cov khoom tsis tsim nyog, thiab feem ntau ntawm cov teeb meem yuav raug daws! Peb yuav ua hauj lwm tawm thaum peb mus.
Vim li cas Consul?
Tau ua pov thawj nws tus kheej zoo. Hauv xyoo 2014-15, peb tau siv nws ua qhov backend rau Vault, uas peb khaws cov passwords.
Tsis poob cov ntaub ntawv. Thaum lub sijhawm siv, Consul tsis poob cov ntaub ntawv thaum muaj xwm txheej ib zaug. Qhov no yog ib qho ntxiv rau qhov kev tswj hwm firewall.
P2P kev sib txuas ua kom nrawm nrawm ntawm kev hloov pauv. Nrog P2P, txhua qhov kev hloov pauv tuaj sai sai, tsis tas yuav tos ntev teev.
Yooj Yim REST API. Peb kuj tau txiav txim siab Apache ZooKeeper, tab sis nws tsis muaj REST API, yog li koj yuav tsum tau nruab ib lub crutches.
Ua haujlwm raws li ob qho tseem ceeb Vault (KV) thiab Phau Ntawv Qhia (Service Discovery). Koj tuaj yeem khaws cov kev pabcuam, catalogs, thiab cov chaw khaws ntaub ntawv ib zaug. Qhov no yooj yim tsis yog rau peb xwb, tab sis kuj rau cov neeg nyob sib ze, vim tias thaum tsim lub ntiaj teb kev pabcuam, peb xav tias loj.
Sau hauv Go, uas yog ib feem ntawm Wargaming pawg. Peb nyiam hom lus no, peb muaj ntau tus neeg tsim tawm Go.
Hwj chim ACL system. Hauv Consul, koj tuaj yeem siv ACLs los tswj leej twg sau dab tsi. Peb lav tias cov kev cai firewall yuav tsis sib tshooj nrog lwm yam thiab peb yuav tsis muaj teeb meem nrog qhov no.
Tab sis Consul kuj muaj nws qhov tsis zoo.
- Tsis scale nyob rau hauv ib tug data center tshwj tsis yog tias koj muaj ib tug lag luam version. Nws tsuas yog scalable los ntawm federation.
- Heev nyob ntawm qhov zoo ntawm lub network thiab server load. Consul yuav tsis ua haujlwm zoo li tus neeg rau zaub mov ntawm lub server tsis khoom yog tias muaj kev lag luam hauv lub network, piv txwv li, tsis sib xws. Qhov no yog vim P2P kev sib txuas thiab hloov kho cov qauv faib khoom.
- Kev soj ntsuam tsis yooj yim. Hauv Consul xwm txheej nws tuaj yeem hais tias txhua yam zoo, tab sis nws tuag ntev dhau los.
Peb daws feem ntau ntawm cov teeb meem no thaum siv Consul, yog vim li cas peb xaiv nws. Lub tuam txhab muaj cov phiaj xwm rau lwm qhov backend, tab sis peb tau kawm los daws cov teeb meem thiab tam sim no nyob nrog Consul.
Consul ua haujlwm li cas
Peb yuav nruab peb mus rau tsib servers nyob rau hauv ib tug conditional data center. Ib lossis ob lub servers yuav tsis ua haujlwm: lawv yuav tsis tuaj yeem tsim pawg pawg thiab txiav txim siab tias leej twg yog thiab leej twg tsis yog thaum cov ntaub ntawv tsis sib xws. Ntau tshaj tsib ua rau tsis muaj kev nkag siab, kev tsim khoom yuav poob.
Cov neeg siv khoom txuas mus rau cov servers hauv ib qho kev txiav txim: tib tus neeg sawv cev, tsuas yog nrog tus chij server = false.
Tom qab no, cov neeg siv tau txais cov npe ntawm P2P kev sib txuas thiab tsim kev sib txuas ntawm lawv tus kheej.
Nyob rau hauv lub ntiaj teb no, peb txuas ob peb cov ntaub ntawv chaw. Lawv kuj txuas P2P thiab sib txuas lus.
Thaum peb xav khaws cov ntaub ntawv los ntawm lwm qhov chaw khaws ntaub ntawv, qhov kev thov mus ntawm server mus rau server. Lub tswv yim no hu ua Serf raws tu qauv. Serf raws tu qauv, zoo li Consul, yog tsim los ntawm HashiCorp.
Qee qhov tseem ceeb ntawm Consul
Consul muaj cov ntaub ntawv piav qhia nws ua haujlwm li cas. Kuv yuav muab qhov tseeb uas tsim nyog paub xwb.
Consul servers xaiv tus tswv los ntawm cov neeg pov npav. Consul xaiv tus tswv los ntawm cov npe ntawm cov servers rau txhua qhov chaw khaws ntaub ntawv, thiab txhua qhov kev thov tsuas yog mus rau nws, tsis hais tus naj npawb ntawm cov servers. Master freezing tsis ua rau rov xaiv dua. Yog tias tus tswv tsis raug xaiv, kev thov yuav tsis muab kev pabcuam los ntawm leej twg.
Koj puas xav tau kab rov tav scaling? Soorry, tsis muaj.
Ib qho kev thov mus rau lwm lub chaw cov ntaub ntawv mus los ntawm tus tswv mus rau tus tswv, tsis hais tus neeg rau zaub mov twg nws tuaj. Tus tswv xaiv tau txais 100% ntawm qhov thauj khoom, tshwj tsis yog rau qhov thauj khoom ntawm kev thov tom ntej. Tag nrho cov servers hauv cov chaw khaws ntaub ntawv muaj cov ntaub ntawv hloov tshiab, tab sis tsuas yog ib qho teb.
Tib txoj hauv kev los ntsuas qhov ntsuas yog txhawm rau ua kom muaj qhov tsis zoo ntawm tus neeg siv khoom.
Hauv hom stale, koj tuaj yeem teb yam tsis muaj pawg. Qhov no yog hom uas peb muab cov ntaub ntawv sib xws, tab sis nyeem me ntsis sai dua li niaj zaus, thiab txhua tus neeg rau zaub mov teb. Lawm, kaw tsuas yog los ntawm tus tswv.
Consul tsis luam cov ntaub ntawv ntawm cov chaw zov me nyuam. Thaum lub koomhaum koom ua ke, txhua tus neeg rau zaub mov yuav tsuas muaj nws cov ntaub ntawv xwb. Rau lwm tus, nws ib txwm tig mus rau lwm tus.
Atomicity ntawm kev ua haujlwm tsis tau lees paub sab nraud ntawm kev lag luam. Nco ntsoov tias koj tsis yog tib tug uas tuaj yeem hloov tau. Yog tias koj xav tau nws txawv, ua kev lag luam nrog lub xauv.
Kev ua haujlwm thaiv kev thaiv tsis tuaj yeem lav qhov xauv. Qhov kev thov mus los ntawm tus tswv mus rau tus tswv, thiab tsis ncaj qha, yog li tsis muaj kev lees paub tias kev thaiv yuav ua haujlwm thaum koj thaiv, piv txwv li, hauv lwm qhov chaw khaws ntaub ntawv.
ACL kuj tsis lav kev nkag tau (ntau zaus). ACL tej zaum yuav tsis ua haujlwm vim nws tau khaws cia rau hauv ib lub koom haum cov ntaub ntawv chaw zov me nyuam - hauv ACL data center (Primary DC). Yog tias DC tsis teb koj, ACL yuav tsis ua haujlwm.
Ib tug kws khov kho yuav ua rau tag nrho lub koom haum khov. Piv txwv li, muaj 10 lub chaw cov ntaub ntawv nyob rau hauv ib lub koom haum, thiab ib tug muaj ib tug phem network, thiab ib tug tswv tsis ua hauj lwm. Txhua tus neeg uas sib txuas lus nrog nws yuav raug daig hauv lub voj voog: muaj kev thov, tsis muaj lus teb rau nws, xov khov. Tsis muaj txoj hauv kev paub tias qhov no yuav tshwm sim thaum twg, tsuas yog hauv ib teev lossis ob lub koomhaum tag nrho yuav poob. Tsis muaj ib yam dab tsi uas koj ua tau txog nws.
Cov xwm txheej, pawg pawg thiab kev xaiv tsa raug tswj hwm los ntawm ib txoj xov sib cais. Kev xaiv tsa rov qab yuav tsis tshwm sim, qhov xwm txheej yuav tsis qhia dab tsi. Koj xav tias koj muaj Consul nyob, koj nug, thiab tsis muaj dab tsi tshwm sim - tsis muaj lus teb. Tib lub sijhawm, cov xwm txheej qhia tau hais tias txhua yam zoo.
Peb tau ntsib qhov teeb meem no thiab yuav tsum tau tsim kho qhov tshwj xeeb ntawm cov chaw zov me nyuam kom tsis txhob muaj nws.
Kev lag luam version ntawm Consul Enterprise tsis muaj qee qhov tsis zoo saum toj no. Nws muaj ntau yam haujlwm tseem ceeb: xaiv cov neeg xaiv tsa, faib, ntsuas. Muaj tsuas yog ib qho "tab sis" - daim ntawv tso cai rau kev faib tawm yog kim heev.
Lub neej nyiag nkas: rm -rf /var/lib/consul - kho rau txhua yam kab mob ntawm tus neeg sawv cev. Yog tias ib yam dab tsi tsis ua haujlwm rau koj, tsuas yog rho tawm koj cov ntaub ntawv thiab rub tawm cov ntaub ntawv los ntawm daim ntawv theej. Feem ntau, Consul yuav ua haujlwm.
BEFW
Tam sim no cia peb tham txog qhov peb tau ntxiv rau Consul.
yog ib tug acronym rau BackEndFKuv yuav musWtag nrho. Kuv yuav tsum sau npe rau cov khoom lag luam thaum kuv tsim qhov chaw cia khoom kom tso thawj qhov kev sim ua rau nws. Lub npe no tseem nyob.
Txoj cai templates
Cov cai tau sau rau hauv iptables syntax.
- -N PEB
- -P INPUT DROP
- -A INPUT -m xeev - xeev RELATED, ESTABLISHED -j ACCEPT
- -A INPUT -i lo -j ACCEPT
- -A INPUT -j BEFW
Txhua yam mus rau hauv BEFW saw, tshwj tsis yog ESTABLISHED, RELATED thiab localhost. Cov template tuaj yeem yog txhua yam, qhov no tsuas yog piv txwv xwb.
BEFW pab tau li cas?
Kev pabcuam
Peb muaj ib qho kev pabcuam, nws ib txwm muaj qhov chaw nres nkoj, ib qho ntawm nws khiav. Los ntawm peb cov node, peb tuaj yeem nug tus neeg sawv cev hauv zos thiab paub tias peb muaj qee yam kev pabcuam. Koj tuaj yeem tso cov cim npe.
Txhua qhov kev pabcuam uas tau khiav thiab sau npe nrog Consul hloov mus rau hauv txoj cai iptables. Peb muaj SSH - qhib chaw nres nkoj 22. Cov ntawv Bash yog qhov yooj yim: curl thiab iptables, tsis muaj dab tsi ntxiv.
Cov Neeg
Yuav ua li cas qhib kev nkag tsis tau rau txhua tus, tab sis xaiv? Ntxiv cov npe IP rau KV cia los ntawm lub npe kev pabcuam.
Piv txwv li, peb xav kom txhua tus neeg ntawm kaum lub network tuaj yeem nkag mus rau SSH_TCP_22 kev pabcuam. Ntxiv ib daim teb me me TTL? thiab tam sim no peb muaj daim ntawv tso cai ib ntus, piv txwv li, rau ib hnub.
Kev nkag
Peb txuas cov kev pabcuam thiab cov neeg siv khoom: peb muaj kev pabcuam, KV cia yog npaj rau txhua tus. Tam sim no peb muab kev nkag tsis tau rau txhua tus, tab sis xaiv.
Pawg
Yog tias peb sau ntau txhiab tus IP rau kev nkag mus txhua zaus, peb yuav nkees. Wb tuaj nrog pab pawg - cais subset hauv KV. Cia peb hu nws Alias ββββ(los yog pab pawg) thiab khaws cov pab pawg nyob ntawd raws li tib txoj cai.
Cia peb txuas: tam sim no peb tuaj yeem qhib SSH tsis yog rau P2P, tab sis rau tag nrho pawg lossis ob peb pawg. Ib yam li ntawd, muaj TTL - koj tuaj yeem ntxiv rau ib pab pawg thiab tshem tawm ntawm pab pawg ib ntus.
Kev koom ua ke
Peb qhov teeb meem yog tib neeg qhov tseem ceeb thiab automation. Txog tam sim no peb tau daws nws li no.
Peb ua haujlwm nrog Puppet, thiab hloov txhua yam uas cuam tshuam nrog lub kaw lus (daim ntawv thov code) rau lawv. Puppetdb (PostgreSQL tsis tu ncua) khaws cov npe ntawm cov kev pabcuam uas tau ua haujlwm nyob rau ntawd, lawv tuaj yeem pom los ntawm hom kev pabcuam. Nyob ntawd koj tuaj yeem nrhiav seb leej twg thov qhov twg. Peb kuj muaj kev thov rub thiab sib koom ua ke thov rau qhov no.
Peb sau befw-sync, ib qho kev daws teeb meem yooj yim uas pab hloov cov ntaub ntawv. Ua ntej, sync cookies yog nkag los ntawm puppetdb. HTTP API tau teeb tsa nyob ntawd: peb thov cov kev pabcuam peb muaj dab tsi, yuav tsum ua dab tsi. Tom qab ntawd lawv thov rau Consul.
Puas muaj kev sib koom ua ke? Yog: lawv tau sau cov cai thiab tso cai rub Cov Lus Thov kom lees txais. Koj puas xav tau ib qho chaw nres nkoj lossis ntxiv ib tus tswv tsev rau qee pawg? Rub Thov, tshuaj xyuas - tsis muaj ntxiv "Nrhiav 200 lwm ACLs thiab sim ua qee yam txog nws."
Optimization
Pinging localhost nrog txoj cai khoob khoob yuav siv 0,075 ms.
Cia peb ntxiv 10 iptables chaw nyob rau cov saw no. Raws li qhov tshwm sim, ping yuav nce 000 zaug: iptables yog linear tag nrho, ua txhua qhov chaw nyob yuav siv sij hawm.
Rau lub firewall uas peb tsiv ntau txhiab tus ACLs, peb muaj ntau txoj cai, thiab qhov no qhia txog latency. Qhov no tsis zoo rau kev ua si raws tu qauv.
Tab sis yog peb muab 10 chaw nyob hauv ipset Lub ping yuav txawm txo.
Lub ntsiab lus yog tias "O" (algorithm complexity) rau ipset yeej ib txwm sib npaug rau 1, txawm tias muaj pes tsawg txoj cai. Muaj tseeb, muaj kev txwv - tsis tuaj yeem muaj ntau tshaj 65535 txoj cai. Tam sim no peb nyob nrog qhov no: koj tuaj yeem muab lawv, nthuav lawv, ua ob ipsets hauv ib qho.
Cia
Ib qho laj thawj txuas ntxiv ntawm cov txheej txheem iteration yog khaws cov ntaub ntawv hais txog cov neeg siv khoom rau kev pabcuam hauv ipset.
Tam sim no peb muaj tib SSH, thiab peb tsis sau 100 IPs ib zaug, tab sis teeb lub npe ntawm ipset uas peb xav tau kev sib txuas lus, thiab cov cai hauv qab no DROP. Nws tuaj yeem hloov mus rau hauv ib txoj cai "Leej twg tsis nyob ntawm no, DROP", tab sis nws yog qhov tseeb dua.
Tam sim no peb muaj cov cai thiab cov txheej txheem. Lub luag haujlwm tseem ceeb yog ua kom muaj kev teeb tsa ua ntej sau txoj cai, vim tias txwv tsis pub iptables yuav tsis sau txoj cai.
Tus qauv tswv yim
Hauv daim duab kos duab, txhua yam kuv hais zoo li no.
Peb cog lus rau Puppet, txhua yam raug xa mus rau tus tswv tsev, cov kev pabcuam ntawm no, ipset muaj, thiab txhua tus neeg uas tsis tau sau npe nyob ntawd tsis tso cai.
Tso cai & tsis lees paub
Txhawm rau cawm lub ntiaj teb sai sai lossis cuam tshuam ib tus neeg sai sai, thaum pib ntawm txhua txoj hlua peb ua ob ipsets: rules_allow ΠΈ rules_deny. Nws ua haujlwm li cas?
Piv txwv li, ib tug neeg tab tom tsim kev thauj khoom ntawm peb lub vev xaib nrog bots. Yav dhau los, koj yuav tsum nrhiav nws tus IP los ntawm lub cav, coj mus rau network engineers, kom lawv thiaj li nrhiav tau qhov chaw ntawm cov tsheb khiav thiab txwv nws. Nws zoo li txawv tam sim no.
Peb xa nws mus rau Consul, tos 2,5 vib nas this, thiab nws ua tiav. Txij li thaum Consul faib sai sai los ntawm P2P, nws ua haujlwm txhua qhov chaw, hauv txhua qhov chaw ntawm lub ntiaj teb.
Ib zaug kuv ua li cas nres WOT vim qhov yuam kev nrog firewall. rules_allow - qhov no yog peb kev pov hwm tiv thaiv cov xwm txheej zoo li no. Yog tias peb ua yuam kev nyob qhov twg nrog firewall, ib yam dab tsi raug thaiv qhov chaw, peb tuaj yeem xa ib qho xwm txheej 0.0/0kom sai khaws txhua yam. Tom qab ntawd peb yuav kho txhua yam ntawm tes.
Lwm cov teeb
Koj tuaj yeem ntxiv lwm yam teeb tsa hauv qhov chaw $IPSETS$.
Rau dab tsi? Qee zaum ib tus neeg xav tau ipset, piv txwv li, ua raws li kev kaw ntawm qee qhov ntawm pawg. Txhua tus tuaj yeem nqa ib qho teeb meem, npe rau lawv, thiab lawv yuav raug khaws los ntawm Consul. Nyob rau tib lub sijhawm, cov teeb tsa tuaj yeem koom nrog iptables cov cai lossis ua haujlwm pab pawg NOOP: Kev sib raug zoo yuav raug tswj xyuas los ntawm daemon.
Cov Neeg Siv
Yav dhau los, nws zoo li no: tus neeg siv txuas nrog lub network thiab tau txais cov kev txwv los ntawm kev sau npe. Ua ntej qhov tshwm sim ntawm lub cim tshiab firewalls, Cisco tsis paub yuav ua li cas nkag siab tias tus neeg siv nyob qhov twg thiab IP nyob qhov twg. Yog li ntawd, kev nkag tau tsuas yog tso cai los ntawm lub hostname ntawm lub tshuab.
Peb ua dab tsi? Peb tau daig thaum lub sijhawm peb tau txais qhov chaw nyob. Feem ntau qhov no yog dot1x, Wi-Fi lossis VPN - txhua yam mus dhau RADIUS. Rau txhua tus neeg siv, peb tsim ib pab pawg los ntawm tus neeg siv lub npe thiab tso tus IP rau hauv nws nrog TTL uas sib npaug rau nws cov dhcp.lease - sai li sai tau thaum nws tas sijhawm, txoj cai yuav ploj mus.
Tam sim no peb tuaj yeem qhib kev nkag mus rau cov kev pabcuam, zoo li lwm pab pawg, los ntawm tus neeg siv lub npe. Peb tau tshem qhov mob tawm ntawm hostnames thaum lawv hloov, thiab peb tau coj lub nra ntawm network engineers vim lawv tsis xav tau Cisco lawm. Tam sim no engineers lawv tus kheej sau npe nkag rau ntawm lawv cov servers.
Rwb thaiv tsev
Nyob rau tib lub sij hawm, peb pib dismantle lub rwb thaiv tsev. Cov neeg saib xyuas kev pabcuam tau muab cov khoom muag, thiab peb txheeb xyuas tag nrho peb cov tes hauj lwm. Cia peb faib lawv rau hauv tib pab pawg, thiab ntawm cov servers tsim nyog cov pab pawg tau ntxiv, piv txwv li, tsis kam lees. Tam sim no tib qhov kev sib cais ntawm kev sib cais xaus rau hauv cov cai_tsis lees paub ntawm kev tsim khoom, tab sis tsis yog hauv kev tsim khoom nws tus kheej.
Cov tswv yim ua haujlwm sai thiab yooj yim: peb tshem tawm tag nrho ACLs ntawm cov servers, tshem tawm cov khoom kho vajtse, thiab txo cov VLANs cais.
Tswj kev ncaj ncees
Yav dhau los, peb muaj qhov tshwj xeeb txhais tau hais tias thaum ib tus neeg hloov pauv txoj cai firewall manually. Kuv tau sau ib daim ntawv loj loj rau kev kuaj xyuas cov cai ntawm firewall, nws nyuaj heev. Kev ncaj ncees yog tam sim no tswj los ntawm BEFW. Nws mob siab ua kom cov kev cai uas nws ua tsis hloov. Yog tias ib tug neeg hloov cov cai ntawm firewall, nws yuav hloov txhua yam rov qab. "Kuv tau teeb tsa tus neeg sawv cev sai sai kom kuv tuaj yeem ua haujlwm hauv tsev" - tsis muaj ntau txoj kev xaiv.
BEFW tswj cov ipset los ntawm cov kev pabcuam thiab cov npe hauv befw.conf, cov cai ntawm cov kev pabcuam hauv BEFW saw. Tab sis nws tsis saib xyuas lwm cov saw thiab cov cai thiab lwm yam ipsets.
Kev tiv thaiv kev sib tsoo
BEFW ib txwm khaws lub xeem paub zoo ncaj qha hauv state.bin binary qauv. Yog hais tias ib yam dab tsi mus tsis ncaj ncees lawm, nws ib txwm rolls rov qab mus rau lub state.bin.
Qhov no yog kev pov hwm tiv thaiv kev lag luam tsis ruaj khov, thaum nws tsis xa cov ntaub ntawv lossis ib tus neeg ua yuam kev thiab siv cov cai uas tsis tuaj yeem siv. Txhawm rau kom ntseeg tau tias peb tsis raug tso tseg yam tsis muaj firewall, BEFW yuav rov qab mus rau lub xeev tshiab yog tias muaj qhov yuam kev tshwm sim ntawm txhua theem.
Hauv cov xwm txheej tseem ceeb, qhov no yog qhov lav tias peb yuav raug tso tseg nrog lub foob pob ua haujlwm. Peb qhib tag nrho cov tes hauj lwm grey nyob rau hauv kev cia siab tias tus admin yuav tuaj thiab kho nws. Ib hnub kuv yuav muab qhov no tso rau hauv kev teeb tsa, tab sis tam sim no peb tsuas yog muaj peb lub network grey: 10/8, 172/12 thiab 192.168/16. Hauv peb tus Consul, qhov no yog qhov tseem ceeb uas pab peb txhim kho ntxiv.
Demo: thaum lub sij hawm tsab ntawv ceeb toom, Ivan qhia lub demo hom ntawm BEFW. Nws yooj yim dua los saib qhov ua qauv qhia . Demo qhov chaws muaj .
kev cov nyom
Kuv mam li qhia koj txog cov kab uas peb ntsib.
ipset ntxiv teeb 0.0.0.0/0. Yuav ua li cas yog tias koj ntxiv 0.0.0.0/0 rau ipset? Puas yuav tag nrho IPs ntxiv? Kev siv Internet puas yuav muaj?
Tsis yog, peb yuav tau txais kab laum uas ua rau peb poob ob teev. Ntxiv mus, tus kab mob tsis tau ua haujlwm txij li xyoo 2016, nws nyob hauv RedHat Bugzilla raws li tus lej #1297092, thiab peb pom nws los ntawm kev sib tsoo - los ntawm tus tsim tawm daim ntawv tshaj tawm.
Tam sim no nws yog txoj cai nruj ntawm BEFW uas 0.0.0.0/0 hloov mus ua ob qhov chaw nyob: 0.0.0.0/1 ΠΈ 128.0.0.0/1.
ipset restore teeb <file. ipset ua dab tsi thaum koj qhia nws rau restore? Koj puas xav tias nws ua haujlwm zoo ib yam li iptables? Nws puas yuav rov qab tau cov ntaub ntawv?
Tsis muaj dab tsi zoo li ntawd - nws ua kev sib koom ua ke, thiab qhov chaw nyob qub tsis mus qhov twg, koj tsis thaiv kev nkag.
Peb pom muaj kab mob thaum kuaj kev sib cais. Tam sim no muaj ib tug heev complex system - es tsis txhob restore tuav create tempntawd restore flush temp ΠΈ restore temp. Thaum kawg ntawm kev sib pauv: rau atomicity, vim yog koj ua nws ua ntej flush thiab lub sijhawm no qee pob ntawv tuaj txog, nws yuav raug muab pov tseg thiab qee yam yuav mus tsis ncaj ncees lawm. Yog li ntawd muaj me ntsis ntawm cov khawv koob dub nyob ntawd.
consul kv get -datacenter=lwm. Raws li kuv tau hais, peb xav tias peb tab tom thov qee cov ntaub ntawv, tab sis peb yuav tau txais cov ntaub ntawv lossis qhov yuam kev. Peb tuaj yeem ua qhov no ntawm Consul hauv zos, tab sis qhov no ob qho tib si yuav khov.
Tus neeg siv Consul hauv zos yog wrapper hla HTTP API. Tab sis nws tsuas yog dai thiab tsis teb rau Ctrl + C, lossis Ctrl + Z, lossis txhua yam, nkaus xwb kill -9 nyob rau hauv lub console tom ntej. Peb ntsib qhov no thaum peb tab tom tsim ib pawg loj. Tab sis peb tseem tsis tau muaj kev daws teeb meem; peb tab tom npaj kho qhov yuam kev no hauv Consul.
Tus thawj coj Consul tsis teb. Peb tus tswv hauv lub chaw khaws ntaub ntawv tsis teb, peb xav tias: "Tej zaum qhov kev xaiv rov xaiv dua yuav ua haujlwm tam sim no?"
Tsis yog, nws yuav tsis ua haujlwm, thiab kev saib xyuas yuav tsis pom dab tsi: Tus Kws Tshaj Lij yuav hais tias muaj qhov kev cog lus, tus thawj coj tau pom, txhua yam zoo.
Peb yuav ua li cas nrog qhov no? service consul restart hauv cron txhua teev. Yog tias koj muaj 50 servers, tsis muaj teeb meem loj. Thaum muaj 16 ntawm lawv, koj yuav nkag siab tias nws ua haujlwm li cas.
xaus
Yog li ntawd, peb tau txais cov txiaj ntsig hauv qab no:
- 100% kev pabcuam ntawm txhua lub tshuab Linux.
- Ceev.
- Automation.
- Peb tso cov cuab yeej kho vajtse thiab network engineers los ntawm kev ua cev qhev.
- Kev sib koom ua ke tau tshwm sim uas yuav luag tsis muaj kev txwv: txawm tias nrog Kubernetes, txawm tias nrog Ansible, txawm tias nrog Python.
Daim ntawv: Consul, uas peb tam sim no yuav tsum tau nyob, thiab tus nqi siab heev ntawm kev ua yuam kev. Ua piv txwv, ib zaug thaum 6 teev tsaus ntuj (lub sijhawm tseem ceeb hauv tebchaws Russia) Kuv tau kho qee yam hauv cov npe ntawm tes hauj lwm. Peb tsuas yog tsim cov rwb thaiv tsev ntawm BEFW thaum lub sijhawm. Kuv ua yuam kev nyob qhov twg, zoo li kuv qhia lub npog ntsej muag tsis raug, tab sis txhua yam poob hauv ob vib nas this. Kev saib xyuas teeb pom kev, tus neeg txhawb nqa ntawm lub luag haujlwm los khiav: "Peb muaj txhua yam!" Tus thawj coj ntawm lub tuam tsev tau tig grey thaum nws piav qhia rau lub lag luam vim li cas qhov no tshwm sim.
Tus nqi ntawm kev ua yuam kev yog siab heev uas peb tau tuaj nrog peb tus kheej txoj kev tiv thaiv kev nyuaj. Yog tias koj siv qhov no rau ntawm qhov chaw tsim khoom loj, koj tsis tas yuav muab tus tswv token tshaj Consul rau txhua tus. Qhov no yuav xaus phem.
Tus nqi Kuv sau code rau 400 teev ib leeg. Kuv pab neeg ntawm 4 tus neeg siv 10 teev hauv ib hlis ntawm kev txhawb nqa rau txhua tus. Piv nrog rau tus nqi ntawm ib lub cim tshiab firewall, nws yog dawb.
Kev npaj. Txoj kev npaj mus sij hawm ntev yog nrhiav lwm txoj kev thauj mus los los hloov lossis ntxiv Consul. Tej zaum nws yuav yog Kafka lossis qee yam zoo sib xws. Tab sis nyob rau hauv lub xyoo tom ntej peb yuav nyob ntawm Consul.
Cov phiaj xwm tam sim no: kev koom ua ke nrog Fail2ban, nrog kev saib xyuas, nrog nftables, tejzaum nws nrog lwm yam kev faib tawm, kev ntsuas, kev soj ntsuam siab, kev ua kom zoo. Kubernetes kev txhawb nqa kuj yog qhov chaw hauv cov phiaj xwm, vim tias tam sim no peb muaj ob peb pawg thiab lub siab xav.
Ntau yam ntawm cov phiaj xwm:
- nrhiav kev tsis txaus ntseeg hauv kev khiav tsheb;
- network map tswj;
- Kubernetes txhawb nqa;
- assembling tej pob khoom rau tag nrho cov tshuab;
- Web-UI.
Peb niaj hnub ua haujlwm ntawm kev nthuav dav cov kev teeb tsa, nce metrics thiab optimization.
Koom nrog qhov project. Qhov project tau ua kom txias, tab sis, hmoov tsis, nws tseem yog ib qhov project. Los rau thiab sim ua ib yam dab tsi: cog lus, sim, qhia ib yam dab tsi, muab koj qhov kev ntsuam xyuas.
Lub caij no peb tab tom npaj rau , uas yuav muaj nyob rau lub Plaub Hlis 6 thiab 7 nyob rau hauv St. Petersburg, thiab peb caw developers ntawm high-load systems . Cov neeg hais lus paub twb paub yuav ua li cas, tab sis rau cov neeg tshiab hais lus peb xav kom tsawg kawg . Kev koom nrog hauv lub rooj sib tham ua tus hais lus muaj ntau yam zoo. Koj tuaj yeem nyeem qhov twg, piv txwv li, thaum kawg .
Tau qhov twg los: www.hab.com