Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?

Nco tseg. txhais.: Cov ntaub ntawv no yog los ntawm ib txoj haujlwm kev kawm kawm 8s yog cov lus teb rau cov lus nug nrov thaum tsim Kubernetes-based infrastructure. Peb cia siab tias cov lus piav qhia meej ntawm qhov zoo thiab qhov tsis zoo ntawm txhua qhov kev xaiv yuav pab koj xaiv qhov zoo tshaj plaws rau koj qhov project.

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?

TL; DR: Tib txheej ntawm cov khoom ua haujlwm tuaj yeem khiav ntawm ob peb pawg loj (txhua pawg yuav muaj ntau cov haujlwm ua haujlwm) lossis rau ntau qhov me me (nrog rau cov haujlwm me me hauv txhua pawg).

Hauv qab no yog ib lub rooj uas ntsuas qhov zoo thiab qhov tsis zoo ntawm txhua txoj hauv kev:

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?

Thaum siv Kubernetes ua lub platform rau kev khiav haujlwm, ntau cov lus nug tseem ceeb feem ntau tshwm sim txog qhov tsis sib xws ntawm kev teeb tsa pawg:

  • Kuv yuav siv pes tsawg pawg?
  • Kuv yuav ua kom lawv loj npaum li cas?
  • Txhua pawg yuav tsum suav nrog dab tsi?

Hauv tsab xov xwm no, kuv yuav sim teb tag nrho cov lus nug no los ntawm kev txheeb xyuas qhov zoo thiab qhov tsis zoo ntawm txhua txoj hauv kev.

Nqe lus nug

Raws li tus tsim software, koj yuav tsim thiab ua haujlwm ntau daim ntawv thov tib lub sijhawm.

Tsis tas li ntawd, ntau zaus ntawm cov ntawv thov no zoo li yuav khiav hauv ib puag ncig sib txawv - piv txwv li, cov no yuav yog dev, xeem ΠΈ khoom.

Qhov tshwm sim yog tag nrho cov matrix ntawm kev siv thiab ib puag ncig:

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?
Kev siv thiab ib puag ncig

Cov piv txwv saum toj no sawv cev rau 3 daim ntawv thov thiab 3 ib puag ncig, ua rau tag nrho ntawm 9 qhov kev xaiv tau.

Txhua daim ntawv thov piv txwv yog ib lub chaw xa tawm tus kheej uas tuaj yeem ua haujlwm nrog nws tus kheej ntawm lwm tus.

nco ntsoov tias ua piv txwv tuaj yeem muaj ntau yam, xws li frontend, backend, database, thiab lwm yam. Nyob rau hauv cov ntaub ntawv ntawm daim ntawv thov microservices, qhov piv txwv yuav suav nrog tag nrho cov microservices.

Yog li ntawd, cov neeg siv Kubernetes muaj ntau cov lus nug:

  • Puas yuav tsum tau muab tso rau hauv ib pawg?
  • Puas tsim nyog muaj ib pawg cais rau txhua daim ntawv thov piv txwv?
  • Los yog kab tias kev sib xyaw ua ke ntawm cov txheej txheem saum toj no yuav tsum tau siv?

Tag nrho cov kev xaiv no muaj txiaj ntsig zoo, txij li Kubernetes yog qhov hloov tau yooj yim uas tsis txwv tus neeg siv lub peev xwm.

Nov yog qee qhov ua tau:

  • ib pawg loj loj;
  • ntau me me tshwj xeeb pawg;
  • ib pawg rau ib daim ntawv thov;
  • ib pawg rau ib puag ncig.

Raws li qhia hauv qab no, thawj ob txoj hauv kev yog nyob ntawm qhov kawg ntawm qhov ntsuas ntawm cov kev xaiv:

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?
Los ntawm ob peb pawg loj (sab laug) mus rau ntau qhov me me (txoj cai)

Feem ntau, ib pawg yog suav tias yog "loj" dua li lwm qhov yog tias nws muaj ntau dua ntawm cov nodes thiab pods. Piv txwv li, ib pawg nrog 10 nodes thiab 100 pods loj dua ib pawg nrog 1 node thiab 10 pods.

Zoo, cia peb pib!

1. Ib pawg loj loj

Thawj qhov kev xaiv yog tso tag nrho cov haujlwm ua haujlwm hauv ib pawg:

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?
Ib pawg loj

Nyob rau hauv txoj kev no, pawg yog siv raws li universal infrastructure platform - koj tsuas yog xa txhua yam koj xav tau hauv pawg Kubernetes uas twb muaj lawm.

Namespaces Kubernetes tso cai rau qee qhov ntawm pawg kom sib cais los ntawm ib leeg, yog li txhua daim ntawv thov tuaj yeem muaj nws tus kheej lub npe.

Cia peb saib qhov zoo thiab qhov tsis zoo ntawm txoj kev no.

+ Kev siv cov peev txheej zoo

Nrog ib pab pawg, koj tsuas yog xav tau ib daim qauv ntawm tag nrho cov peev txheej xav tau los khiav thiab tswj cov pab pawg Kubernetes.

Piv txwv li, qhov no yog qhov tseeb rau tus tswv ntawm cov nodes. Feem ntau, txhua lub Kubernetes pawg muaj 3 tus tswv, yog li rau ib pawg ntawm lawv tus lej yuav nyob li ntawd (rau kev sib piv, 10 pawg yuav xav tau 30 tus tswv nodes).

Lub subtlety saum toj no tseem siv tau rau lwm cov kev pabcuam ua haujlwm thoob plaws tag nrho pawg, xws li cov khoom sib npaug, Ingress controllers, authentication, loggging thiab saib xyuas cov tshuab.

Hauv ib pawg, tag nrho cov kev pabcuam no tuaj yeem siv tau ib zaug rau txhua qhov haujlwm (tsis tas yuav tsim cov ntawv luam ntawm lawv, zoo li muaj ntau pawg).

+ Pheej yig

Raws li qhov tshwm sim ntawm cov saum toj no, cov pawg tsawg feem ntau pheej yig dua vim tias tsis muaj cov nqi them nyiaj ntau dua.

Qhov no yog qhov tseeb tshwj xeeb tshaj yog rau cov tswv ntawm cov nodes, uas tuaj yeem raug nqi ntau ntawm cov nyiaj tsis hais lawv yuav ua li cas (hauv tsev lossis hauv huab).

Qee qhov kev tswj xyuas Kubernetes, xws li Google Kubernetes Cav (GKE) los yog Azure Kubernetes Service (AKS), muab cov txheej txheem tswj dawb. Hauv qhov no, qhov teeb meem ntawm tus nqi tsis tshua muaj mob.

Kuj tseem muaj cov kev pabcuam tswj hwm uas them tus nqi tas rau kev ua haujlwm ntawm txhua lub Kubernetes pawg (piv txwv li, Amazon Elastic Kubernetes Service, EKS).

+ Kev tswj hwm zoo

Tswj ib pawg yog yooj yim dua li tswj ob peb.

Kev tswj hwm yuav suav nrog cov haujlwm hauv qab no:

  • Kubernetes version hloov tshiab;
  • teeb tsa CI/CD pipeline;
  • txhim kho CNI plugin;
  • teeb tsa tus neeg siv qhov system authentication;
  • installation ntawm ib tug access controller;

thiab ntau lwm tus…

Nyob rau hauv cov ntaub ntawv ntawm ib pawg, koj yuav tau ua tag nrho cov no ib zaug xwb.

Rau ntau pawg, kev ua haujlwm yuav tsum tau rov ua dua ntau zaus, uas yuav xav tau qee qhov kev ua haujlwm ntawm cov txheej txheem thiab cov cuab yeej los xyuas kom meej qhov sib xws thiab sib xws hauv cov txheej txheem.

Thiab tam sim no ob peb lo lus hais txog cons.

- Ib qho kev ua tsis tiav

Thaum tsis kam ib leeg xwb pawg yuav tsum tsis ua haujlwm tam sim ntawd tag nrho ua haujlwm!

Muaj ntau txoj hauv kev tuaj yeem ua yuam kev:

  • Kev hloov kho Kubernetes ua rau muaj kev phom sij tsis txaus ntseeg;
  • Ib pawg dav dav (piv txwv li, CNI plugin) pib tsis ua haujlwm raws li qhov xav tau;
  • ib qho ntawm pawg pawg tsis raug teeb tsa kom raug;
  • tsis ua hauj lwm nyob rau hauv lub hauv paus infrastructure.

Ib qho xwm txheej zoo li no tuaj yeem ua rau muaj kev puas tsuaj loj rau tag nrho cov haujlwm uas tau tuav hauv pawg sib koom.

- Tsis muaj rwb thaiv tsev nruj

Kev khiav hauv ib pawg sib koom txhais tau hais tias cov ntawv thov sib qhia cov khoom siv kho vajtse, kev muaj peev xwm sib tham, thiab kev ua haujlwm ntawm cov pawg nodes.

Hauv kev nkag siab, ob lub thawv nrog ob daim ntawv thov sib txawv khiav ntawm tib lub node zoo li ob txheej txheem khiav ntawm tib lub tshuab khiav tib lub OS kernel.

Linux ntim muab qee hom kev sib cais, tab sis nws tsis yog ze li qhov muaj zog raws li tau muab los ntawm, hais, cov tshuab virtual. Nyob rau hauv qhov tseeb, ib tug txheej txheem nyob rau hauv ib lub thawv yog tib cov txheej txheem khiav ntawm lub host operating system.

Qhov no tuaj yeem yog qhov teeb meem kev nyab xeeb: qhov kev npaj no theoretically tso cai rau cov ntawv thov tsis cuam tshuam nrog kev sib txuas lus (xws li txhob txwm los yog yuam kev).

Tsis tas li ntawd, tag nrho cov haujlwm ua haujlwm hauv Kubernetes pawg qhia qee qhov kev pabcuam thoob plaws pawg xws li DNS - qhov no tso cai rau cov ntawv thov kom pom Cov Kev Pabcuam ntawm lwm cov ntawv thov hauv pawg.

Tag nrho cov ntsiab lus saum toj no yuav muaj lub ntsiab lus sib txawv nyob ntawm daim ntawv thov kev ruaj ntseg.

Kubernetes muab ntau yam cuab yeej los tiv thaiv teeb meem kev nyab xeeb xws li PodSecurityPolicies ΠΈ NetworkPolicies. Txawm li cas los xij, kev teeb tsa lawv kom raug yuav tsum muaj qee qhov kev paub dhau los; ntxiv rau, lawv tsis tuaj yeem kaw tag nrho qhov kev nyab xeeb kiag li.

Nws yog ib qho tseem ceeb uas yuav tsum nco ntsoov tias Kubernetes yog thawj tsim los rau sib koom, tsis yog rau kev cais thiab kev nyab xeeb.

- Tsis muaj kev txwv nruj heev

Muab ntau qhov kev sib koom ua ke hauv Kubernetes pawg, muaj ntau txoj hauv kev uas cov ntawv thov sib txawv tuaj yeem ua rau ib leeg ntiv taw.

Piv txwv li, ib daim ntawv thov yuav monopolize cov peev txheej sib koom (xws li CPU lossis lub cim xeeb) thiab tsis kam lees lwm cov ntawv thov uas khiav ntawm tib lub node nkag mus rau nws.

Kubernetes muab ntau lub tswv yim los tswj tus cwj pwm no, xws li kev thov thiab kev txwv (saib kab lus β€œ CPU txwv thiab nruj throttling hauv Kubernetes "- kwv yees. txhais.), ResourceQuotas ΠΈ LimitRanges. Txawm li cas los xij, raws li nyob rau hauv cov ntaub ntawv ntawm kev ruaj ntseg, lawv configuration yog heev tsis tseem ceeb thiab lawv tsis muaj peev xwm tiv thaiv kiag li tag nrho unforeseen phiv.

- Ntau tus neeg siv

Nyob rau hauv rooj plaub ntawm ib pawg, koj yuav tsum qhib kev nkag mus rau nws rau ntau tus neeg. Thiab qhov lawv tus lej ntau dua, qhov kev pheej hmoo siab dua uas lawv yuav "tawg" ib yam dab tsi.

Hauv pawg koj tuaj yeem tswj tau leej twg tuaj yeem ua dab tsi siv Lub luag haujlwm-raws li kev nkag mus tswj (RBAC) (saib tsab xov xwm β€œ Cov neeg siv thiab tso cai RBAC hauv Kubernetes "- kwv yees. txhais.). Txawm li cas los xij, nws yuav tsis tiv thaiv cov neeg siv los ntawm "tawg" ib yam dab tsi hauv thaj tsam ntawm lawv lub luag haujlwm.

- Cov pawg tsis tuaj yeem loj hlob mus ib txhis

Cov pawg uas siv rau txhua qhov haujlwm yuav zoo li loj heev (raws li tus naj npawb ntawm cov nodes thiab pods).

Tab sis ntawm no lwm qhov teeb meem tshwm sim: pawg hauv Kubernetes tsis tuaj yeem loj hlob mus ib txhis.

Muaj qhov theoretical txwv ntawm pawg loj. Hauv Kubernetes nws yog kwv yees li 5000 nodes, 150 txhiab pods thiab 300 txhiab ntim.

Txawm li cas los xij, hauv lub neej tiag tiag, teeb meem tuaj yeem pib ua ntej - piv txwv li, tsuas yog nrog 500 knots.

Qhov tseeb yog tias cov pawg loj tso cov khoom hnyav ntawm Kubernetes tswj txheej. Hauv lwm lo lus, ua kom cov pawg nce thiab khiav kom zoo yuav tsum tau ceev faj.

Qhov teeb meem no tau tshawb nrhiav hauv ib tsab xov xwm ntsig txog ntawm tus thawj blog hu ua "Architecting Kubernetes pawg - xaiv tus neeg ua haujlwm ntawm qhov loj me".

Tab sis cia peb xav txog qhov sib txawv: ntau pawg me.

2. Ntau qhov me me, tshwj xeeb pawg

Nrog rau txoj hauv kev no, koj siv ib pawg cais rau txhua qhov koj siv:

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?
Ntau pawg me me

Rau lub hom phiaj ntawm tsab xov xwm no, hauv qab deployable element hais txog ib qho piv txwv ntawm ib daim ntawv thov - piv txwv li, ib tug dev version ntawm ib daim ntawv thov nyias.

Lub tswv yim no siv Kubernetes ua tus tshwj xeeb lub sijhawm ua haujlwm rau tus kheej daim ntawv thov piv txwv.

Cia peb saib qhov zoo thiab qhov tsis zoo ntawm txoj kev no.

+ Txwv "lub voj voog tawg"

Thaum ib pawg ua tsis tiav, cov txiaj ntsig tsis zoo tsuas yog txwv rau cov haujlwm uas tau muab tso rau hauv pawg ntawd. Tag nrho lwm cov haujlwm ua haujlwm tseem tsis tau kov.

+ Rwb thaiv tsev

Cov chaw ua haujlwm tuav hauv ib pawg neeg tsis sib koom cov peev txheej xws li processor, nco, kev khiav haujlwm, network, lossis lwm yam kev pabcuam.

Qhov tshwm sim yog kev sib cais nruj ntawm cov ntawv thov tsis cuam tshuam, uas tuaj yeem muaj txiaj ntsig zoo rau lawv txoj kev nyab xeeb.

+ Cov neeg siv tsawg

Muab hais tias txhua pawg tsuas muaj ib qho kev txwv ntawm kev ua haujlwm, tus naj npawb ntawm cov neeg siv nkag mus rau nws raug txo.

Cov neeg tsawg dua tau nkag mus rau pawg, qhov tsawg dua qhov kev pheej hmoo tias qee yam yuav "lov".

Cia peb saib cov cons.

- Kev siv tsis zoo ntawm cov peev txheej

Raws li tau hais ua ntej lawm, txhua tus Kubernetes pawg yuav tsum muaj cov txheej txheem kev tswj hwm tshwj xeeb: cov tswv ntawm cov txheej txheem, kev tswj cov txheej txheem, kev saib xyuas thiab kev daws teeb meem.

Nyob rau hauv cov ntaub ntawv ntawm ib tug loj tus naj npawb ntawm cov pab pawg me, ib feem loj ntawm cov peev txheej yuav tsum tau muab faib rau kev tswj hwm.

- Nqis

Kev siv tsis zoo ntawm cov peev txheej tau txais cov nqi siab.

Piv txwv li, tuav 30 tus tswv nodes es tsis txhob ntawm peb nrog tib lub computer fais fab yuav tsum cuam tshuam cov nqi.

- Kev nyuaj hauv kev tswj hwm

Kev tswj ntau pawg Kubernetes yog qhov nyuaj dua li kev tswj hwm ib leeg xwb.

Piv txwv li, koj yuav tau teeb tsa kev lees paub thiab kev tso cai rau txhua pawg. Kubernetes version kuj tseem yuav tau hloov kho ob peb zaug.

Koj yuav xav tau siv automation los ua kom tag nrho cov haujlwm no ua haujlwm zoo dua.

Tam sim no cia saib cov xwm txheej tsawg dua.

3. Ib pawg rau ib daim ntawv thov

Hauv txoj hauv kev no, koj tsim ib pawg cais rau txhua qhov xwm txheej ntawm ib daim ntawv thov tshwj xeeb:

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?
Cluster rau ib daim ntawv thov

Txoj kev no tuaj yeem suav hais tias yog qhov kev nthuav dav ntawm lub hauv paus ntsiab lus "cais pawg rau ib pab neeg”, txij li feem ntau pab pawg engineers tab tom tsim ib lossis ntau daim ntawv thov.

Cia peb saib qhov zoo thiab qhov tsis zoo ntawm txoj kev no.

+ Cov pawg tuaj yeem hloov kho rau daim ntawv thov

Yog tias daim ntawv thov muaj cov kev xav tau tshwj xeeb, lawv tuaj yeem siv rau hauv pawg tsis muaj kev cuam tshuam rau lwm pawg.

Cov kev xav tau no tuaj yeem suav nrog cov neeg ua haujlwm GPU, qee qhov CNI plugins, kev pabcuam mesh, lossis qee qhov kev pabcuam.

Txhua pawg tuaj yeem hloov kho raws li daim ntawv thov khiav hauv nws kom nws tsuas muaj yam xav tau.

- Ib puag ncig sib txawv hauv ib pawg

Qhov tsis zoo ntawm txoj hauv kev no yog tias daim ntawv thov piv txwv los ntawm ntau qhov chaw sib koom ua ke hauv tib pawg.

Piv txwv li, cov prod version ntawm daim ntawv thov khiav hauv tib pawg li dev version. Qhov no kuj txhais tau tias cov neeg tsim khoom lag luam ua haujlwm hauv tib pawg uas tsim cov ntawv thov ua haujlwm.

Yog hais tias, vim yog cov kev ua ntawm developers los yog glitches nyob rau hauv lub dev version, ib tug tsis ua hauj lwm tshwm sim nyob rau hauv pawg, ces cov prod version tej zaum yuav raug kev txom nyem ib yam nkaus - ib tug loj drawback ntawm no mus kom ze.

Thiab thaum kawg, qhov kawg scenario ntawm peb daim ntawv teev npe.

4. Ib pawg ib puag ncig

Qhov xwm txheej no suav nrog kev faib ib pawg cais rau txhua qhov chaw:

Tsim Kubernetes pawg: pes tsawg yuav tsum muaj?
Ib pawg rau ib puag ncig

Piv txwv li, tej zaum koj yuav muaj pawg dev, xeem ΠΈ khoom, nyob rau hauv uas koj yuav khiav tag nrho cov piv txwv ntawm daim ntawv thov rau ib tug tshwj xeeb ib puag ncig.

Ntawm no yog cov pros thiab cons ntawm txoj kev no.

+ Kev rho tawm ntawm cov khoom ib puag ncig

Nyob rau hauv txoj kev no, txhua qhov chaw ib puag ncig yog cais los ntawm ib leeg. Txawm li cas los xij, hauv kev xyaum qhov no tseem ceeb tshwj xeeb tshaj yog nyob rau hauv ib puag ncig khoom.

Ntau cov qauv ntawm daim ntawv thov tam sim no tsis muaj dab tsi tshwm sim hauv lwm pawg thiab ib puag ncig.

Txoj kev no, yog tias muaj teeb meem tam sim ntawd tshwm sim hauv pawg dev, cov khoom tsim tawm ntawm cov ntawv thov yuav ua haujlwm txuas ntxiv zoo li tsis muaj dab tsi tshwm sim.

+ Cov pawg tuaj yeem hloov kho rau ib puag ncig

Txhua pawg tuaj yeem hloov kho rau nws ib puag ncig. Piv txwv li, koj tuaj yeem:

  • nruab cov cuab yeej rau kev txhim kho thiab kev debugging hauv pawg dev;
  • nruab cov qauv xeem thiab cov cuab yeej hauv pawg xeem;
  • siv cov cuab yeej muaj zog dua thiab cov kev sib txuas hauv network hauv pawg khoom.

Qhov no tso cai rau koj los txhim kho kev ua haujlwm ntawm ob daim ntawv thov kev txhim kho thiab kev ua haujlwm.

+ Txwv kev nkag mus rau pawg tsim khoom

Qhov yuav tsum tau ua haujlwm ncaj qha nrog cov khoom lag luam tsis tshua muaj tshwm sim, yog li koj tuaj yeem txwv lub voj voog ntawm cov neeg uas muaj kev nkag mus rau nws.

Koj tuaj yeem mus ntxiv thiab tsis lees txais tib neeg nkag mus rau pawg no tag nrho, thiab ua txhua qhov kev xa tawm siv lub cuab yeej siv CI / CD. Xws li ib txoj hauv kev yuav txo qhov kev pheej hmoo ntawm tib neeg yuam kev raws nraim qhov nws muaj feem cuam tshuam.

Thiab tam sim no ob peb lo lus hais txog cons.

- Tsis muaj kev sib cais ntawm daim ntawv thov

Lub ntsiab tsis zoo ntawm txoj hauv kev yog qhov tsis muaj kev kho vajtse thiab kev sib cais ntawm kev siv.

Cov ntawv thov tsis cuam tshuam sib faib cov peev txheej hauv pawg: cov txheej txheem tseem ceeb, processor, nco, thiab qee qhov kev pabcuam.

Raws li tau hais, qhov no tuaj yeem ua rau muaj kev phom sij.

- Tsis muaj peev xwm los txheeb xyuas daim ntawv thov kev vam meej

Yog tias daim ntawv thov muaj cov cai tshwj xeeb, ces lawv yuav tsum txaus siab rau txhua pawg.

Piv txwv li, yog tias daim ntawv thov xav tau GPU, ces txhua pawg yuav tsum muaj tsawg kawg ib tus neeg ua haujlwm nrog GPU (txawm tias nws tsuas yog siv los ntawm daim ntawv thov ntawd).

Vim li ntawd, peb pheej pheej hmoo cov nqi ntau dua thiab siv tsis tau cov peev txheej.

xaus

Yog tias koj muaj cov ntawv thov tshwj xeeb, lawv tuaj yeem muab tso rau hauv ob peb pawg loj lossis ntau qhov me me.

Tsab xov xwm tham txog qhov zoo thiab qhov tsis zoo ntawm ntau txoj hauv kev, xws li los ntawm ib pawg thoob ntiaj teb mus rau ntau qhov me me thiab tshwj xeeb:

  • ib pawg loj dav dav;
  • ntau me me tshwj xeeb pawg;
  • ib pawg rau ib daim ntawv thov;
  • ib pawg rau ib puag ncig.

Yog li koj yuav tsum coj li cas?

Raws li ib txwm muaj, cov lus teb nyob ntawm cov ntaub ntawv siv: koj yuav tsum ntsuas qhov zoo thiab qhov tsis zoo ntawm kev sib txawv thiab xaiv qhov kev xaiv zoo tshaj plaws.

Txawm li cas los xij, qhov kev xaiv tsis txwv rau cov piv txwv saum toj no - koj tuaj yeem siv lawv ua ke!

Piv txwv li, koj tuaj yeem npaj ob peb pawg rau txhua pab pawg: pawg kev loj hlob (nyob rau hauv uas yuav muaj ib puag ncig dev ΠΈ xeem) thiab pawg rau ntau lawm (qhov chaw tsim khoom yuav nyob).

Raws li cov ntaub ntawv hauv tsab xov xwm no, koj tuaj yeem kho qhov zoo thiab qhov tsis zoo raws li qhov xwm txheej tshwj xeeb. Hmoov zoo!

PS

Nyeem kuj ntawm peb blog:

Tau qhov twg los: www.hab.com

Ntxiv ib saib