ProHoster > Pūnaewele > Nā Administration > hale hana VxLAN. Mahele 1

hale hana VxLAN. Mahele 1

Aloha, habr. ʻO wau ke alakaʻi papa no ka papa Network Engineer ma OTUS.
Ma ka pō o ka hoʻomaka ʻana o kahi kākau inoa hou no ka papa "ʻenehana pūnaewele", Ua hoʻomākaukau au i nā ʻatikala ma ka ʻenehana VxLAN EVPN.

Nui ka nui o nā mea e pili ana i ka hana o VxLAN EVPN, no laila makemake wau e hōʻiliʻili i nā hana like ʻole a me nā hana no ka hoʻoponopono ʻana i nā pilikia i kahi kikowaena data hou.

hale hana VxLAN. Mahele 1

Ma ka hapa mua o ka moʻo ma ka ʻenehana VxLAN EVPN, makemake wau e nānā i kahi ala e hoʻonohonoho ai i ka pilina L2 ma waena o nā pūʻali ma luna o kahi lole pūnaewele.

E hana ʻia nā hiʻohiʻona āpau ma Cisco Nexus 9000v, i hui ʻia ma ka topology Spine-Leaf. ʻAʻole mākou e noho i ka hoʻonohonoho ʻana i ka pūnaewele Underlay ma kēia ʻatikala.

  1. ʻupena lalo
  2. Ke nānā nei ʻo BGP no ka helu wahi-ʻohana l2vpn evpn
  3. Hoʻonohonoho i ka NVE
  4. Kāohi-arp

ʻupena lalo

ʻO ka topology i hoʻohana ʻia penei:

hale hana VxLAN. Mahele 1

E hoʻonohonoho i ka helu ʻana ma nā mea hana a pau:

Spine-1 - 10.255.1.101
Spine-2 - 10.255.1.102

Leaf-11 - 10.255.1.11
Leaf-12 - 10.255.1.12
Leaf-21 - 10.255.1.21

Host-1 - 192.168.10.10
Host-2 - 192.168.10.20

E nānā kākou aia ka pilina IP ma waena o nā mea hana a pau:

Leaf21# sh ip route
<........>
10.255.1.11/32, ubest/mbest: 2/0                      ! Leaf-11 доступен чеерз два Spine
    *via 10.255.1.101, Eth1/4, [110/81], 00:00:03, ospf-UNDERLAY, intra
    *via 10.255.1.102, Eth1/3, [110/81], 00:00:03, ospf-UNDERLAY, intra
10.255.1.12/32, ubest/mbest: 2/0                      ! Leaf-12 доступен чеерз два Spine
    *via 10.255.1.101, Eth1/4, [110/81], 00:00:03, ospf-UNDERLAY, intra
    *via 10.255.1.102, Eth1/3, [110/81], 00:00:03, ospf-UNDERLAY, intra
10.255.1.21/32, ubest/mbest: 2/0, attached
    *via 10.255.1.22, Lo0, [0/0], 00:02:20, local
    *via 10.255.1.22, Lo0, [0/0], 00:02:20, direct
10.255.1.101/32, ubest/mbest: 1/0
    *via 10.255.1.101, Eth1/4, [110/41], 00:00:06, ospf-UNDERLAY, intra
10.255.1.102/32, ubest/mbest: 1/0
    *via 10.255.1.102, Eth1/3, [110/41], 00:00:03, ospf-UNDERLAY, intra

E nānā kāua ua hana ʻia ka waihona VPC a ua hala nā hoʻololi ʻelua i ka nānā kūlike a ua like nā hoʻonohonoho ma nā node ʻelua:

Leaf11# show vpc 

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 0
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router    : Disabled

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
5     Po5           up     success     success               1

Nānā BGP

ʻO ka hope, hiki iā mākou ke neʻe i ka hoʻonohonoho ʻana i ka pūnaewele Overlay.

Ma ke ʻano he ʻatikala, pono e hoʻonohonoho i kahi pūnaewele ma waena o nā pūʻali, e like me ka mea i hōʻike ʻia ma ke kiʻikuhi ma lalo nei:

hale hana VxLAN. Mahele 1

No ka hoʻonohonoho ʻana i kahi pūnaewele Overlay, pono ʻoe e ʻae iā BGP ma nā hoʻololi Spine a me Leaf me ke kākoʻo no ka ʻohana l2vpn evpn:

feature bgp
nv overlay evpn

A laila, pono ʻoe e hoʻonohonoho i ka BGP peering ma waena o Leaf a me Spine. No ka hoʻomaʻamaʻa ʻana i ka hoʻonohonoho ʻana a me ka hoʻonui ʻana i ka hāʻawi ʻana i ka ʻike alahele, hoʻonohonoho mākou iā Spine ma ke ʻano he kikowaena Route-Reflector. E kākau mākou i ka Leaf āpau i ka config ma o nā templates i mea e hoʻonui ai i ka hoʻonohonoho.

No laila ke ʻano o nā hoʻonohonoho ma Spine penei:

router bgp 65001
  template peer LEAF 
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
      route-reflector-client
  neighbor 10.255.1.11
    inherit peer LEAF
  neighbor 10.255.1.12
    inherit peer LEAF
  neighbor 10.255.1.21
    inherit peer LEAF

Ua like ke ʻano o ka hoʻonohonoho ʻana ma ka Leaf switch:

router bgp 65001
  template peer SPINE
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.255.1.101
    inherit peer SPINE
  neighbor 10.255.1.102
    inherit peer SPINE

Ma Spine, e nānā i ka nānā ʻana me nā hoʻololi Leaf a pau:

Spine1# sh bgp l2vpn evpn summary
<.....>
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.255.1.11     4 65001       7       8        6    0    0 00:01:45 0
10.255.1.12     4 65001       7       7        6    0    0 00:01:16 0
10.255.1.21     4 65001       7       7        6    0    0 00:01:01 0

E like me kāu e ʻike ai, ʻaʻohe pilikia me ka BGP. E neʻe kākou i ka hoʻonohonoho ʻana iā VxLAN. E hana ʻia ka hoʻonohonoho hou aʻe ma ka ʻaoʻao o nā hoʻololi Leaf. Hoʻohana wale ʻia ʻo Spine ma ke ʻano he kumu nui o ka pūnaewele a pili wale i ka hoʻoili ʻana i nā kaʻa. Loaʻa nā hana āpau ma ka encapsulation a me ka wehewehe ʻana i ke ala ma nā hoʻololi Leaf wale nō.

Hoʻonohonoho i ka NVE

NVE - ka mea hoʻohana pili pūnaewele

Ma mua o ka hoʻomaka ʻana i ka hoʻonohonoho, e hoʻolauna i kekahi mau huaʻōlelo:

VTEP - Vitual Tunnel End Point, ka mea e hoʻomaka ai a hoʻopau paha ka tunnel VxLAN. ʻAʻole pono ʻo VTEP i kahi ʻenehana pūnaewele. Hiki i kahi kikowaena kākoʻo ʻenehana VxLAN ke hana ma ke ʻano he kikowaena. I kā mākou topology, ʻo nā hoʻololi Leaf āpau he VTEP.

VNI - Hōʻike Pūnaewele Pūnaewele - ʻike pūnaewele i loko o VxLAN. Hiki iā ʻoe ke kiʻi i kahi hoʻohālikelike me VLAN. Eia naʻe, aia kekahi mau ʻokoʻa. I ka hoʻohana ʻana i kahi lole, lilo nā VLAN i mea kū hoʻokahi wale nō i loko o hoʻokahi hoʻololi Leaf a ʻaʻole hoʻouna ʻia ma luna o ka pūnaewele. Akā hiki ke hoʻopili ʻia kēlā me kēia VLAN me kahi helu VNI i hoʻouna ʻia ma luna o ka pūnaewele. He aha kona ʻano a pehea e hoʻohana ai e kūkākūkā ʻia ma lalo nei.

E ho'ā i ka hiʻona no ka ʻenehana VxLAN e hana a me ka hiki ke hoʻohui i nā helu VLAN me kahi helu VNI:

feature nv overlay
feature vn-segment-vlan-based

E hoʻonohonoho i ka interface NVE, nona ke kuleana no ka hana ʻana o VxLAN. Aia ke kuleana no ka hoʻopili ʻana i nā kiʻi ma nā poʻomanaʻo VxLAN. Hiki iā ʻoe ke kahakiʻi i kahi hoʻohālikelike me ka interface Tunnel no GRE:

interface nve1
  no shutdown
  host-reachability protocol bgp ! используем BGP для передачи маршрутной информации
  source-interface loopback0    ! интерфейс  с которого отправляем пакеты loopback0

Ma ka Leaf-21 hoʻololi i hana ʻia nā mea āpau me ka pilikia ʻole. Eia naʻe, inā mākou e nānā i ka hopena o ke kauoha show nve peers, alaila, nele. Maanei pono ʻoe e hoʻi i ka hoʻonohonoho VPC. ʻIke mākou ua hana ʻelua ʻo Leaf-11 a me Leaf-12 a hoʻohui ʻia e kahi kikowaena VPC. Hāʻawi kēia iā mākou i kēia kūlana:

Hoʻouna ʻo Host-2 i hoʻokahi kiʻi i Leaf-21 e hoʻouna ʻia ma luna o ka pūnaewele i Host-1. Eia naʻe, ʻike ʻo Leaf-21 e loaʻa ana ka helu MAC o Host-1 ma o ʻelua VTEP i ka manawa hoʻokahi. He aha ka Leaf-21 e hana ai i kēia hihia? Ma hope o nā mea a pau, ʻo ia ke ʻano hiki ke ʻike ʻia kahi loop i ka pūnaewele.

No ka hoʻoponopono ʻana i kēia kūlana, pono mākou i ka Leaf-11 a me ka Leaf-12 e hana pū me hoʻokahi mea hana i loko o ka hale hana. Hoʻoholo maʻalahi. Ma ka Loopback interface kahi a mākou e kūkulu nei i ka tunnel, e hoʻohui i ka helu lua. Pono e like ka helu helu lua ma nā VTEP ʻelua.

interface loopback0
 ip add 10.255.1.10/32 secondary

No laila, mai ka manaʻo o nā VTEP ʻē aʻe, loaʻa iā mākou kēia topology:

hale hana VxLAN. Mahele 1

ʻO ia, i kēia manawa e kūkulu ʻia ka tunnel ma waena o ka IP address o Leaf-21 a me ka IP virtual ma waena o ʻelua Leaf-11 a me Leaf-12. I kēia manawa, ʻaʻohe pilikia e aʻo i ka leka uila MAC mai ʻelua mau mea hana a hiki ke neʻe aku ke kaʻa mai kekahi VTEP i kekahi. ʻO wai o nā VTEP ʻelua e hoʻoponopono i ke kaʻa e hoʻoholo ʻia me ka hoʻohana ʻana i ka papa kuhikuhi ma Spine:

Spine1# sh ip route
<.....>
10.255.1.10/32, ubest/mbest: 2/0
    *via 10.255.1.11, Eth1/1, [110/41], 1d01h, ospf-UNDERLAY, intra
    *via 10.255.1.12, Eth1/2, [110/41], 1d01h, ospf-UNDERLAY, intra
10.255.1.11/32, ubest/mbest: 1/0
    *via 10.255.1.11, Eth1/1, [110/41], 1d22h, ospf-UNDERLAY, intra
10.255.1.12/32, ubest/mbest: 1/0
    *via 10.255.1.12, Eth1/2, [110/41], 1d01h, ospf-UNDERLAY, intra

E like me kāu e ʻike ai ma luna nei, loaʻa koke ka helu helu 10.255.1.10 ma o ʻelua Next-hops.

I kēia pae, ua ʻike mākou i ka pilina kumu. E neʻe kākou i ka hoʻonohonoho ʻana i ka interface NVE:
E ho'ā koke mākou iā Vlan 10 a hoʻopili iā ia me VNI 10000 ma kēlā me kēia Leaf no nā pūʻali. E hoʻonohonoho i kahi tunnel L2 ma waena o nā pūʻali

vlan 10                 ! Включаем VLAN на всех VTEP подключенных к необходимым хостам
  vn-segment 10000      ! Ассоциируем VLAN с номер VNI 

interface nve1
  member vni 10000      ! Добавляем VNI 10000 для работы через интерфейс NVE. для инкапсуляции в VxLAN
    ingress-replication protocol bgp    ! указываем, что для распространения информации о хосте используем BGP

I kēia manawa e nānā kākou i nā hoa a me ka papa no BGP EVPN:

Leaf21# sh nve peers
Interface Peer-IP          State LearnType Uptime   Router-Mac
--------- ---------------  ----- --------- -------- -----------------
nve1      10.255.1.10      Up    CP        00:00:41 n/a                 ! Видим что peer доступен с secondary адреса

Leaf11# sh bgp l2vpn evpn

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.255.1.11:32777    (L2VNI 10000)        ! От кого именно пришел этот l2VNI
*>l[3]:[0]:[32]:[10.255.1.10]/88                                   ! EVPN route-type 3 - показывает нашего соседа, который так же знает об l2VNI10000
                      10.255.1.10                       100      32768 i
*>i[3]:[0]:[32]:[10.255.1.20]/88
                      10.255.1.20                       100          0 i
* i                   10.255.1.20                       100          0 i

Route Distinguisher: 10.255.1.21:32777
* i[3]:[0]:[32]:[10.255.1.20]/88
                      10.255.1.20                       100          0 i
*>i                   10.255.1.20                       100          0 i

Ma luna aʻe mākou e ʻike ai i nā ala ala EVPN-type 3 wale nō. Ke kamaʻilio nei kēia ʻano o nā ala e pili ana i nā hoa (Leaf), akā ʻauhea ko mākou mau pūʻali?
A ʻo ka mea ʻo ia ka ʻike e pili ana i nā host MAC e hoʻouna ʻia ma o EVPN ala-ʻano 2

I mea e ʻike ai i kā mākou mea hoʻokipa, pono ʻoe e hoʻonohonoho i ka EVPN ala-ʻano 2:

evpn
  vni 10000 l2
    route-target import auto   ! в рамках данной статьи используем автоматический номер для route-target
    route-target export auto

E ping kāua mai Host-2 a i Host-1:

Firewall2# ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1): 56 data bytes
36 bytes from 192.168.10.2: Destination Host Unreachable
Request 0 timed out
64 bytes from 192.168.10.1: icmp_seq=1 ttl=254 time=215.555 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=254 time=38.756 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=254 time=42.484 ms
64 bytes from 192.168.10.1: icmp_seq=4 ttl=254 time=40.983 ms

A ma lalo hiki iā mākou ke ʻike i kēlā ala-type 2 i hōʻike ʻia i ka papa BGP me ka helu MAC o nā pūʻali - 5001.0007.0007 a me 5001.0008.0007

Leaf11# sh bgp l2vpn evpn
<......>

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.255.1.11:32777    (L2VNI 10000)
*>l[2]:[0]:[0]:[48]:[5001.0007.0007]:[0]:[0.0.0.0]/216                      !  evpn route-type 2 и mac адрес хоста 1
                      10.255.1.10                       100      32768 i
*>i[2]:[0]:[0]:[48]:[5001.0008.0007]:[0]:[0.0.0.0]/216                      ! evpn route-type 2 и mac адрес хоста 2
* i                   10.255.1.20                       100          0 i
*>l[3]:[0]:[32]:[10.255.1.10]/88
                      10.255.1.10                       100      32768 i
Route Distinguisher: 10.255.1.21:32777
* i[2]:[0]:[0]:[48]:[5001.0008.0007]:[0]:[0.0.0.0]/216
                      10.255.1.20                       100          0 i
*>i                   10.255.1.20                       100          0 i

A laila, hiki iā ʻoe ke ʻike i ka ʻike kikoʻī e pili ana i ka Update, kahi āu i loaʻa ai ka ʻike e pili ana i ka MAC Host. Ma lalo iho ʻaʻole nā ​​​​mea hoʻopuka kauoha a pau.

Leaf21# sh bgp l2vpn evpn 5001.0007.0007

BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.255.1.11:32777        !  отправил Update с MAC Host. Не виртуальный адрес VPC, а адрес Leaf
BGP routing table entry for [2]:[0]:[0]:[48]:[5001.0007.0007]:[0]:[0.0.0.0]/216,
 version 1507
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not i
n HW

  Path type: internal, path is valid, not best reason: Neighbor Address, no labe
led nexthop
  AS-Path: NONE, path sourced internal to AS
    10.255.1.10 (metric 81) from 10.255.1.102 (10.255.1.102)    ! с кем именно строим VxLAN тоннель
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000         ! Номер VNI, который ассоциирован с VLAN, в котором находится Host
      Extcommunity: RT:65001:10000 SOO:10.255.1.10:0 ENCAP:8        ! Тут видно, что RT сформировался автоматически на основе номеров AS и VNI
      Originator: 10.255.1.11 Cluster list: 10.255.1.102
<........>

E ʻike kākou i ke ʻano o nā kiʻi i ka wā e hele ai lākou ma ka hale hana:

hale hana VxLAN. Mahele 1

Kāohi-ARP

Nui, loaʻa iā mākou kahi pilina L2 ma waena o nā pūʻali a ʻo ia paha ka hopena. Eia naʻe, ʻaʻole maʻalahi nā mea a pau. ʻOiai he liʻiliʻi nā mea hoʻokipa, ʻaʻohe pilikia. Akā, e noʻonoʻo kākou i nā kūlana i loaʻa iā mākou he mau haneli a he mau tausani. He aha ka pilikia e hiki iā mākou ke alo?

ʻO kēia pilikia ʻo BUM (Broadcast, Unknown Unicast, Multicast) traffic. Ma ke kaʻina o kēia ʻatikala, e noʻonoʻo mākou i ke koho o ka hakakā ʻana i ka lele hoʻolaha.
ʻO ka mea hoʻolaha nui Broadcast ma nā pūnaewele Ethernet ʻo ia nā mea hoʻokipa ma o ka protocol ARP.

Hoʻohana ʻo Nexus i kēia ʻano hana e hakakā ai i nā noi ARP - suppress-arp.
Ke hana nei kēia hiʻohiʻona penei:

  1. Hoʻouna ʻo Host-1 i kahi noi APR i ka helu hoʻolaha o kāna pūnaewele.
  2. Hiki ke noi i ka hoʻololi Leaf a ma kahi o ka hāʻawi ʻana i kēia noi i ka lole i ka Host-2, pane ʻo Leaf iā ia iho a hōʻike i ka IP pono a me MAC.

No laila, ʻaʻole i hele ka noi Broadcast i ka hale hana. Akā pehea e hana ai kēia inā ʻike wale ʻo Leaf i ka helu MAC?

He mea maʻalahi nā mea a pau, EVPN ala-ʻano 2, ma kahi o ka helu MAC, hiki ke hoʻouna i kahi hui MAC/IP. No ka hana ʻana i kēia, pono ʻoe e hoʻonohonoho i kahi leka uila IP ma ka VLAN ma Leaf. Aia ka nīnau, he aha kaʻu IP e hoʻonoho ai? Ma ka nexus hiki ke hana i ka helu helu (hoʻokahi) ma nā hoʻololi a pau:

feature interface-vlan

fabric forwarding anycast-gateway-mac 0001.0001.0001    ! задаем virtual mac для создания распределенного шлюза между всеми коммутаторами

interface Vlan10
  no shutdown
  ip address 192.168.10.254/24          ! на всех Leaf задаем одинаковый IP
  fabric forwarding mode anycast-gateway    ! говорим использовать Virtual mac

No laila, mai ka manaʻo o nā mea hoʻokipa, e like ke ʻano o ka pūnaewele:

hale hana VxLAN. Mahele 1

E nānā i ka BGP l2ala evpn

Leaf11# sh bgp l2vpn evpn
<......>

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.255.1.11:32777    (L2VNI 10000)
*>l[2]:[0]:[0]:[48]:[5001.0007.0007]:[0]:[0.0.0.0]/216
                      10.255.1.21                       100      32768 i
*>i[2]:[0]:[0]:[48]:[5001.0008.0007]:[0]:[0.0.0.0]/216
                      10.255.1.10                       100          0 i
* i                   10.255.1.10                       100          0 i
* i[2]:[0]:[0]:[48]:[5001.0008.0007]:[32]:[192.168.10.20]/248
                      10.255.1.10                       100          0 i
*>i                   10.255.1.10                       100          0 i

<......>

Route Distinguisher: 10.255.1.21:32777
* i[2]:[0]:[0]:[48]:[5001.0008.0007]:[0]:[0.0.0.0]/216
                      10.255.1.20                       100          0 i
*>i                   10.255.1.20                       100          0 i
* i[2]:[0]:[0]:[48]:[5001.0008.0007]:[32]:[192.168.10.20]/248
*>i                   10.255.1.20                       100          0 i

<......>

Mai ka puka o ke kauoha, hiki ke ʻike ʻia ma ka EVPN route-type 2, i ka hoʻohui ʻana i ka MAC, ʻike mākou i kēia manawa i ka IP address o ka mea hoʻokipa.

E hoʻi kāua i ka hoʻonohonoho suppress-arp. Hiki ke hoʻonohonoho ʻia kēia hoʻonohonoho no kēlā me kēia VNI.

interface nve1
  member vni 10000   
    suppress-arp

A laila aia kekahi pilikia:

  • Pono kēia hiʻohiʻona i kahi i loko o ka hoʻomanaʻo TCAM. E hāʻawi wau i kahi laʻana o ka hoʻonohonoho ʻana no suppress-arp:

hardware access-list tcam region arp-ether 256

Pono kēia hoʻonohonoho ʻana i ʻelua-ākea. ʻO ia hoʻi, inā hoʻonoho ʻoe i ka 256, a laila pono e hoʻokuʻu ʻia ʻo 512 ma TCAM. ʻO ka hoʻonohonoho ʻana i ka TCAM ʻaʻole ia ma mua o ke ʻano o kēia ʻatikala, ʻoiai ʻo ka hoʻonohonoho ʻana i ka TCAM e hilinaʻi wale ʻia i ka hana i hāʻawi ʻia iā ʻoe a ʻokoʻa paha mai kahi pūnaewele i kekahi.

  • Pono e hoʻokō ʻia ka suppress-arp ma nā hoʻololi Leaf āpau. Eia nō naʻe, hiki ke paʻakikī i ka wā e hoʻonohonoho ai i nā hui lau i loaʻa ma kahi kikowaena VPC. I ka hoʻololi ʻana i ka TCAM, e uhaʻi ʻia ka kūlike ma waena o nā pālua a hiki ke lawe ʻia kahi node mai ka lawelawe. Eia hou, pono paha e hoʻomaka hou ka mea hana e hoʻohana i ka hoʻololi hoʻololi TCAM.

ʻO ka hopena, pono ʻoe e noʻonoʻo pono inā pono e hoʻokō i kēia hoʻonohonoho ma kahi hale hana hana i kou kūlana.

Hoʻopau kēia i ka hapa mua o ka pōʻai. Ma ka ʻāpana aʻe, e noʻonoʻo mākou i ke ala ʻana ma o kahi hale hana VxLAN me ka hoʻokaʻawale ʻana i ka pūnaewele ma waena o nā VRF like ʻole.

A i kēia manawa ke kono nei au i nā mea a pau e webinar manuahi, i loko e haʻi aku au iā ʻoe e pili ana i ka papa. E loaʻa i nā poʻe 20 mua e hoʻopaʻa inoa no kēia webinar kahi palapala hōʻemi ma o ka leka uila i loko o 1-2 mau lā ma hope o ka hoʻolaha.

Source: www.habr.com

Pākuʻi i ka manaʻo hoʻopuka