{"id":73232,"date":"2020-03-08T08:42:18","date_gmt":"2020-03-08T05:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/vxlan-v-nsx-v-trablshutim-underlay"},"modified":"2020-03-08T08:42:18","modified_gmt":"2020-03-08T05:42:18","slug":"vxlan-v-nsx-v-trablshutim-underlay","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vxlan-v-nsx-v-trablshutim-underlay","title":{"rendered":"VXLAN in NSX-V \u2014 troubleshooting underlay","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Greetings, and first a bit of sentiment. I sometimes envy my colleagues who work remotely \u2014 it\u2019s truly wonderful to have the ability to work from anywhere in the world connected to the Internet, take vacations at any time, have project responsibilities and deadlines, rather than being stuck in the office from 8 to 5. My position and work duties nearly exclude the possibility of being away from the data center for long. However, interesting cases do occasionally arise, like the one described below \u2014 and I realize that there are few positions where there is such room for the creative expression of an internal troubleshooter.
\n<\/a><\/noindex>
\nA small disclaimer \u2014 at the time of writing, the case is not fully resolved, but considering the vendors' response time, a complete solution may take months, and I want to share my findings now. I hope dear readers will forgive me for this haste. But enough about that \u2014 what\u2019s going on with the case? <\/p>\n

First, some background: there is a company (where I work as a network engineer) that hosts client solutions in a private VMWare cloud. Most new solutions connect to VXLAN segments managed by NSX-V \u2014 I won't evaluate how much time this solution has gifted me, simply put \u2014 a lot. I even managed to train colleagues on configuring NSX ESG, and small client solutions are deployed without my involvement. An important note \u2014 our control plane is with unicast replication. Hypervisors are redundantly connected via two interfaces to different physical Juniper QFX5100 switches (assembled in a Virtual Chassis) and use a route-based policy based on the originating virtual port \u2014 for completeness.<\/p>\n

Client solutions are very diverse: from Windows IIS, where all web server components are installed on one machine, to quite large configurations \u2014 for instance, load-balanced Apache web fronts + LB MariaDB in Galera + share servers synchronized with GlusterFS. Almost every server needs to be monitored separately, and not all components have public addresses \u2014 if you've faced this challenge and have a more elegant solution, I would appreciate your advice.
\nMy monitoring solution consists of 'connecting' a firewall (Fortigate) to each internal client network (+SNAT and, of course, strict restrictions on the allowed traffic type) and monitoring the internal addresses \u2014 thus achieving a certain unification and simplification of monitoring. The monitoring itself takes place from the PRTG server cluster. The monitoring scheme is approximately as follows:<\/p>\n

\"VXLAN
\n
\nAs long as we only operated with VLANs, everything was quite usual and worked like clockwork. After implementing NSX-V and VXLAN, we faced the question \u2014 can we continue monitoring in the old way? At the time of this question, the 'quickest' solution was to deploy NSX ESG and connect the VXLAN trunk interface to the VTEP network. Quick in quotes \u2014 because using the GUI to configure client networks, SNAT, and firewall rules might unify management in a single vSphere interface, but in my opinion, it is rather cumbersome and, among other things, limits the toolkit for troubleshooting. Those who have used NSX ESG as a replacement for a 'real' firewall will probably agree. Although, perhaps, such a solution would be more stable \u2014 after all, everything happens within the same vendor.<\/p>\n

Another solution is to use NSX DLR in bridging mode between VLAN and VXLAN. Here, I think everything is clear \u2014 the benefit of using VXLAN is simply lost since in this case, you still have to stretch VLAN to the monitoring installation. By the way, while working on this solution, I encountered an issue where the DLR bridge did not send packets to the virtual machine that was on the same host. I know, I know \u2014 books and guides on NSX-V clearly state that a separate cluster must be allocated for NSX Edge, but that's in the books... Anyway, after a couple of months with support, we didn't solve the problem. In principle, I understood the logic of the action \u2014 the hypervisor kernel module responsible for VXLAN encapsulation was not engaged if the DLR and the monitored server were on the same host, since the traffic does not leave the host and logically should be connected to the VXLAN segment \u2014 encapsulation is not needed. We ended up with the virtual interface vdrPort with support, which logically combines uplinks and also performs bridging\/encapsulation \u2014 it was there that a discrepancy in incoming traffic was noted, which I took on for processing in the current case. But as it was said, I did not fully pursue this case since I was reassigned to another project and, to be honest, the branch was originally a dead end, and I had no particular desire to develop it. If I\u2019m not mistaken, the problem was observed in NSX versions 6.1.4 and 6.2. <\/p>\n

And here \u2014 bingo! Fortinet announces native support for VXLAN<\/a><\/noindex>. And not just point-to-point or VXLAN-over-IPSec, not software bridging VLAN-VXLAN \u2014 all this began to be implemented back in version 5.4 (and is presented by other vendors<\/a><\/noindex>), and true support for unicast control plane. During the implementation of the solution, I encountered another issue \u2014 the monitored servers would sometimes <\/p>\n

After successfully testing the traffic, it was decided to implement this solution. In production, it turned out that after a day or two, everything monitored through VXLAN gradually disconnected. Deactivating\/activating the interface helped, but only temporarily. Knowing the sluggishness of the manufacturer's support, I took on the troubleshooting myself\u2014after all, my company, my network\u2014my responsibility. <\/p>\n

Spoiler alert: the course of troubleshooting. Anyone tired of letters and boasting\u2014feel free to skip to the post-analysis.<\/p>\n

Troubleshooting Progress<\/b>Thank you for continuing to read\u2014let\u2019s continue!<\/p>\n

So, monitoring works for some time, then it disconnects by itself. This means that there are most likely no problems with the firewall policies. However, since I have encountered hanging system processes in Fortigate versions 5.6+, we will first check 'diagnose debug flow'\u2014as expected, the traffic is allowed and leaves the interface, and, predictably, nothing comes back in response. This means we dig further down the stack. Unfortunately, I will have to hide the addresses even if they are RFC1918, but I hope to provide enough description for understanding. The server inside the VXLAN has the address x.x.x.15, the Fortigate interface x.x.x.254, and all other addresses belong to the VTEP network.<\/p>\n

For the successful transmission of VXLAN encapsulated packets, correct information in several tables is required. For overlay, this includes ARP and OVSDB, and for underlay, it's ARP and CAM. In the case of Fortigate, VXLAN FDB is the OVSDB. Let's start there:<\/p>\n

 fortigate (root) #diag sys vxlan fdb list vxlan-LS\nmac=00:50:56:8f:3f:5a state=0x0002 flags=0x00 remote_ip=x.x.x.47 port=4789 vni=5008 ifindex=7\n<\/code><\/pre>\n
\nEverything is fairly straightforward here \u2014 the MAC address of the virtual machine must be on the VTEP with the address x.x.x.47. Looking at the contents and settings of the ESXI cluster, I find that the MAC of the virtual machine is correct, and the VTEP address is too. I check the CAM\/ARP table on the Fortigate \u2014 once again, everything matches the settings of the ESXI host:<\/p>\n
fortigate (root) #get sys arp | grep x.x.x.47\nx.x.x.47 0 00:50:56:65:f6:2c dmz\n<\/code><\/pre>\n
\nThe tables are correct and traffic is flowing \u2014 maybe the issue is not with the Fortigate? I deliberately skipped the traffic switching analysis on Juniper \u2014 logically, the next step in troubleshooting should be there, but my network is simple \u2014 just one VLAN for VTEP and all components are connected directly. Plus, I remember the case with the DLR bridge, VDR, and disappearing traffic \u2014 I\u2019m going to sniff on the ESXi host, and at the same time, I\u2019m creating a case with VMware. Below, the MAC \"97:6e\" belongs to Fortinet, vmnic1 \u2014 this is the interface that has VTEP with the address x.x.x.47, sniffing in both directions \"\u2014dir 2\".<\/p>\n
pktcap-uw --uplink vmnic1 --vni 5008  --mac 90:6c:ac:a9:97:6e --dir 2 -o \/tmp\/monitor.pcap\n<\/code><\/pre>\n
\n\"VXLAN
\n
\nProgress \u2014 in the sniff, I see an ARP request and the incoming response. I only bring the ARP response, and everything is correct there. I should mention, however, that during this time, the monitoring server is pinging address y.y.y.15 \u2014 where is the ICMP traffic? I recall that I have two uplinks. One could argue and say that the virtual source port remains the same (my teaming policy), meaning the same uplink should be selected for the same vNIC, but since I'm on the host, checking the other uplink is not an issue:<\/p>\n
pktcap-uw --uplink vmnic4 --vni 5008  --mac 90:6c:ac:a9:97:6e --dir 2 -o \/tmp\/monitor.pcap\n<\/code><\/pre>\n
\n\"VXLAN
\n
\nRequests come from Fortigate, but there is no response. So, the problem is not with Fortigate. Well, I think, it's the same issue with the disappearing traffic on the VDR again; it will take weeks to redirect the case to the right direction. A couple of days later, cooling down, and unwilling to accept the stalemate, I decided to dig up more sniff data for support to speed up the process. And then, 'by chance', my gaze falls on the Ethernet encapsulation underlay. The king is not real, and the MAC address of the VTEP does not match its IP. I reset it, sniff it, dig in \u2014 that's right and wrong. I'll provide the ARP table alongside to make it easier to compare. Note the first Ethernet encapsulation in the image above:<\/p>\n
fortigate (root) #get sys arp | grep x.x.x.47\nx.x.x.47 0 00:50:56:65:f6:2c dmz\nfortigate (root) #get sys arp | grep x.x.x.42\nx.x.x.42 0 00:50:56:6a:78:86 dmz\n<\/code><\/pre>\n
\nSo, what do we have in the end \u2014 after migrating the virtual machine, Fortigate tries to send traffic to the VTEP from the (correct) VXLAN FDB but uses the wrong DST MAC, and the traffic is expectedly dropped by the receiving hypervisor interface. Moreover, in one out of four cases, this MAC belonged to the original hypervisor from which the machine migration began.<\/p>\n
Yesterday I received a letter from Fortinet support \u2014 a bug 615586 has been opened for my case. I don't know whether to be happy or sad: on one hand, the problem is not with the settings, on the other hand, the fix will only come with the next firmware update, at best. My self-importance is also fueled by another bug that I discovered last month, though that time it was in the HTML5 GUI of vSphere. It's like having a local QA department for vendors...<\/p>\n
I\u2019ll venture to suggest the following:<\/p>\n
1 \u2014 multicast control plane is unlikely to be affected by the described problem \u2014 since the MAC addresses of the VTEP are derived from the group IP address to which the interface is subscribed. <\/p>\n
2 \u2014 the Fortigate issue is likely related to session offloading to the Network Processor (approximately analogous to CEF) \u2014 if each packet is passed through the CPU, the tables containing the correct \u2014 at least visually \u2014 information will be used. Supporting this assumption is the fact that toggling the interface or waiting some time, more than 5 minutes, helps close\/open the issue. <\/p>\n
3 \u2014 changing the teaming policy, for example to explicit failover, or implementing LAG will not resolve the problem, as there was observable 'sticking' of the original hypervisor's MAC in the encapsulated packets.<\/p>\n
In light of this, I can share that I recently discovered blog<\/a><\/noindex>, where one of the articles claimed that stateful firewalls and cacheable data transmission methods are crutches. Well, I\u2019m not experienced enough in IT to assert such a thing, and I don\u2019t automatically agree with all the claims made in blog articles. However, something tells me there is some truth in Ivan's words.<\/p>\n
Thank you for your attention! I would be happy to answer questions and hear constructive criticism.
\n
Source: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041f\u0440\u0438\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e, \u0438 \u0441\u043f\u0435\u0440\u0432\u0430 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u043b\u0438\u0440\u0438\u043a\u0438. \u042f \u0438\u043d\u043e\u0433\u0434\u0430 \u0437\u0430\u0432\u0438\u0434\u0443\u044e \u043a\u043e\u043b\u043b\u0435\u0433\u0430\u043c, \u0440\u0430\u0431\u043e\u0442\u0430\u044e\u0449\u0438\u043c \u0443\u0434\u0430\u043b\u0451\u043d\u043d\u043e \u2014 \u0432\u0435\u0434\u044c \u044d\u0442\u043e \u043f\u0440\u0435\u043a\u0440\u0430\u0441\u043d\u043e \u0438\u043c\u0435\u0442\u044c \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u044c \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0438\u0437 \u043b\u044e\u0431\u043e\u0433\u043e \u043a\u043e\u043d\u0446\u0430 \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0451\u043d\u043d\u043e\u0433\u043e \u043a Internet \u043c\u0438\u0440\u0430, \u043a\u0430\u043d\u0438\u043a\u0443\u043b\u044b \u0432 \u043b\u044e\u0431\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u0441\u0442\u044c \u0437\u0430 \u043f\u0440\u043e\u0435\u043a\u0442\u044b \u0438 \u0434\u0435\u0434\u043b\u0430\u0439\u043d\u044b, \u0430 \u043d\u0435 \u043d\u0430\u0445\u043e\u0436\u0434\u0435\u043d\u0438\u0435 \u0432 \u043e\u0444\u0438\u0441\u0435 \u0441 8 \u0434\u043e 17. \u041c\u043e\u044f \u043f\u043e\u0437\u0438\u0446\u0438\u044f \u0438 \u0440\u0430\u0431\u043e\u0447\u0438\u0435 \u043e\u0431\u044f\u0437\u0430\u043d\u043d\u043e\u0441\u0442\u0438 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438 \u0438\u0441\u043a\u043b\u044e\u0447\u0430\u044e\u0442 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u044c \u0434\u043e\u043b\u0433\u043e\u0433\u043e \u043e\u0442\u0441\u0443\u0442\u0441\u0442\u0432\u0438\u044f \u0432 \u0434\u0430\u0442\u0430\u0446\u0435\u043d\u0442\u0440\u0435. […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":73233,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-73232","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47VXLAN in NSX-V \u2014 troubleshooting underlay | ProHoster","description":"Greetings, and first a little bit of reflection.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vxlan-v-nsx-v-trablshutim-underlay","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47VXLAN \u0432 NSX-V \u2014 \u0442\u0440\u0430\u0431\u043b\u0448\u0443\u0442\u0438\u043c underlay | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e, \u0438 \u0441\u043f\u0435\u0440\u0432\u0430 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u043b\u0438\u0440\u0438\u043a\u0438.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vxlan-v-nsx-v-trablshutim-underlay","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-08T05:42:18+00:00","article:modified_time":"2020-03-08T05:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"73232","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 18:36:23","updated":"2022-09-27 15:26:08","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/73232","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=73232"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/73232\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/73233"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=73232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=73232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=73232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}