{"id":35972,"date":"2019-10-31T22:08:54","date_gmt":"2019-10-31T19:08:54","guid":{"rendered":"https:\/\/prohoster.info\/blog\/avtomatizatsiya-dlya-samyh-malenkih-chast-nulevaya-planirovanie\/"},"modified":"2019-10-31T22:08:54","modified_gmt":"2019-10-31T19:08:54","slug":"avtomatizatsiya-dlya-samyh-malenkih-chast-nulevaya-planirovanie","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/avtomatizatsiya-dlya-samyh-malenkih-chast-nulevaya-planirovanie","title":{"rendered":"Automation for the smallest. Zero part. Planning","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>The SDSM has ended, but the uncontrollable desire to write remains. <\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/fs.linkmeup.ru\/images\/adsm\/0\/kdpv.jpg\"><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/2da9a566e92a782cf1205a6da716d4c4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/a><\/noindex><\/p>\n<p>For many years, our brother suffered from routine tasks, crossing his fingers before commits and losing sleep due to nightly rollbacks.<br \/>\nBut dark times come to an end.<\/p>\n<p>With this article, I will start a series on how <i>to me<\/i> automation is perceived. <br \/>\nAlong the way, we will look into automation stages, variable storage, design formalization, RestAPI, NETCONF, YANG, YDK, and we will be doing a lot of programming. <br \/>\n<i>To me<\/i> it means that a) this is not an objective truth, b) it is not an unconditionally best approach, c) my perspective may even change while moving from the first to the last article \u2014 honestly, from the draft stage to publication, I completely rewrote everything twice. <\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h1>Content<\/h1>\n<p><\/p>\n<ol>\n<li> <b>Objectives<\/b>\n<ol>\n<li>The network is like a single organism<\/li>\n<li>Configuration testing<\/li>\n<li>Versioning<\/li>\n<li>Monitoring and self-recovery of services<\/li>\n<\/ol>\n<p>\n <\/li>\n<li> <b>Tools<\/b>\n<ol>\n<li>Inventory system<\/li>\n<li>IP space management system<\/li>\n<li>Network service description system<\/li>\n<li>Device initialization mechanism<\/li>\n<li>Vendor-agnostic configuration model<\/li>\n<li>Vendor-specific interface driver<\/li>\n<li>Configuration delivery mechanism to the device<\/li>\n<li>CI\/CD<\/li>\n<li>Backup mechanism and anomaly detection<\/li>\n<li>Monitoring system<\/li>\n<\/ol>\n<p>\n <\/li>\n<li> <b>Conclusion<\/b>\n <\/li>\n<\/ol>\n<p>\nI will try to conduct ADSCM in a format slightly different from SDSM. There will still be extensive numbered articles, and between them, I will publish small notes from everyday experience. I will attempt to fight perfectionism and not polish each of them. <\/p>\n<blockquote><p> How funny it is to walk the same path for the second time.<\/p>\n<p> Initially, I had to write articles about networks myself because they were lacking in the RuNet.<\/p>\n<p> Now I couldn't find a comprehensive document that systematizes approaches to automation and analyzes the aforementioned technologies with simple practical examples.<\/p>\n<p> Perhaps I am mistaken, so feel free to share links to valuable resources. However, this will not change my determination to write because the main goal is to learn something myself, and making life easier for others is a pleasant bonus that tickles the urge to spread knowledge.\n<\/p><\/blockquote>\n<p>We will attempt to take a medium-sized data center, LAN DC, and work through the entire automation scheme. <br \/>\nI will be doing some things for the first time alongside you. <\/p>\n<blockquote><p> I won't be original in the ideas and tools described here. Dmitry Figol has an excellent <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=MLeiShwb7Qo&amp;list=PLSwGHYY8t8JiGI9powOXDHTAXDSs7FZqt\">channel with streams on this topic.<\/a><\/noindex>. <br \/>\n The articles will intersect with these in many aspects. \n<\/p><\/blockquote>\n<p>In the LAN DC, there are 4 data centers, about 250 switches, half a dozen routers, and a couple of firewalls. <br \/>\nIt's not Facebook, but it's enough to think deeply about automation. <br \/>\nHowever, there is an opinion that if you have more than one device, automation is already necessary. <br \/>\nIn reality, it's hard to imagine someone living without at least a handful of knee scripts. <br \/>\nThough I\u2019ve heard of companies where IP addresses are tracked in Excel, and each of the thousands of network devices is configured manually with its unique configuration. This can certainly be passed off as modern art, but an engineer's feelings will definitely be hurt. <\/p>\n<h1>Objectives<\/h1>\n<p>\nNow we will set the most abstract goals:<\/p>\n<ul>\n<li>The network is like a single organism<\/li>\n<li>Configuration testing<\/li>\n<li>Versioning the state of the network<\/li>\n<li>Monitoring and self-recovery of services<\/li>\n<\/ul>\n<p>\nLater in this article, we will discuss what tools we will use, and in the following ones, the goals and means in detail.<\/p>\n<h2>The network is like a single organism<\/h2>\n<p>\nThe defining phrase of the cycle, though it may seem insignificant at first glance: <b>we will configure the network, not individual devices.<\/b>. <br \/>\nIn recent years, we've seen a shift towards treating the network as a single entity, hence the arrival of <i>Software Defined Networking<\/i>, <i>Intent Driven Networks<\/i> and <i>Autonomous Networks<\/i>. <br \/>\nAfter all, what do applications fundamentally need from the network: connectivity between points A and B (and sometimes C) and isolation from other applications and users. <\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/f8b43928ccc6d487722ce7724f5c4829.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThus, our task in this series is to <b>build a system<\/b>, supporting the current configuration <b>of the entire network<\/b>, which is already decomposed into the current configuration on each device according to its role and location. <br \/>\n<b>System<\/b> Network management implies that to make changes, we refer to it, and it then calculates the necessary state for each device and configures it. <br \/>\nIn this way, we minimize human CLI access to almost zero \u2014 any changes in device settings or network design must be formalized and documented \u2014 and only then applied to the necessary network elements. <\/p>\n<blockquote><p> For instance, if we decide that from this moment forward, rack switches in Kazan should announce two networks instead of one, we<\/p>\n<ol>\n<li>First, we document the changes in the systems<\/li>\n<li>We generate the target configuration for all network devices<\/li>\n<li>We launch the network configuration update program, which calculates what needs to be removed on each node, what to add, and brings the nodes to the required state.<\/li>\n<\/ol>\n<p>\n At this stage, we manually make changes only during the first step.\n<\/p><\/blockquote>\n<h2>Configuration testing<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"http:\/\/www.wikisummaries.org\/wiki\/Visible_Ops\">It is known<\/a><\/noindex>, that 80% of problems occur during configuration changes \u2014 indirect evidence of this is that during the New Year holidays, everything is usually calm. <br \/>\nI have personally witnessed dozens of global downtimes due to human error: an incorrect command executed, not on the right configuration branch, forgetting the community, globally deleting MPLS on a router, configuring five devices, but missing an error on the sixth, committing old changes made by someone else. There are countless scenarios. <\/p>\n<p>Automation will allow us to make fewer mistakes, but on a larger scale. This way, we could brick not just one device, but the entire network at once. <\/p>\n<p>Since time immemorial, our ancestors checked the correctness of the changes made with keen eyes, steel nerves, and the operability of the network after deployment. <br \/>\nThose ancestors whose work led to downtimes and catastrophic losses left fewer descendants and should eventually die out, but evolution is a slow process, and therefore, not everyone still tests changes in the lab beforehand. <br \/>\nHowever, at the forefront of progress are those who automated the configuration testing process, and its further application to the network. In other words \u2014 borrowed the CI\/CD procedure (<noindex><a rel=\"nofollow\" href=\"https:\/\/img.devrant.com\/devrant\/rant\/r_1535091_ErSUL.jpg\">Continuous Integration, Continuous Deployment<\/a><\/noindex>) from developers.<br \/>\nIn one of the sections, we will examine how to implement this using a version control system, probably GitHub. <\/p>\n<blockquote><p> Once you come to terms with the idea of network CI\/CD, the method of validating configuration by applying it to a live network will seem to you like early medieval ignorance. About as ridiculous as hitting a warhead with a hammer.\n<\/p><\/blockquote>\n<p>An organic continuation of the ideas about <b>network management system and CI\/CD becomes full-fledged configuration versioning.<\/b> network management and CI\/CD is becoming complete versioning of the configuration. <\/p>\n<h2>Versioning<\/h2>\n<p>\nWe will assume that any changes, even the smallest, even on one unnoticed device, will cause the entire network to transition from one state to another.<br \/>\nAnd we never execute commands on the device; we change the state of the network instead. <br \/>\nSo, shall we call these states versions?<\/p>\n<p>Let's say the current version is 1.0.0.<br \/>\nHas the Loopback interface IP address changed on one of the ToRs? This is a minor version \u2014 it will receive the number 1.0.1.<br \/>\nIf we've revised the route import policies in BGP \u2014 that's a bit more serious \u2014 and already 1.1.0.<br \/>\nDecided to get rid of IGP and switch only to BGP \u2014 that\u2019s a radical design change \u2014 2.0.0.<\/p>\n<p>At the same time, different data centers can have different versions \u2014 the network is evolving, new equipment is being installed, sometimes new spine levels are added, and sometimes they are not, etc.<\/p>\n<p>About <noindex><a rel=\"nofollow\" href=\"https:\/\/semver.org\">semantic versioning<\/a><\/noindex> we will discuss it in a separate article.<\/p>\n<p>I repeat \u2014 any change (except for debugging commands) is a version update. Administrators must be notified of any deviations from the current version.<\/p>\n<p>The same goes for rolling back changes \u2014 this isn't the cancellation of the last commands, it\u2019s not a rollback by the device's operating system \u2014 it\u2019s bringing the entire network back to a new (previous) version. <\/p>\n<h2>Monitoring and self-recovery of services<\/h2>\n<p>\nThis obvious task in modern networks is reaching a new level.<br \/>\nOften, large service providers practice the approach of very quickly terminating a failed service and raising a new one, instead of figuring out what happened.<br \/>\n\u201cVery\u201d means that one needs to slather on monitoring from all sides, which will detect the slightest deviations from the norm within seconds.<br \/>\nAnd here, the usual metrics, like interface load or node availability, are no longer sufficient. Manual monitoring by the on-duty personnel is also not enough.<br \/>\nFor many things, there should be a <noindex><a rel=\"nofollow\" href=\"https:\/\/www.irisns.com\/self-healing-network-tomorrow-look-like\/\">Self-Healing<\/a><\/noindex> \u2014 the monitors lit up red and went to apply a band-aid on where it hurts.<\/p>\n<p>Here, we monitor not only individual devices but also the health of the entire network, both as a white box, which is relatively clear, and as a black box, which is more complex.<\/p>\n<p>\nWhat do we need to implement such ambitious plans?<\/p>\n<ul>\n<li>We need a list of all devices in the network, their locations, roles, models, and software versions. <br \/>\n <i>kazan-leaf-1.lmu.net, Kazan, leaf, Juniper QFX 5120, R18.3.<\/i>\n <\/li>\n<li>We need a system for describing network services. <br \/>\n <i>IGP, BGP, L2\/3VPN, Policy, ACL, NTP, SSH.<\/i><\/li>\n<li>Be able to initialize the device. <br \/>\n <i>Hostname, Mgmt IP, Mgmt Route, Users, RSA-Keys, LLDP, NETCONF<\/i><\/li>\n<li>Configure the device and revert the configuration to the desired (including the old) version.<\/li>\n<li>Test the configuration<\/li>\n<li>Periodically check the status of all devices for deviations from the current state and report to the appropriate parties.<br \/>\n <i>At night, someone quietly added a rule to the ACL<\/i>.<\/li>\n<li>Monitor performance.<\/li>\n<\/ul>\n<p><\/p>\n<h1>Tools<\/h1>\n<p>\nIt sounds complex enough to start breaking down the project into components.<\/p>\n<p>And there will be ten of them:<\/p>\n<ol>\n<li>Inventory system<\/li>\n<li>IP space management system<\/li>\n<li>Network service description system<\/li>\n<li>Device initialization mechanism<\/li>\n<li>Vendor-agnostic configuration model<\/li>\n<li>Vendor-specific interface driver<\/li>\n<li>Configuration delivery mechanism to the device<\/li>\n<li>CI\/CD<\/li>\n<li>Backup mechanism and anomaly detection<\/li>\n<li>Monitoring system<\/li>\n<\/ol>\n<blockquote><p> This is, by the way, an example of how the view of the cycle's goals changed \u2014 the draft of the components had 4.\n<\/p><\/blockquote>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/91132e29d359a4042b213529511f6d7d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIn the illustration, I depicted all components and the device itself. <br \/>\nIntersecting components interact with each other. <br \/>\nThe larger the block, the more attention needs to be paid to this component. <\/p>\n<p><\/p>\n<h2>Component 1. Inventory System<\/h2>\n<p>\nClearly, we want to know what equipment is where and what it is connected to. <br \/>\nThe inventory system is an integral part of any enterprise. <br \/>\nMost often, for network devices, the enterprise has a separate inventory system that addresses more specific tasks. <br \/>\nIn this series of articles, we will refer to it as DCIM \u2014 Data Center Infrastructure Management. Although the term DCIM, strictly speaking, encompasses much more.<\/p>\n<p>For our purposes, we will store the following information about the device:<\/p>\n<ul>\n<li>Inventory number<\/li>\n<li>Name\/description <\/li>\n<li>Model (<i>Huawei CE12800, Juniper QFX5120, etc.<\/i>)<\/li>\n<li>Characteristics (<i>boards, interfaces, etc.<\/i>)<\/li>\n<li>Role (<i>Leaf, Spine, Border Router, etc.<\/i>)<\/li>\n<li>Location (<i>region, city, data center, rack, unit<\/i>)<\/li>\n<li>Interconnects between devices<\/li>\n<li>Network topology<\/li>\n<\/ul>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/fac708b1bf3640a943f90247bff8216e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIt's clear that we ourselves want to know all this. <br \/>\nBut will this help with automation? <br \/>\nAbsolutely. <br \/>\nFor example, we know that in this data center on Leaf switches, if it is Huawei, ACLs for filtering certain traffic should be applied to VLAN, and if it is Juniper \u2014 then on unit 0 of the physical interface. <br \/>\nOr we need to deploy a new Syslog server on all borders of the region.<\/p>\n<p>We will also store virtual network devices, such as virtual routers or route reflectors. We can add DNS servers, NTP, Syslog, and generally everything related to the network. <\/p>\n<h2>Component 2. IP Space Management System<\/h2>\n<p>\nYes, even in our time, there are groups of people who keep track of prefixes and IP addresses in an Excel file. However, the modern approach is still to use a database, with a frontend on nginx\/apache, an API, and extensive capabilities for managing IP addresses and networks with VRF segmentation.<br \/>\nIPAM \u2013 IP Address Management. <\/p>\n<p>For our tasks, we will store the following information in it:<\/p>\n<ul>\n<li>VLAN<\/li>\n<li>VRF<\/li>\n<li>Networks\/Subnets<\/li>\n<li>an IP address<\/li>\n<li>Binding addresses to devices, networks to locations, and VLAN numbers<\/li>\n<\/ul>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/6d476d48f95ed663d84cae052a86d129.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIt is also clear that we want to ensure that when allocating a new IP address for the Loopback of the ToR, we don't stumble upon the fact that it has already been assigned to someone. Or that we have used the same prefix twice at different ends of the network. <br \/>\nBut how will this help with automation? <br \/>\nEasy. <br \/>\nWe request a prefix from the system with the role of Loopbacks that has available IP addresses to allocate \u2014 if found, we allocate the address; if not, we request the creation of a new prefix. <br \/>\nOr when creating a device configuration, we can learn from the same system which VRF the interface should be in. <br \/>\nAnd when launching a new server, the script will go into the system to find out which server switch it is in, which port, and what subnet is assigned to the interface \u2014 from that, the server address will be allocated. <\/p>\n<p>\nIt makes sense to combine DCIM and IPAM into one system to avoid duplicating functions and servicing two similar entities. <br \/>\nAnd we will do just that. <\/p>\n<h2>Component 3. Network Services Description System<\/h2>\n<p>\nWhile the first two systems store variables that still need to be used somehow, the third describes how each device role should be configured. <br \/>\nIt is worth highlighting two different types of network services: <\/p>\n<ul>\n<li>Infrastructure<\/li>\n<li>Client.<\/li>\n<\/ul>\n<p>The first provides basic connectivity and device management. This includes VTY, SNMP, NTP, Syslog, AAA, routing protocols, CoPP, etc. <br \/>\nThe second organizes services for the client: MPLS L2\/L3VPN, GRE, VXLAN, VLAN, L2TP, etc. <br \/>\nOf course, there are also edge cases \u2014 where to categorize MPLS LDP, BGP? Routing protocols can also be used for clients. But that is not critical. <\/p>\n<p>Both types of services break down into configuration primitives:<\/p>\n<ul>\n<li>Physical and logical interfaces (tag\/untag, mtu)<\/li>\n<li>IP addresses and VRF (IP, IPv6, VRF)<\/li>\n<li>ACLs and traffic processing policies<\/li>\n<li>Protocols (IGP, BGP, MPLS)<\/li>\n<li>Routing policies (prefix lists, communities, ASN filters).<\/li>\n<li>Service services (SSH, NTP, LLDP, Syslog\u2026)<\/li>\n<li>Etc.<\/li>\n<\/ul>\n<p>\nI can't quite figure out how exactly we will do this. We'll address it in a separate article.<\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/092bc3a4b875f4a6422ee13fb580ba4f.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIf we relate it to real life, we could describe that<br \/>\nThe Leaf switch must have BGP sessions with all connected Spine switches, import connected networks into the process, and accept only networks from a specific prefix from the Spine switches. Limit CoPP IPv6 ND to 10 pps, etc. <br \/>\nIn turn, the Spine switches maintain sessions with all connected Leaf switches, acting as route reflectors, and only accept routes of a certain length and with a specific community from them.<\/p>\n<h2>Component 4. Device Initialization Mechanism<\/h2>\n<p>\nUnder this heading, I am combining many actions that must happen for the device to appear on the radar and be remotely accessible.<\/p>\n<ol>\n<li>Add the device to the inventory system.<\/li>\n<li>Allocate a management IP address.<\/li>\n<li>Configure basic access to it: <br \/>\nHostname, management IP address, route to the management network, users, SSH keys, protocols \u2014 telnet\/SSH\/NETCONF<\/li>\n<\/ol>\n<p>There are three approaches here:<\/p>\n<ul>\n<li>Everything entirely manual. The device is brought to the stand where an ordinary organic person will add it to the systems, connect via the console, and configure it. This might work for small static networks.<\/li>\n<li>ZTP \u2014 Zero Touch Provisioning. The hardware arrives, powers on, obtains an address via DHCP, goes to a special server, and self-configures. <\/li>\n<li>Infrastructure of console servers where the initial configuration takes place through the console port in automatic mode.<\/li>\n<\/ul>\n<p>We'll talk about all three in a separate article.<\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/6d8d061e53b508edbbcbd98fe496ca91.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h2>Component 5. Vendor-Agnostic Configuration Model<\/h2>\n<p>\nUntil now, all systems have been disparate patches providing variables and declarative descriptions of what we would like to see in the network. But sooner or later, we will have to deal with specifics. <br \/>\nAt this stage, for each specific device, primitives, services, and variables are combined into a configuration model that effectively describes the complete configuration of that device, but in a vendor-independent manner. <br \/>\nWhat does this step provide? Why not form the device configuration directly, which can simply be uploaded? <br \/>\nIn fact, it allows us to solve three tasks:<\/p>\n<ol>\n<li> Do not adapt to a specific device interaction interface. Whether it's CLI, NETCONF, RESTCONF, or SNMP, the model will be the same.<\/li>\n<li> Do not maintain the number of templates\/scripts based on the number of vendors in the network; if the design changes, update the same thing in multiple places.<\/li>\n<li> Download the configuration from the device (backup), structure it into the exact same model, and directly compare the target configuration with the existing one to calculate the delta and prepare a configuration patch that will change only the necessary parts or identify discrepancies.<\/li>\n<\/ol>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/a154c42ebe7978fcc5b2ce4f235516c9.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAs a result of this stage, we obtain a vendor-independent configuration.<\/p>\n<h2>Component 6. Vendor-specific driver interface<\/h2>\n<p>\nOne should not indulge in hopes that configuring Cisco will ever be as straightforward as configuring Juniper, simply by sending them exactly the same calls. Despite the rising popularity of whiteboxes and the emergence of support for NETCONF, RESTCONF, and OpenConfig, the specific content delivered by these protocols varies from vendor to vendor, and this is one of their competitive advantages that they will not easily relinquish. <br \/>\nIt's roughly the same as OpenContrail and OpenStack, which have RestAPI as their NorthBound interface, expecting completely different calls. <\/p>\n<p>So, in the fifth step, the vendor-independent model must take the form in which it will be sent to the hardware. <br \/>\nAnd here, anything goes (not really): CLI, NETCONF, RESTCONF, SNMP, simply due to necessity.<\/p>\n<p>Therefore, we will need a driver that will transform the result of the previous step into the required format for a specific vendor: a set of CLI commands or XML structure.<\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/6ed8e935183075fda60a209477f84937.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h2>Component 7. Mechanism for delivering configuration to the device<\/h2>\n<p>\nWe have generated the configuration, but it still needs to be delivered to the devices \u2014 and obviously, not manually. <br \/>\n<b>Firstly<\/b>, we face the question of which transport to use? The choice today is quite vast:<\/p>\n<ul>\n<li>CLI (telnet, ssh)<\/li>\n<li>SNMP<\/li>\n<li>NETCONF<\/li>\n<li>RESTCONF<\/li>\n<li>REST API<\/li>\n<li>OpenFlow (although it stands out from the list, as it's a way to deliver FIB rather than configuration)<\/li>\n<\/ul>\n<p>Let's clarify things here. CLI is legacy. SNMP... ahem.<br \/>\nRESTCONF is still an unknown creature; REST API is supported by almost no one. Therefore, we will focus on NETCONF in this iteration.<\/p>\n<p>As the reader may already understand, we have settled on the interface by this point \u2014 the result of the previous step has already been presented in the format of the chosen interface.<\/p>\n<p><b>Secondly<\/b>, but what tools will we use to do this?<br \/>\nThere are also many choices here:<\/p>\n<ul>\n<li>A custom script or platform. We will equip ourselves with ncclient and asyncIO and do it ourselves. How hard can it be to build a deployment system from scratch?<\/li>\n<li>Ansible, with its rich library of network modules.<\/li>\n<li>Salt, with its limited network functionality and integration with Napalm.<\/li>\n<li>Napalm itself, which knows a couple of vendors and that's it, farewell.<\/li>\n<li>Nornir \u2014 another tool we'll dissect in the future.<\/li>\n<\/ul>\n<p>Here a favorite has not yet been chosen \u2014 we will have to explore.<\/p>\n<p>What else is important here? The consequences of applying the configuration. <br \/>\nWhether it succeeded or not. Whether access to the device remains or not. <br \/>\nIt seems that a commit with confirmation and validation of what has been loaded onto the device will help here. <br \/>\nThis, combined with the correct implementation of NETCONF, significantly narrows the range of suitable devices \u2014 normal commits are not supported by many manufacturers. But this is just one of the necessary conditions in <noindex><a rel=\"nofollow\" href=\"https:\/\/ru.wikipedia.org\/wiki\/%D0%97%D0%B0%D0%BF%D1%80%D0%BE%D1%81_%D0%BF%D1%80%D0%B5%D0%B4%D0%BB%D0%BE%D0%B6%D0%B5%D0%BD%D0%B8%D1%8F\">RFP<\/a><\/noindex>. After all, no one worries that no Russian vendor will meet the condition of a 32*100GE interface. Or do they?<\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/59d8ad3b55b9c107642550bb06fc759a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h2>Component 8. CI\/CD<\/h2>\n<p>\nBy this point, we already have the configuration ready for all the devices in the network.<br \/>\nI say \"for all\" because we are talking about versioning the state of the network. And even if it\u2019s necessary to change the settings of just one switch, changes are calculated for the entire network. Obviously, they can be zero for most nodes.<\/p>\n<p>But, as already mentioned above, we are not some kind of barbarians to deploy everything to production at once. <br \/>\nThe generated configuration must go through the CI\/CD Pipeline first.<\/p>\n<blockquote><p>CI\/CD stands for Continuous Integration, Continuous Deployment. This approach allows the team to not just release a new major version every six months, completely replacing the old one, but to regularly implement new functionality incrementally (Deployment) in small portions, each of which is thoroughly tested for compatibility, security, and functionality (Integration).\n<\/p><\/blockquote>\n<p>For this, we have a version control system that tracks changes in configuration, a lab where we check whether the client service remains intact, a monitoring system that verifies this fact, and the final step \u2014 deploying changes to the production network.<\/p>\n<p>With the exception of debugging commands, absolutely all changes to the network must go through the CI\/CD Pipeline \u2014 this is our key to a peaceful life and a long, happy career.<\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/101a5ff87c7a9c7b6776aca9daf762a4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h2>Component 9. Backup and deviation detection system<\/h2>\n<p>\nWell, there's no need to go over backups again. <br \/>\nWe will just store them on a cron schedule or when there is a change in configuration in Git.<\/p>\n<p>Now the second part is more interesting \u2014 someone has to keep an eye on these backups. In some cases, this person must restore everything to its previous state, and in others, they need to notify someone about any irregularities. <br \/>\nFor example, if a new user appears who is not listed in the variables, it needs to be removed quickly to avoid any hacks. And if there is a new firewall rule \u2014 it\u2019s better not to change it; someone might just have enabled debugging, or maybe a careless new service wrote its configuration incorrectly, and people have already started using it.<\/p>\n<p>We cannot escape some minor delta on the scale of the entire network, regardless of any automation systems and the firm hand of management. For troubleshooting issues, no one will still make changes to the configuration in the systems. Moreover, the configuration model may not even anticipate them. <\/p>\n<blockquote><p>For instance, a firewall rule for counting packets to a specific IP, used for localizing problems \u2014 is quite a standard temporary configuration.\n<\/p><\/blockquote>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/390f78139d336f1fc2c60573bccfe76c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h2>Component 10. Monitoring system<\/h2>\n<p>\nAt first, I didn\u2019t plan to address the topic of monitoring \u2014 it is indeed a vast, contentious, and complex subject. But it turned out that it is an integral part of automation. We can't overlook it even without practical experience. <\/p>\n<p>Further developing the thought \u2014 it is an organic part of the CI\/CD process. After deploying the configuration to the network, we need to be able to determine if everything is now in order. <br \/>\nAnd it's not only about usage graphs of interfaces or node availability, but also about more nuanced aspects \u2014 the presence of necessary routes, attributes, the number of BGP sessions, OSPF neighbors, and the end-to-end operability of the higher-level services.<br \/>\nHave the syslogs stopped being sent to the external server, has the SFlow agent failed, have drops started to increase in the queues, and has the connectivity between any pair of prefixes been disrupted?<\/p>\n<p>In a separate article, we will ponder this as well.<\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/08aeee4939081abb21e61abadd0ce5de.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><img decoding=\"async\" alt=\"Automation for the smallest. Zero part. Planning\" src=\"\/wp-content\/uploads\/2019\/07\/13c186070c85fd00f7b6923df154f1b6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<h1>Conclusion<\/h1>\n<p>\nI chose one of the modern data center network designs as a base \u2014 L3 Clos Fabric with BGP as the routing protocol.<br \/>\nThis time, we will build the network using Juniper, because now the JunOs interface is just a breeze.<\/p>\n<p>We will complicate our lives by using only Open Source tools and a multi-vendor network \u2014 so besides Juniper, I'll choose another lucky winner as we go along.<\/p>\n<p>The plan for upcoming publications is roughly as follows:<br \/>\nFirst, I will talk about virtual networks. Firstly, because I want to, and secondly, because without this, the design of the infrastructure network will not be very clear.<br \/>\nThen we will discuss the design of the network itself: topology, routing, policies.<br \/>\nWe will assemble a lab setup.<br \/>\nWe will reflect and perhaps practice initializing a device in the network.<br \/>\nNext, we will go into intimate details about each component.<\/p>\n<p>And yes, I don't promise to gracefully conclude this cycle with a ready solution. \ud83d\ude42<\/p>\n<h5>Useful links<\/h5>\n<p><\/p>\n<ul>\n<li>Before delving into the series, it's worth reading Natasha Samoylenko's book <noindex><a rel=\"nofollow\" href=\"https:\/\/natenka.gitbook.io\/pyneng\/\">Python for Network Engineers.<\/a><\/noindex>. And perhaps also go through <noindex><a rel=\"nofollow\" href=\"https:\/\/natenka.github.io\/pyneng-online\/\">a course<\/a><\/noindex>.<\/li>\n<li>It will also be useful to read <noindex><a rel=\"nofollow\" href=\"https:\/\/tools.ietf.org\/html\/rfc7938\">RFC<\/a><\/noindex> on the design of data center fabrics from Facebook authored by Petr Lapukhov.<\/li>\n<li>Documentation on architecture will provide insight into how Overlay-based SDN works. <noindex><a rel=\"nofollow\" href=\"https:\/\/tungstenfabric.github.io\/website\/Tungsten-Fabric-Architecture.html\">Tungsten Fabric<\/a><\/noindex> (previously Open Contrail).<\/li>\n<\/ul>\n<h5>Thanks<\/h5>\n<p>\nRomanorge. For comments and edits.<br \/>\nArtem Chernobay. For the KDPV.<br \/>\n<br \/>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/453516\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421\u0414\u0421\u041c \u0437\u0430\u043a\u043e\u043d\u0447\u0438\u043b\u0441\u044f, \u0430 \u0431\u0435\u0441\u043a\u043e\u043d\u0442\u0440\u043e\u043b\u044c\u043d\u043e\u0435 \u0436\u0435\u043b\u0430\u043d\u0438\u0435 \u043f\u0438\u0441\u0430\u0442\u044c \u2014 \u043e\u0441\u0442\u0430\u043b\u043e\u0441\u044c. \u0414\u043e\u043b\u0433\u0438\u0435 \u0433\u043e\u0434\u044b \u043d\u0430\u0448 \u0431\u0440\u0430\u0442 \u0441\u0442\u0440\u0430\u0434\u0430\u043b \u043e\u0442 \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u0440\u0443\u0442\u0438\u043d\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u044b, \u0441\u043a\u0440\u0435\u0449\u0438\u0432\u0430\u043b \u043f\u0430\u043b\u044c\u0446\u044b \u043f\u0435\u0440\u0435\u0434 \u043a\u043e\u043c\u043c\u0438\u0442\u043e\u043c \u0438 \u043d\u0435\u0434\u043e\u0441\u044b\u043f\u0430\u043b \u0438\u0437-\u0437\u0430 \u043d\u043e\u0447\u043d\u044b\u0445 \u0440\u043e\u043b\u0431\u044d\u043a\u043e\u0432. \u041d\u043e \u0442\u0451\u043c\u043d\u044b\u043c \u0432\u0440\u0435\u043c\u0435\u043d\u0430\u043c \u043f\u0440\u0438\u0445\u043e\u0434\u0438\u0442 \u043a\u043e\u043d\u0435\u0446. \u042d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0451\u0439 \u044f \u043d\u0430\u0447\u043d\u0443 \u0441\u0435\u0440\u0438\u044e \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u043d\u0435 \u0432\u0438\u0434\u0438\u0442\u0441\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u044f. \u041f\u043e \u0445\u043e\u0434\u0443 \u0434\u0435\u043b\u0430 \u0440\u0430\u0437\u0431\u0435\u0440\u0451\u043c\u0441\u044f \u0441 \u044d\u0442\u0430\u043f\u0430\u043c\u0438 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438, \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0435\u043c \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0445, \u0444\u043e\u0440\u043c\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0435\u0439 \u0434\u0438\u0437\u0430\u0439\u043d\u0430, \u0441 RestAPI, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":26891,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-35972","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.1.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0421\u0414\u0421\u041c \u0437\u0430\u043a\u043e\u043d\u0447\u0438\u043b\u0441\u044f, \u0430 \u0431\u0435\u0441\u043a\u043e\u043d\u0442\u0440\u043e\u043b\u044c\u043d\u043e\u0435 \u0436\u0435\u043b\u0430\u043d\u0438\u0435 \u043f\u0438\u0441\u0430\u0442\u044c \u2014 \u043e\u0441\u0442\u0430\u043b\u043e\u0441\u044c.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/avtomatizatsiya-dlya-samyh-malenkih-chast-nulevaya-planirovanie\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.1.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0410\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u044f \u0434\u043b\u044f \u0441\u0430\u043c\u044b\u0445 \u043c\u0430\u043b\u0435\u043d\u044c\u043a\u0438\u0445. \u0427\u0430\u0441\u0442\u044c \u043d\u0443\u043b\u0435\u0432\u0430\u044f. \u041f\u043b\u0430\u043d\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0421\u0414\u0421\u041c \u0437\u0430\u043a\u043e\u043d\u0447\u0438\u043b\u0441\u044f, \u0430 \u0431\u0435\u0441\u043a\u043e\u043d\u0442\u0440\u043e\u043b\u044c\u043d\u043e\u0435 \u0436\u0435\u043b\u0430\u043d\u0438\u0435 \u043f\u0438\u0441\u0430\u0442\u044c \u2014 \u043e\u0441\u0442\u0430\u043b\u043e\u0441\u044c.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/avtomatizatsiya-dlya-samyh-malenkih-chast-nulevaya-planirovanie\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:08:54+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:08:54+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Automation for the smallest. Part zero. Planning | ProHoster","description":"The SDSM has ended, but the uncontrollable desire to write remains.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/avtomatizatsiya-dlya-samyh-malenkih-chast-nulevaya-planirovanie","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0410\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u044f \u0434\u043b\u044f \u0441\u0430\u043c\u044b\u0445 \u043c\u0430\u043b\u0435\u043d\u044c\u043a\u0438\u0445. \u0427\u0430\u0441\u0442\u044c \u043d\u0443\u043b\u0435\u0432\u0430\u044f. \u041f\u043b\u0430\u043d\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 | ProHoster","og:description":"\u0421\u0414\u0421\u041c \u0437\u0430\u043a\u043e\u043d\u0447\u0438\u043b\u0441\u044f, \u0430 \u0431\u0435\u0441\u043a\u043e\u043d\u0442\u0440\u043e\u043b\u044c\u043d\u043e\u0435 \u0436\u0435\u043b\u0430\u043d\u0438\u0435 \u043f\u0438\u0441\u0430\u0442\u044c \u2014 \u043e\u0441\u0442\u0430\u043b\u043e\u0441\u044c.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/avtomatizatsiya-dlya-samyh-malenkih-chast-nulevaya-planirovanie","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:08:54+00:00","article:modified_time":"2019-10-31T19:08:54+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"35972","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 01:28:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 09:58:22","updated":"2026-01-22 01:28:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35972","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=35972"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35972\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/26891"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=35972"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=35972"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=35972"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}