Multivan and routing on Mikrotik RouterOS

Introduction

Taking up the article, in addition to vanity, was prompted by the depressing frequency of questions on this topic in the profile groups of the Russian-speaking telegram community. The article is aimed at novice Mikrotik RouterOS (hereinafter referred to as ROS) administrators. It deals only with the multivan, with an emphasis on routing. As a bonus, there are minimally sufficient settings to ensure safe and convenient operation. Those who are looking for disclosure of the topics of queues, load balancing, vlans, bridges, multi-stage deep analysis of the state of the channel and the like - may not waste time and effort reading.

Initial data

As a test subject, a five-port Mikrotik router with ROS version 6.45.3 was selected. It will route traffic between two local networks (LAN1 and LAN2) and three providers (ISP1, ISP2, ISP3). The channel to ISP1 has a static "gray" address, ISP2 - "white", obtained via DHCP, ISP3 - "white" with PPPoE authorization. The connection diagram is shown in the figure:

The task is to configure the MTK router based on the scheme so that:

1. Provide automatic switching to a backup provider. The main provider is ISP2, the first reserve is ISP1, the second reserve is ISP3.
2. Organize LAN1 network access to the Internet only through ISP1.
3. Provide the ability to route traffic from local networks to the Internet through the selected provider based on the address-list.
4. Provide for the possibility of publishing services from the local network to the Internet (DSTNAT)
5. Set up a firewall filter to provide the minimum sufficient security from the Internet.
6. The router could issue its own traffic through any of the three providers, depending on the chosen source address.
7. Ensure that response packets are routed to the channel from which they came (including LAN).

Comment. We will configure the router “from scratch” in order to guarantee the absence of surprises in the starting configurations “out of the box” that change from version to version. Winbox was chosen as a configuration tool, where changes will be visually displayed. The settings themselves will be set by commands in the Winbox terminal. The physical connection for configuration is made by a direct connection to the Ether5 interface.

A bit of reasoning about what a multivan is, is it a problem or are cunning smart people around weaving conspiracy networks

An inquisitive and attentive administrator, setting up such or a similar scheme on his own, suddenly suddenly realizes that it is already working normally. Yes, yes, without your custom routing tables and other route rules, which most articles on this topic are full of. Let's check?

Can we configure addressing on interfaces and default gateways? Yes:

On ISP1, the address and gateway were registered with distance=2 и check-gateway=ping.
On ISP2, the default dhcp client setting - accordingly, distance will be equal to one.
On ISP3 in the pppoe client settings when add-default-route=yes put default-route-distance=3.

Do not forget to register NAT on the exit:

/ip firewall nat add action=masquerade chain=srcnat out-interface-list=WAN

As a result, users of local sites have fun downloading cats through the main ISP2 provider and there is a channel reservation using the mechanism check gateway See note 1

Point 1 of the task is implemented. Where is the multivan with its marks? No…

Further. You need to release specific clients from the LAN via ISP1:

/ip firewall mangle add action=route chain=prerouting dst-address-list=!BOGONS
passthrough=yes route-dst=100.66.66.1 src-address-list=Via_ISP1
/ip firewall mangle add action=route chain=prerouting dst-address-list=!BOGONS
passthrough=no route-dst=100.66.66.1 src-address=192.168.88.0/24

Items 2 and 3 of the task have been implemented. Labels, stamps, route rules, where are you?!

Need to give access to your favorite OpenVPN server with the address 172.17.17.17 for clients from the Internet? Please:

/ip cloud set ddns-enabled=yes

As a peer, we give the client the output result: “:put [ip cloud get dns-name]"

We register port forwarding from the Internet:

/ip firewall nat add action=dst-nat chain=dstnat dst-port=1194
in-interface-list=WAN protocol=udp to-addresses=172.17.17.17

Item 4 is ready.

We set up a firewall and other security for point 5, at the same time we are glad that everything is already working for users and reach for a container with a favorite drink ...
A! Tunnels are forgotten.

l2tp-client, configured by google article, has risen to your favorite Dutch VDS? Yes.
l2tp-server with IPsec has risen and clients by DNS-name from IP Cloud (see above.) cling? Yes.
Leaning back in our chair, sipping a drink, we lazily consider points 6 and 7 of the task. We think - do we need it? All the same, it works like that (c) ... So, if it is still not needed, then that's it. Multivan implemented.

What is a multivan? This is the connection of several Internet channels to one router.

You don’t have to read the article further, because what can be there besides a show off of dubious applicability?

For those who remain, who are interested in points 6 and 7 of the task, and also feel the itch of perfectionism, we dive deeper.

The most important task of implementing a multivan is the correct traffic routing. Namely: regardless of which (or which) See. note 3 the ISP's channel(s) look at the default route on our router, it should return a response to the exact channel the packet came from. The task is clear. Where is the problem? Indeed, in a simple local network, the task is the same, but no one bothers with additional settings and does not feel trouble. The difference is that any routable node on the Internet is accessible through each of our channels, and not through a strictly specific one, as in a simple LAN. And the “trouble” is that if a request came to us for the IP address of ISP3, then in our case the answer will go through the ISP2 channel, since the default gateway is directed there. Leaves and will be discarded by the provider as incorrect. The problem has been identified. How to solve it?

The solution is divided into three stages:

1. Presetting. At this stage, the basic settings of the router will be set: local network, firewall, address lists, hairpin NAT, etc.
2. Multivan. At this stage, the necessary connections will be marked and sorted into routing tables.
3. Connecting to an ISP. At this stage, the interfaces that provide connection to the Internet will be configured, routing and the Internet channel reservation mechanism will be activated.

1. Presetting

1.1. We clear the router configuration with the command:

/system reset-configuration skip-backup=yes no-defaults=yes

agree with "Dangerous! Reset anyway? [y/N]:” and, after rebooting, we connect with Winbox via MAC. At this stage, the configuration and user base are cleared.

1.2. Create a new user:

/user add group=full name=knight password=ultrasecret comment=”Not horse”

log in under it and delete the default one:

/user remove admin

Comment. It is the removal and not disabling of the default user that the author considers safer and recommends for use.

1.3. We create basic interface lists for the convenience of operating in a firewall, discovery settings and other MAC servers:

/interface list add name=WAN comment="For Internet"
/interface list add name=LAN comment="For Local Area"

Signing interfaces with comments

/interface ethernet set ether1 comment="to ISP1"
/interface ethernet set ether2 comment="to ISP2"
/interface ethernet set ether3 comment="to ISP3"
/interface ethernet set ether4 comment="to LAN1"
/interface ethernet set ether5 comment="to LAN2"

and fill in the interface lists:

/interface list member add interface=ether1 list=WAN comment=ISP1
/interface list member add interface=ether2 list=WAN comment=ISP2 
/interface list member add interface=ether3 list=WAN comment="to ISP3"
/interface list member add interface=ether4 list=LAN  comment="LAN1"
/interface list member add interface=ether5 list=LAN  comment="LAN2"

Comment. Writing understandable comments is worth the time spent on this, plus it greatly facilitates troubleshooting and understanding the configuration.

The author considers it necessary, for security reasons, to add the ether3 interface to the “WAN” interface list, despite the fact that the ip protocol will not go through it.

Do not forget that after the PPP interface is raised on ether3, it will also need to be added to the interface list “WAN”

1.4. We hide the router from neighborhood detection and control from provider networks via MAC:

/ip neighbor discovery-settings set discover-interface-list=!WAN
/tool mac-server set allowed-interface-list=LAN
/tool mac-server mac-winbox set allowed-interface-list=LAN

1.5. We create the minimum sufficient set of firewall filter rules to protect the router:

/ip firewall filter add action=accept chain=input comment="Related Established Untracked Allow" 
connection-state=established,related,untracked

(the rule provides permission for established and related connections that are initiated from both connected networks and the router itself)

/ip firewall filter add action=accept chain=input comment="ICMP from ALL" protocol=icmp

(ping and not only ping. All icmp is allowed in. Very useful for finding MTU problems)

/ip firewall filter add action=drop chain=input comment="All other WAN Drop" in-interface-list=WAN

(the rule that closes the input chain forbids everything else that comes from the Internet)

/ip firewall filter add action=accept chain=forward 
comment="Established, Related, Untracked allow" 
connection-state=established,related,untracked

(the rule allows established and related connections that pass through the router)

/ip firewall filter add action=drop chain=forward comment="Invalid drop" connection-state=invalid

(the rule resets connections with connection-state=invalid passing through the router. It is strongly recommended by Mikrotik, but in some rare situations it can block useful traffic)

/ip firewall filter add action=drop chain=forward comment="Drop all from WAN not DSTNATed"  
connection-nat-state=!dstnat connection-state=new in-interface-list=WAN

(the rule forbids packets that come from the Internet and have not passed the dstnat procedure to pass through the router. This will protect local networks from intruders who, being in the same broadcast domain with our external networks, will register our external IPs as a gateway and, thus, try to “explore” our local networks.)

Comment. Let us assume that the networks LAN1 and LAN2 are trusted and the traffic between them and from them is not filtered.

1.6. Create a list with a list of non-routable networks:

/ip firewall address-list
add address=0.0.0.0/8 comment=""This" Network" list=BOGONS
add address=10.0.0.0/8 comment="Private-Use Networks" list=BOGONS
add address=100.64.0.0/10 comment="Shared Address Space. RFC 6598" list=BOGONS
add address=127.0.0.0/8 comment=Loopback list=BOGONS
add address=169.254.0.0/16 comment="Link Local" list=BOGONS
add address=172.16.0.0/12 comment="Private-Use Networks" list=BOGONS
add address=192.0.0.0/24 comment="IETF Protocol Assignments" list=BOGONS
add address=192.0.2.0/24 comment=TEST-NET-1 list=BOGONS
add address=192.168.0.0/16 comment="Private-Use Networks" list=BOGONS
add address=198.18.0.0/15 comment="Network Interconnect Device Benchmark Testing"
 list=BOGONS
add address=198.51.100.0/24 comment=TEST-NET-2 list=BOGONS
add address=203.0.113.0/24 comment=TEST-NET-3 list=BOGONS
add address=224.0.0.0/4 comment=Multicast list=BOGONS
add address=192.88.99.0/24 comment="6to4 Relay Anycast" list=BOGONS
add address=240.0.0.0/4 comment="Reserved for Future Use" list=BOGONS
add address=255.255.255.255 comment="Limited Broadcast" list=BOGONS

(This is a list of addresses and networks that are not routable to the Internet and will be followed accordingly.)

Comment. The list is subject to change, so I advise you to periodically check the relevance.

1.7. Set up DNS for the router itself:

/ip dns set servers=1.1.1.1,8.8.8.8

Comment. In the current version of ROS, dynamic servers take precedence over static ones. The name resolution request is sent to the first server in order in the list. The transition to the next server is carried out when the current one is unavailable. The timeout is large - more than 5 seconds. Returning back, when the “fallen server” is resumed, does not automatically occur. Given this algorithm and the presence of a multivan, the author recommends not using servers provided by providers.

1.8. Set up a local network.
1.8.1. We configure static IP addresses on LAN interfaces:

/ip address add interface=ether4 address=192.168.88.254/24 comment="LAN1 IP"
/ip address add interface=ether5 address=172.16.1.0/23 comment="LAN2 IP"

1.8.2. We set the rules for routes to our local networks through the main routing table:

/ip route rule add dst-address=192.168.88.0/24 table=main comment=”to LAN1”
/ip route rule add dst-address=172.16.0.0/23 table=main comment="to LAN2"

Comment. This is one of the quick and easy ways to access LAN addresses with sources of external IP addresses of router interfaces that do not go through the default route.

1.8.3. Enable Hairpin NAT for LAN1 and LAN2:

/ip firewall nat add action=src-nat chain=srcnat comment="Hairpin to LAN1" 
out-interface=ether4 src-address=192.168.88.0/24 to-addresses=192.168.88.254
/ip firewall nat add action=src-nat chain=srcnat comment="Hairpin to LAN2" 
out-interface=ether5 src-address=172.16.0.0/23 to-addresses=172.16.1.0

Comment. This allows you to access your resources (dstnat) via an external IP while being inside the network.

2. Actually, the implementation of the very correct multivan

To solve the problem of “answering where they asked from”, we will use two ROS tools: connection mark и routing mark. connection mark allows you to mark the desired connection and then work with this label as a condition for applying routing mark. And already with routing mark possible to work in ip route и route rules. We figured out the tools, now you need to decide which connections to mark - once, exactly where to mark - two.

With the first one, everything is simple - we must mark all the connections that come to the router from the Internet via the appropriate channel. In our case, these will be three labels (by the number of channels): “conn_isp1”, “conn_isp2” and “conn_isp3”.

The nuance with the second is that incoming connections will be of two types: transit and those that are intended for the router itself. The connection mark mechanism works in the table mangrove. Consider the movement of the package on a simplified diagram, kindly compiled by the specialists of the mikrotik-trainings.com resource (not advertising):

Following the arrows, we see that the packet arriving at “input interface”, goes through the chain “Prerouting” and only then is it divided into transit and local in the block “Routing decision". Therefore, to kill two birds with one stone, we use Connection Mark in the table Mangle Pre-routing chains Prerouting.

Comment. In ROS, “Routing mark” labels are listed as “Table” in the Ip/Routes/Rules section, and as “Routing Mark” in other sections. This may introduce some confusion into understanding, but, in fact, this is the same thing, and is an analogue of rt_tables in iproute2 on linux.

2.1. We mark incoming connections from each of the providers:

/ip firewall mangle add action=mark-connection chain=prerouting 
comment="Connmark in from ISP1" connection-mark=no-mark in-interface=ether1  new-connection-mark=conn_isp1 passthrough=no

/ip firewall mangle add action=mark-connection chain=prerouting 
comment="Connmark in from ISP2" connection-mark=no-mark in-interface=ether2  new-connection-mark=conn_isp2 passthrough=no

/ip firewall mangle add action=mark-connection chain=prerouting 
comment="Connmark in from ISP3" connection-mark=no-mark in-interface=pppoe-isp3  new-connection-mark=conn_isp3 passthrough=no

Comment. In order not to mark already marked connections, I use the connection-mark=no-mark condition instead of connection-state=new because I think this is more correct, as well as the rejection of drop invalid connections in the input filter.

passthrough=no - because in this implementation method, re-marking is excluded and, to speed up, you can interrupt the enumeration of rules after the first match.

It should be borne in mind that we do not interfere in any way with routing yet. Now there are only stages of preparation. The next stage of implementation will be the processing of transit traffic that returns over the established connection from the destination in the local network. Those. those packets that (see the diagram) passed through the router along the way:

“Input Interface”=>”Prerouting”=>”Routing Decision”=>”Forward”=>”Post Routing”=>”Output Interface” and got to their addressee in the local network.

Important! In ROS, there is no logical division into external and internal interfaces. If we trace the path of the response packet according to the above diagram, then it will follow the same logical path as the request:

“Input Interface”=>”Prerouting”=>”Routing Decision”=>”Forward”=>”Post Routing”=>”Output Interface” just for a request"Input Interface” was the ISP interface, and for the answer - LAN

2.2. We direct response transit traffic to the corresponding routing tables:

/ip firewall mangle add action=mark-routing chain=prerouting 
comment="Routemark transit out via ISP1" connection-mark=conn_isp1 
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp1 passthrough=no

/ip firewall mangle add action=mark-routing chain=prerouting 
comment="Routemark transit out via ISP2" connection-mark=conn_isp2 
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp2 passthrough=no

/ip firewall mangle add action=mark-routing chain=prerouting 
comment="Routemark transit out via ISP3" connection-mark=conn_isp3 
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp3 passthrough=no

Comment. in-interface-list=!WAN - we work only with traffic from the local network and dst-address-type=!local that does not have the destination address of the address of the interfaces of the router itself.

The same for local packets that came to the router along the way:

“Input Interface”=>”Prerouting”=>”Routing Decision”=>”Input”=>”Local Process”

Important! The answer will go in the following way:

”Local Process”=>”Routing Decision”=>”Output”=>”Post Routing”=>”Output Interface”

2.3. We direct response local traffic to the corresponding routing tables:

/ip firewall mangle add action=mark-routing chain=output 
comment="Routemark local out via ISP1" connection-mark=conn_isp1 dst-address-type=!local 
new-routing-mark=to_isp1 passthrough=no

/ip firewall mangle add action=mark-routing chain=output 
comment="Routemark local out via ISP2" connection-mark=conn_isp2 dst-address-type=!local 
new-routing-mark=to_isp2 passthrough=no

/ip firewall mangle add action=mark-routing chain=output 
comment="Routemark local out via ISP3" connection-mark=conn_isp3 dst-address-type=!local 
new-routing-mark=to_isp3 passthrough=no

At this stage, the task of preparing to send a response to the Internet channel from which the request came can be considered solved. Everything is marked, labeled and ready to be routed.
An excellent “side” effect of this setup is the ability to work with DSNAT port forwarding from both (ISP2, ISP3) providers at the same time. Not at all, since on ISP1 we have a non-routable address. This effect is important, for example, for a mail server with two MXs that look at different Internet channels.

To eliminate the nuances of the operation of local networks with external IP routers, we use the solutions from paragraphs. 1.8.2 and 3.1.2.6.

In addition, you can use a tool with markings to solve paragraph 3 of the problem. We implement it like this:

2.4. We direct traffic from local clients from the routing lists to the appropriate tables:

/ip firewall mangle add action=mark-routing chain=prerouting 
comment="Address List via ISP1" dst-address-list=!BOGONS new-routing-mark=to_isp1 
passthrough=no src-address-list=Via_ISP1

/ip firewall mangle add action=mark-routing chain=prerouting 
comment="Address List via ISP2" dst-address-list=!BOGONS new-routing-mark=to_isp2 
passthrough=no src-address-list=Via_ISP2

/ip firewall mangle add action=mark-routing chain=prerouting 
comment="Address List via ISP3" dst-address-list=!BOGONS new-routing-mark=to_isp3 
passthrough=no src-address-list=Via_ISP3

As a result, it looks something like this:

3. Set up a connection to the ISP and enable branded routing

3.1. Set up a connection to ISP1:
3.1.1. Configure a static IP address:

/ip address add interface=ether1 address=100.66.66.2/30 comment="ISP1 IP"

3.1.2. Set up static routing:
3.1.2.1. Add a default "emergency" route:

/ip route add comment="Emergency route" distance=254 type=blackhole

Comment. This route allows traffic from local processes to pass the Route Decision stage, regardless of the state of the links of any of the providers. The nuance of outgoing local traffic is that in order for the packet to move at least somewhere, the main routing table must have an active route to the default gateway. If not, then the package will simply be destroyed.

As a tool extension check gateway For a deeper analysis of the channel state, I suggest using the recursive route method. The essence of the method is that we tell the router to look for a path to its gateway not directly, but through an intermediate gateway. 4.2.2.1, 4.2.2.2 and 4.2.2.3 will be chosen as such “test” gateways for ISP1, ISP2 and ISP3 respectively.

3.1.2.2. Route to the “verification” address:

/ip route add check-gateway=ping comment="For recursion via ISP1"  
distance=1 dst-address=4.2.2.1 gateway=100.66.66.1 scope=10

Comment. We lower the scope value to the default in ROS target scope in order to use 4.2.2.1 as a recursive gateway in the future. I emphasize: the scope of the route to the “test” address must be less than or equal to the target scope of the route that will refer to the test one.

3.1.2.3. Recursive default route for traffic without routing mark:

/ip route add comment="Unmarked via ISP1" distance=2 gateway=4.2.2.1

Comment. The distance=2 value is used because ISP1 is declared as the first backup according to the task conditions.

3.1.2.4. Recursive default route for traffic with routing mark “to_isp1”:

/ip route add comment="Marked via ISP1 Main" distance=1 gateway=4.2.2.1 
routing-mark=to_isp1

Comment. Actually, here we are finally starting to enjoy the fruits of the preparatory work that was carried out in paragraph 2.

On this route, all traffic that has the mark route “to_isp1” will be directed to the gateway of the first provider, regardless of which default gateway is currently active for the main table.

3.1.2.5. First fallback recursive default route for ISP2 and ISP3 tagged traffic:

/ip route add comment="Marked via ISP2 Backup1" distance=2 gateway=4.2.2.1 
routing-mark=to_isp2
/ip route add comment="Marked via ISP3 Backup1" distance=2 gateway=4.2.2.1 
routing-mark=to_isp3

Comment. These routes are needed, among other things, to reserve traffic from local networks that are members of the address list “to_isp*”'

3.1.2.6. We register the route for the local traffic of the router to the Internet through ISP1:

/ip route rule add comment="From ISP1 IP to Inet" src-address=100.66.66.2 table=to_isp1

Comment. In combination with the rules from paragraph 1.8.2, it provides access to the desired channel with a given source. This is critical for building tunnels that specify the local side IP address (EoIP, IP-IP, GRE). Since the rules in ip route rules are executed from top to bottom, until the first match of the conditions, then this rule should be after the rules from clause 1.8.2.

3.1.3. We register the NAT rule for outgoing traffic:

/ip firewall nat add action=src-nat chain=srcnat comment="NAT via ISP1"  
ipsec-policy=out,none out-interface=ether1 to-addresses=100.66.66.2

Comment. NATim everything that goes out, except for what gets into the IPsec policies. I try not to use action=masquerade unless absolutely necessary. It is slower and more resource intensive than src-nat because it calculates the NAT address for each new connection.

3.1.4. We send clients from the list who are prohibited from accessing through other providers directly to the ISP1 provider's gateway.

/ip firewall mangle add action=route chain=prerouting comment="Address List via ISP1 only" 
dst-address-list=!BOGONS passthrough=no route-dst=100.66.66.1 
src-address-list=Via_only_ISP1 place-before=0

Comment. action=route has a higher priority and is applied before other routing rules.

place-before=0 - places our rule first in the list.

3.2. Set up a connection to ISP2.

Since the ISP2 provider gives us the settings via DHCP, it is reasonable to make the necessary changes with a script that starts when the DHCP client is triggered:

/ip dhcp-client
add add-default-route=no disabled=no interface=ether2 script=":if ($bound=1) do={r
    n    /ip route add check-gateway=ping comment="For recursion via ISP2" distance=1 
           dst-address=4.2.2.2/32 gateway=$"gateway-address" scope=10r
    n    /ip route add comment="Unmarked via ISP2" distance=1 gateway=4.2.2.2;r
    n    /ip route add comment="Marked via ISP2 Main" distance=1 gateway=4.2.2.2 
           routing-mark=to_isp2;r
    n    /ip route add comment="Marked via ISP1 Backup1" distance=2 gateway=4.2.2.2 
           routing-mark=to_isp1;r
    n    /ip route add comment="Marked via ISP3 Backup2" distance=3 gateway=4.2.2.2 
           routing-mark=to_isp3;r
    n    /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none 
           out-interface=$"interface" to-addresses=$"lease-address" comment="NAT via ISP2" 
           place-before=1;r
    n    if ([/ip route rule find comment="From ISP2 IP to Inet"] ="") do={r
    n        /ip route rule add comment="From ISP2 IP to Inet" 
               src-address=$"lease-address" table=to_isp2 r
    n    } else={r
    n       /ip route rule set [find comment="From ISP2 IP to Inet"] disabled=no 
              src-address=$"lease-address"r
    n    }      r
    n} else={r
    n   /ip firewall nat remove  [find comment="NAT via ISP2"];r
    n   /ip route remove [find comment="For recursion via ISP2"];r
    n   /ip route remove [find comment="Unmarked via ISP2"];r
    n   /ip route remove [find comment="Marked via ISP2 Main"];r
    n   /ip route remove [find comment="Marked via ISP1 Backup1"];r
    n   /ip route remove [find comment="Marked via ISP3 Backup2"];r
    n   /ip route rule set [find comment="From ISP2 IP to Inet"] disabled=yesr
    n}r
    n" use-peer-dns=no use-peer-ntp=no

The script itself in the Winbox window:

Comment. The first part of the script is triggered when the lease is successfully obtained, the second - after the lease is released.See note 2

3.3. We set up a connection to the ISP3 provider.

Since the settings provider gives us dynamic, it is reasonable to make the necessary changes with scripts that start after the ppp interface has been raised and after the fall.

3.3.1. First we configure the profile:

/ppp profile
add comment="for PPPoE to ISP3" interface-list=WAN name=isp3_client 
on-down="/ip firewall nat remove  [find comment="NAT via ISP3"];r
    n/ip route remove [find comment="For recursion via ISP3"];r
    n/ip route remove [find comment="Unmarked via ISP3"];r
    n/ip route remove [find comment="Marked via ISP3 Main"];r
    n/ip route remove [find comment="Marked via ISP1 Backup2"];r
    n/ip route remove [find comment="Marked via ISP2 Backup2"];r
    n/ip route rule set [find comment="From ISP3 IP to Inet"] disabled=yes;" 
on-up="/ip route add check-gateway=ping comment="For recursion via ISP3" distance=1 
    dst-address=4.2.2.3/32 gateway=$"remote-address" scope=10r
    n/ip route add comment="Unmarked via ISP3" distance=3 gateway=4.2.2.3;r
    n/ip route add comment="Marked via ISP3 Main" distance=1 gateway=4.2.2.3 
    routing-mark=to_isp3;r
    n/ip route add comment="Marked via ISP1 Backup2" distance=3 gateway=4.2.2.3 
    routing-mark=to_isp1;r
    n/ip route add comment="Marked via ISP2 Backup2" distance=3 gateway=4.2.2.3 
    routing-mark=to_isp2;r
    n/ip firewall mangle set [find comment="Connmark in from ISP3"] 
    in-interface=$"interface";r
    n/ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none 
    out-interface=$"interface" to-addresses=$"local-address" comment="NAT via ISP3" 
    place-before=1;r
    nif ([/ip route rule find comment="From ISP3 IP to Inet"] ="") do={r
    n   /ip route rule add comment="From ISP3 IP to Inet" src-address=$"local-address" 
    table=to_isp3 r
    n} else={r
    n   /ip route rule set [find comment="From ISP3 IP to Inet"] disabled=no 
    src-address=$"local-address"r
    n};r
    n"

The script itself in the Winbox window:

Comment. Line
/ip firewall mangle set [find comment="Connmark in from ISP3"] in-interface=$"interface";
allows you to correctly handle the renaming of the interface, since it works with its code and not the display name.

3.3.2. Now, using the profile, create a ppp connection:

/interface pppoe-client add allow=mschap2 comment="to ISP3" disabled=no 
interface=ether3 name=pppoe-isp3 password=isp3_pass profile=isp3_client user=isp3_client

As a final touch, let's set the clock:

/system ntp client set enabled=yes server-dns-names=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org

For those who read to the end

The proposed way to implement a multivan is the personal preference of the author and is not the only possible one. The ROS toolkit is extensive and flexible, which, on the one hand, causes difficulties for beginners, and, on the other hand, is the reason for its popularity. Learn, try, discover new tools and solutions. For example, as an application of the acquired knowledge, it is possible to replace the tool in this implementation of the multivan check-gateway with recursive routes to netwatch.

Notes

1. check-gateway - a mechanism that allows you to deactivate the route after two consecutive unsuccessful checks of the gateway for availability. The check is performed once every 10 seconds, plus the response timeout. In total, the actual switching timing lies in the range of 20-30 seconds. If such switching timing is not sufficient, there is an option to use the tool netwatch, where the check timer can be set manually. check-gateway does not fire on intermittent packet loss on the link.

   Important! Deactivating a primary route will deactivate all other routes that refer to it. Therefore, for them to specify check-gateway=ping not necessary.

2. It happens that a failure occurs in the DHCP mechanism, which looks like a client stuck in the renew state. In this case, the second part of the script will not work, but it will not prevent traffic from walking correctly, since the state tracks the corresponding recursive route.
3. ECMP (Equal Cost Multi-Path) - in ROS it is possible to set a route with several gateways and the same distance. In this case, connections will be distributed across channels using the round robin algorithm, in proportion to the number of specified gateways.

For the impetus to write the article, help in shaping its structure and placement of accents - personal gratitude to Evgeny @jscar

Source: habr.com