Basics of Static Routing in Mikrotik RouterOS

Routing is the process of finding the best path for transmitting packets over TCP/IP networks. Any device connected to an IPv4 network contains a process and routing tables.

This article is not a HOWTO, it describes static routing in RouterOS with examples, I deliberately omitted the rest of the settings (for example, srcnat for accessing the Internet), so understanding the material requires a certain level of knowledge of networks and RouterOS.

Switching and routing

Switching is the process of exchanging packets within one Layer2 segment (Ethernet, ppp, ...). If the device sees that the recipient of the packet is on the same Ethernet subnet with it, it learns the mac address using the arp protocol and transmits the packet directly, bypassing the router. A ppp (point-to-point) connection can have only two participants and the packet is always sent to one address 0xff.

Routing is the process of transferring packets between Layer2 segments. If a device wants to send a packet whose recipient is outside the Ethernet segment, it looks into its routing table and passes the packet to the gateway, which knows where to send the packet next (or may not know, the original sender of the packet is not aware of this).

The easiest way to think of a router is as a device connected to two or more Layer2 segments and capable of passing packets between them by determining the best route from the routing table.

If you understand everything, or you already knew it, then read on. For the rest, I strongly recommend that you familiarize yourself with a small, but very capacious article.

Routing in RouterOS and PacketFlow

Almost all the functionality related to static routing is in the package system... Package routing adds support for dynamic routing algorithms (RIP, OSPF, BGP, MME), Routing Filters and BFD.

Main menu for setting up routing: [IP]->[Route]. Complex schemes may require packets to be pre-labeled with a routing mark in: [IP]->[Firewall]->[Mangle] (chains PREROUTING и OUTPUT).

There are three places on PacketFlow where IP packet routing decisions are made:

1. Routing packets received by the router. At this stage, it is decided whether the packet will go to the local process or will be sent further to the network. Transit packages receive output interface
2. Routing local outgoing packets. Outgoing packets receive output interface
3. Additional routing step for outgoing packets, allows you to change the routing decision in [Output|Mangle]

• The packet path in blocks 1, 2 depends on the rules in [IP]->[Route]
• The packet path in points 1, 2 and 3 depends on the rules in [IP]->[Route]->[Rules]
• The package path in blocks 1, 3 can be influenced using [IP]->[Firewall]->[Mangle]

RIB, FIB, Routing Cache

Routing Information Base
The base in which routes are collected from dynamic routing protocols, routes from ppp and dhcp, static and connected routes. This database contains all routes, except those filtered by the administrator.

Conditionally, we can assume that [IP]->[Route] displays RIB.

Forwarding Information Base

The base in which the best routes from RIB are collected. All routes in the FIB are active and are used to forward packets. If the route becomes inactive (disabled by the administrator (system), or the interface through which the packet should be sent is not active), the route is removed from the FIB.

To make a routing decision, the FIB table uses the following information about an IP packet:

• source address
• Destination address
• source interface
• Routing mark
• ToS (DSCP)

Getting into the FIB package goes through the following stages:

• Is the package intended for a local router process?
• Is the packet subject to system or user PBR rules?
  • If yes, then the packet is sent to the specified routing table
• The packet is sent to the main table

Conditionally, we can assume that [IP]->[Route Active=yes] displays FIB.

Routing Cache
Route caching mechanism. The router remembers where the packets were sent and if there are similar ones (presumably from the same connection) it lets them go along the same route, without checking in the FIB. The route cache is periodically cleared.

For RouterOS administrators, they did not make tools for viewing and managing the Routing Cache, but when it can be disabled in [IP]->[Settings].

This mechanism was removed from the linux 3.6 kernel, but RouterOS still uses kernel 3.3.5, perhaps Routing cahce is one of the reasons.

Add route dialog

[IP]->[Route]->[+]

1. Subnet for which you want to create a route (default: 0.0.0.0/0)
2. Gateway IP or interface to which the packet will be sent (there may be several, see ECMP below)
3. Gateway Availability Check
4. Record type
5. Distance (metric) for a route
6. Routing table
7. IP for local outgoing packets via this route
8. The purpose of Scope and Target Scope is written at the end of the article.

Route flags

• X - The route is disabled by the administrator (disabled=yes)
• A - The route is used to send packets
• D - Route added dynamically (BGP, OSPF, RIP, MME, PPP, DHCP, Connected)
• C - The subnet is connected directly to the router
• S - Static route
• r,b,o,m - Route added by one of the dynamic routing protocols
• B,U,P - Filtering route (drops packets instead of transmitting)

What to specify in gateway: ip-address or interface?

The system allows you to specify both, while it does not swear and does not give hints if you did something wrong.

IP Address
The gateway address must be accessible over Layer2. For Ethernet, this means that the router must have an address from the same subnet on one of the active ip interfaces, for ppp, that the gateway address is specified on one of the active interfaces as the subnet address.
If the accessibility condition for Layer2 is not met, the route is considered inactive and does not fall into the FIB.

Interface
Everything is more complicated and the behavior of the router depends on the type of interface:

• PPP (Async, PPTP, L2TP, SSTP, PPPoE, OpenVPN *) connection assumes only two participants and the packet will always be sent to the gateway for transmission, if the gateway detects that the recipient is itself, then it will transfer the packet to its local process.
  
• Ethernet assumes the presence of many participants and will send requests to the arp interface with the address of the recipient of the packet, this is expected and quite normal behavior for connected routes.
  But when you try to use the interface as a route for a remote subnet, you will get the following situation: the route is active, ping to the gateway passes, but does not reach the recipient from the specified subnet. If you look at the interface through a sniffer, you will see arp requests with addresses from a remote subnet.
  

Try to specify the ip address as the gateway whenever possible. The exception is connected routes (created automatically) and PPP (Async, PPTP, L2TP, SSTP, PPPoE, OpenVPN*) interfaces.

OpenVPN does not contain a PPP header, but you can use the OpenVPN interface name to create a route.

More Specific Route

Basic routing rule. The route that describes the smaller subnet (with the largest subnet mask) takes precedence in the packet's routing decision. The position of the entries in the routing table is not relevant to the choice - the main rule is More Specific.

All routes from the specified scheme are active (located in FIB). point to different subnets and do not conflict with each other.

If one of the gateways becomes unavailable, the associated route will be considered inactive (removed from the FIB) and packets will be searched from the remaining routes.

The route with subnet 0.0.0.0/0 is sometimes given special meaning and is called the "Default Route" or "Gateway of last resort". In fact, there is nothing magical about it and it simply includes all possible IPv4 addresses, but these names describe its task well - it indicates the gateway where to forward packets for which there are no other, more accurate routes.

The maximum possible subnet mask for IPv4 is /32, this route points to a specific host and can be used in the routing table.

Understanding More Specific Route is fundamental to any TCP/IP device.

Distance

Distances (or Metrics) are required for administrative filtering of routes to a single subnet accessible through multiple gateways. A route with a lower metric is considered a priority and will be included in the FIB. If a route with a lower metric ceases to be active, then it will be replaced by a route with a higher metric in the FIB.

If there are several routes to the same subnet with the same metric, the router will add only one of them to the FIB table, guided by its internal logic.

The metric can take a value from 0 to 255:

• 0 - Metric for connected routes. Distance 0 cannot be set by the administrator
• 1-254 - Metrics available to the administrator for setting routes. Metrics with a lower value have a higher priority
• 255 - Metric available to the administrator for setting routes. Unlike 1-254, a route with a metric of 255 always remains inactive and does not fall into the FIB
• specific metrics. Routes derived from dynamic routing protocols have standard metric values

check gateway

Check gateway is a MikroTik RoutesOS extension for checking the availability of the gateway via icmp or arp. Once every 10 seconds (cannot be changed), a request is sent to the gateway, if the response is not received twice, the route is considered unavailable and is removed from the FIB. If check gateway has disabled the check route continues and the route will become active again after one successful check.

Check gateway disables the entry in which it is configured and all other entries (in all routing tables and ecmp routes) with the specified gateway.

In general, check gateway works fine as long as there are no problems with packet loss to the gateway. Check gateway does not know what is happening with communication outside the checked gateway, this requires additional tools: scripts, recursive routing, dynamic routing protocols.

Most VPN and tunnel protocols contain built-in tools for checking connection activity, enabling check gateway for them is an additional (but very small) load on the network and device performance.

ECMP routes

Equal-Cost Multi-Path - sending packets to the recipient using several gateways simultaneously using the Round Robin algorithm.

An ECMP route is created by the administrator by specifying multiple gateways for one subnet (or automatically, if there are two equivalent OSPF routes).

ECMP is used for load balancing between two channels, in theory, if there are two channels in the ecmp route, then for each packet the outgoing channel should be different. But the Routing cache mechanism sends packets from the connection along the route that the first packet took, as a result, we get a kind of balancing based on connections (per-connection loading balancing).

If you disable Routing Cache, then the packets in the ECMP route will be shared correctly, but there is a problem with NAT. The NAT rule processes only the first packet from the connection (the rest are processed automatically), and it turns out that packets with the same source address leave different interfaces.

Check gateway does not work in ECMP routes (RouterOS bug). But you can get around this limitation by creating additional validation routes that will disable entries in ECMP.

Filtering by means of Routing

The Type option determines what to do with the package:

• unicast - send to the specified gateway (interface)
• blackhole - discard a packet
• prohibit, unreachable - discard the packet and send an icmp message to the sender

Filtering is usually used when it is necessary to secure the sending of packets along the wrong path, of course, you can filter this through the firewall.

A couple of examples

To consolidate the basic things about routing.

Typical home router

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1

1. Static route to 0.0.0.0/0 (default route)
2. Connected route on the interface with the provider
3. Connected route on LAN interface

Typical home router with PPPoE

1. Static route to default route, added automatically. it is specified in connection properties
2. Connected route for PPP connection
3. Connected route on LAN interface

Typical home router with two providers and redundancy

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.20.20.1 distance=2

1. Static route to default route through the first provider with metric 1 and gateway availability check
2. Static route to default route through second provider with metric 2
3. Connected routes

Traffic to 0.0.0.0/0 goes through 10.10.10.1 while this gateway is available, otherwise it switches to 10.20.20.1

Such a scheme can be considered a channel reservation, but it is not without drawbacks. If a break occurs outside the provider's gateway (for example, inside the operator's network), your router will not know about it and will continue to consider the route as active.

Typical home router with two providers, redundancy and ECMP

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.20.20.1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.10.10.1,10.20.20.1 distance=1

1. Static routes for checking chack gateway
2. ECMP route
3. Connected routes

Routes to check are blue (the color of inactive routes), but this does not interfere with the check gateway. The current version (6.44) of RoS gives automatic priority to the ECMP route, but it is better to add test routes to other routing tables (option routing-mark)

On Speedtest and other similar sites, there will be no increase in speed (ECMP divides traffic by connections, not by packets), but p2p applications should download faster.

Filtering via Routing

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1
add dst-address=192.168.200.0/24 gateway=10.30.30.1 distance=1
add dst-address=192.168.200.0/24 gateway=10.10.10.1 distance=2 type=blackhole

1. Static route to default route
2. Static route to 192.168.200.0/24 over ipip tunnel
3. Forbidding static route to 192.168.200.0/24 through ISP router

A filtering option in which tunnel traffic will not go to the provider's router when the ipip interface is disabled. Such schemes are rarely required, because you can implement blocking through the firewall.

Routing loop
Routing loop - a situation when a packet runs between routers before the ttl expires. Usually it is the result of a configuration error, in large networks it is treated by the implementation of dynamic routing protocols, in small ones - with care.

It looks something like this:

An example (simplest) of how to get a similar result:

The Routing loop example is of no practical use, but it shows that routers have no idea about their neighbor's routing table.

Policy Base Routing and Additional Routing Tables

When choosing a route, the router uses only one field from the packet header (Dst. Address) - this is basic routing. Routing based on other conditions, such as source address, type of traffic (ToS), balancing without ECMP, belongs to Policy Base Routing (PBR) and uses additional routing tables.

More Specific Route is the main route selection rule within the routing table.

By default, all routing rules are added to the main table. The administrator can create an arbitrary number of additional routing tables and route packets to them. Rules in different tables do not conflict with each other. If the package does not find a suitable rule in the specified table, it will go to the main table.

Example with distribution via Firewall:

• 192.168.100.10 -> 8.8.8.8
  1. Traffic from 192.168.100.10 gets labeled via-isp1 в [Prerouting|Mangle]
  2. At the Routing stage in the table via-isp1 searches for a route to 8.8.8.8
  3. Route found, traffic is sent to gateway 10.10.10.1
• 192.168.200.20 -> 8.8.8.8
  1. Traffic from 192.168.200.20 gets labeled via-isp2 в [Prerouting|Mangle]
  2. At the Routing stage in the table via-isp2 searches for a route to 8.8.8.8
  3. Route found, traffic is sent to gateway 10.20.20.1
• If one of the gateways (10.10.10.1 or 10.20.20.1) becomes unavailable, then the packet will go to the table main and will look for a suitable route there

Terminology issues

RouterOS has certain terminology issues.
When working with rules in [IP]->[Routes] the routing table is indicated, although it is written that the label:

В [IP]->[Routes]->[Rule] everything is correct, in the label condition in the table action:

How to send a packet to a specific routing table

RouterOS provides several tools:

• Rules [IP]->[Routes]->[Rules]
• Route markers (action=mark-routing) In [IP]->[Firewall]->[Mangle]
• VRF

Rules [IP]->[Route]->[Rules]
Rules are processed sequentially, if the packet matches the conditions of the rule, it does not pass further.

Routing Rules allow you to expand the possibilities of routing, relying not only on the recipient address, but also on the source address and interface on which the packet was received.

Rules consist of conditions and an action:

• Conditions. Practically repeat the list of signs by which the package is checked in the FIB, only ToS is missing.
• Actions
  • lookup - send a packet to a table
  • lookup only in table - lock the package in the table, if the route is not found, the package will not go to the main table
  • drop - drop a packet
  • unreacheable - discard the packet with sender notification

In FIB, traffic to local processes is processed bypassing the rules [IP]->[Route]->[Rules]:

Marking [IP]->[Firewall]->[Mangle]
Routing labels allow you to set the gateway for a packet using almost any Firewall conditions:

Practically, because not all of them make sense, and some may work unstably.

There are two ways to label a package:

• Immediately put routing mark
• Put first connection-mark, then based on connection-mark put routing mark

In an article about firewalls, I wrote that the second option is preferable. reduces the load on the cpu, in the case of marking routes - this is not entirely true. These marking methods are not always equivalent and are usually used to solve various problems.

Примеры использования

Let's move on to the examples of using Policy Base Routing, they are much easier to show why all this is needed.

MultiWAN and return outgoing (Output) traffic
A common problem with a MultiWAN configuration: Mikrotik is available from the Internet only through an "active" provider.

The router does not care what ip the request came to, when generating a response, it will look for a route in the routing table where the route through isp1 is active. Further, such a packet will most likely be filtered along the way to the recipient.

Another interesting point. If a "simple" source nat is configured on the ether1 interface: /ip fi nat add out-interface=ether1 action=masquerade the package will go online with src. address=10.10.10.100, which makes things even worse.

There are several ways to fix the problem, but any of them will require additional routing tables:

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1 check-gateway=ping distance=1
add dst-address=0.0.0.0/0 gateway=10.20.20.1 check-gateway=ping distance=2
add dst-address=0.0.0.0/0 gateway=10.10.10.1 routing-mark=over-isp1
add dst-address=0.0.0.0/0 gateway=10.20.20.1 routing-mark=over-isp2

Using [IP]->[Route]->[Rules]
Specify the routing table that will be used for packets with the specified Source IP.

/ip route rule
add src-address=10.10.10.100/32 action=lookup-only-in-table table=over-isp1
add src-address=10.20.20.200/32 action=lookup-only-in-table table=over-isp2

Can be used action=lookup, but for local outgoing traffic, this option completely excludes connections from the wrong interface.

• The system generates a response packet with Src. Address: 10.20.20.200
• The Routing Decision(2) step checks [IP]->[Routes]->[Rules] and the packet is sent to the routing table over-isp2
• According to the routing table, the packet must be sent to the gateway 10.20.20.1 via the ether2 interface

This method does not require a working Connection Tracker, unlike using the Mangle table.

Using [IP]->[Firewall]->[Mangle]
The connection starts with an incoming packet, so we mark it (action=mark-connection), for outgoing packets from a marked connection, set the routing label (action=mark-routing).

/ip firewall mangle
#Маркировка входящих соединений
add chain=input in-interface=ether1 connection-state=new action=mark-connection new-connection-mark=from-isp1
add chain=input in-interface=ether2 connection-state=new action=mark-connection new-connection-mark=from-isp2
#Маркировка исходящих пакетов на основе соединений
add chain=output connection-mark=from-isp1 action=mark-routing new-routing-mark=over-isp1 passthrough=no
add chain=output connection-mark=from-isp2 action=mark-routing new-routing-mark=over-isp2 passthrough=no

If several ips are configured on one interface, you can add to the condition dst-address to be sure.

• A packet opens the connection on the ether2 interface. The package goes into [INPUT|Mangle] which says to mark all packets from the connection as from-isp2
• The system generates a response packet with Src. Address: 10.20.20.200
• At the Routing Decision(2) stage, the packet, in accordance with the routing table, is sent to the gateway 10.20.20.1 via the ether1 interface. You can verify this by logging the packages in [OUTPUT|Filter]
• At the stage [OUTPUT|Mangle] connection label is checked from-isp2 and the packet receives a route label over-isp2
• The Routing Adjusment(3) step checks for the presence of a routing label and sends it to the appropriate routing table
• According to the routing table, the packet must be sent to the gateway 10.20.20.1 via the ether2 interface

MultiWAN and return dst-nat traffic

An example is more complicated, what to do if there is a server (for example, web) behind the router on a private subnet and you need to provide access to it through any of the providers.

/ip firewall nat
add chain=dstnat proto=tcp dst-port=80,443 in-interface=ether1 action=dst-nat to-address=192.168.100.100
add chain=dstnat proto=tcp dst-port=80,443 in-interface=ether2 action=dst-nat to-address=192.168.100.100

The essence of the problem will be the same, the solution is similar to the Firewall Mangle option, only other chains will be used:

/ip firewall mangle
add chain=prerouting connection-state=new in-interface=ether1 protocol=tcp dst-port=80,443 action=mark-connection new-connection-mark=web-input-isp1
add chain=prerouting connection-state=new in-interface=ether2 protocol=tcp dst-port=80,443 action=mark-connection new-connection-mark=web-input-isp2
add chain=prerouting connection-mark=web-input-isp1 in-interface=ether3 action=mark-routing new-routing-mark=over-isp1 passthrough=no
add chain=prerouting connection-mark=web-input-isp2 in-interface=ether3 action=mark-routing new-routing-mark=over-isp2 passthrough=no

The diagram does not show NAT, but I think everything is clear.

MultiWAN and outbound connections

You can use the PBR capabilities to create multiple vpn (SSTP in the example) connections from different router interfaces.

Additional routing tables:

/ip route
add dst-address=0.0.0.0/0 gateway=192.168.100.1 routing-mark=over-isp1
add dst-address=0.0.0.0/0 gateway=192.168.200.1 routing-mark=over-isp2
add dst-address=0.0.0.0/0 gateway=192.168.0.1 routing-mark=over-isp3

add dst-address=0.0.0.0/0 gateway=192.168.100.1 distance=1
add dst-address=0.0.0.0/0 gateway=192.168.200.1 distance=2
add dst-address=0.0.0.0/0 gateway=192.168.0.1 distance=3

Package markings:

/ip firewall mangle
add chain=output dst-address=10.10.10.100 proto=tcp dst-port=443 action=mark-routing new-routing-mark=over-isp1 passtrough=no
add chain=output dst-address=10.10.10.101 proto=tcp dst-port=443 action=mark-routing new-routing-mark=over-isp2 passtrough=no
add chain=output dst-address=10.10.10.102 proto=tcp dst-port=443 action=mark-routing new-routing-mark=over-isp3 passtrough=no

Simple NAT rules, otherwise the packet will leave the interface with the wrong Src. address:

/ip firewall nat
add chain=srcnat out-interface=ether1 action=masquerade
add chain=srcnat out-interface=ether2 action=masquerade
add chain=srcnat out-interface=ether3 action=masquerade

Parsing:

• Router creates three SSTP processes
• At the Routing Decision (2) stage, a route is selected for these processes based on the main routing table. From the same route, the packet receives Src. Address bound to ether1 interface
• В [Output|Mangle] packets from different connections receive different labels
• Packets enter the tables corresponding to the labels at the Routing Adjusment stage and receive a new route for sending packets
• But packages still have Src. Address from ether1, on stage [Nat|Srcnat] the address is substituted according to the interface

Interestingly, on the router you will see the following connection table:

Connection Tracker works out earlier [Mangle] и [Srcnat], so all connections come from the same address, if you look in more detail, then in Replay Dst. Address there will be addresses after NAT:

On the VPN server (I have one on the test bench), you can see that all connections come from the correct addresses:

Hold up the way
There is an easier way, you can simply specify a specific gateway for each of the addresses:

/ip route
add dst-address=10.10.10.100 gateway=192.168.100.1
add dst-address=10.10.10.101 gateway=192.168.200.1
add dst-address=10.10.10.102 gateway=192.168.0.1

But such routes will affect not only outgoing but also transit traffic. Plus, if you do not need traffic to the vpn server to go through inappropriate communication channels, then you will have to add 6 more rules to [IP]->[Routes]с type=blackhole. In the previous version - 3 rules in [IP]->[Route]->[Rules].

Distribution of user connections by communication channels

Simple, everyday tasks. Again, additional routing tables will be needed:

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1 dist=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.20.20.1 dist=2 check-gateway=ping

add dst-address=0.0.0.0/0 gateway=10.10.10.1 dist=1 routing-mark=over-isp1
add dst-address=0.0.0.0/0 gateway=10.20.20.1 dist=1 routing-mark=over-isp2

Using [IP]->[Route]->[Rules]

/ip route rules
add src-address=192.168.100.0/25 action=lookup-only-in-table table=over-isp1
add src-address=192.168.100.128/25 action=lookup-only-in-table table=over-isp2

If use action=lookup, then when one of the channels is disabled, the traffic will go to the main table and go through the working channel. Whether this is necessary or not depends on the task.

Using the markings in [IP]->[Firewall]->[Mangle]
A simple example with lists of ip addresses. In principle, almost any conditions can be used. The only caveat of layer7, even when paired with connection labels, it may seem that everything is working correctly, but some of the traffic will still go the wrong way.

/ip firewall mangle
add chain=prerouting src-address-list=users-over-isp1 dst-address-type=!local action=mark-routing new-routing-mark=over-isp1
add chain=prerouting src-address-list=users-over-isp2 dst-address-type=!local action=mark-routing new-routing-mark=over-isp2

You can “lock” users in one routing table through [IP]->[Route]->[Rules]:

/ip route rules
add routing-mark=over-isp1 action=lookup-only-in-table table=over-isp1
add routing-mark=over-isp2 action=lookup-only-in-table table=over-isp2

Either through [IP]->[Firewall]->[Filter]:

/ip firewall filter
add chain=forward routing-mark=over-isp1 out-interface=!ether1 action=reject
add chain=forward routing-mark=over-isp2 out-interface=!ether2 action=reject

Retreat pro dst-address-type=!local
Additional condition dst-address-type=!local it is necessary that traffic from users reach the local processes of the router (dns, winbox, ssh, ...). If several local subnets are connected to the router, it is necessary to ensure that the traffic between them does not go to the Internet, for example, using dst-address-table.

In the example using [IP]->[Route]->[Rules] there are no such exceptions, but traffic reaches local processes. The fact is that getting into the FIB package marked in [PREROUTING|Mangle] has a route label and goes into a routing table other than main, where there is no local interface. In the case of Routing Rules, first it is checked whether the packet is intended for a local process and only at the User PBR stage does it go to the specified routing table.

Using [IP]->[Firewall]->[Mangle action=route]
This action only works in [Prerouting|Mangle] and allows you to direct traffic to the specified gateway without using additional routing tables, by specifying the gateway address directly:

/ip firewall mangle
add chain=prerouting src-address=192.168.100.0/25 action=route gateway=10.10.10.1
add chain=prerouting src-address=192.168.128.0/25 action=route gateway=10.20.20.1

Action route has a lower priority than routing rules ([IP]->[Route]->[Rules]). In the case of route marks, everything depends on the position of the rules, if the rule with action=route worth more than action=mark-route, then it will be used (regardless of the flag passtrough), otherwise marking the route.
There is very little information on the wiki about this action and all conclusions are obtained experimentally, in any case, I did not find options when using this option gives advantages over others.

PPC based dynamic balancing

Per Connection Classifier - is a more flexible analogue of ECMP. Unlike ECMP, it divides traffic by connections more strictly (ECMP knows nothing about connections, but when paired with Routing Cache, something similar is obtained).

PCC takes specified fields from the ip header, converts them to a 32-bit value, and divides by denominator. The remainder of the division is compared with the specified the remainder and if they match, then the specified action is applied. Details. Sounds crazy, but it works.

Example with three addresses:

192.168.100.10: 192+168+100+10 = 470 % 3 = 2
192.168.100.11: 192+168+100+11 = 471 % 3 = 0
192.168.100.12: 192+168+100+12 = 472 % 3 = 1

An example of dynamic distribution of traffic by src.address between three channels:

#Таблица маршрутизации
/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1 dist=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.20.20.1 dist=2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.30.30.1 dist=3 check-gateway=ping

add dst-address=0.0.0.0/0 gateway=10.10.10.1 dist=1 routing-mark=over-isp1
add dst-address=0.0.0.0/0 gateway=10.20.20.1 dist=1 routing-mark=over-isp2
add dst-address=0.0.0.0/0 gateway=10.30.30.1 dist=1 routing-mark=over-isp3

#Маркировка соединений и маршрутов
/ip firewall mangle
add chain=prerouting in-interface=br-lan dst-address-type=!local connection-state=new per-connection-classifier=src-address:3/0 action=mark-connection new-connection-mark=conn-over-isp1
add chain=prerouting in-interface=br-lan dst-address-type=!local connection-state=new per-connection-classifier=src-address:3/1 action=mark-connection new-connection-mark=conn-over-isp2
add chain=prerouting in-interface=br-lan dst-address-type=!local connection-state=new per-connection-classifier=src-address:3/2 action=mark-connection new-connection-mark=conn-over-isp3

add chain=prerouting in-interface=br-lan connection-mark=conn-over-isp1 action=mark-routing new-routing-mark=over-isp1
add chain=prerouting in-interface=br-lan connection-mark=conn-over-isp2 action=mark-routing new-routing-mark=over-isp2
add chain=prerouting in-interface=br-lan connection-mark=conn-over-isp3 action=mark-routing new-routing-mark=over-isp3

When marking routes, there is an additional condition: in-interface=br-lan, without it under action=mark-routing response traffic from the Internet will get and, in accordance with the routing tables, will go back to the provider.

Switching communication channels

Check ping is a good tool, but it only checks the connection with the nearest IP peer, provider networks usually consist of a large number of routers and a connection break can occur outside the nearest peer, and then there are backbone telecom operators who may also have problems, in general check ping does not always show up-to-date information about access to the global network.
If providers and large corporations have the BGP dynamic routing protocol, then home and office users have to independently figure out how to check Internet access through a specific communication channel.

Typically, scripts are used that, through a certain communication channel, check the availability of an ip address on the Internet, while choosing something reliable, for example, google dns: 8.8.8.8. 8.8.4.4. But in the Mikrotik community, a more interesting tool has been adapted for this.

A few words about recursive routing
Recursive routing is necessary when building Multihop BGP peering and got into the article about the basics of static routing only due to cunning MikroTik users who figured out how to use recursive routes paired with check gateway to switch communication channels without additional scripts.

It's time to understand the scope / target scope options in general terms and how the route is bound to the interface:

1. The route looks up an interface to send the packet based on its scope value and all entries in the main table with less than or equal target scope values
2. From the found interfaces, the one through which you can send a packet to the specified gateway is selected
3. The interface of the found connected entry is selected to send the packet to the gateway

In the presence of a recursive route, everything happens the same, but in two stages:

• 1-3 One more route is added to the connected routes, through which the specified gateway can be reached
• 4-6 Finding the route connected route for the "intermediate" gateway

All manipulations with the recursive search occur in the RIB, and only the final result is transferred to the FIB: 0.0.0.0/0 via 10.10.10.1 on ether1.

An example of using recursive routing to switch routes

Configuration:

/ip route
add dst-address=0.0.0.0/0 gateway=8.8.8.8 check-gateway=ping distance=1 target-scope=10
add dst-address=8.8.8.8 gateway=10.10.10.1 scope=10
add dst-address=0.0.0.0/0 gateway=10.20.20.1 distance=2

You can check that packets will be sent to 10.10.10.1:

Check gateway knows nothing about recursive routing and simply sends pings to 8.8.8.8, which (based on the main table) is accessible through gateway 10.10.10.1.

If there is a loss of communication between 10.10.10.1 and 8.8.8.8, then the route is disconnected, but packets (including test pings) to 8.8.8.8 continue to go through 10.10.10.1:

If the link to ether1 is lost, then an unpleasant situation occurs when packets before 8.8.8.8 go through the second provider:

This is a problem if you are using NetWatch to run scripts when 8.8.8.8 is not available. If the link is broken, NetWatch will simply work through the backup communication channel and assume that everything is fine. Solved by adding an additional filter route:

/ip route
add dst-address=8.8.8.8 gateway=10.20.20.1 distance=100 type=blackhole

There is on habré article, where the situation with NetWatch is considered in more detail.

And yes, when using such a reservation, the address 8.8.8.8 will be hardwired to one of the providers, so choosing it as a dns source is not a good idea.

A few words about Virtual Routing and Forwarding (VRF)

VRF technology is designed to create several virtual routers within one physical one, this technology is widely used by telecom operators (usually in conjunction with MPLS) to provide L3VPN services to clients with overlapping subnet addresses:

But VRF in Mikrotik is organized on the basis of routing tables and has a number of disadvantages, for example, local ip addresses of the router are available from all VRFs, you can read more here to register:.

vrf configuration example:

/ip route vrf
add interfaces=ether1 routing-mark=vrf1
add interfaces=ether2 routing-mark=vrf2

/ip address
add address=192.168.100.1/24 interface=ether1 network=192.168.100.0
add address=192.168.200.1/24 interface=ether2 network=192.168.200.0

From the device connected to ether2, we see that ping goes to the router address from another vrf (and this is a problem), while ping does not go to the Internet:

To access the Internet, you need to register an additional route that accesses the main table (in vrf terminology, this is called route leaking):

/ip route
add distance=1 gateway=172.17.0.1@main routing-mark=vrf1
add distance=1 gateway=172.17.0.1%wlan1 routing-mark=vrf2

Here are two ways of route leaking: using the routing table: 172.17.0.1@main and using interface name: 172.17.0.1%wlan1.

And set up marking for return traffic in [PREROUTING|Mangle]:

/ip firewall mangle
add chain=prerouting in-interface=ether1 action=mark-connection new-connection-mark=from-vrf1 passthrough=no
add chain=prerouting connection-mark=from-vrf1 routing-mark=!vrf1 action=mark-routing new-routing-mark=vrf1 passthrough=no 
add chain=prerouting in-interface=ether2 action=mark-connection new-connection-mark=from-vrf2 passthrough=no
add chain=prerouting connection-mark=from-vrf2 routing-mark=!vrf1 action=mark-routing new-routing-mark=vrf2 passthrough=no 

Subnets with the same address
Organization of access to subnets with the same addressing on the same router using VRF and netmap:

Basic configuration:

/ip route vrf
add interfaces=ether1 routing-mark=vrf1
add interfaces=ether2 routing-mark=vrf2

/ip address
add address=192.168.100.1/24 interface=ether1 network=192.168.100.0
add address=192.168.100.1/24 interface=ether2 network=192.168.100.0
add address=192.168.0.1/24 interface=ether3 network=192.168.0.0

firewall rules:

#Маркируем пакеты для отправки в правильную таблицу маршрутизации
/ip firewall mangle
add chain=prerouting dst-address=192.168.101.0/24 in-interface=ether3 action=mark-routing new-routing-mark=vrf1 passthrough=no
add chain=prerouting dst-address=192.168.102.0/24 in-interface=ether3 action=mark-routing new-routing-mark=vrf2 passthrough=no

#Средствами netmap заменяем адреса "эфимерных" подсетей на реальные подсети
/ip firewall nat
add chain=dstnat dst-address=192.168.101.0/24 in-interface=ether3 action=netmap to-addresses=192.168.100.0/24
add chain=dstnat dst-address=192.168.102.0/24 in-interface=ether3 action=netmap to-addresses=192.168.100.0/24

Routing rules for return traffic:

#Указание имени интерфейса тоже может считаться route leaking, но по сути тут создается аналог connected маршрута
/ip route
add distance=1 dst-address=192.168.0.0/24 gateway=ether3 routing-mark=vrf1
add distance=1 dst-address=192.168.0.0/24 gateway=ether3 routing-mark=vrf2

Adding routes received via dhcp to a given routing table
VRF can be interesting if you need to automatically add a dynamic route (for example, from a dhcp client) to a specific routing table.

Adding interface to vrf:

/ip route vrf
add interface=ether1 routing-mark=over-isp1

Rules for sending traffic (outgoing and transit) through the table over-isp1:

/ip firewall mangle
add chain=output out-interface=!br-lan action=mark-routing new-routing-mark=over-isp1 passthrough=no
add chain=prerouting in-interface=br-lan dst-address-type=!local action=mark-routing new-routing-mark=over-isp1 passthrough=no

Additional, fake route for outbound routing to work:

/interface bridge
add name=bare

/ip route
add dst-address=0.0.0.0/0 gateway=bare

This route is only needed so that local outgoing packets can pass through the Routing decision (2) before [OUTPUT|Mangle] and get the routing label, if there are other active routes on the router before 0.0.0.0/0 in the main table, it is not required.

Chains connected-in и dynamic-in в [Routing] -> [Filters]

Route filtering (inbound and outbound) is a tool that is usually used in conjunction with dynamic routing protocols (and therefore only available after installing the package routing), but there are two interesting chains in the incoming filters:

• connected-in — filtering connected routes
• dynamic-in - filtering dynamic routes received by PPP and DCHP

Filtering allows you not only to discard routes, but also to change a number of options: distance, routing-mark, comment, scope, target scope, ...

This is a very precise tool and if you can do something without Routing Filters (but not scripts), then do not use Routing Filters, do not confuse yourself and those who will configure the router after you. In the context of dynamic routing, Routing Filters will be used much more frequently and more productively.

Setting the Routing Mark for Dynamic Routes
An example from a home router. I have two VPN connections configured and the traffic in them should be wrapped in accordance with the routing tables. At the same time, I want the routes to be created automatically when the interface is activated:

#При создании vpn подключений указываем создание default route и задаем дистанцию
/interface pptp-client
add connect-to=X.X.X.X add-default-route=yes default-route-distance=101 ...
add connect-to=Y.Y.Y.Y  add-default-route=yes default-route-distance=100 ...

#Фильтрами отправляем маршруты в определенные таблицы маршрутизации на основе подсети назначения и дистанции
/routing filter
add chain=dynamic-in distance=100 prefix=0.0.0.0/0 action=passthrough set-routing-mark=over-vpn1
add chain=dynamic-in distance=101 prefix=0.0.0.0/0 action=passthrough set-routing-mark=over-vpn2

I don’t know why, probably a bug, but if you create a vrf for the ppp interface, then the route to 0.0.0.0/0 will still get into the main table. Otherwise, everything would be even easier.

Disabling Connected Routes
Sometimes this is required:

/route filter
add chain=connected-in prefix=192.168.100.0/24 action=reject

Debugging tools

RouterOS provides a number of tools for debugging routing:

• [Tool]->[Tourch] - allows you to view packets on interfaces
• /ip route check - allows you to see which gateway the packet will be sent to, does not work with routing tables
• /ping routing-table=<name> и /tool traceroute routing-table=<name> - ping and trace using the specified routing table
• action=log в [IP]->[Firewall] - an excellent tool that allows you to trace the path of a packet along the packet flow, this action is available in all chains and tables

Source: habr.com