Load balancing in Zimbra Open-Source Edition using HAProxy

One of the main tasks in building large-scale Zimbra OSE infrastructures is competent load balancing. In addition to increasing the fault tolerance of the service, without load balancing it is impossible to ensure the same responsiveness of the service for all users. In order to solve this problem, load balancers are used - software and hardware solutions that redistribute requests between servers. Among them, there are both rather primitive ones, like RoundRobin, which simply sends every next request to the next server in the list, and there are more advanced ones, for example, HAProxy, which is widely used in highly loaded computing infrastructures due to a number of significant advantages. Let's take a look at how the HAProxy load balancer and Zimbra OSE can work together.

So, according to the terms of the problem, we are given the Zimbra OSE infrastructure, which has two Zimbra Proxies, two LDAP and LDAP Replica servers, four mail stores with 1000 mailboxes each, and three MTAs. Given that we are dealing with a mail server, it will receive three types of traffic that need to be balanced: HTTP for downloading the web client, as well as POP and SMTP for sending email. In this case, HTTP traffic will go to Zimbra Proxy servers with ip addresses 192.168.0.57 and 192.168.0.58, and SMTP traffic will go to MTA servers with ip addresses 192.168.0.77 and 192.168.0.78.

As already mentioned, to ensure even distribution of requests between servers, we will use the HAProxy load balancer, which will run on the input node of the Zimbra infrastructure running Ubuntu 18.04. Installing haproxy on this operating system is done using the command sudo apt-get install haproxy. After that, it is necessary in the file /etc/default/haproxy change parameter ENABLED = 0 on ENABLED = 1. Now, in order to make sure that haproxy is working, just enter the command service haproxy. In the event that this service is running, it will be clear from the output of the command.

One of the main shortcomings of HAProxy is that by default it does not transmit the IP address of the connected client, replacing it with its own. This can lead to situations when letters sent by attackers cannot be identified by IP address in order to add it to the blacklist. However, this issue can be resolved. To do this, you need to edit the file /opt/zimbra/common/conf/master.cf.in on Postfix servers and add the following lines to it:

26      inet  n       -       n       -       1       postscreen
        -o postscreen_upstream_proxy_protocol=haproxy
 
466    inet  n       -       n       -       -       smtpd
%%uncomment SERVICE:opendkim%%  -o content_filter=scan:[%%zimbraLocalBindAddress%%]:10030
        -o smtpd_tls_wrappermode=yes
        -o smtpd_sasl_auth_enable=yes
        -o smtpd_client_restrictions=
        -o smtpd_data_restrictions=
        -o smtpd_helo_restrictions=
        -o smtpd_recipient_restrictions=
        -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
        -o syslog_name=postfix/smtps
        -o milter_macro_daemon_name=ORIGINATING
        -o smtpd_upstream_proxy_protocol=haproxy
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_filter=[%%zimbraLocalBindAddress%%]:10027
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_options=speed_adjust
 
588 inet n      -       n       -       -       smtpd
%%uncomment SERVICE:opendkim%%  -o content_filter=scan:[%%zimbraLocalBindAddress%%]:10030
        -o smtpd_etrn_restrictions=reject
        -o smtpd_sasl_auth_enable=%%zimbraMtaSaslAuthEnable%%
        -o smtpd_tls_security_level=%%zimbraMtaTlsSecurityLevel%%
        -o smtpd_client_restrictions=permit_sasl_authenticated,reject
        -o smtpd_data_restrictions=
        -o smtpd_helo_restrictions=
        -o smtpd_recipient_restrictions=
        -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
        -o syslog_name=postfix/submission
        -o milter_macro_daemon_name=ORIGINATING
        -o smtpd_upstream_proxy_protocol=haproxy
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_filter=[%%zimbraLocalBindAddress%%]:10027
%%uncomment LOCAL:postjournal_enabled%% -o smtpd_proxy_options=speed_adjust

Due to this, we will open ports 26, 466 and 588, which will receive incoming traffic from HAProxy. After the files have been saved, restart Postfix on all servers using the zmmtactl restart command.

After that, let's start configuring HAProxy. To do this, first create a backup copy of the settings file cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak. Then open the source file in a text editor /etc/haproxy/haproxy.cfg and begin to gradually add the necessary settings to it. The first block will be adding a server that takes logs, setting the maximum number of simultaneous connections allowed, as well as specifying the name and group of the user to which the executable process will belong.

global
    user daemon
    group daemon
    daemon
    log 127.0.0.1 daemon
    maxconn 5000
    chroot /var/lib/haproxy

The figure of 5000 simultaneous connections appeared for a reason. Since there are 4000 mailboxes in our infrastructure, it is necessary to provide for the possibility that all of them will go to their work mail at the same time. In addition, it is necessary to leave a small margin in case their number increases.

Now let's add a block with default settings:

defaults
        timeout client 1m
        log global
        mode tcp
        timeout server 1m
        timeout connect 5s

This block sets the maximum client and server wait time to close the connection when it expires, and also sets the HAProxy operation mode. In our case, the load balancer works in TCP mode, that is, it simply transfers TCP packets without analyzing their contents.

Next, we will add rules for connections on various ports. For example, if port 25 is used for SMTP connections and mail transfer, then it makes sense to forward connections to it to the MTAs that we have in our infrastructure. If the connection is on port 80, then this is an http request that needs to be forwarded to the Zimbra Proxy.

Rule for port 25:

frontend smtp-25
bind *:27
default_backend backend-smtp-25
 
backend backend-smtp-25
server mta1 192.168.0.77:26 send-proxy
server mta2 192.168.0.78:26 send-proxy

Rule for port 465:

frontend smtp-465
bind *:467
default_backend backend-smtp-465

backend backend-smtp-465
server mta1 192.168.0.77:466 send-proxy
server mta2 192.168.0.78:466 send-proxy

Rule for port 587:

frontend smtp-587
bind *:589
default_backend backend-smtp-587
 
backend backend-smtp-587
server mail1 192.168.0.77:588 send-proxy
server mail2 192.168.0.78:588 send-proxy

Rule for port 80:

frontend http-80
bind    *:80
default_backend http-80
 
backend http-80
mode tcp
server zproxy1 192.168.0.57:80 check
server zproxy2 192.168.0.58:80 check

Rule for port 443:

frontend https
bind  *:443
default_backend https-443
 
backend https-443
mode tcp
server zproxy1 192.168.0.57:80 check
server zproxy2 192.168.0.58:80 check

Please note that in the rules for forwarding TCP packets to the MTA, next to their addresses is the parameter send-proxy. This is necessary so that, in accordance with the changes we made earlier to the Postfix settings, the original IP address of its sender is also sent along with TCP packets.

Now that all the necessary changes have been made to HAProxy, you can restart the service with the command service haproxy restart and start using it.

For all questions related to Zextras Suite, you can contact the Representative of Zextras Ekaterina Triandafilidi by e-mail [email protected]

Source: habr.com