I’m still a huge fan of Zen (Zevenet) Load Balancer, and it’s been serving our Exchange servers for a couple of years. HOWEVER, Zen’s features are a bit limited and not enough for us anymore. It does a very good job with basic load balancing, but unfortunately it lacks in the logging department. This was a deal breaker for us, as we weren’t able to get the client IP’s logged. If you’re using any load balancer in Source NAT (SNAT)/reverse proxy configuration (most certainly you are), this is the default behavior. You’ll then end up with the load balancer IP instead of the client IP in the (Exchange IIS) logs. Also, with Zen LB, you’ll get NO client logs at all on the load balancer itself. This is not ideal. Furthermore, when I was testing http mode instead of L4xNAT in Zen, I couldn’t get it working as intended (including x-forward-for). I configured it with no persistency/affinity, but Outlook wouldn’t simply start. I scratched my head with this for far too long, and I had to give up. This seems like a bug in Zen if you ask me. (Exchange 2013 is btw designed to work without session affinity/persistence, see https://blogs.technet.microsoft.com/exchange/2014/03/05/load-balancing-in-exchange-2013/).
This left us no other choice than to look for alternative load balancers. We finally settled for HAProxy, as the price tag was just right (free). That said, I ALMOST had to throw in the towel with HAProxy also, as the Mac Outlook clients wouldn’t work no matter what. Luckily I got it sorted out in the end though, more about that later on.
The SNAT/client IP dilemma is a big deal, but the whole architecture/algorithm/method behind it isn’t really the point of this blog post. None the less, you should read at least some homework. Here are a couple of nice articles describing the dilemma and also different load balancing methods:
HAProxy: Microsoft Exchange 2013 architectures (check the limitations in reverse-proxy mode/source NAT)
Logging Client IP Address in IIS When Using Load Balancing with Source NAT
What are the best load balancing methods and algorithms?
Now that you know a little bit more about the LB methods, it’s time to think about how to configure HAProxy itself. HAProxy with Exchange 2013 (2016), that is.
First off, you’ll need L7 load balancing (http), so you’ll get the possibility to insert the x-forward-for header. This is the trick for getting the client IP in the logs (See https://www.loadbalancer.org/blog/iis-and-x-forwarded-for-header/ for information how to configure x-forward-for on the Exchange servers). I found a very nice blog post describing how to install and configure HAProxy for Exchange, “Highly Available L7 Load Balancing for Exchange 2013 with HAProxy”. I have to say it’s TOTALLY AWESOME. I couldn’t have done the configuration without help from this blog post. No other article I found on the Internet had such deep detail and demonstration regarding L7 load balancing for Exchange. That said, I added some custom stuff to the configuration file, and I also configured/installed HAProxy + keepalived a bit different (easier) than in the above guide. Well, enough talking. Here are my steps, based on Zoltan’s blog post:
- I didn’t install a PKI infrastructure. Most of you probably have a working PKI infrastructure in production already.
- I use public certificates on Exchange so I have no need for an internal PKI (in this specific case).
- I didn’t install any Exchange servers either (obviously).
- I installed CentOS 7 (two servers) in cooperation with our Linux team. This way I got the optimal installation (no extra crap, and centralized management with Puppet).
- I started following the guide from part 5, http://ezoltan.blogspot.com.au/2014/10/highly-available-l7-load-balancing-for.html
- Please read the chapter Brief HAProxy Certificate Primer to get an idea of what you’re trying to accomplish!
- Take note that we’re not doing SSL offloading, we’re in fact doing SSL bridging
Certificate preparation
- I’m using a public certificate, so there’s no need to update the root and intermediate cert store stuff (yet). (In fact the intermediate certificate needed some configuration, more about that later on).
- I started following the guide more to the letter from the chapter Upload the Exchange Certificate and Private Key onwards
- I extracted the private key from the pfx file and removed the password protection from the private key. When asked, I entered the password gotten from the public certificate provider. I then entered a new PEM pass phrase (just pick a new one):
- Now it’s time to remove the password protection from the private key. You’ll be prompted for the PEM pass phrase entered in the above step:
- We’re now ready for the final stage – extracting the certificate from the pfx file:
- We now have all files needed, and we just need to combine the certificate and the private key files, as HAProxy doesn’t allow use of separate files.
- cat exchange_certificate.pem exchange_private_key_nopassword.pem > exchange_certificate_and_key_nopassword.pem
- Moved the file to its final destination:
- mv exchange_certificate_and_key_nopassword.pem /etc/ssl/certs/
- I’ll now copy/paste Zoltan; “Well, let’s give ourselves a pat on the shoulder, we deserve it. We are through the most difficult part, at least in my opinion, of this lab. Well done!” 🙂 I have to agree with the guy here…
HAProxy installation
I did NOT compile HAProxy from scratch, instead I just yum installed the HAProxy package. Much more straight forward. Here are my steps:
- yum install haproxy
- I used a “dummy” haproxy.conf –file (for Exchange). It was modified down the road, see the working example from the config file chapter later on.
- Apart from the info in the blog, you should also edit another line in rsyslog.conf. Add local0.none to the following line:
- *.info;mail.none;authpriv.none;cron.none,local0.none /var/log/messages
- There’s no need to create a log rotate-script when yum installing HAProxy, it gets created automatically!
- I edited the log rotate-script and changed the rotate parameter to “30”. One month is a suitable time for us to keep the logs.
- Same thing goes for HAProxy automatic startup at boot. There’s no need for complicated scripts, just execute the following command instead:
- systemctl enable haproxy:
- I then configured the firewall.
- You should now do some testing. I had previously done some tests in my virtual environment, so I had no need for any tests right now.
- Thus far, everything is working:
- Happy days!
HAProxy HA
Now that we have one node working (master), it’s time to think about High Availability. I continued following the excellent guide at http://ezoltan.blogspot.com.au/2014/10/highly-available-l7-load-balancing-for_48.html. Yet again had some changes in my environment. Comments below:
- I did not compile keepalived from scratch, instead I yum installed it.
- Added net.ipv4.ip_nonlocal_bind=1 to /etc/sysctl.conf
- No need to create keepalived.conf, it’s pre-created when yum installed.
- Edited keepalived.conf to match our environment (copy pasted the blog config and edited email / interface / IP parameters)
- Edited the hosts file following the blog information
- There’s yet again no need for a startup script when you have yum installed. Keepalived will autostart at every boot when you execute the following:
- systemctl enable keepalived
- I made firewall changes based on the blog.
- Did all the tests, everything worked like a charm 🙂
- Made the same changes to the other server/node (backup)
- priority was set to a lower value than on master node
- other small changes which are written in Zoltan’s blog
- Did keepalived-testing following the blog. Everything was fine! 🙂
Testing
This has to be my favorite chapter from Zoltan’s blog. I have to say he does a VERY good job explaining all the testing and troubleshooting. READ IT CAREFULLY! I myself also noticed “weird” clients connecting to the “unspecified protocol” back-end “be_ex2013”. If all ACL’s match, there should be no traffic passing through this back-end. I found some autodiscover and ews urls that had mixed upper/lower case letters however. This was easily fixed by adding a few more ACL’s. I’ll paste the config (including the ACL’s) in the next chapter so you’ll get a better understanding of the whole thing.
Config file, haproxy.conf
Well, this is the most important part of the whole setup, no doubt. Also the most time consuming. Here’s the config, I’ll explain it in more detail after the paste:
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#log 127.0.0.1 local2 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
daemon# turn on stats unix socket
stats socket /var/run/haproxy.stat
#--------------------------
# SSL tuning / hardening
#--------------------------
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-server-options no-sslv3
ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
tune.ssl.default-dh-param 2048
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------# Regarding timeout client and timeout server:
# https://discourse.haproxy.org/t/high-number-of-connection-resets-during-transfers-exchange-2013/1158/4defaults
mode http
log global
option httplog
option dontlognull
option forwardfor except 127.0.0.0/8
option redispatch
# option contstats
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 15m # this value should be rather high with Exchange
timeout server 15m # this value should be rather high with Exchange
timeout http-keep-alive 10s
timeout check 10s
maxconn 100000
#-------------------------------------------------------
# Stats section
#-------------------------------------------------------listen stats x.x.x.x:444 # VIP
stats enable
stats refresh 300s
stats show-node
stats auth admin:xxxxxx
stats hide-version
stats uri /stats
#---------------------------------------------------------------------
# Main front-end which proxies to the back-ends
#---------------------------------------------------------------------frontend fe_ex2013
# http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload
http-response set-header X-Frame-Options SAMEORIGIN
http-response set-header X-Content-Type-Options nosniff
mode http
bind *:80
bind *:443 ssl crt /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem
redirect scheme https code 301 if !{ ssl_fc } # redirect 80 -> 443 (for owa)
acl autodiscover url_beg /Autodiscover
acl autodiscover url_beg /autodiscover
acl mapi url_beg /mapi
acl rpc url_beg /rpc
acl owa url_beg /owa
acl owa url_beg /OWA
acl eas url_beg /Microsoft-Server-ActiveSync
acl ecp url_beg /ecp
acl ews url_beg /EWS
acl ews url_beg /ews
acl oab url_beg /OAB
use_backend be_ex2013_autodiscover if autodiscover
use_backend be_ex2013_mapi if mapi
use_backend be_ex2013_rpc if rpc
use_backend be_ex2013_owa if owa
use_backend be_ex2013_eas if eas
use_backend be_ex2013_ecp if ecp
use_backend be_ex2013_ews if ews
use_backend be_ex2013_oab if oab
default_backend be_ex2013
#------------------------------
# Back-end section
#------------------------------backend be_ex2013_autodiscover
mode http
balance roundrobin
option httpchk GET /autodiscover/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013_mapi
mode http
balance roundrobin
option httpchk GET /mapi/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013_rpc
mode http
balance roundrobin
option httpchk GET /rpc/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013_owa
mode http
balance roundrobin
option httpchk GET /owa/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013_eas
mode http
balance roundrobin
option httpchk GET /microsoft-server-activesync/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013_ecp
mode http
balance roundrobin
option httpchk GET /ecp/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013_ews
mode http
balance roundrobin
option httpchk GET /ews/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013_oab
mode http
balance roundrobin
option httpchk GET /oab/healthcheck.htm
option log-health-checks
http-check expect status 200
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
backend be_ex2013
mode http
balance roundrobin
server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
#######################################
# End of Exchange's own protocols,
# STMP and IMAP next
########################################
frontend fe_exchange_smtp
mode tcp
option tcplog
bind x.x.x.x:25 name smtp # VIP
default_backend be_exchange_smtp
backend be_exchange_smtp
mode tcp
option tcplog
balance roundrobin
option log-health-checks
server exchange1 1.1.1.1:25 weight 10 check
server exchange2 2.2.2.2:25 weight 20 check#only port 25 needed in our case. The port is open (only) against our Postfix server, which handles the outgoing mail traffic (MTA). In other words, we're using an external send connector in Exchange.
frontend fe_exchange_imaps
mode tcp
option tcplog
# bind x.x.x.x:143 name imap # NO unencrypted imap traffic allowed...
bind x.x.x.x:993 name imaps # ssl crt /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem <-- No need, certificate is read straight from the Exchange servers.
default_backend be_exchange_imaps
backend be_exchange_imaps
mode tcp
option tcplog
# balance roundrobin
balance leastconn
option log-health-checks
# stick store-request src
# stick-table type ip size 200k expire 30m
# option tcp-check
# tcp-check connect port 143
# tcp-check expect string * OK
# tcp-check connect port 993 ssl
# tcp-check expect string * OK
server exchange1 1.1.1.1:993 weight 10 check
server exchange2 2.2.2.2:993 weight 20 check
Explanations:
For obvious reasons, all IP addresses are censored.
SSL tuning / hardening section was added by me for additional security. As said in many blog posts before, I care about server security. More information regarding this can be found at: http://www.mattzuba.com/2015/07/hardening-haproxy-for-an-a-rating/ for example. At the same time the http-response set-header X-Frame-Options SAMEORIGIN and http-response set-header X-Content-Type-Options nosniff options were added. I still left the option # http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload commented though. These options are required when aiming for a A+ score on Qualys SSL Labs SSL Server test.
Btw, HAProxy can also protect you from DDOS attacks and other attacks. Have a look at
https://www.haproxy.com/blog/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/
https://www.loadbalancer.org/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2/
for more information.
Stats section was added to get some nice stats. You can uncomment the the line #option contstats to get continuous statistics. At the moment it require a manual webpage refresh.
I added http to https redirect on the main front-end with the line: redirect scheme https code 301 if !{ ssl_fc }
I added new ACL’s so the back-ends would fetch every combination of Autodiscover and so forth:
acl autodiscover url_beg /Autodiscover
acl autodiscover url_beg /autodiscover
acl owa url_beg /owa
acl owa url_beg /OWA
acl ews url_beg /EWS
acl ews url_beg /ews
I added a SMTP and IMAP section. We only need port 25 (smtp) to be open, as Postfix is taking care of sending the emails. Exchange only hands them over to Postfix (send connector). IMAPs on the other hand is open to the world, even though we don’t have many users accessing Exchange using imap. (Unencrypted imap isn’t allowed at all). You can use different techniques for configuring imaps (available in comments), but I’ve chosen to use the easiest one. Also note that SMTP and IMAP are using Layer 4 (tcp) load balancing, opposite to Exchanges own protocols. That said, you can still get the imap client IP’s straight from haproxy’s log file (instead of IIS on Exchange).
Problem solving
Apart from mega problems with Mac (sigh), we also had a hiccup with non-domain joined Windows clients. I had to scratch my head quite a bit before finding the reason why a client would prompt for credentials even though the certificate was correctly applied in HAProxy. Well, it turned out to be a certificate problem after all, and Zoltan briefly even described it in Part 5 of his blog:
“By the way, even if you acquire a certificate from a commercial CA, there are no guarantees that its intermediate and root CA certificates will come pre-canned with CentOS 7 (or any Linux flavour as a matter of fact), so it is good to know the procedure anyway, just in case”.
I happened to miss this part though, so I was very confused. At the moment I don’t even remember what pointed me in the right direction, but the solution was indeed to add the intermediate certificate from our certificate provider into the .pem file created earlier:
cat /etc/pki/ca-trust/source/DigiCertCA.crt > /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem
DigiCertCA is automatically deployed to our Linux servers with puppet, the “only” problem was that HAProxy didn’t understand how to use it without the above trick. More information: https://www.happyassassin.net/2015/01/14/trusting-additional-cas-in-fedora-rhel-centos-dont-append-to-etcpkitlscertsca-bundle-crt-or-etcpkitlscert-pem/. Well, it now makes much more sense why it worked on domain joined Windows clients but not on other clients – the domain joined Windows clients were automatically receiving the intermediate certificate via group policies. Oh well. Live and learn 🙂
Finally, the BIGGEST problem of them all, was Outlook for Mac. What a pain in the ass. Even with the above intermediate certificate in place it wouldn’t work. I was completely clueless, and it seemed the whole Internet was equally confused. (I asked in forums for a solution to my Mac dilemma, without success). What helped me in the end, was a colleague noticing that Outlook for Mac is connecting to Exchange using old protocols. Even the newest Outlook for Mac 2016. SHAME ON YOU MICROSOFT! This however opened up a new world of googling for me. That said, here’s some information regarding the dilemma (which ISN’T a HAProxy problem btw):
https://support.microsoft.com/en-ph/help/2955530/outlook-for-mac-clients-cannot-connect-to-exchange-server
https://outlook.uservoice.com/forums/293343-outlook-for-mac/suggestions/15120408-add-support-for-tls-1-0-1-1-and-1-2
https://www.quora.com/Is-there-a-way-to-connect-Mac-Office-Outlook-2016-with-Microsoft-Exchange-Server-2007
https://answers.microsoft.com/en-us/mac/forum/macoffice2011-macoutlook/outlook-2011-to-use-sslv3/7e777e6b-9e92-4a89-8874-d357c4bdf6ef?auth=1
and the solution:
In other words, edit the registry on the Exchange servers to enable Compatible mode. And there you have it, all Mac clients can finally connect through HAProxy 🙂
Final thoughts
As said before, HAProxy’s logging mechanism is amazing. It does produce quite huge logs though – hundreds of MB’s per day. Luckily the logs are very nicely compressed, and the daily log rotated log takes up less than 100MB HDD space. That said, be sure allocate enough disk space for the logs! Analyzing the logs in depth can be a little bit cryptic, but luckily there’s documentation available. For example, check out
https://cdn.haproxy.com/wp-content/uploads/2017/07/aloha_load_balancer_memo_log.pdf for an amazing chart with all the log value codes and explanations..
The termination flags can be found in HAProxy’s own documentation: https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4-option%20dontlognull. Search for the text “The most common termination flags combinations are indicated below. They are alphabetically sorted, with the lowercase set just after the upper case for easier finding and understanding” and you’re good to go.
Here’s a random screenshot from haproxy.log:
You’ll now get all the client IP’s from these logs. For obvious reasons they’re censored in the screenshot though. Furthermore, (and perhaps even more important), you can now also check the IIS logs on the Exchange servers and notice that a column (the last one) has been added for x-forward-for:
2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=ce9fb341-8f0b-4315-b3d9-3e77591e0a18@abo.fi&CorrelationID=<empty>;&ClientId=I9XPTDBZEATAAVKDG&cafeReqId=d3cdb726-44ae-4237-9bbc-2c5e5f434e13; 443 – x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 2 5 15 1.2.3.148
2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=57c56614-6647-470d-a620-f3b1f5e2dc8f@abo.fi&CorrelationID=<empty>;&ClientId=SHDGSCWGKIOLIHQCGSKA&cafeReqId=259f647e-646b-4a62-88c9-5de42df747e7; 443 - x.x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 2 5 15 1.2.3.149
2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=43aacf1f-5def-4d3d-9fdb-899ba3c49ec3@abo.fi&CorrelationID=<empty>;&ClientId=IQWDWWOIDUSFPCVQTPW&cafeReqId=4d9bc40c-3d2d-4771-a0bb-024d32ea509c; 443 - x.x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 2 5 0 1.2.3.30
2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=57c56614-6647-470d-a620-f3b1f5e2dc8f@abo.fi&CorrelationID=<empty>;&ClientId=SHDGSCWGKIOLIHQCGSKA&cafeReqId=c064ab51-5e49-4c86-9e22-ba7a33ca7223; 443 - x.x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 1 2148074254 0 1.2.3.14
The last column (“1.2.3.x” ) in the above log snippet is the real client IP (obviously censored yet again).
And there we have it. Logging problem solved! 🙂 Now we can finally start tracking down the bad guys…
A random blog from a sysadmin 