Exchange 2013/2016: Switching from Zen Load Balancer to HAProxy

I’m still a huge fan of Zen (Zevenet) Load Balancer, and it’s been serving our Exchange servers for a couple of years. HOWEVER, Zen’s features are a bit limited and not enough for us anymore. It does a very good job with basic load balancing, but unfortunately it lacks in the logging department. This was a deal breaker for us, as we weren’t able to get the client IP’s logged. If you’re using any load balancer in Source NAT (SNAT)/reverse proxy configuration (most certainly you are), this is the default behavior. You’ll then end up with the load balancer IP instead of the client IP in the (Exchange IIS) logs. Also, with Zen LB, you’ll get NO client logs at all on the load balancer itself. This is not ideal. Furthermore, when I was testing http mode instead of L4xNAT in Zen, I couldn’t get it working as intended (including x-forward-for). I configured it with no persistency/affinity, but Outlook wouldn’t simply start. I scratched my head with this for far too long, and I had to give up. This seems like a bug in Zen if you ask me. (Exchange 2013 is btw designed to work without session affinity/persistence, see https://blogs.technet.microsoft.com/exchange/2014/03/05/load-balancing-in-exchange-2013/).

This left us no other choice than to look for alternative load balancers. We finally settled for HAProxy, as the price tag was just right (free). That said, I ALMOST had to throw in the towel with HAProxy also, as the Mac Outlook clients wouldn’t work no matter what. Luckily I got it sorted out in the end though, more about that later on.

The SNAT/client IP dilemma is a big deal, but the whole architecture/algorithm/method behind it isn’t really the point of this blog post. None the less, you should read at least some homework. Here are a couple of nice articles describing the dilemma and also different load balancing methods:

HAProxy: Microsoft Exchange 2013 architectures (check the limitations in reverse-proxy mode/source NAT)
Logging Client IP Address in IIS When Using Load Balancing with Source NAT
What are the best load balancing methods and algorithms?

Now that you know a little bit more about the LB methods, it’s time to think about how to configure HAProxy itself. HAProxy with Exchange 2013 (2016), that is.

First off, you’ll need L7 load balancing (http), so you’ll get the possibility to insert the x-forward-for header. This is the trick for getting the client IP in the logs (See https://www.loadbalancer.org/blog/iis-and-x-forwarded-for-header/ for information how to configure x-forward-for on the Exchange servers). I found a very nice blog post describing how to install and configure HAProxy for Exchange, “Highly Available L7 Load Balancing for Exchange 2013 with HAProxy”. I have to say it’s TOTALLY AWESOME. I couldn’t have done the configuration without help from this blog post. No other article I found on the Internet had such deep detail and demonstration regarding L7 load balancing for Exchange. That said, I added some custom stuff to the configuration file, and I also configured/installed HAProxy + keepalived a bit different (easier) than in the above guide. Well, enough talking. Here are my steps, based on Zoltan’s blog post:

  • I didn’t install a PKI infrastructure. Most of you probably have a working PKI infrastructure in production already.
    • I use public certificates on Exchange so I have no need for an internal PKI (in this specific case).
  • I didn’t install any Exchange servers either (obviously).
  • I installed CentOS 7 (two servers) in cooperation with our Linux team. This way I got the optimal installation (no extra crap, and centralized management with Puppet).
  • I started following the guide from part 5, http://ezoltan.blogspot.com.au/2014/10/highly-available-l7-load-balancing-for.html
    • Please read the chapter Brief HAProxy Certificate Primer to get an idea of what you’re trying to accomplish!
    • Take note that we’re not doing SSL offloading, we’re in fact doing SSL bridging

 

Certificate preparation

  • I’m using a public certificate, so there’s no need to update the root and intermediate cert store stuff (yet). (In fact the intermediate certificate needed some configuration, more about that later on).
  • I started following the guide more to the letter from the chapter Upload the Exchange Certificate and Private Key onwards
  • I extracted the private key from the pfx file and removed the password protection from the private key. When asked, I entered the password gotten from the public certificate provider. I then entered a new PEM pass phrase (just pick a new one):

           haproxy_extract_private_key

  • Now it’s time to remove the password protection from the private key. You’ll be prompted for the PEM pass phrase entered in the above step:

           haproxy_remove_pw_from_private_key

  • We’re now ready for the final stage – extracting the certificate from the pfx file:

           haproxy_extract_certificate_from_pfx_file

  • We now have all files needed, and we just need to combine the certificate and the private key files, as HAProxy doesn’t allow use of separate files.
    • cat exchange_certificate.pem exchange_private_key_nopassword.pem > exchange_certificate_and_key_nopassword.pem
  • Moved the file to its final destination:
    • mv exchange_certificate_and_key_nopassword.pem /etc/ssl/certs/
    • I’ll now copy/paste Zoltan; “Well, let’s give ourselves a pat on the shoulder, we deserve it. We are through the most difficult part, at least in my opinion, of this lab. Well done!” 🙂 I have to agree with the guy here…

 

HAProxy installation

I did NOT compile HAProxy from scratch, instead I just yum installed the HAProxy package. Much more straight forward. Here are my steps:

  • yum install haproxy
  • I used a “dummy” haproxy.conf –file (for Exchange). It was modified down the road, see the working example from the config file chapter later on.
  • Apart from the info in the blog, you should also edit another line in rsyslog.conf. Add local0.none to the following line:
    • *.info;mail.none;authpriv.none;cron.none,local0.none          /var/log/messages
  • There’s no need to create a log rotate-script when yum installing HAProxy, it gets created automatically!
    • I edited the log rotate-script and changed the rotate parameter to “30”. One month is a suitable time for us to keep the logs.
  • Same thing goes for HAProxy automatic startup at boot. There’s no need for complicated scripts, just execute the following command instead:
    • systemctl enable haproxy:

              haproxy_autostart_centos

  • I then configured the firewall.
  • You should now do some testing. I had previously done some tests in my virtual environment, so I had no need for any tests right now.
  • Thus far, everything is working:

          haproxy_service_status_ok

  • Happy days!

 

HAProxy HA

Now that we have one node working (master), it’s time to think about High Availability. I continued following the excellent guide at http://ezoltan.blogspot.com.au/2014/10/highly-available-l7-load-balancing-for_48.html. Yet again had some changes in my environment. Comments below:

  • I did not compile keepalived from scratch, instead I yum installed it.
  • Added net.ipv4.ip_nonlocal_bind=1 to /etc/sysctl.conf
  • No need to create keepalived.conf, it’s pre-created when yum installed.
  • Edited keepalived.conf to match our environment (copy pasted the blog config and edited email / interface / IP parameters)
  • Edited the hosts file following the blog information
  • There’s yet again no need for a startup script when you have yum installed. Keepalived will autostart at every boot when you execute the following:
    • systemctl enable keepalived
  • I made firewall changes based on the blog.
  • Did all the tests, everything worked like a charm 🙂
  • Made the same changes to the other server/node (backup)
    • priority was set to a lower value than on master node
    • other small changes which are written in Zoltan’s blog
  • Did keepalived-testing following the blog. Everything was fine! 🙂

 

Testing

This has to be my favorite chapter from Zoltan’s blog. I have to say he does a VERY good job explaining all the testing and troubleshooting. READ IT CAREFULLY! I myself also noticed “weird” clients connecting to the “unspecified protocol” back-end “be_ex2013”. If all ACL’s match, there should be no traffic passing through this back-end. I found some autodiscover and ews urls that had mixed upper/lower case letters however. This was easily fixed by adding a few more ACL’s. I’ll paste the config (including the ACL’s) in the next chapter so you’ll get a better understanding of the whole thing.

 

Config file, haproxy.conf

Well, this is the most important part of the whole setup, no doubt. Also the most time consuming. Here’s the config, I’ll explain it in more detail after the paste:

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------

global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #

    log         127.0.0.1 local2 info
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/run/haproxy.stat     


#--------------------------
# SSL tuning / hardening
#--------------------------
    ssl-default-bind-options no-sslv3
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-server-options no-sslv3
    ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    tune.ssl.default-dh-param 2048
   
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------

# Regarding timeout client and timeout server:
# https://discourse.haproxy.org/t/high-number-of-connection-resets-during-transfers-exchange-2013/1158/4

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  forwardfor       except 127.0.0.0/8
    option                  redispatch
#   option                  contstats
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          15m # this value should be rather high with Exchange
    timeout server          15m # this value should be rather high with Exchange
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 100000


#-------------------------------------------------------
# Stats section
#-------------------------------------------------------

listen stats x.x.x.x:444 # VIP
        stats enable
        stats refresh 300s
        stats show-node
        stats auth admin:xxxxxx
        stats hide-version
        stats uri  /stats


#---------------------------------------------------------------------
# Main front-end which proxies to the back-ends
#---------------------------------------------------------------------

frontend fe_ex2013
# http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload
  http-response set-header X-Frame-Options SAMEORIGIN
  http-response set-header X-Content-Type-Options nosniff
  mode http
  bind *:80
  bind *:443 ssl crt /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem
  redirect scheme https code 301 if !{ ssl_fc }   # redirect 80 -> 443 (for owa)
  acl autodiscover url_beg /Autodiscover
  acl autodiscover url_beg /autodiscover
  acl mapi url_beg /mapi
  acl rpc url_beg /rpc
  acl owa url_beg /owa
  acl owa url_beg /OWA
  acl eas url_beg /Microsoft-Server-ActiveSync
  acl ecp url_beg /ecp
  acl ews url_beg /EWS
  acl ews url_beg /ews
  acl oab url_beg /OAB
  use_backend be_ex2013_autodiscover if autodiscover
  use_backend be_ex2013_mapi if mapi
  use_backend be_ex2013_rpc if rpc
  use_backend be_ex2013_owa if owa
  use_backend be_ex2013_eas if eas
  use_backend be_ex2013_ecp if ecp
  use_backend be_ex2013_ews if ews
  use_backend be_ex2013_oab if oab
  default_backend be_ex2013

 

#------------------------------
# Back-end section
#------------------------------

backend be_ex2013_autodiscover
  mode http
  balance roundrobin
  option httpchk GET /autodiscover/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013_mapi
  mode http
  balance roundrobin
  option httpchk GET /mapi/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013_rpc
  mode http
  balance roundrobin
  option httpchk GET /rpc/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013_owa
  mode http
  balance roundrobin
  option httpchk GET /owa/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013_eas
  mode http
  balance roundrobin
  option httpchk GET /microsoft-server-activesync/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013_ecp
  mode http
  balance roundrobin
  option httpchk GET /ecp/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013_ews
  mode http
  balance roundrobin
  option httpchk GET /ews/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013_oab
  mode http
  balance roundrobin
  option httpchk GET /oab/healthcheck.htm
  option log-health-checks
  http-check expect status 200
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt


backend be_ex2013
  mode http
  balance roundrobin
  server exchange1 1.1.1.1:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt
  server exchange2 2.2.2.2:443 check ssl inter 15s verify required ca-file /etc/ssl/certs/ca-bundle.crt

 

#######################################
# End of Exchange's own protocols,
# STMP and IMAP next
########################################


frontend fe_exchange_smtp
    mode tcp
    option tcplog
    bind x.x.x.x:25 name smtp # VIP
    default_backend be_exchange_smtp
 
backend be_exchange_smtp
    mode tcp
    option tcplog
    balance roundrobin
    option log-health-checks
    server exchange1 1.1.1.1:25 weight 10 check
    server exchange2 2.2.2.2:25 weight 20 check

#only port 25 needed in our case. The port is open (only) against our Postfix server, which handles the outgoing mail traffic (MTA). In other words, we're using an external send connector in Exchange.

 
frontend fe_exchange_imaps
    mode tcp
    option tcplog
#   bind x.x.x.x:143 name imap  # NO unencrypted imap traffic allowed...
    bind x.x.x.x:993 name imaps # ssl crt /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem  <-- No need, certificate is read straight from the Exchange servers.
    default_backend be_exchange_imaps
 
backend be_exchange_imaps
    mode tcp
    option tcplog
#   balance roundrobin
    balance leastconn
    option log-health-checks
#   stick store-request src
#   stick-table type ip size 200k expire 30m
#   option tcp-check
#   tcp-check connect port 143
#   tcp-check expect string * OK
#   tcp-check connect port 993 ssl
#   tcp-check expect string * OK
    server exchange1 1.1.1.1:993 weight 10 check
    server exchange2 2.2.2.2:993 weight 20 check

 

 

 

Explanations:

For obvious reasons, all IP addresses are censored.

SSL tuning / hardening section was added by me for additional security. As said in many blog posts before, I care about server security. More information regarding this can be found at: http://www.mattzuba.com/2015/07/hardening-haproxy-for-an-a-rating/ for example. At the same time the http-response set-header X-Frame-Options SAMEORIGIN and http-response set-header X-Content-Type-Options nosniff options were added. I still left the option # http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload commented though. These options are required when aiming for a A+ score on Qualys SSL Labs SSL Server test.

Btw, HAProxy can also protect you from DDOS attacks and other attacks. Have a look at

https://www.haproxy.com/blog/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/
https://www.loadbalancer.org/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2/

for more information.

 

Stats section was added to get some nice stats. You can uncomment the the line  #option contstats to get continuous statistics. At the moment it require a manual webpage refresh.

I added http to https redirect on the main front-end with the line: redirect scheme https code 301 if !{ ssl_fc }

I added new ACL’s so the back-ends would fetch every combination of Autodiscover and so forth:

acl autodiscover url_beg /Autodiscover
acl autodiscover url_beg /autodiscover

acl owa url_beg /owa
acl owa url_beg /OWA

acl ews url_beg /EWS
acl ews url_beg /ews

 

I added a SMTP and IMAP section. We only need port 25 (smtp) to be open, as Postfix is taking care of sending the emails. Exchange only hands them over to Postfix (send connector). IMAPs on the other hand is open to the world, even though we don’t have many users accessing Exchange using imap. (Unencrypted imap isn’t allowed at all). You can use different techniques for configuring imaps (available in comments), but I’ve chosen to use the easiest one. Also note that SMTP and IMAP are using Layer 4 (tcp) load balancing, opposite to Exchanges own protocols. That said, you can still get the imap client IP’s straight from haproxy’s log file (instead of IIS on Exchange).

 

 

Problem solving

Apart from mega problems with Mac (sigh), we also had a hiccup with non-domain joined Windows clients. I had to scratch my head quite a bit before finding the reason why a client would prompt for credentials even though the certificate was correctly applied in HAProxy. Well, it turned out to be a certificate problem after all, and Zoltan briefly even described it in Part 5 of his blog:

By the way, even if you acquire a certificate from a commercial CA, there are no guarantees that its intermediate and root CA certificates will come pre-canned with CentOS 7 (or any Linux flavour as a matter of fact), so it is good to know the procedure anyway, just in case”.

I happened to miss this part though, so I was very confused. At the moment I don’t even remember what pointed me in the right direction, but the solution was indeed to add the intermediate certificate from our certificate provider into the .pem file created earlier:

cat /etc/pki/ca-trust/source/DigiCertCA.crt  > /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem

DigiCertCA is automatically deployed to our Linux servers with puppet, the “only” problem was that HAProxy didn’t understand how to use it without the above trick. More information: https://www.happyassassin.net/2015/01/14/trusting-additional-cas-in-fedora-rhel-centos-dont-append-to-etcpkitlscertsca-bundle-crt-or-etcpkitlscert-pem/. Well, it now makes much more sense why it worked on domain joined Windows clients but not on other clients – the domain joined Windows clients were automatically receiving the intermediate certificate via group policies. Oh well. Live and learn 🙂

Finally, the BIGGEST problem of them all, was Outlook for Mac. What a pain in the ass. Even with the above intermediate certificate in place it wouldn’t work. I was completely clueless, and it seemed the whole Internet was equally confused. (I asked in forums for a solution to my Mac dilemma, without success). What helped me in the end, was a colleague noticing that Outlook for Mac is connecting to Exchange using old protocols. Even the newest Outlook for Mac 2016. SHAME ON YOU MICROSOFT! This however opened up a new world of googling for me. That said, here’s some information regarding the dilemma (which ISN’T a HAProxy problem btw):

https://support.microsoft.com/en-ph/help/2955530/outlook-for-mac-clients-cannot-connect-to-exchange-server
https://outlook.uservoice.com/forums/293343-outlook-for-mac/suggestions/15120408-add-support-for-tls-1-0-1-1-and-1-2
https://www.quora.com/Is-there-a-way-to-connect-Mac-Office-Outlook-2016-with-Microsoft-Exchange-Server-2007
https://answers.microsoft.com/en-us/mac/forum/macoffice2011-macoutlook/outlook-2011-to-use-sslv3/7e777e6b-9e92-4a89-8874-d357c4bdf6ef?auth=1

and the solution:

https://support.microsoft.com/en-us/help/980436/ms10-049-vulnerabilities-in-schannel-could-allow-remote-code-execution

haproxy_allowInsecureRenegotiation

In other words, edit the registry on the Exchange servers to enable Compatible mode. And there you have it, all Mac clients can finally connect through HAProxy 🙂

 

 

Final thoughts

As said before, HAProxy’s logging mechanism is amazing. It does produce quite huge logs though – hundreds of MB’s per day. Luckily the logs are very nicely compressed, and the daily log rotated log takes up less than 100MB HDD space. That said, be sure allocate enough disk space for the logs! Analyzing the logs in depth can be a little bit cryptic, but luckily there’s documentation available. For example, check out

https://cdn.haproxy.com/wp-content/uploads/2017/07/aloha_load_balancer_memo_log.pdf for an amazing chart with all the log value codes and explanations..

The termination flags can be found in HAProxy’s own documentation: https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4-option%20dontlognull. Search for the text “The most common termination flags combinations are indicated below. They are alphabetically sorted, with the lowercase set just after the upper case for easier finding and understanding” and you’re good to go.

Here’s a random screenshot from haproxy.log:

haproxy_log_example_output

 

You’ll now get all the client IP’s from these logs. For obvious reasons they’re censored in the screenshot though. Furthermore, (and perhaps even more important), you can now also check the IIS logs on the Exchange servers and notice that a column (the last one) has been added for x-forward-for:

2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=ce9fb341-8f0b-4315-b3d9-3e77591e0a18@abo.fi&CorrelationID=<empty>;&ClientId=I9XPTDBZEATAAVKDG&cafeReqId=d3cdb726-44ae-4237-9bbc-2c5e5f434e13; 443 – x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 2 5 15 1.2.3.148
2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=57c56614-6647-470d-a620-f3b1f5e2dc8f@abo.fi&CorrelationID=<empty>;&ClientId=SHDGSCWGKIOLIHQCGSKA&cafeReqId=259f647e-646b-4a62-88c9-5de42df747e7; 443 - x.x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 2 5 15 1.2.3.149
2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=43aacf1f-5def-4d3d-9fdb-899ba3c49ec3@abo.fi&CorrelationID=<empty>;&ClientId=IQWDWWOIDUSFPCVQTPW&cafeReqId=4d9bc40c-3d2d-4771-a0bb-024d32ea509c; 443 - x.x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 2 5 0 1.2.3.30
2018-06-30 23:59:42 x.x.x.x POST /mapi/emsmdb/ MailboxId=57c56614-6647-470d-a620-f3b1f5e2dc8f@abo.fi&CorrelationID=<empty>;&ClientId=SHDGSCWGKIOLIHQCGSKA&cafeReqId=c064ab51-5e49-4c86-9e22-ba7a33ca7223; 443 - x.x.x.x Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4639;+Pro) - 401 1 2148074254 0 1.2.3.14

The last column (“1.2.3.x” ) in the above log snippet is the real client IP (obviously censored yet again).

And there we have it. Logging problem solved! 🙂 Now we can finally start tracking down the bad guys…

Test Lab Guide: Windows Server 2016 with Integrated Exchange 2016, SfB Server 2015 and SharePoint 2016

WARNING! This is a pretty long and detailed blog post 🙂

I decided to upgrade (or actually reinstall) my test lab with the most recent version of Windows Server (including the most recent versions of Exchange, SfB Server and SharePoint). All my server virtual machines are built from a clonedGolden Windows Server Image” in VMware workstation, and I also use the same principle for my clients. This way you can deploy new servers/clients very fast, and they will take up much less disk space compared to installing from scratch.

This “custom” TLG is based on:

Windows Server 2012 R2 Test Lab Guide (including the Basic PKI add-on from here) and
Test Lab Guide: Configure an Integrated Exchange, Lync, and SharePoint Test Lab

with my own Exchange add-ons including:

  • A script for configuring the virtual directories
  • Certificate from domain CA
  • Zevenet Load Balancer (formerly known as Zen)
  • A second server (EX2)
  • Another script for copying the virtual directories from an existing server to a new one
  • Database Availability Group (DAG) between EX1 and EX2
  • Moving a user from one database to another

More about these later on.

I’ll start with an overview of the whole TLG, including my own add-ons:

tlg2016_overview

Fig 1. Test Lab overview – a modified picture from the TLG. (I also configured the Internet subnet (131.107.0.0/24), even though not visible in this picture).

 

The whole project started by following the Windows Server 2012 R2 Test Lab Guide. I then added the Basic PKI infrastructure. These Test Lab Guides were actually “translatable” straight from Windows Server 2012 R2 to Windows Server 2016. I got a “Duplicate IP address error” on one of the servers however, but it was easily solved by following: http://support.huawei.com/enterprise/en/knowledge/KB1000068724. (I have no idea why I got this error (hasn’t happened before), but then again it doesn’t matter now that it was solved).

I then moved over to the Test Lab Guide: Configure an Integrated Exchange, Lync, and SharePoint Test Lab. Step 1 was already done so I moved over to Step 2 and 3 – Installing and configuring a new server named SQL1. Step 3 includes a link to a separate SQL Server 2012 Test Lab Guide, and this TLG also happen to be more or less translatable straight to SQL Server 2016. So yeah, I actually have no further comments about the installation. Step 4 guides you through the Client2-installation, but there’s really nothing to comment about this installation either (pretty basic stuff).

 

Exchange 2016, EX1

It was now time for the Exchange server, EX1. Note to self: Use at least 6GB ram for the VM or the memory will run out. This installation also has a separate guide:

https://social.technet.microsoft.com/wiki/contents/articles/24277.test-lab-guide-install-exchange-server-2013-on-the-windows-2012-r2-base-configuration.aspx.

It’s fine for the most part, however instead of downloading the evaluation version of Exchange I suggest you download the newest Exchange Server 2016 CU instead. This way you’ll get the newest updates from scratch. And yes, all setup files are included in the CU so you can use it as a “clean install”. The prerequisites for Exchange 2016 (on Windows Server 2016) are a bit different compared to Exchange 2013 (on Windows Server 2012 R2) also. The only thing you need to download and install “separately” is Microsoft Unified Communications Managed API 4.0, Core Runtime 64-bit. There’s no need for Microsoft Knowledge Base article KB3206632 if you have a recent/patched version of Windows Server 2016. After that just copy/paste the PowerShell command from the prerequisites page:

Install-WindowsFeature NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation, RSAT-ADDS

Then run setup.exe and install Exchange. Use the default options. After completion, follow Step 6: Demonstrate EX1 as an email server in the Exchange TLG. I did NOT try to send an email message from Chris to Janet at this stage though, as I wanted to try this after the Load Balancer was installed. But for now, happy days, Exchange installed!

 

Script for configuring the virtual directories

Usually when installing an Exchange server you change the virtual directories/namespace to something other than the server hostname (default). This namespace is the same name that should be included in the certificate. (I didn’t like the fact that this TLG use self-signed certificates so I added my own subchapter about getting a certificate from a domain CA, see next chapter). In a production environment you should plan the namespace and certificate prior to installation, but in this TLG it doesn’t matter that much. I decided to go with the namespace “exchange.corp.contoso.com”. (Autodiscover (DNS url) should also be included in the certificate (request), which it is by default). Anyhow, I first added the mentioned A records (exchange and autodiscover) to DNS. I pointed them to 10.0.0.11 at this stage (but that will change after the Load Balancer installation). I then changed the virtual directories according to the above plan. For this I used a nice script found from:

https://gallery.technet.microsoft.com/office/Set-all-virtual-directories-f4ec71d3

This script is very nice. The only thing that got me worried was the fact that it tried to change the PowerShell virtual directory. Afaik you shouldn’t change that. Anyway, no big deal, I just answered “no” (seen in screenshot) when the script asked me to change this. Here are a couple of screenshots from the script in action:

tlg2016_set_allvdirs_script1

Fig 2. set-allvdirs.ps1 script

tlg2016_set_allvdirs_script2

Fig 3. set-allvdirs.ps1 script, continued

 

Certificate from domain CA

After all the virtual directories were set, it was time to get a new certificate which reflect the above changes. I headed over to trusty practical 365 to refresh my memory. This time I used EAC when requesting a new certificate. I changed the domains to reflect my newly configured environment. I added exchange.corp.contoso.com and autodiscover.corp.contoso.com and removed all the other hostnames. The other options were pretty basic so nothing special there. I then saved the certificate request and copied it over to my domain CA. However, when I tried to process the certificate request on the CA I was greeted with an error message:

tlg2016_cert_req_error_from_ca

Fig 4. Certificate Request Processor error

A bit of investigation led me to the following url: http://mytechweblog.blogspot.fi/2012/11/the-request-contains-no-certificate.html, which had a solution:

“certreq -submit -attrib “CertificateTemplate: WebServer” WebServerCertReq.txt”

tlg2016_cert_req_manual_submit

Fig 5. Certification request with manual submit.

This solution worked for me, nice! I then saved the .crt file and imported it into Exchange from the same place in EAC where I made the request. However, shortly after this I noticed that EAC and OWA still gave certificate errors. This was strange, but then again nothing new. I had a look in IIS/Bindings, and surely the wrong certificate had been assigned. I corrected this so the newly requested certificate was in use:

tlg2016_cert_assignment_from_IIS_on_ex1

Fig 6. Exchange certificate from domain CA.

 

Zevenet Load Balancer

It was now time to install the Zevenet Load Balancer. The reason for installing the Load Balancer at this stage had to do with the fact that I had now preconfigured all the Exchange virtual directories + autodiscover in DNS. This also meant that it’ll be very easy to point DNS at the Load Balancer instead of the Exchange server/CAS further down the road.

I headed over to https://www.zevenet.com/products/community/ and downloaded the newest version. I installed it following my own old blog post. The main difference this time was that I didn’t bother to use clustered servers. (I already know that it works and we’re using clustered LB’s in production). After installation I did the initial configuration:

tlg2016_zen_new_virtual_network_interface

Fig 7. New virtual network interface. The new VIP was set to 10.0.0.61 (the server IP is 10.0.0.60).

 

I then created a new farm, which listens on port 443 on the newly created virtual network interface IP (VIP):

tlg2016_zen_new_farm

Fig 8. New Farm

 

After this I edited the farm and configured the “real IP”:

tlg2016_zen_edit_real_ip_servers_configuration

Fig 9. Real IP’s. In my case, 10.0.0.11 is the “real” IP for EX1.

 

I then converted the Exchange certificate (in a Linux VM) for use within Zevenet LB:

openssl pkcs12 -in file.pfx -out file.pem -nodes

Source: https://stackoverflow.com/questions/15413646/converting-pfx-to-pem-using-openssl

 

It was then time to import it into Zevenet LB:

tlg2016_zen_manage_certificates

Fig 10. Certificate imported

 

After this I made changes in DNS so that all traffic would go through the Load Balancer:

tlg2016_zen_dns_edited

Fig 11. Editing DNS.

 

Now it was finally time to check that Outlook was working correctly from client1 (or 2):

tlg2016_outlook_connection_status_through_zen

Fig 12. Outlook Connection Status

Well yes, it was. Perfect! 🙂

That quite much summarizes the Load Balancer part. Now moving over to the installation of the second Exchange server, EX2.

 

 

Exchange 2016 – second server, EX2

Now that everything was working as it should with EX1, it was time to add another exchange server to the environment. There were no special notes about this installation, I just followed the same guide as with the first one. One thing that was different however, was the script. I now used a script that could automatically copy the virtual directories from an existing Exchange server during deployment. The script can be found at:

http://www.expta.com/2016/07/new-set-autodiscoverscp-v2-script-is-on.html

I’ll copy/paste some information:

The script is designed to be run during installation. Normally, you would run this script from an existing Exchange server of the same version while the new server is being installed.

That sounded almost too good to be true and I had to try it. That said, I had a test-run from EX1 while EX2 was installing:

tlg2016_set_autodiscover_scp_script1

Fig 13. Set-Autodiscover.ps1 script. Looks promising…

…but it wasn’t:

tlg2016_set_autodiscover_scp_script2

Fig 14. Can’t set the virtual directories.

I had the script running during the whole installation of EX2, but no luck. I suspected that it would be better running the script immediately after the installation instead. That said, I had a go just after the finished installation of EX2:

tlg2016_set_autodiscover_scp_script3

Fig 15. Running the script immediately after the EX2 installation.

Yes, much better this time. All the virt dirs were set within a couple of seconds, and I’d say this “lag” would be fine for a production environment as well. I would also like to “thank” the script for reminding me to install a certificate on this second server. That said, I opened up EAC and chose the new EX2 server from the pull-down menu under certificates. I then chose “import” and used the same certificate I made for EX1. It got imported nicely:

tlg2016_imported_cert_ex2

Fig 16. Imported domain certificate

 

Be sure to enable all needed services on the newly imported certificate also:

tlg2016_setting_services_on_imported_certificate_ex2

Fig 17. Overwriting existing SMTP certificate. At the same time I also chose to enable the IIS, POP and IMAP services.

 

Checking certificates from EMS:

tlg2016_checking_certs_from_ems

Fig 18. Checking active certificates. Looks good!

 

…and while you’re at it, check that OWA won’t give you certificate errors:

tlg2016_checking_that_cert_ís_ok_from_owa

Fig 19. OWA

It doesn’t. All good! (The new certificate wasn’t yet active In Fig 16, therefore the status url bar is red).

Only thing left to do now was to add this second IP (10.0.0.12) to “real servers” in Zevenet Load Balancer. After this change, the “dual exchange server setup” was ready for use.

 

DAG

With both EX1 and EX2 up ‘n running, it was time to configure a Database Availability Group (DAG) between the servers. I’ve done this many times before, and I’ve always used the same guide whether it’s for Exchange 2013 or Exchange 2016. The guide I’ve used is:

https://practical365.com/exchange-server/installing-an-exchange-server-2013-database-availability-group/

It’s very straight forward without any extra bs. Some notes:

  • I’m using the “SP1 (APP1)” server as the witness server.
  • I pre-staged a computer account in AD named “EXDAG”
  • I did not configure a dedicated replication network. (Overkill for a test lab).
  • I did not move the Default Mailbox Databases from the default folder path onto storage volumes dedicated to databases and transaction log files. (Again, a little overkill for a test lab).

tlg2016_manage_db_availability_group_membership

Fig 20. Manage Database Availability Membership

 

After this step was done I configured database copies following yet another good (and familiar) follow-up guide from the same series:

https://practical365.com/exchange-server/exchange-2013-dag-database-copies/

I’ve got no additional comments about the database copies, they work just as intended/written in the guide 🙂 Below you’ll find some related screenshots:

tlg2016_add_mailbox_db_copy

Fig 21. Add Mailbox Database Copy

 

tlg2016_databases_overview_after_db_copy_and_dag

Fig 22. Database overview with database copies.

 

Moving users

I moved the user “Janet” from the original database on EX1 over to the database on EX2. This way I “spread out” my (two 🙂 ) users so their mailboxes are situated on different servers. This is good for failover testing and so forth.

tlg2016_move_user_janet_to_ex2

Fig 23. Moving Janet to another database (server).

 

tlg2016_move_user_janet_to_ex2_2

Fig 24. Moving Janet to another database (server), continued.

 

The above steps now completes the whole Exchange-part of the TLG.

 

 

Skype for Business Server 2015, LYNC1

It was now time to move over to the Lync-part of the TLG. The first change was actually the software itself – I’m installing Skype for Business Server 2015 instead of Lync Server 2013. As with other software in this lab, the prerequisites are way different for SfB Server compared to Lync Server. I used a combination of

https://blogs.perficient.com/microsoft/2017/08/skype-for-business-how-to-install-on-windows-server-2016/ and
http://www.garethjones294.com/install-skype-for-business-server-2015-on-windows-server-2016-step-by-step/

as a base for my deployment. Some additional notes:

  • Note to self: Use at least 3GB ram for the VM.
  • (Newest) Cumulative Update has to be installed, otherwise SfB Server won’t work at all on Windows Server 2016.
  • As I installed SfB Server in an isolated network (no internet access), I also had to define the source (which is the Windows Server 2016 DVD) in the PowerShell prerequisite command:

          tlg2016_sfb_install_prereq_from_powershell

          Fig 25. Prerequisites installation for SfB Server 2015 on Windows Server 2016.

 

I then continued following the TLG guide again, and moved over to the chapter “To prepare Active Directory”. Some notes:

  • Installed newest version of offline-Silverlight manually.
  • Chose not to check for updates.
  • Added the DNS SRV records, but they didn’t work when I tested them (probably outdated info in the TLG). This was no big deal, as lyncdiscoverinternal can be used instead for example. You could also Google for updated information, but I didn’t feel it was necessary for this TLG.
  • Everything went fine until “21. From the Topology Builder Action menu, select Publish Topology.” I was greeted with:

          tlg2016_sfb_publishing_topology

          Fig 26. Publishing Topology error.

          tlg2016_sfb_publishing_topology_error

          Fig 27. Publishing Topology error, continued

Well, after some investigation (googling), it turned out this just had to do with UAC: http://terenceluk.blogspot.fi/2013/03/publishing-new-lync-server-2013.html. Surely, after running the deployment wizard again as an administrator (run as), it worked!

 

I now moved over to the “To install Lync Server 2013 core components” -part of the TLG. Notes:

  • I was only running step 1 and 2 at this stage.
  • The IIS URL Rewrite Module problem was well known, I’ve even blogged about it.
  • After step 2 was done, it was time to install the newest CU for SfB, otherwise SfB Server won’t run at all on Windows Server 2016.
    • Remember to run the SfB Management Shell as an Administrator.
    • Everything went smoothly with the CU installation!
  • I moved over to Step 3 – Deployment Wizard/SfB Server 2015 Core Components.
    • Everything went fine!
  • Step 4 was different for SfB compared to Lync. You can’t start services from the Deployment Wizard in SfB Server 2015.
    • Instead, you start them from the SfB Management Shell with the command “Start-CsWindowsService
    • The command didn’t run as planned though:

                tlg2016_sfb_start-cswindowsservices_error

                 Fig 28. SfB Server 2015 Deployment Log.

    • I tried to manually start the “Skype for Business Front-end” service from “Services” in Windows.
      • Did not work either, got stuck in “starting…”
      • Tried old school method and rebooted the server.
        • Worked, all services were now up ‘n running after reboot 🙂
  • I moved over to the “To enable users in the Lync Server Control Panel”-part of the TLG and enabled the users.
  • Yep, all done, working! 🙂

 

 

SharePoint 2016, SP1

SharePoint was the last (and the “easiest”) software candidate on the list. Yet again the prerequisites were different compared to the 2013 version of the TLG. My notes:

  • SP1 is the server name, not Service Pack 1 🙂
  • I tried various offline methods for the prerequisite installation. What a headache. Spare yourself the pain and DO NOT try to install the prerequisites without an active internet connection. I repeat, DO NOT try it.
  • I then installed the prerequisites with “Install software prerequisites” from default.hta. Everything went smoothly.
  • I continued following the TLG and the “To prepare DC1 and SQL1 –part. Nothing to add or comment here.
  • I continued following the TLG and the “To install SharePoint Server 2013” –part. Nothing to add or comment here.
  • Happy days, SP1 installed!

 

 

Configure integration between EX1, LYNC1, and SP1

As a last step in this TLG, I configured server integration between the servers. I would advise you to stay away from the TLG script and use newer information instead. The script has failed me before, and surely it failed this time also when I tried it. In other words, skip the script.

As a first step though, check the SP1/APP1 certificate. The TLG tells you to add a https site binding and select the certificate with the name sp1.corp.contoso.com. This won’t work, at least not for me (never has). Instead, when creating the new https binding, choose the certificate that has been issued to the SP1/APP1 server (never mind the confusing “friendly” name):

tlg2016_sp_checking_cert

Fig 29. Checking SSL certificate in SharePoint/IIS

I got a warning about the certificate already being used for the Default Web Site, but this can be ignored (at least in this TLG).

 

Now we’re ready to move over to some “fresh” information about integration. For starters, have a look at:

Exchange <-> SfB: http://lyncdude.com/2015/10/06/the-complete-skype-for-business-exchange-2016-integration-guide-part-i/index.html
Exchange –> SharePoint and SfB: https://technet.microsoft.com/en-us/library/jj649094(v=exchg.160).aspx
SfB –> SharePoint: https://technet.microsoft.com/en-us/library/jj204975.aspx
SharePoint –> Exchange: https://technet.microsoft.com/en-us/library/jj655399.aspx
SharePoint –> SfB: https://technet.microsoft.com/en-us/library/jj670179.aspx

All links are compatible with the 2016 versions also. Here are the results from my own environment:

 

Skype for Business:

tlg2016_oauth_sfb_check_current_cert

Fig 30. Checking current OAuth certificate and OAuth configuration.

tlg2016_oauth_sfb_setting_cs_auth_configuration

Fig 31. Setting OAuth configuration and checking the configuration.

 

tlg2016_oauth_sfb_to_ex

Fig 32. SfB –> Exchange integration

tlg2016_oauth_sfb_to_sp

Fig 33. SfB –> SharePoint integration

 

tlg2016_oauth_sfb_to_ex_and_sp_check

Fig 34. Checking partner applications. Both Exchange and SharePoint are integration partners.

 

Exchange:

tlg2016_oauth_ex_to_sfb

Fig 35. Exchange –> SfB integration

tlg2016_oauth_ex_to_sp

Fig 36. Exchange –> SharePoint integration

 

tlg2016_oauth_ex_to_sfb_and_sp_checking

Fig 37. Checking partner applications. Both SfB (Lync) and SharePoint are integration partners.

 

SharePoint:

tlg2016_oauth_sp_to_ex

Fig 38. SharePoint –> Exchange integration

tlg2016_oauth_sp_to_sfb

Fig 39. SharePoint –> SfB integration

 

tlg2016_oauth_sp_to_ex_and_sfb_checking

Fig 40. Checking partner applications. Both Exchange and SfB are integration partners.

 

The integration chapter above now finalizes this whole TLG. It was a fun project and I hope someone will find this information useful.