Changing Outlook Connectivity towards MAPI over HTTP

We had a big Microsoft Office 2016 upgrade project this fall. The main reason for this was a non-consistent Office environment with versions dating all the way back to MS Office 2007. The whole upgrade process was done using a System Center Configuration Manger application package, which I also happen to be the author for. The upgrade process went mostly fine, even though it was quite a long process involving many (stubborn) users and computers.

Anyhow, now that a new MS Office version was deployed, it was finally time to think about changing the Outlook protocol and authentication to a more modern one. RPC over HTTP (and basic/ntlm authentication) was getting dated, and the main reason for not using MAPI over HTTP were the old clients. That problem is now gone 🙂

So, what’s up with this MAPI-thing then? I’ll just copy/paste a few things and you can read more about MAPI in the provided links.

“MAPI over HTTP is a new transport used to connect Outlook and Exchange. MAPI/HTTP was first delivered with Exchange 2013 SP1 and Outlook 2013 SP1 and begins gradually rolling out in Office 365 in May. It is the long term replacement for RPC over HTTP connectivity (commonly referred to as Outlook Anywhere). MAPI/HTTP removes the complexity of Outlook Anywhere’s dependency on the legacy RPC technology”.

“The primary goal of MAPI/HTTP is provide a better user experience across all types of connections by providing faster connection times to Exchange – yes, getting email to users faster. Additionally MAPI/HTTP will improve the connection resiliency when the network drops packets in transit. Let’s quantify a few of these improvements your users can expect. These results represent what we have seen in our own internal Microsoft user testing.”

Source: https://blogs.technet.microsoft.com/exchange/2014/05/09/outlook-connectivity-with-mapi-over-http/

MAPI over HTTP is also the default protocol in Exchange 2016 which clearly shows the way Microsoft is going. More information about MAPI:

https://technet.microsoft.com/en-us/library/dn635177(v=exchg.150).aspx
http://markgossa.blogspot.fi/2015/11/exchange-2013-and-exchange-2016-mapi.html
http://searchexchange.techtarget.com/definition/MAPI-over-HTTP-Messaging-Application-Programming-Interface-over-HTTP

What are the requirements for MAPI then?

 

Main Prerequisites

Complete the following steps to prepare the clients and servers to support MAPI over HTTP.

  1. Upgrade Outlook clients to Outlook 2013 SP1 or Outlook 2010 SP2 and updates KB2956191 and KB2965295 (April 14, 2015).

  2. Upgrade Client Access and Mailbox servers to the latest Exchange 2013 cumulative update (CU). For information about how to upgrade, see Upgrade Exchange 2013 to the latest cumulative update or service pack.

Source: https://technet.microsoft.com/en-us/library/dn635177(v=exchg.150).aspx

No problem, these prerequisites were now in order for us.

 

Other Prerequisites

Check/Set the MAPI Virtual Directories

Clients can’t connect if you don’t have working MAPI Virtual Directories. Just follow the above TechNet article and you’ll be fine. Our original MAPI virtual directories looked like this:

exchange_mapi_virt_dirs_before_change

Sorry for the blur. All I can say is that the InternalUrl had the Exchange server’s hostname specified instead of the (single) namespace. Changing and verifying the new url’s is done in the following screenshot:

exchange_mapi_virt_dirs_changing_urls

I changed the ExternalUrl on one of our Exchange servers. I then used the same command for the InternalUrl, replacing the word “External” with “Internal”. I also made the same change to our second Exchange server. The end the result would have both external and internal url’s listed, like so:

exchange_mapi_virt_dirs_after_change

Again, sorry for the blur. We’re using a single namespace so all url’s are basically identical.

 

Enable MAPI over HTTP in your Exchange Organization

This is easily done with one command:

Set-OrganizationConfig -MapiHttpEnabled $true

In theory, this isn’t needed if the user mailboxes already have MAPI enabled:

“To enable or disable MAPI over HTTP at the mailbox level, use the set-Casmailbox cmdlet with the MapiHttpEnabled parameter. The default value is Null, which means the mailbox will follow organization-level settings. The other options are True to enable MAPI over HTTP and False to disable. In either case, the setting would override any organization-level settings. If MAPI over HTTP is enabled at the organization level but disabled for a mailbox, that mailbox will use Outlook Anywhere connections”.

Source: https://technet.microsoft.com/en-us/library/mt634322(v=exchg.160).aspx

Checking if enabled on user level (we had it enabled already):

exchange_checking_mapi_user_mailbox

That said, it’s still a good idea to enable it organization wide (if you have old migrated room mailboxes or public folders). See the following link:

https://social.technet.microsoft.com/Forums/ie/en-US/f4df02af-20cd-423c-8c59-1ea563b7b940/exchange-2013-mapihttp-public-folder-and-room-mailboxes-still-using-rpchttp?forum=exchangesvrdeploy

We enabled it organization wide and It was now time to do some tests.

 

Testing

Still following the TechNet article:

exchange_checking_mapi_selftest

All good 🙂

The change isn’t instant. A recycle of the MSExchangeAutodiscoverAppPool make things happen faster however. After a MSExchangeAutodiscoverAppPool recycle and a coffee break plus a restart of my own Outlook client, I had a look at the Outlook Connection Status:

exchange_outlook_connection_status_with_mapi

Well, well, well. All connections are now using MAPI over HTTP instead of the old RPC over HTTP.

You’ll see several changes:

  • First change is in the “Server name” column. It’s now a real server name instead of a mailbox GUID.
    • Notice also that the server path includes /mapi/.
  • The protocol (column) has changed from RPC/HTTP to just HTTP.
  • The “Authn” column have changed values from NTLM to Nego* (Kerberos)

 

Further check-ups

Now that we’re using up2date clients, there’s no need to use old authentication methods for Outlook Anywhere either. (Outlook Anywhere authentication and MAPI authentication are configured/can be configured separately btw). The main reason for sticking with the old Basic/NTLM authentication (at least externally) is when using an Exchange 2010/2013 co-existence environment (and sticking with old “best practices”). This hasn’t been the case for us for a long time. Another reason for still sticking with “Basic” is, well, old clients. A third reason would be a mix of both old clients and an Exchange 2010/2013 co-existence environment. Luckily for us, we now have a “pure” Exchange 2013 environment and (almost) all Windows clients are using Outlook 2016.

Even though Outlook clients are now configured to connect via the MAPI protocol, there will still be some clients connecting via RPC over HTTP. This will probably be the case for quite some time, as the clients are gradually moving towards MAPI. (The change isn’t instant as already stated). Many users don’t restart their Outlook client so often either, which in turn means that they’re using RPC over HTTP until restart (at least from my experience). Correct me if I’m wrong.

As a side note, the MAPI protocol is only used for Windows Outlook clients. The Mac version of Outlook use EWS, and mobile phones use ActiveSync. So yes, you’ll still have many different active protocols in your organization. Don’t disable them 🙂

If you want to read more about the Outlook Anywhere authentication types, have a look at http://msexchangeguru.com/2013/01/10/e2013-outlook-anywhere/ for example.

A checkup of our old settings for Outlook Anywhere:

exchange_checking_outlook_anywhere_auth_methods

As you can see, the servers were still using Basic for external authentication method. Time to change that! This time I’ll use EAC:

exchange_outlook_anywhere_changing_external_auth_from_eac

You’ll get a warning that you shouldn’t enable Negotiate if you’re having Exchange servers older than version 2013 in your environment. Again, no problem!

I did the same for the internal url (must be done from EMS):

exchange_outlook_anywhere_changing_internal_auth_from_ems

Like so. All authentication methods pimped to modern standards 🙂

 

This should cause no problems at all (and still haven’t). A good referral:

For Exchange 2007/2010

Set-OutlookAnywhere -Identity ‘SERVER\Rpc (Default Web Site)’ -SSLOffloading $true -ClientAuthenticationMethod NTLM -IISAuthenticationMethods Basic,NTLM

For Exchange 2013+ with backwards compatibility with Outlook 2010 and 2007

Set-OutlookAnywhere -Identity ‘SERVER\Rpc (Default Web Site)’ -SSLOffloading $true -ExternalClientAuthenticationMethod NTLM -InternalClientAuthenticationMethod NTLM -IISAuthenticationMethods Basic,NTLM,Negotiate

For Exchange 2013+ with Outlook 2013+

Set-OutlookAnywhere -Identity ‘SERVER\Rpc (Default Web Site)’ -SSLOffloading $true -ExternalClientAuthenticationMethod Negotiate -InternalClientAuthenticationMethod Negotiate -IISAuthenticationMethods Basic,NTLM,Negotiate

Source: https://community.spiceworks.com/topic/1979514-outlook-auto-discover-issue

 

Lastly, if you’re having trouble with the famous and dreadful Outlook credential-popup, changing the authentication methods should help.

“Basic authentication: If you select this authentication type, Outlook will prompt for username and password while attempting a connection with Exchange”.

Source: http://msexchangeguru.com/2013/01/10/e2013-outlook-anywhere/ 

Similar information can be found at:

http://www.sysadminlab.net/exchange/outlook-anywhere-basic-vs-ntlm-authentication-explained
https://community.spiceworks.com/topic/1979514-outlook-auto-discover-issue

for example. I’ve seen this mentioned on other sites as well, I just can’t remember them right now. We’ve also seen the “popup dilemma” here, but luckily gotten rid of it by now.

 

Get-ActiveExchangeUsers script

If you want to get some nice details about how your users connect (per protocol), I’d suggest you grab the script from

https://gallery.technet.microsoft.com/office/Get-ActiveExchangeUsersps1-f1642024

As seen in my screenshots below, not everyone is connected over MAPI (yet). Old RPC connections are still used, and that will probably be the situation for a while. If someone wiser than me care to explain why this is the case, please do.

Get-ActiveExchangeUsers_script_server1

Get-ActiveExchangeUsers_script_server2

The user activity was fairly low when these screenshots were taken.

 

Checking logs

Finally, if you feel like deep diving into the logs, the MAPI-stuff gets logged in:

C:\Program Files\Microsoft\Exchange Server\V15\Logging\MAPI Client Access for the CAS and
C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Mapi for the Mailbox.

If you see any suspicious things in here, do Google (or use common sense 🙂 ).

Advertisements

Moving SpamAssassin’s Spam-flagged Mail Automatically into Outlook’s Junk E-mail Folder

More and more of our users are moving towards a pure Exchange environment. This means that both email and calendar functions are used from Exchange. This is the default behavior for most companies nowadays, but we’ve been using a 3rd party imap server for email. The (slow) migration towards Exchange also brings challenges. One of many challenges is regarding the spam management (on the client). Should it be an automatic or a manual process for the users?

On imap, we have a postfix server in front of the imap server. The postfix server is configured to take care of spam and antivirus using SpamAssassin+Amavis. (Postfix routes email via the SpamAssassin/Amavis server before it “lands” on the imap (and Exchange) “back-ends”). This is still true for Exchange – which is a good thing. There’s no need to install (or buy a license for) a separate Exchange Edge Transport server to handle spam and antivirus when we already have a VERY efficient Spam/AV-solution deployed.

However, Exchange have no idea what to do with these spam-flagged emails by default – it’s delivering them straight into Outlook’s inbox. Here’s an example of a spam-flagged email (without Exchange server-side filters/transport rules):

Exchange_spamexample_without_filters

Fig 1. A Spam-flagged email (by SpamAssassin) from a user inbox. No server-side configuration done yet.

Even though this email is already flagged as spam (***Spam***), it’s not very “nice” having them arrive in your inbox. What you want is an automated process/feature which moves spam automatically into Outlook’s “Junk E-mail”-folder.

On a side note, not all spam in our organization reaches the users. If an email has a spam score over 12, it won’t be delivered to a user mailbox. A spam score between 5-11 is classified as spam, so these emails will be delivered to the users however. Of course emails with a score of 5 or lower will also be delivered, but these emails aren’t flagged as spam 🙂 If you want to get technical and read about spam scores in SpamAssassin, go ahead:

https://en.wikipedia.org/wiki/SpamAssassin
https://litmus.com/help/testing/spam-scoring/
https://help.aweber.com/hc/en-us/articles/204032316-What-is-a-good-Spam-Assassin-Score

 

Well, how do you automatically sort the spam emails to the Junk E-mail folder then? You can do this on the client-side (Outlook), but the more logical solution is to filter these on the server-side (Exchange). I followed a good guide from http://exchangepedia.com/blog/2008/01/assigning-scl-to-messages-scanned-by.html and got it working. Some info regarding SCL:

https://technet.microsoft.com/en-us/library/aa995744(v=exchg.150).aspx
https://technet.microsoft.com/en-us/library/dn798345(v=exchg.150).aspx

I hadn’t touched the default values (no need) so they looked like this:

Exchange_contentfilterSCLconfig

Fig 2. Content filtering default configuration.

 

I then followed the guide to make a new Transport Rule. I made it from EAC, but it can easily be created from EMS as well:

Exchange_spam-status-rule

Fig 3. Creating a new Transport Rule.

I made sure that SCL was set to a HIGHER value than the SCLJunkThreshold. In our case “9”. It won’t work otherwise, even though some other guide tells you to put it at a lower value.

 

It’s also a good idea to check that Junk E-mail configuration is enabled on the mailboxes. Here I do a check on my own mailbox:

Exchange_junkemailconfiguration

Fig 4. Check Junk E-mail configuration. It’s enabled.

Good. Everything is in order. Now we should test and see if a spam email arrives in my Junk E-mail folder instead of the Inbox. I spam-bombed myself to test, and yes, the spam arrives correctly in the Junk E-mail folder:

Exchange_spamexample_with_filter

Fig 5. Spam in the Junk.

 

Let’s see what the spam looks like in more detail:

Exchange_spamexample_with_filters

Fig 6. A Spam-flagged email, now with the server-side configuration done.

As you can see, Outlook now provides more information. When a message arrives/is situated in the Junk E-mail folder, it has disabled links and is also converted into plain text.

Happy days – spam is now automatically moved into the Junk E-mail folder for each user 🙂

 

(If you for whatever reason would want to configure this whole procedure on the client side instead of Exchange, you could follow https://www.jyu.fi/itp/en/email/how-to/exchange-spam-filtering for example).

Disabled (AD) users still searchable in Lync/SfB

Well, this was a bit of a nightmare to debug. I started the whole project with some googling on how to (automatically) remove AD-disabled users from Lync (SfB) Server. This procedure is by no means automatic, and If you disable a user in AD they can still use Lync (at least for some time). Have a look at https://ucgeek.co/2014/04/lync-users-can-login-after-domain-account-disabled/ for some deeper details.

My concern wasn’t about the fact that the users still could use Lync however, it was more about the fact that they are “hanging around” like ghosts on the Lync Server for no good reason. In plain English – users should be disabled if their AD account is also disabled.

I’ll now present MANY different ways to get rid of the unwanted users in the Lync/SfB Address Book. One method might work for someone, and another method might work for someone else. Hell, you might even need to use all the methods combined 🙂

 

Method 1: Disable AD-disabled users on Lync/SfB Server

This idea led me to some PowerShell scripts. Here are a couple of examples that will disable users on the Lync Server if they are disabled in Active Directory:

http://www.markc.me.uk/blog/files/ADDisabled.html
https://www.ucunleashed.com/265
https://trogjels.wordpress.com/2012/08/16/lync-disable-ad-disabled-users/

This worked just fine and I got all the accounts removed from the Lync Server. A little addition to the above links is that I now use this script from a Scheduled Task:

disableADdisabledaccount_task_scheduler

 

The PowerShell “script” itself looks like this:

# disable_ADdisabled.ps1 , source http://www.markc.me.uk/blog/files/ADDisabled.html
# J.S. 13.9.2017
Import-Module ‘C:\Program Files\Common Files\Microsoft Lync Server 2013\Modules\Lync\Lync.psd1’
Get-CsAdUser -ResultSize Unlimited | Where-Object {$_.UserAccountControl -match “AccountDisabled” -and $_.Enabled} | Disable-CsUser

Well, still no luck. You think your users wouldn’t show up when searching in Lync (Address Book) if they are no longer activated on Lync Server (and also has the –HiddenFromAddressListsEnabled flag, see next chapter). Well, they do. At least for us. This was getting pretty weird. I tried googling but all I could come up with was “Update-CsAddressBook” and so forth. Examples:

http://lyncinsider.com/lync-server-2010/updating-lync-contacts-sync-error-fixes-part-2-of-3/
http://www.fots.nl/lync-address-book-sync-process-information/
http://blog.schertz.name/2010/09/updating-the-lync-2010-address-book/
https://social.technet.microsoft.com/Forums/sharepoint/en-US/2e0d1200-b924-42dd-8700-55893b8cfc03/deleted-lync-user-still-in-list-showing-presence-unknown?forum=officeitpro
https://technet.microsoft.com/en-us/library/gg429695(v=ocs.15).aspx

No. Still no luck. I could see the “old” users which aren’t enabled in AD nor on the Lync Server. This was getting frustrated. Better turn to my old friend Event Viewer. Yes, some hits indeed:

Event ID 21054, Error:
Users are not indexed in the database that should be.

Expected indexed user count: 0
Actual indexed user count: 1999
Cause: User replication issue.
Resolution:
Run Update-CsAddressBook to synchronize all accounts.

Well, what command did I just run? Did it help? No. Then again if you Google the message you’ll soon notice that this error is nothing to worry about:

“As long as “Objects not indexed that should be” and “Abandoned Objects” are zero, the database is okay.” I did run the Debug-csAddressBookReplication and “my” objects were zero. Fine.

Source: https://social.technet.microsoft.com/Forums/lync/en-US/49de4752-a73e-4a55-98d6-a48409a02d94/alert-lync-users-are-not-indexed-in-the-database-that-should-be?forum=lyncdeploy (you can find this information in many other sources as well…)

 

Method 2: HiddenFromAddressListsEnabled PowerShell command

There are many different flavors of this command, but the one we’re using is something like the one found at https://stackoverflow.com/questions/7974204/powershell-command-to-hide-user-from-exchange-address-lists (and the answer “I use this as a daily scheduled task to hide users disabled in AD from the Global Address List”). We also use it in a Scheduled Task.

This HiddenFromAddressListsEnabled –method was applied at an earlier stage, as we had “problems” with Outlook / Exchange showing disabled users in Global Address List (GAL). The above PowerShell-trick solved the problem on the Outlook/Exchange-side. It SHOULD however also work with Lync/SfB if you are using the Unified Contact Store, UCS. More information about UCS in the next chapter. However, it didn’t work for us. We still saw disabled users in Lync’s Address Book 😦

 

Method 3: Rollback from UCS

What now? I’m ok with just ignoring the event in Event Viewer, but I still have the same problem. My disabled users are visible in Lync’s Address Book. This got me thinking about the Exchange <-> Lync relationship. I know they are a tight fit. Like all MS products. The https://social.technet.microsoft.com/Forums/sharepoint/en-US/2e0d1200-b924-42dd-8700-55893b8cfc03/deleted-lync-user-still-in-list-showing-presence-unknown?forum=officeitpro post had some hints at the end also. Many, MANY Google searches later got me to https://support.office.com/en-us/article/Set-Personal-options-c09b21ac-7334-49cf-a510-d8c432fcaf01

And there it was. The answer I was (partly) looking for:

You specify how Skype for Business interacts with other Office programs.

  1. In Personal options, under Personal information manager, select either Microsoft Exchange or Microsoft Outlook or None from the drop-down list. When you select Microsoft Exchange or Microsoft Outlook, the Skype for Business search feature uses the Microsoft Outlook Contact list as a source of contacts, in addition to the global address list.

    When you select None, the Skype for Business search feature returns contacts only from the global address list. It doesn’t use either the Windows Address Book or the Outlook Contact list.

Source: https://support.office.com/en-us/article/Set-Personal-options-c09b21ac-7334-49cf-a510-d8c432fcaf01

So, was my solution setting this setting to “None”? You’d think so. Think again. This setting is different in the SfB client. You CAN’T set it to none. Have a look at https://answers.microsoft.com/en-us/msoffice/forum/msoffice_sfb-mso_winother/skype-for-business-2016-personal-information/67d14b5e-23d7-4d23-ab6e-5710739cb0d3 for example.

Well, as my integration is controlled “by my administrator” (which would be myself thank you very much), it got me thinking that it HAS to be a server setting. Now the question was, WHICH setting specifically? Now I honestly don’t remember what I googled to find this, but in the end it got me to http://techgenix.com/integrating-exchange-server-2013-and-skype-business-server-2015-part6/

Well well well. This was all new to me. I ran the command Get-CSUserServicePolicy and found out that UcsAllowed was set to true. It was indeed enabled for ALL our users. I also checked from the SfB client itself, and yes, (not surprisingly) same result:

UCS_screenshot_SfB_configuration_information

 

So, what is UCS?

Detailed Description

The unified contact store introduced in Lync Server 2013 gives administrators the option of storing a user’s contacts in Exchange instead of in Skype for Business Server 2015; in turn that allows the user to access the same set of contacts in Outlook and Outlook Web Access as well as. (Alternatively, you can continue to store contacts in Skype for Business Server 2015. In that case, users will have to maintain two separate sets of contacts: one for use with Outlook and Outlook Web Access, and one for use with Skype for Business.)

Source: https://technet.microsoft.com/en-us/library/jj205414.aspx

Now the big problem here is that our user accounts used in different systems aren’t disabled at the same time. Users aren’t automatically disabled in Exchange if they get disabled in AD (at least not yet, but this will probably change when our imap-server retires). See the pattern here? What this means is that if a user is still “active” (from Exchange’s point of view), it ALSO lists this person in LYNC’s Address Book. This also means that I’ve found the root problem for our “ghost users”.

The solution is NOT using UCS, and rollback to the system where the contacts are stored on the Lync Server instead of Exchange. Well yeah, you COULD disable UCS, but it feels like going the wrong way. If you still want to test however, the information for this procedure is available here:

https://technet.microsoft.com/en-us/library/jj205224(v=ocs.15).aspx (To rollback unified contact store contacts from Exchange 2013 to Lync Server 2013).
https://technet.microsoft.com/en-us/library/jj688083.aspx

I myself only tried it for a single user as a test, and it worked just fine.

Step one:

New-CsUserServicesPolicy -Identity “NoUCS” -UcsAllowed $False

Step two:

Grant-CsUserServicesPolicy -Identity “mytestuser” -PolicyName “NoUCS”

Step three:

Invoke-CsUcsRollback -Identity “mytestuser”

This removes UCS Address Book from the user and the contacts will be retrieved from the Lync Server instead. This method did INDEED WORK. I did NOT see the disabled users anymore in Lync’s Address Book.

I thought this was the perfect solution/method. However, it feels like going against Microsoft’s thoughts about a more unified Address Book on all MS platforms. I therefore buried this idea.

 

Method 4: ABSConfig

Even further googling led me to:

http://www.justin-morris.net/how-to-hide-users-from-the-lync-address-book/
https://social.technet.microsoft.com/Forums/lync/en-US/269159b0-3850-4bd6-83ed-1a00516417b8/using-the-abs-configuration-tool-to-limit-the-address-book-to-accounts-which-have-a-sip-address?forum=ocsaddressbook

which use yet another method to hide unwanted users from the Address Book, namely ABSConfig. I changed the configuration to this:

lync_server2013_abs_configuration_tool

…and was then greeted with:

lync_server2013_abs_configuration_tool_changes_applied

Ok, so I did the following:

  1. Restart RtcSrv = Restarted Lync Server Front-End service
  2. Update-CsUserDatabase (https://social.technet.microsoft.com/Forums/lync/en-US/5cb81b13-bed4-4068-af9b-d917a65094a8/lync-server-user-regeneration?forum=ocsaddressbook)
  3. Update-CsAddressBook

Was this finally the solution to all Address Book problems? Well yes, actually it was 🙂

To sum it up: In our case the problem was solved by using methods 1,2 and 4. Results may vary though – what worked for us might not work for you.

Alternative Witness Server for Exchange 2013 DAG

As stated in my previous blog posts, we’re using a two node DAG spanning across two datacenters/two AD Sites. The problem with this scenario is the witness server, or should I say the location of the witness server. I learned this the hard way, as we had minor problems with one of our datacenters. It also happened to be the datacenter where the witness server is located. This resulted in unresponsive/non-working email for some users, even though the HA aspect of Exchange via the Load Balancer was working fine.

I’ll borrow some pictures and text from http://searchexchange.techtarget.com/tip/How-Dynamic-Quorum-keeps-Exchange-2013-clusters-running-during-failure as I’m too lazy to rewrite everything. (We’re using dynamic quorum by default btw, as our servers are Windows Server 2012 R2).

“In a planned data center shutdown — where power to the data center and a host is often cut cleanly — we would have the opportunity to change the FSW to a host in the secondary data center. This allows for maintenance, but it does not help the inevitable event where an air conditioning unit overheats one weekend, servers begin to shut down, email stops working — and somebody has to get everything up and running again.

Dynamic Quorum with Windows Server 2012 and Exchange 2013 protects not only against this scenario above, but also against scenarios where the majority of nodes in a cluster fail. In another example, we see that in the primary site, we’ve lost both one Exchange node and the FSW (Fig 1). (This happened to us).

In our example, Dynamic Quorum can protect against a data center failure while the Exchange DAG remains online. This means that when the circumstances are right (we’ll come to that in a moment), a power failure in your primary data center can occur and Exchange can continue to stay up and running. This can even happen for smaller environments without the need to place the FSW in a third site.”

exchange_dag_witness1

Fig 1. Loss of the first node and FSW with Dynamic Quorum.

 

The key caveat is that the cluster must shut down cleanly. In the previous example, where the first data center failed, we relied on a mechanism to coordinate data center shutdown. This doesn’t need to be complicated, and a well-designed data center often will have this built in.

This can also protect against another scenario where there are three-node Exchange DAGs in a similar configuration — with two Exchange nodes present in the first data center and a single node present in a second data center. As the two nodes in the first data center shut down cleanly, Dynamic Quorum will ensure the remaining node keeps the DAG online.”

Some similar information can also be found at:

https://practical365.com/exchange-server/windows-server-2012-dynamic-quorum/
http://techgenix.com/exchange-2013-dag-dynamic-quorum-part1/ for example.

 

Well, this would all be too good to be true if it wasn’t for the “the cluster must shut down cleanly” -part. This got me thinking about alternatives. What about a third Exchange server and skipping the witness server altogether? Well, it doesn’t work any better as stated above. It’s the same dilemma if two of the nodes looses power. The solution as I can see it is (briefly) explained in the below article, DAC – Database Activation Coordination mode. This, together with an alternative witness server is the recipe for a better disaster plan. With DAC and an alternative witness server in place, you can force the exchange servers in a AD-Site to connect to the local witness server. It requires some manual work (in case disaster strikes) though, but it’s doable.

 

DAC

So, what’s up with the DAC mode and the alternative witness server? Lets have a look. First, let’s do some homework and have a look at DAC:

https://technet.microsoft.com/en-us/library/dd979790.aspx
https://practical365.com/exchange-server/exchange-best-practices-datacenter-activation-coordination-mode/
https://blogs.technet.microsoft.com/exchange/2011/05/31/exchange-2010-high-availability-misconceptions-addressed/

DAC mode is used to control the database mount on startup behavior of a DAG. This control is designed to prevent split brain from occurring at the database level during a datacenter switchback. Split brain, also known as split brain syndrome, is a condition that results in a database being mounted as an active copy on two members of the same DAG that are unable to communicate with one another. Split brain is prevented using DAC mode, because DAC mode requires DAG members to obtain permission to mount databases before they can be mounted”.

Source: https://technet.microsoft.com/en-us/library/dd979790.aspx

Datacenter Activation Coordination (DAC) mode has nothing whatsoever to do with failover. DAC mode is a property of the DAG that, when enabled, forces starting DAG members to acquire permission from other DAG members in order to mount mailbox databases. DAC mode was created to handle the following basic scenario:

  • You have a DAG extended to two datacenters.
  • You lose the power to your primary datacenter, which also takes out WAN connectivity between your primary and secondary datacenters.
  • Because primary datacenter power will be down for a while, you decide to activate your secondary datacenter and you perform a datacenter switchover.
  • Eventually, power is restored to your primary datacenter, but WAN connectivity between the two datacenters is not yet functional.
  • The DAG members starting up in the primary datacenter cannot communicate with any of the running DAG members in the secondary datacenter”.

Source: https://blogs.technet.microsoft.com/exchange/2011/05/31/exchange-2010-high-availability-misconceptions-addressed/

In short: Enable DAC mode on your Exchange servers if using more than two nodes.

 

Alternative witness server

Now that we have some basic understanding about DAC, let’s look at the Alternative witness server (AWS):

https://www.rutter-net.com/blog/news/alternate-file-share-witness-correcting-the-confusion
https://blogs.technet.microsoft.com/exchange/2011/05/31/exchange-2010-high-availability-misconceptions-addressed/

I think it’s quite well summarized in the first article:

The confusion lies in the event of datacenter activation; that the alternate file share witness would automatically come online as a means to provide quorum to the surviving DAG members and keep the databases mounted. So in many ways, some people view it as redundancy to the file share witness for an even numbered DAG.

In reality, the alternate file share witness is only invoked when an admin goes through procedures of activating the mailbox servers who lost quorum. DAC mode dramatically simplifies the process and when the “Restore-DatabaseAvailabilityGroup” cmdlet is executed during a datacenter activation, the alternate file share witness will be activated.”

The second article also has some nice overall information about High Availability Misconceptions. I suggest you read it.

In short: Manual labor is required even though you have configured an alternative witness server.

 

Datacenter switchover

So, what to do when disaster strikes? First, have a look at the TechNet article “Datacenter switchovers”:

https://technet.microsoft.com/en-us/library/dd351049.aspx

Then have a look at:

https://smtpport25.wordpress.com/2010/12/10/exchange-2010-dag-local-and-site-drfailover-and-fail-back/ for some serious deep diving into the subject. This has to be one of the most comprehensive articles about DAG/Failover/DAC/you name it on the Internet.

I’ll summarize the TechNet and the smtpport25 articles into actions:

From TechNet:

“There are four basic steps that you complete to perform a datacenter switchover, after making the initial decision to activate the second datacenter:

  1. Terminate a partially running datacenter   This step involves terminating Exchange services in the primary datacenter, if any services are still running. This is particularly important for the Mailbox server role because it uses an active/passive high availability model. If services in a partially failed datacenter aren’t stopped, it’s possible for problems from the partially failed datacenter to negatively affect the services during a switchover back to the primary datacenter”.

The sub-chapter Terminating a Partially Failed Datacenter has details on how to do this, and smtpport25 has even more information. If you start reading from “Figure 19” onwards in the smtpport25 article you’ll find this:

In figure 20. Marked in red has the details about started mailbox servers and Stopped Mailbox Servers. Started mailbox servers are the servers which are available for DAG for bringing the Database online. Stopped mailbox Servers are no longer participating in the DAG. There may be servers which are offline or down because of Datacenter failures. When we are restoring the service on secondary site, ideally all the servers which are in primary should be marked as stopped and they should not use when the services are brought online”.

So, in other words we should move the primary servers into Stopped State. To do that, use the PowerShell command:

Stop-DatabaseAvailabilityGroup -Identity DAG1 -Mailboxserver AMBX1 –Configurationonly

Stop-DatabaseAvailabilityGroup -Identity DAG1 -Mailboxserver AMBX2 –Configurationonly

Source: https://smtpport25.wordpress.com/2010/12/10/exchange-2010-dag-local-and-site-drfailover-and-fail-back/

Then, TechNet and smtpport25 have different information:

TechNet tells you to:

“2.The second datacenter must now be updated to represent which primary datacenter servers are stopped. This is done by running the same Stop-DatabaseAvailabilityGroup command with the ConfigurationOnly parameter using the same ActiveDirectorySite parameter and specifying the name of the Active Directory site in the failed primary datacenter. The purpose of this step is to inform the servers in the second datacenter about which mailbox servers are available to use when restoring service”.

The above should be enough if the DAG is in DAC mode (which it is).

Smtpport25 however doesn’t mention DAC mode at all in this case, instead they use the non-DAC mode approach from TechNet, with a little twist:

  • First, stop the cluster service on the secondary site/datacenter, Net stop Clussvc
  • Then, restore DAG on the secondary site, Restore-DatabaseAvailabilityGroup -Identity DAG01 -ActiveDirectorySite BSite

I honestly don’t know which of the solutions are correct, and I hope I won’t have to find out in our production environment anytime soon 🙂

 

Next step would be to Activate the Mailboxes Servers, again following different information whether the DAG is in DAC mode or not. I won’t paste all the text here as it is available in the TechNet article.

Then, following on to the chapter Activating Client Access Services:

  • Activate Client Access services   This involves using the URL mapping information and the Domain Name System (DNS) change methodology to perform all required DNS updates. The mapping information describes what DNS changes to perform. The amount of time required to complete the update depends on the methodology used and the Time to Live (TTL) settings on the DNS record (and whether the deployment’s infrastructure honors the TTL).

We do not need to perform this step as we’re using Zen Load Balancer 🙂

And lastly, I won’t copy/paste information regarding Restoring Service to the Primary Datacenter, it’s already nicely written in the TechNet or smtpport25 article. I sure do hope I won’t have to use the commands though 🙂

Update for Conferencing Server fails to install on Skype for Business Server 2015

I finally had some time to install and play around with Skype for Business Server 2015 and Exchange Server 2016 in my test environment. One of the features I was looking forward to in SfB Server 2015 was the ability to use offline messaging. I followed lots and lots of different articles, and this feature should be enabled out of the box from CU3(?) onwards. You can enable it manually on older CU’s also, following

https://technet.microsoft.com/en-us/library/mt756734.aspx
http://communicationsknowledge.blogspot.fi/2016/11/how-to-enable-offline-instant-messaging.html

for example. There are many other guides as well, but they’re all about the same. However, I learned the hard way that this feature is not yet enabled in the on-premise version of SfB Server 2015. Correct me if I’m wrong though. Some info regarding this (without an answer):

https://social.technet.microsoft.com/Forums/en-US/61648721-8693-4805-8fe3-82ca8a88aa34/when-will-be-be-offline-im-available-for-sfb-onpremise?forum=sfbfr

After some serious Googling I was still unable to enable offline messaging though. This basically means that all of my updating was for “nothing”. Well, not completely. At least I learned how to force-update SfB Server 2015 🙂

 

First I installed the latest CU for SfB Server 2015 from https://support.microsoft.com/en-us/help/3061064/updates-for-skype-for-business-server-2015 (in my case February 2017 Cumulative Server Update Installer) as this should’ve “fixed” lots of offline messaging stuff. Well, installing this CU wasn’t so straight forward after all. I followed the installation method to apply the update to a back end server or Standard Edition server. This is where the problems started.

The first problem was regarding the Url rewrite module 2.0 for IIS on Windows Server 2016. It wouldn’t install and you’ll receive an error while running the SfB Server 2015 Update Installer. Luckily I found a solution at http://stackoverflow.com/questions/35319942/url-rewrite-does-not-install-on-windows-10-iis-10. (You should edit the windows registry before running the Update Installer). Well, the problems didn’t stop here. After this problem was fixed I was greeted with:

sfb_server_update_installer_error

 

and after a while the Update installer told me that the Conferencing Server update failed:

sfb_server_update_installer_error2

(never mind the versions in the screenshot, they are wrong as I had to “borrow” this picture from the ucsorted.com link below).

 

If you read the log it’ll tell you “ERROR 1603: OCSMCU.msp had errors installing”. I did some googling and found these:

https://social.technet.microsoft.com/Forums/ie/en-US/42e284fb-ae07-424c-9ed3-07b6a85748da/skype-for-business-server-components-install-fails-when-patching-ocsmcumsi?forum=sfbfr
https://ucsorted.com/2016/11/19/error-returned-while-installing-ocsmcu-msi-code-1603/

The TechNet forum post suggest uninstalling some updates which I couldn’t even find on my system. This made the whole “fix” rather useless. The other link also talks about problematic updates, or the fact that you shouldn’t let SfB Server update itself at all before installing a new CU. This seemed very weird and the solution to reinstall everything from scratch also seemed very far fetched. There had to be a better solution.

 

Well, using my own common sense I started thinking about alternatives. After a while I figured why not uninstall the Conferencing Server component from the SfB Server 2015 installation and replace it with the stock one from the installation media? That said, I uninstalled the problematic component (even though SfB tells you it’s “dangerous”):

sfb_server_conferencing_server_uninstallation

After uninstallation of the Conferencing Server component, I copied the OCSMCU.msi file from the original install media (X:\Setup\amd64\Setup) and ran it. It installed without problems. It was now time to run the Skype for Business Server 2015 Update Installer again:

sfb_server_update_installer_success_cmd_prompt

Success 🙂

 

And just in case I ran the Skype for Business Server 2015 Update Installer once again to see the result:

sfb_server_update_installer_success_green

All dots are green, success indeed 🙂

 

I then continued following the installation instructions and ran:

Stop-CsWindowsService
net start w3svc

followed by:

Install-CsDatabase -Update –LocalDatabases:

sfb_server_install_csdatabase_after_update

sfb_server_install_csdatabase_after_update2

All good and everything seems to be working 🙂

I do wonder why there always seem to be problems with every update for Lync/SfB Server… oh well.

Lync Server 2013 Health Checking

I thought I’d write a tiny follow up to my previous blog post Health Checking / Monitoring Exchange Server 2013/2016. I got exited about all the health checking stuff, so naturally I had to do some research about Lync/SfB Server as well. That  said, I found a very comprehensive script written by Steve Moore. You can get the script from:

https://blogs.msdn.microsoft.com/canberrapfe/2014/12/11/lync-server-2013-health-check-script/

I’ve tested the script on both Lync Server 2013 and SfB Server 2015 with success. To put it briefly, it’s awesome! 🙂

As with the Exchange scripts, I wanted to use a scheduled task for running the script. I tried the same methods as with Exchange, but it failed. It turned out that this script wouldn’t accept the $true or $false parameters when running from a scheduled task. From the instructions:

Running the Script

To run the script you can use the following command line:

.\SystemStatus.ps1 -Action status -ServerFile c:\scripts\systemstatus\servers.txt -PoolName lync-pool.contoso.org -SendEmail $true -Environment Contoso -EventData $True

 

However, running the script from cmd (not PowerShell) or task scheduler you’ll get an error:

“Cannot convert value “System.String” to type “System.Boolean”, parameters of this type only accept booleans or numbers, use $true, $false, 1 or 0 instead”

My solution was to hard-code the parameters in the script itself, like so:

Lync-serverhealth-script-edit

There was no need to edit PoolName but I did it anyways – it’s one less parameter to define when running the script. Same goes for Environment. The interesting parameter however was [bool]$SendEmail=$False. I changed this to $True so I didn’t have to specify it as a parameter when running the script. As a bonus I also removed CC and BCC from

Send-MailMessage -To $Recipients -From $FromAddress -CC $CC -BCC $BCC -Subject $strMailSubject -BodyAsHtml:$True -SmtpServer $SMTPHost -Body $strBody #Send the email

as I have no need to send CC’s or BCC’s. In other words:

Send-MailMessage -To $Recipients -From $FromAddress -Subject $strMailSubject -BodyAsHtml:$True -SmtpServer $SMTPHost -Body $strBody #Send the email

 

With these changes in place, it was easy setting up Task Scheduler:

  • General: Run whether user is logged on or not. Run with highest privileges
  • Trigger: Weekly, at 00:00 every Wednesday of every week.
  • Action: Start a program
    • Program/script: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    • Add arguments (optional): -NoProfile -ExecutionPolicy Bypass -File “c:\software\scripts\SystemStatus.ps1” -Action status -ServerFile c:\software\scripts\servers.txt

servers.txt include the name of our front-end server.

And that’s it. You’ll then get a sexy report emailed back to you every Wednesday looking something like this:

Lync-serverhealth-report-screenshot

Just some non-critical warnings and nothing major to worry about. All good!

Happy monitoring to you all! 🙂

Health Checking / Monitoring Exchange Server 2013/2016

I‘ve never wrote about monitoring / health checking before so here we go. There are maaaaany different ways of monitoring servers, so I’ll just present my way of monitoring things (in the Exchange environment). If you’re using SCOM or Nagios, you’re already halfway there. The basic checks in SCOM or Nagios will warn you about low disk space and high CPU load and so forth. But what if a Exchange service or a DAG renders errors for example? Exchange is a massive beast to master, so in the end you’ll need decent monitoring tools to make your life easier (before disaster strikes).

We’re using Nagios for basic monitoring which is working great. That said, from time to time I’ve noticed some small problems that Nagios won’t report. I have then resorted to PowerShell commands or the windows event log. These problems would probably have been noticed (in time) if we had decent/additional Exchange-specific monitoring in place. There are Exchange-plugins available for Nagios (I’ve tried a few), but they aren’t as sophisticated as custom PowerShell scripts made by “Exchange experts”. It’s also much easier running a script from Task Scheduler than configuring Nagios. At least that’s my opinion.

Anyhow, our monitoring/health checking consist of three scripts, namely:

add-pssnapin *exchange* -erroraction SilentlyContinue
$body=Get-HealthReport -Server “yourserver” | where {$_.alertvalue -ne “Healthy” -and $_.AlertValue -ne “Disabled”} | Format-Table -Wrap -AutoSize; Send-MailMessage -To “me@ourdomain.com” -From “HealthSetReport@yourserver” -Subject “HealthSetReport, yourserver” -Body ($body | out-string ) -SmtpServer yoursmtpserver.domain.com

 

I have all of these scripts set up as scheduled tasks. You’d think that setting up a scheduled task is easy. Well, not in my opinion. I had to try many different techniques but at least it’s working now.

For Paul’s script I’m using the following settings:

ExServerHealth_schedtask_general

  • For “Triggers” I’m using 02:00 daily.
  • For “Actions” I’m using:
    • Start a program: powershell.exe
    • Add arguments (optional): -NoProfile -ExecutionPolicy Bypass -File “G:\software\scripts\Test-ExchangeServerHealth.ps1” –SendEmail
    • Start in (optional): G:\software\scripts

 

The same method wouldn’t work for Steve’s script though. I used the same “Run with highest privileges” setting, but running the PowerShell command similar to the above wouldn’t work. (This was easily tested running from a cmd promt manually (instead of powershell)). My solution:

  • Triggers: 01:00 every Saturday (yeah, I don’t feel the need to run this every night. Paul’s script will report the most important things anyways).
  • Actions:
    • Start a program: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    • Add arguments (optional): -NonInteractive -WindowStyle Hidden -command “. ‘C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1’; Connect-ExchangeServer -auto; G:\software\scripts\Get-ExchangeEnvironmentReport.ps1 -HTMLReport G:\software\scripts\environreport.html -SendMail:$true -MailFrom:environreport@yourserver -MailTo:me@ourdomain.com -MailServer:yoursmtpserver.domain.com
    • Start in (optional): (empty)

 

My own script:

  • Triggers: 00:00 every Sunday (yeah, I don’t feel the need to run this every night. Paul’s script will report the most important things anyways).
  • Actions:
    • Start a program: powershell.exe
    • Add arguments (optional): -NoProfile -ExecutionPolicy Bypass -File “G:\software\scripts\Get-HealthSetReport.ps1”
    • Start in (optional): G:\software\scripts

 

Sources:

http://practical365.com/exchange-server/powershell-script-exchange-server-health-check-report/ (Paul’s script)
http://www.stevieg.org/2011/06/exchange-environment-report/ (Steve’s script)

https://technet.microsoft.com/en-us/library/jj218724%28v=exchg.160%29.aspx
http://www.msexchange.org/kbase/ExchangeServerTips/ExchangeServer2013/Powershell/scheduling-exchange-powershell-task.html
http://blog.enowsoftware.com/solutions-engine/bid/186014/Introduction-to-Managed-Availability-How-to-Check-Recover-and-Maintain-Your-Exchange-Organization-Part-II

 

Exchange also checks its own health. Let me copy/paste some information:

One of the interesting features of Exchange Server 2013 is the way that Managed Availability communicates the health of individual Client Access protocols (eg OWA, ActiveSync, EWS) by rendering a healthcheck.htm file in each CAS virtual directory. When the protocol is healthy you can see it yourself by navigating to a URL such as https://mail.exchangeserverpro.net/owa/healthcheck.htm.
When the protocol is unhealthy the page is unavailable, and instead of the HTTP 200 result above you will see a “Page not found” or HTTP 404 result instead.

Source: http://practical365.com/exchange-server/testing-exchange-server-2013-client-access-server-health-with-powershell/

Further reading: https://blogs.technet.microsoft.com/exchange/2014/03/05/load-balancing-in-exchange-2013/ and the chapter about Health Probe Checking

We have no need to implement this at the “Exchange server-level” though, as these checks are already done in our Zen Load Balancer (described in this blog post). I guess you could call this the “fourth script” for checking/monitoring server health.

 

Reports

So, what all this adds up to are some nice reports. I’ll get a daily mail (generated at 02:00) looking like this (and hopefully always will 🙂 ):

ExServerHealth_screenshot_from_report

Daily report generated from Test-ExchangeServerHealth.ps1 script

 

ExEnvironmentreport_screenshot_from_report

Weekly report generated from Get-ExchangeEnvironmentReport.ps1 script

 

ExServerHealthSet_screenshot_from_report

Weekly report generated from Get-HealthSetReport.ps1 script (The problem was already taken care of 🙂 )

 

As you can see from the screenshots above, these checks are all reporting different aspects of the Exchange server health.

This quite much covers all the necessary monitoring in my opinion and with these checks in place you can probably also sleep better during the nights 🙂 Even though these checks are comprehensive, I’m still planning on even more checks. My next step will be an attempt at Real-time event log monitoring using NSClient++ / Nagios. Actually the Nagios thought was buried and I will instead focus on the ELK stack which is a VERY comprehensive logging solution.