Disabled (AD) users still searchable in Lync/SfB

Well, this was a bit of a nightmare to debug. I started the whole project with some googling on how to (automatically) remove AD-disabled users from Lync (SfB) Server. This procedure is by no means automatic, and If you disable a user in AD they can still use Lync (at least for some time). Have a look at https://ucgeek.co/2014/04/lync-users-can-login-after-domain-account-disabled/ for some deeper details.

My concern wasn’t about the fact that the users still could use Lync however, it was more about the fact that they are “hanging around” like ghosts on the Lync Server for no good reason. In plain English – users should be disabled if their AD account is also disabled.

I’ll now present MANY different ways to get rid of the unwanted users in the Lync/SfB Address Book. One method might work for someone, and another method might work for someone else. Hell, you might even need to use all the methods combined 🙂

 

Method 1: Disable AD-disabled users on Lync/SfB Server

This idea led me to some PowerShell scripts. Here are a couple of examples that will disable users on the Lync Server if they are disabled in Active Directory:

http://www.markc.me.uk/blog/files/ADDisabled.html
https://www.ucunleashed.com/265
https://trogjels.wordpress.com/2012/08/16/lync-disable-ad-disabled-users/

This worked just fine and I got all the accounts removed from the Lync Server. A little addition to the above links is that I now use this script from a Scheduled Task:

disableADdisabledaccount_task_scheduler

 

The PowerShell “script” itself looks like this:

# disable_ADdisabled.ps1 , source http://www.markc.me.uk/blog/files/ADDisabled.html
# J.S. 13.9.2017
Import-Module ‘C:\Program Files\Common Files\Microsoft Lync Server 2013\Modules\Lync\Lync.psd1’
Get-CsAdUser -ResultSize Unlimited | Where-Object {$_.UserAccountControl -match “AccountDisabled” -and $_.Enabled} | Disable-CsUser

Well, still no luck. You think your users wouldn’t show up when searching in Lync (Address Book) if they are no longer activated on Lync Server (and also has the –HiddenFromAddressListsEnabled flag, see next chapter). Well, they do. At least for us. This was getting pretty weird. I tried googling but all I could come up with was “Update-CsAddressBook” and so forth. Examples:

http://lyncinsider.com/lync-server-2010/updating-lync-contacts-sync-error-fixes-part-2-of-3/
http://www.fots.nl/lync-address-book-sync-process-information/
http://blog.schertz.name/2010/09/updating-the-lync-2010-address-book/
https://social.technet.microsoft.com/Forums/sharepoint/en-US/2e0d1200-b924-42dd-8700-55893b8cfc03/deleted-lync-user-still-in-list-showing-presence-unknown?forum=officeitpro
https://technet.microsoft.com/en-us/library/gg429695(v=ocs.15).aspx

No. Still no luck. I could see the “old” users which aren’t enabled in AD nor on the Lync Server. This was getting frustrated. Better turn to my old friend Event Viewer. Yes, some hits indeed:

Event ID 21054, Error:
Users are not indexed in the database that should be.

Expected indexed user count: 0
Actual indexed user count: 1999
Cause: User replication issue.
Resolution:
Run Update-CsAddressBook to synchronize all accounts.

Well, what command did I just run? Did it help? No. Then again if you Google the message you’ll soon notice that this error is nothing to worry about:

“As long as “Objects not indexed that should be” and “Abandoned Objects” are zero, the database is okay.” I did run the Debug-csAddressBookReplication and “my” objects were zero. Fine.

Source: https://social.technet.microsoft.com/Forums/lync/en-US/49de4752-a73e-4a55-98d6-a48409a02d94/alert-lync-users-are-not-indexed-in-the-database-that-should-be?forum=lyncdeploy (you can find this information in many other sources as well…)

 

Method 2: HiddenFromAddressListsEnabled PowerShell command

There are many different flavors of this command, but the one we’re using is something like the one found at https://stackoverflow.com/questions/7974204/powershell-command-to-hide-user-from-exchange-address-lists (and the answer “I use this as a daily scheduled task to hide users disabled in AD from the Global Address List”). We also use it in a Scheduled Task.

This HiddenFromAddressListsEnabled –method was applied at an earlier stage, as we had “problems” with Outlook / Exchange showing disabled users in Global Address List (GAL). The above PowerShell-trick solved the problem on the Outlook/Exchange-side. It SHOULD however also work with Lync/SfB if you are using the Unified Contact Store, UCS. More information about UCS in the next chapter. However, it didn’t work for us. We still saw disabled users in Lync’s Address Book 😦

 

Method 3: Rollback from UCS

What now? I’m ok with just ignoring the event in Event Viewer, but I still have the same problem. My disabled users are visible in Lync’s Address Book. This got me thinking about the Exchange <-> Lync relationship. I know they are a tight fit. Like all MS products. The https://social.technet.microsoft.com/Forums/sharepoint/en-US/2e0d1200-b924-42dd-8700-55893b8cfc03/deleted-lync-user-still-in-list-showing-presence-unknown?forum=officeitpro post had some hints at the end also. Many, MANY Google searches later got me to https://support.office.com/en-us/article/Set-Personal-options-c09b21ac-7334-49cf-a510-d8c432fcaf01

And there it was. The answer I was (partly) looking for:

You specify how Skype for Business interacts with other Office programs.

  1. In Personal options, under Personal information manager, select either Microsoft Exchange or Microsoft Outlook or None from the drop-down list. When you select Microsoft Exchange or Microsoft Outlook, the Skype for Business search feature uses the Microsoft Outlook Contact list as a source of contacts, in addition to the global address list.

    When you select None, the Skype for Business search feature returns contacts only from the global address list. It doesn’t use either the Windows Address Book or the Outlook Contact list.

Source: https://support.office.com/en-us/article/Set-Personal-options-c09b21ac-7334-49cf-a510-d8c432fcaf01

So, was my solution setting this setting to “None”? You’d think so. Think again. This setting is different in the SfB client. You CAN’T set it to none. Have a look at https://answers.microsoft.com/en-us/msoffice/forum/msoffice_sfb-mso_winother/skype-for-business-2016-personal-information/67d14b5e-23d7-4d23-ab6e-5710739cb0d3 for example.

Well, as my integration is controlled “by my administrator” (which would be myself thank you very much), it got me thinking that it HAS to be a server setting. Now the question was, WHICH setting specifically? Now I honestly don’t remember what I googled to find this, but in the end it got me to http://techgenix.com/integrating-exchange-server-2013-and-skype-business-server-2015-part6/

Well well well. This was all new to me. I ran the command Get-CSUserServicePolicy and found out that UcsAllowed was set to true. It was indeed enabled for ALL our users. I also checked from the SfB client itself, and yes, (not surprisingly) same result:

UCS_screenshot_SfB_configuration_information

 

So, what is UCS?

Detailed Description

The unified contact store introduced in Lync Server 2013 gives administrators the option of storing a user’s contacts in Exchange instead of in Skype for Business Server 2015; in turn that allows the user to access the same set of contacts in Outlook and Outlook Web Access as well as. (Alternatively, you can continue to store contacts in Skype for Business Server 2015. In that case, users will have to maintain two separate sets of contacts: one for use with Outlook and Outlook Web Access, and one for use with Skype for Business.)

Source: https://technet.microsoft.com/en-us/library/jj205414.aspx

Now the big problem here is that our user accounts used in different systems aren’t disabled at the same time. Users aren’t automatically disabled in Exchange if they get disabled in AD (at least not yet, but this will probably change when our imap-server retires). See the pattern here? What this means is that if a user is still “active” (from Exchange’s point of view), it ALSO lists this person in LYNC’s Address Book. This also means that I’ve found the root problem for our “ghost users”.

The solution is NOT using UCS, and rollback to the system where the contacts are stored on the Lync Server instead of Exchange. Well yeah, you COULD disable UCS, but it feels like going the wrong way. If you still want to test however, the information for this procedure is available here:

https://technet.microsoft.com/en-us/library/jj205224(v=ocs.15).aspx (To rollback unified contact store contacts from Exchange 2013 to Lync Server 2013).
https://technet.microsoft.com/en-us/library/jj688083.aspx

I myself only tried it for a single user as a test, and it worked just fine.

Step one:

New-CsUserServicesPolicy -Identity “NoUCS” -UcsAllowed $False

Step two:

Grant-CsUserServicesPolicy -Identity “mytestuser” -PolicyName “NoUCS”

Step three:

Invoke-CsUcsRollback -Identity “mytestuser”

This removes UCS Address Book from the user and the contacts will be retrieved from the Lync Server instead. This method did INDEED WORK. I did NOT see the disabled users anymore in Lync’s Address Book.

I thought this was the perfect solution/method. However, it feels like going against Microsoft’s thoughts about a more unified Address Book on all MS platforms. I therefore buried this idea.

 

Method 4: ABSConfig

Even further googling led me to:

http://www.justin-morris.net/how-to-hide-users-from-the-lync-address-book/
https://social.technet.microsoft.com/Forums/lync/en-US/269159b0-3850-4bd6-83ed-1a00516417b8/using-the-abs-configuration-tool-to-limit-the-address-book-to-accounts-which-have-a-sip-address?forum=ocsaddressbook

which use yet another method to hide unwanted users from the Address Book, namely ABSConfig. I changed the configuration to this:

lync_server2013_abs_configuration_tool

…and was then greeted with:

lync_server2013_abs_configuration_tool_changes_applied

Ok, so I did the following:

  1. Restart RtcSrv = Restarted Lync Server Front-End service
  2. Update-CsUserDatabase (https://social.technet.microsoft.com/Forums/lync/en-US/5cb81b13-bed4-4068-af9b-d917a65094a8/lync-server-user-regeneration?forum=ocsaddressbook)
  3. Update-CsAddressBook

Was this finally the solution to all Address Book problems? Well yes, actually it was 🙂

To sum it up: In our case the problem was solved by using methods 1,2 and 4. Results may vary though – what worked for us might not work for you.

Advertisements

Update for Conferencing Server fails to install on Skype for Business Server 2015

I finally had some time to install and play around with Skype for Business Server 2015 and Exchange Server 2016 in my test environment. One of the features I was looking forward to in SfB Server 2015 was the ability to use offline messaging. I followed lots and lots of different articles, and this feature should be enabled out of the box from CU3(?) onwards. You can enable it manually on older CU’s also, following

https://technet.microsoft.com/en-us/library/mt756734.aspx
http://communicationsknowledge.blogspot.fi/2016/11/how-to-enable-offline-instant-messaging.html

for example. There are many other guides as well, but they’re all about the same. However, I learned the hard way that this feature is not yet enabled in the on-premise version of SfB Server 2015. Correct me if I’m wrong though. Some info regarding this (without an answer):

https://social.technet.microsoft.com/Forums/en-US/61648721-8693-4805-8fe3-82ca8a88aa34/when-will-be-be-offline-im-available-for-sfb-onpremise?forum=sfbfr

After some serious Googling I was still unable to enable offline messaging though. This basically means that all of my updating was for “nothing”. Well, not completely. At least I learned how to force-update SfB Server 2015 🙂

 

First I installed the latest CU for SfB Server 2015 from https://support.microsoft.com/en-us/help/3061064/updates-for-skype-for-business-server-2015 (in my case February 2017 Cumulative Server Update Installer) as this should’ve “fixed” lots of offline messaging stuff. Well, installing this CU wasn’t so straight forward after all. I followed the installation method to apply the update to a back end server or Standard Edition server. This is where the problems started.

The first problem was regarding the Url rewrite module 2.0 for IIS on Windows Server 2016. It wouldn’t install and you’ll receive an error while running the SfB Server 2015 Update Installer. Luckily I found a solution at http://stackoverflow.com/questions/35319942/url-rewrite-does-not-install-on-windows-10-iis-10. (You should edit the windows registry before running the Update Installer). Well, the problems didn’t stop here. After this problem was fixed I was greeted with:

sfb_server_update_installer_error

 

and after a while the Update installer told me that the Conferencing Server update failed:

sfb_server_update_installer_error2

(never mind the versions in the screenshot, they are wrong as I had to “borrow” this picture from the ucsorted.com link below).

 

If you read the log it’ll tell you “ERROR 1603: OCSMCU.msp had errors installing”. I did some googling and found these:

https://social.technet.microsoft.com/Forums/ie/en-US/42e284fb-ae07-424c-9ed3-07b6a85748da/skype-for-business-server-components-install-fails-when-patching-ocsmcumsi?forum=sfbfr
https://ucsorted.com/2016/11/19/error-returned-while-installing-ocsmcu-msi-code-1603/

The TechNet forum post suggest uninstalling some updates which I couldn’t even find on my system. This made the whole “fix” rather useless. The other link also talks about problematic updates, or the fact that you shouldn’t let SfB Server update itself at all before installing a new CU. This seemed very weird and the solution to reinstall everything from scratch also seemed very far fetched. There had to be a better solution.

 

Well, using my own common sense I started thinking about alternatives. After a while I figured why not uninstall the Conferencing Server component from the SfB Server 2015 installation and replace it with the stock one from the installation media? That said, I uninstalled the problematic component (even though SfB tells you it’s “dangerous”):

sfb_server_conferencing_server_uninstallation

After uninstallation of the Conferencing Server component, I copied the OCSMCU.msi file from the original install media (X:\Setup\amd64\Setup) and ran it. It installed without problems. It was now time to run the Skype for Business Server 2015 Update Installer again:

sfb_server_update_installer_success_cmd_prompt

Success 🙂

 

And just in case I ran the Skype for Business Server 2015 Update Installer once again to see the result:

sfb_server_update_installer_success_green

All dots are green, success indeed 🙂

 

I then continued following the installation instructions and ran:

Stop-CsWindowsService
net start w3svc

followed by:

Install-CsDatabase -Update –LocalDatabases:

sfb_server_install_csdatabase_after_update

sfb_server_install_csdatabase_after_update2

All good and everything seems to be working 🙂

I do wonder why there always seem to be problems with every update for Lync/SfB Server… oh well.

Lync Server 2013 Health Checking

I thought I’d write a tiny follow up to my previous blog post Health Checking / Monitoring Exchange Server 2013/2016. I got exited about all the health checking stuff, so naturally I had to do some research about Lync/SfB Server as well. That  said, I found a very comprehensive script written by Steve Moore. You can get the script from:

https://blogs.msdn.microsoft.com/canberrapfe/2014/12/11/lync-server-2013-health-check-script/

I’ve tested the script on both Lync Server 2013 and SfB Server 2015 with success. To put it briefly, it’s awesome! 🙂

As with the Exchange scripts, I wanted to use a scheduled task for running the script. I tried the same methods as with Exchange, but it failed. It turned out that this script wouldn’t accept the $true or $false parameters when running from a scheduled task. From the instructions:

Running the Script

To run the script you can use the following command line:

.\SystemStatus.ps1 -Action status -ServerFile c:\scripts\systemstatus\servers.txt -PoolName lync-pool.contoso.org -SendEmail $true -Environment Contoso -EventData $True

 

However, running the script from cmd (not PowerShell) or task scheduler you’ll get an error:

“Cannot convert value “System.String” to type “System.Boolean”, parameters of this type only accept booleans or numbers, use $true, $false, 1 or 0 instead”

My solution was to hard-code the parameters in the script itself, like so:

Lync-serverhealth-script-edit

There was no need to edit PoolName but I did it anyways – it’s one less parameter to define when running the script. Same goes for Environment. The interesting parameter however was [bool]$SendEmail=$False. I changed this to $True so I didn’t have to specify it as a parameter when running the script. As a bonus I also removed CC and BCC from

Send-MailMessage -To $Recipients -From $FromAddress -CC $CC -BCC $BCC -Subject $strMailSubject -BodyAsHtml:$True -SmtpServer $SMTPHost -Body $strBody #Send the email

as I have no need to send CC’s or BCC’s. In other words:

Send-MailMessage -To $Recipients -From $FromAddress -Subject $strMailSubject -BodyAsHtml:$True -SmtpServer $SMTPHost -Body $strBody #Send the email

 

With these changes in place, it was easy setting up Task Scheduler:

  • General: Run whether user is logged on or not. Run with highest privileges
  • Trigger: Weekly, at 00:00 every Wednesday of every week.
  • Action: Start a program
    • Program/script: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    • Add arguments (optional): -NoProfile -ExecutionPolicy Bypass -File “c:\software\scripts\SystemStatus.ps1” -Action status -ServerFile c:\software\scripts\servers.txt

servers.txt include the name of our front-end server.

And that’s it. You’ll then get a sexy report emailed back to you every Wednesday looking something like this:

Lync-serverhealth-report-screenshot

Just some non-critical warnings and nothing major to worry about. All good!

Happy monitoring to you all! 🙂

Adding Edge and Reverse Proxy Servers to an Existing Lync 2013 Environment

My recent task was to expand our existing Lync environment (Lync Server 2013 Standard) with an Edge and a Reverse proxy server. (This guide probably works for Skype for Business as well). Our old Lync environment had been in test usage for a while (with a rather small test-user group), but with more and more Lync users adding up it was time to expand. As a matter of fact, a Lync environment without Edge and reverse proxy servers is rather useless – you are unable to organize external meetings.

First, let me start off by saying that there are A LOT of “moving parts” involved in configuring a reverse proxy and an Edge server. You must plan for IP addresses, DMZ settings, DNS settings, certificates, firewall settings and so forth. To get a grasp of the whole picture I’m suggesting that you read/watch the following:

https://www.youtube.com/watch?v=2TdoVb1Grak
https://technet.microsoft.com/en-us/library/dn594589.aspx – A very good protocol poster (Skype for Business Server 2015 Protocol Workloads) that helps with the overall picture. It’s also very good for checking firewall requirements/port configurations. Yes, it’s a bit overwhelming but very good in the end 🙂
https://technet.microsoft.com/en-us/library/jj204756%28v=ocs.15%29.aspx – Explains (firewall) ports.

A good place to continue after this would be http://blog.schertz.name/2012/07/lync-edge-server-best-practices/. It’s a very nice all-around document about Edge and Reverse proxy. Pay close attention to the chapter about Best Practices. Our goal in the end was to get something that resembles this picture:

lync-schertz-best-practise-pic-single-topology

Fig 1. Lync Front-end in combination with Edge and Reverse proxy – Simple Topology. (Pic source: http://blog.schertz.name/2012/07/lync-edge-server-best-practices/)

 

Even after a lot of reading, It’s hard to know where to start (in this blog post). There are soooooo many different things going on and a lot of stuff to remember. A good place to start could be certificate planning, which also means that you have to decide which IPs/hostnames you’ll be using in your own environment. Then again, I think the best place to start is planning network infrastructure/topology. First, consider whether you are going with a simple topology (Front-end, Edge server, Reverse proxy – we’re using this) OR a complex topology (multiple Front-ends, Edge servers, Reverse proxies). Second, It’s very important to have a working DMZ, and you should also know if you’ll be using (only) Public IP addresses or public IPs in combination with NATed ones. After you’ve got an answer to these questions it will be much easier planning for the other requirements. With this in mind, I’ll start off with the networking part. I’ll then move over to areas like DNS, certificates, actual Edge server installation, IIS ARR installation and finally some words about mobility and federation.

But first off, here’s a short explanation of what the Edge and Reverse proxy servers bring to the table:

Edge

Includes 4 modules:

  • Access Edge service. The Access Edge service provides a single, trusted connection point for both outbound and inbound Session Initiation Protocol (SIP) traffic.
  • Web Conferencing Edge service. The Web Conferencing Edge service enables external users to join meetings that are hosted on your internal Lync Server 2013 deployment.
  • A/V Edge service. The A/V Edge service makes audio, video, application sharing, and file transfer available to external users. Your users can add audio and video to meetings that include external participants, and they can communicate using audio and/or video directly with an external user in point-to-point sessions. The A/V Edge service also provides support for desktop sharing and file transfer.
  • XMPP Proxy service. The XMPP Proxy service accepts and sends extensible messaging and presence protocol (XMPP) messages to and from configured XMPP Federated partners.

 

Reverse proxy

The reverse proxy is required for the following:

  • To allow users to connect to meetings or dial-in conferences using simple URLs
  • To enable external users to download meeting content
  • To enable external users to expand distribution groups
  • To allow the user to obtain a user-based certificate for client certificate based authentication
  • To enable remote users to download files from the Address Book Server or to submit queries to the Address Book Web Query service
  • To enable remote users to obtain updates to client and device software
  • To enable mobile devices to automatically discover Front End Servers offering mobility services
  • To enable push notifications to mobile devices from the Office 365 or Apple push notification services

Source: https://technet.microsoft.com/en-us/library/gg425779%28v=ocs.15%29.aspx.

 

 

Networking / Network interfaces

I’m now assuming that you have:

  • A working Lync Server Standard/Enterprise 2013 (or Skype for Business) Front-end
  • A soon-to-become (Lync) Reverse Proxy server (Windows Server 2012 R2)
  • A soon-to-become Lync Edge server (Windows Server 2012 R2)
  • Talked to your network guys about the network infrastructure (IPs/DMZ). Hardware (F5) load balancers can be a whole different story for example.
  • Talked to your firewall guys about opening ports. I myself sat down with the Skype for Business Server 2015 Protocol Workloads printout and had a long discussion with a firewall guy. We/he got the job done without any hiccups. (It’s still working fine today 🙂 )

 

On the Reverse Proxy:

Assign one IP for the internal network adapter and one for the external network adapter. Internal and External should be in different subnets. One interface is communicating with the internet and the other one is communicating with your internal network/AD. Have a look at https://technet.microsoft.com/en-us/library/gg429707%28v=ocs.15%29.aspx or https://technet.microsoft.com/en-us/library/jj204691%28v=ocs.15%29.aspx for examples. (I’m not going into much DNS details (yet), but you could name these new IPs lyncproxy-int.domain.com and lyncproxy-ext.domain.com for example).

  • Set the default gateway on the external network adapter only
  • Assign static routes. From my experience, the information regarding this can be a bit difficult to understand. Let me copy/paste the information from the above link:

Important: Similar to the Edge Servers, you set the default gateway on the external network adapter only. The default gateway will be the IP address of the router or external facing firewall that directs traffic to the Internet. For traffic that is destined from the reverse proxy to the internal facing network adaptor, you must use persistent static routes (such as the route command in Windows Server) for all subnets containing servers referenced by the web publishing rules. Setting a persistent route does not cause the computer to become a router. If IP forwarding is not enabled, the computer is acting only to direct specific traffic destined for another network to the appropriate interface. This is essentially setting two gateways – one as the default pointing to the external networks, and one for traffic destined to the internal interface and on to a router or other network.
However, creating persistent routes for all subnets may not be necessary if your network’s routers are configured to summarize routes. Create a persistent route to the network where the router is defined and use the router as the default gateway. If you are not sure how your network is configured and need guidance on what persistent routes need to be created, consult with your company’s Network Engineers.
The reverse proxy must be able to resolve the DNS host (A) records for the internal Director or Front End Server and next hop pool FQDNs used in the web publishing rules. As with the Edge Servers, for security reasons, we recommend that you do not configure a reverse proxy to use a DNS server located in the internal network. This means you either need DNS servers in the perimeter, or you need HOSTS file entries on the reverse proxy that resolves each of these FQDNs to the internal IP address of the servers”.

In plain English this means that you configure the external interface “normally”, as you would with any other external network interface in your infrastructure. You should define the gateway as the “IP address of the router or external facing firewall that directs traffic to the Internet”. Your network guys can help you with this if unsure (also see the next chapter). The internal side on the other hand should not have a default gateway – instead you configure static routes. I’ll try to explain this:

Example network subnets (defined by your network administrators):

External DMZ (16 addresses, all are not needed but room for expansion)

Range:  123.123.123.16/28
Mask:    255.255.255.240
GW:       123.123.123.30

Internal DMZ (16 addresses, all are not needed but room for expansion)

Range:   123.123.123.0/28
Mask:     255.255.255.240
GW:        123.123.123.14

 

Server configuration:

Example external network adapter configuration on the server:

IP:         123.123.123.20
Mask:   255.255.255.240
GW:      123.123.123.30

Example Internal network adapter configuration on the server:

IP:         123.123.123.3
Mask:   255.255.255.240
GW:      no gw

We’re using split-brain DNS so the internal and external DNS names are the same. All IP’s are from a Class B chunk, and they’re all public IP’s that are defined as internal or external in the firewall/DNS. With the above configuration in place, you should now add a route to the internal interface on the reverse proxy server. This is done with the route add command (The –p switch make the changes persistent). Here’s an example using the above IP schema:

C:\>route add -p 123.123.17.0 mask 255.255.240.0 123.123.123.14
C:\>route add -p 123.123.90.0 mask 255.255.240.0 123.123.123.14
C:\>route add -p 123.123.154.0 mask 255.255.240.0 123.123.123.14
C:\>route add -p 123.123.207.0 mask 255.255.240.0 123.123.123.14
C:\>route add -p 123.123.106.0 mask 255.255.224.0 123.123.123.14

The above command example would make all of the above IP ranges take the route against the internal interface. All other IPs would take the external route. The above ranges are also defined as internal in DNS/firewall. Do the same for all of your internal IP ranges. This method is different when using NAT and/or non-split-brain configurations. (In case of NAT, your internal IPs are in the 192.168.x.x, 10.x.x.x, or 172.16.x.x. range). Perhaps a picture will tell more than words:

lync-proxy-int-ext-subnets

Fig 2. Internal and external overview. (Picture source: https://technet.microsoft.com/en-us/library/jj204756%28v=ocs.15%29.aspx)

This should be it for the networking part on the Reverse proxy server. Now we do the same on the Edge server.

 

On the Edge server:

The network configuration on the Edge server follow the same pattern as the Reverse proxy. I’m using three external IPs and one internal IP. This is by best practice design (https://technet.microsoft.com/en-us/library/gg412847%28v=ocs.15%29.aspx). If you are in a limited-budget-external-IP-dilemma, you can also make it work with one external IP (not including that option in this text however).

  • Assign three external IPs
    • one for SIP traffic
    • one for AV traffic
    • one for Web Conferencing
  • Assign one internal IP

I’m not going much into DNS details now either, but you could name these new IPs lyncedge-int.domain.com, lyncedgesip-ext.domain.com, lyncedgeav-ext.domain.com and lyncedgewc-ext.domain.com for example. There’s nothing much to add here. Follow the same procedure as for the reverse proxy when configuring your internal and external network interfaces:

Server configuration:

Example Internal network adapter configuration on the server:

IP:        123.123.123.4
Mask:  255.255.255.240
GW:      no gw

Example external network adapter configuration on the server:

IP1:       123.123.123.21
Mask:   255.255.255.240
GW:      123.123.123.30

IP2:       123.123.123.22
Mask:   255.255.255.240
GW:      123.123.123.30

IP3:       123.123.123.23
Mask:   255.255.255.240
GW:      123.123.123.30

 

Now add the same routes as you did on the reverse proxy. There you have it, we can now move over to the DNS part.

 

 

DNS

I assume that you by now have figured out your topology and configured networking on the involved servers. Good, that’s one step in the right direction. You might have noticed that I haven’t talked much about host names, only IP addresses. This is mostly because you can configure the networking part this far without knowing (almost) any host names. (Of course you most certainly will ask for a hostname at the same time you get an IP address, but anyways).

I have to say that DNS was one of the most confusing/difficult/challenging/painful parts in this whole configuration/deployment. There were tons and tons of misleading/wrong information, and it required countless hours of testing. Anyways, I’ll spare you the DNS-pain and tell you about our configuration in a while. But before I do, I make you read some homework. Here are a couple of interesting links (with or without errors):

https://technet.microsoft.com/en-us/library/dn951397.aspx
https://technet.microsoft.com/en-us/library/gg398758%28v=ocs.15%29.aspx
http://jackstromberg.com/2013/06/lync-2013-dns-settings/
https://technet.microsoft.com/en-us/library/gg425874%28v=ocs.15%29.aspx
https://technet.microsoft.com/en-us/library/gg398287%28v=ocs.15%29.aspx
http://msunified.net/2013/08/07/lync-client-sign-in-and-dns-records-recommendations/

Let me start off by saying that I like the jackstromberg article. All my testing was actually based on the DNS table from that article. However those records were also a bit confusing, and some even unnecessary. Here are our DNS records with comments:

 

Internal DNS:

lync-internalDNS

Fig 3. Internal DNS

No other records are required for our specific configuration/environment (at the moment). SRV records are a thing of the past and only needed when working with Lync 2010 clients. See http://blog.schertz.name/2012/12/lync-2013-client-autodiscover/ for more information. If you are going to use federation however (which we probably are in the future), you SHOULD set up SRV records (though not needed if manually entering servers). See: http://windowsitpro.com/skype-business/understanding-main-types-federation-skype-and-lync-environments. As you can see, Allowed Partner Server (Direct Federation) works without SRV records but specifying the records when you federate will probably still make your life easier.

I will now also make a statement about the lyncdiscover.mydomain.com record in the Internal DNS. You can read on many, MANY places on the Internet that you should have this record present in the internal DNS so that mobility works. I can confirm that our users mobile phones (WP, iOS, Android) work just FINE without this record. The key is to have the external Web Services record present in the internal DNS (webext.mydomain.com) and point it to the reverse proxy. If you DO use lyncdiscover in the internal DNS, ALL traffic will go through the proxy. This is probably not a desirable configuration. Yes, I’ve seen this “live” in our environment so I know what I’m talking about. The “problem” went away after we removed the lyncdiscover.mydomain.com record from the internal DNS. Good info about this:

http://blog.schertz.name/2013/07/understanding-lync-2013-mobility/
http://www.ucprimer.com/tech-blog/5-essential-tips-for-deploying-lync2013-mobility-service
https://technet.microsoft.com/en-us/library/hh690030%28v=ocs.15%29.aspx

Read the above links CAREFULLY and you’ll have a MUCH better understanding, believe me 🙂 Again, this setup works FOR US. I’m not saying that the lyncdiscover.mydomain.com record should be removed from every internal DNS configuration out there.

 

External DNS:

lync-externalDNS

Fig 4. External DNS.

External DNS was much more straight forward. Comments are included in the picture.

The DNS records are (as you can see) a bit different for the external network/outside world compared to the internal network. All external traffic goes through the reverse proxy, which in turn use URL rewrites to connect to the corresponding URLs on the inside network. (I’ll leave the URL rewrite / ISS/ARR discussion for a later chapter).

 

Hosts file:

You have probably noticed that the webext.mydomain.com record is present in BOTH the internal and external DNS. The reason for this is mobile devices. Mobile devices need to access the mobility service, and they do that ONLY from the outside. I’m yet again referring to the Mobility service flow using AutoDiscover (picture) at https://technet.microsoft.com/en-us/library/hh690030%28v=ocs.15%29.aspx. Lyncs own autodiscover feature will know if the client is on the internal or external network based on the lyncdiscover/lyncdiscoverinternal record. However, it’s a whole different story with webext.mydomain.com. If this URL is accessed from either the inside or outside network, the client is unable to know it’s final destination. This is because you’re pointing the client (in both cases) to the reverse proxy, which in turn point to the same URL internally and externally. This means that you’ll end up in an endless loop. To solve this you’ll edit the hosts-file.

You also have to add a local DNS record for lyncdiscover, otherwise this record will remain unresolvable as it’s not present in the internal DNS. This was all a big mystery for me, as the documentation seldom mentioned this dilemma. I got an idea after hours of googling though – the holy hosts-file. Thanks to http://kressmark.blogspot.fi/2015/06/windows-2012-r2-iis-85-and-arr-30-as.html for the idea. This was by no means a big surprise, but you’ll get lost (in DNS) after hours and hours of testing. Believe me.

This means that you’ll have to add local DNS records on the reverse proxy. Fire up notepad and edit the C:\Windows\System32\drivers\etc\hosts –file on the reverse proxy. Add the following:

123.123.123.10    webext.mydomain.com, where 123.123.123.10 is the IP of your Front-end server.
123.123.123.10    lyncdiscover.mydomain.com, where 123.123.123.10 is the IP of your Front-end server.

(Meet and dialin are already resolvable by internal DNS and correctly points to the Front-end).

Now when a client resolves webext (internally or externally), it always gets sent to the reverse proxy. The reverse proxy in turn resolves webext to the front-end via the hosts-file. Lyncdiscover in turn won’t be resolvable internally after it reaches the reverse proxy if no hosts file-record is added. There you have it – all your DNS problems solved 🙂

 

 

Certificates

Certificates and DNS go somewhat hand-in-hand as you need to know which hostnames you’ll be using in the certificates. I’d probably start off by reading again. Some information to get you started:

https://technet.microsoft.com/en-us/library/jj204747%28v=ocs.15%29.aspx
https://technet.microsoft.com/en-us/library/jj205381%28v=ocs.15%29.aspx
http://skype4b.uk/2015/08/29/skype-for-business-certificate-requirements-the-definitive-guide/
http://social.technet.microsoft.com/wiki/contents/articles/24210.demystify-lync-2013-server-internal-certificate-requirements.aspx

Take your time to read and plan – this way you’ll be rewarded in the end. The above links discuss both internal and external certificates. They also discuss the differences between the reverse proxy and the Lync Edge certificates.

We’re not completely going by best practice regarding the certificates. We’re using external certificates on the internal front-end server. This is due to the fact that we already had an external certificate installed on the Front-end. It doesn’t do much harm either, and at least for us it’s not an extra expense.

Without further stories I’ll present our “certificate solution”. We’re using one certificate per interface on the edge server, but you could also use just one certificate will all hosts included. This would be more expensive due to the fact that you’ll have to pay extra for additional SAN-names. (You DON’T need a certificate for the AV-interface on the Edge server).

lync-certificate-chart

Fig 5. Edge and Reverse proxy certificate chart.

To add to this list, our Front-end needed to get its public certificate renewed with the added host webext.domain.com. Before the renewal, it included public certs for lyncdiscover, lyncdiscoverinternal, meet, dialin and sip. Now it includes all those + webext. Webext is needed for the external Web Services (externally accessing the front-end). More about that later on.

The following chapters has information about how to install the certificates on the servers.

 

 

Configuring the Lync Front-end Server for Edge / Installing the Edge Server / Edge Server Certificate Installation

Congratulations If you’ve had the energy to read this far. It’s now finally time to install the Edge server 🙂 Much of the Edge installation/configuration is actually tied to the Front-end server however. You’ll start by making changes to your current topology and then export/publish the topology on the Edge server. Like you’ve probably noticed before, it’s the prep work that takes most of the time. (Internet is full of articles on how to install an Edge server). That said, I happen to like the post at http://www.sherweb.com/blog/installing-lync-2013-edge-server/ and our installation is based on this article. However, there are some differences. In Step 6, the article tells you to request internal and external certificates from the setup itself. We didn’t do it this way because our internal CA isn’t an “online certification authority” (it doesn’t respond to online web requests due to security reasons). Instead we made an offline request and signed it manually on the CA. A little bit more hassle, but worked just fine in the end.

We didn’t use the certificate request wizard for the external certificate either, as those gets created by our “certificate guy”. He uses his own methods and just delivers a fully working certificate. I won’t go into the details, but it works. So in the end, whatever floats your boat and can get you the correct certificates is fine 🙂

If using this “manual method” (alternative to Step 6 in the guide), you must manually install the certificate(s) in the certificate store(s) before continuing. This is by no means difficult. For the internal/external certificates do the following:

  • Fire up “mmc.exe” on the Edge server
  • Add the certificates Snap-in. Select computer account –> local computer and click OK.
  • Right click on the Personal –> Certificates folder.
    • Select All Tasks… Import
    • Imported certificate will show up in this location.
  • Move the different certificates to their corresponding places (Personal, Trusted Root Certification Authorities, Intermediate Certification Authorities). See screenshots/figures below.

lyncedge-cert-personal

Fig 6. Personal certificate (computer)

lyncedge-cert-trusted-root

Fig 7. Trusted Root CA certificate

lyncedge-cert-intermediate

Fig 8. Intermediate CA certificate

  • Done. External certificates are also showing in this list as I have imported them in the same manner.

Now, back to the installation post at http://www.sherweb.com/blog/installing-lync-2013-edge-server/.

Some of my own notes:

  • Followed the guide.
    • Network specifications were OK.
    • Software specifications were OK. Didn’t (need to) install Windows Identity Foundation on the Front-end. (It should be installed as a pre-requirement on the Edge server however).
    • Fired up Topology Builder on the front-end and followed the guide.
      • FQDN of the new edge pool should match the FQDN (CN) on the internal certificate: lyncedge-int.domain.com from certificate chart
      • Access Edge service should match the FQDN (CN) on the external certificate: lyncedgesip-ext.domain.com
      • Web Conferencing Edge service should match the FQDN (CN) on the external certificate: lyncedgewc-ext.domain.com
      • A/V Edge service should match the FQDN (CN) on the external certificate: lyncedgeav-ext.domain.com
        • NOTE: This is a “DNS/certificate-thing”. Whatever certificate CN-record you created for the above services should be used. You can swap CN and SAN records for a more “clean” name, i.e. lyncedgeav-ext.domain.com vs. av.domain.com. (See above DNS/certificate chart and the note about switching places between CN and SAN).
      • Enabled federation (not xmpp yet though)
      • Defined internal and external IP addresses
      • Defined Next hop pool: our front-end
    • Changed the External web services in the Topology builder to match the one we have defined in DNS and in the certificate (would be webext.domain.com in the example).

lync_ext_web_services

Fig 9. External web services

    • Published the topology
    • Exported the configuration (step 4 in guide).
  • Moved over to the Edge server itself
    • Installed Lync server and imported the configuration (step 5 in guide)
    • The certificates were already installed in the certificate store so no need to request certificates (step 6)
      • Defined the existing certificates
    • Started services
    • Done! (no need for step 8 and 9, yet)

 

 

Installing and configuring the Reverse Proxy server / Reverse Proxy server Certificate Installation

I’m now assuming that you have a working Edge server. You can install the reverse proxy server without a working Edge server also, but installing the Edge server first makes it easier to test the reverse proxy functionality right after the installation. First some homework/reading:

https://technet.microsoft.com/en-us/library/jj204691%28v=ocs.15%29.aspx
https://technet.microsoft.com/en-us/library/gg398069%28v=ocs.15%29.aspx
https://blogs.technet.microsoft.com/nexthop/2013/02/19/using-iis-arr-as-a-reverse-proxy-for-lync-server-2013/
http://jackstromberg.com/2014/11/tutorial-deploying-a-reverse-proxy-for-lync-server-2013/
http://kressmark.blogspot.fi/2015/06/windows-2012-r2-iis-85-and-arr-30-as.html
https://thamaraw.com/2013/07/15/publish-lync-server-external-web-services-using-iss-arr/

I decided to go with a combination of the Microsoft guide and the jackstromberg one this time. In the end, it worked perfectly. I had lots of problems and headaches down the line, but this time it had nothing to do with the guides (rather it had to do with typos and a non-working ARR that had to be reinstalled).

Before following the (Microsoft) guide however, we have to install the certificates the same way we did on the Edge server. The reverse proxy certificates are of course different, but I’m assuming that you have requested them at the same time as the Edge certificates. Just follow the Edge steps (fire up mmc.exe and so on) and you’re good to go. One different step is that you have to bind the certificates in IIS, otherwise they won’t be used when clients connect via the reverse proxy. It’s rather easy, let me show you some screenshots:

lync_reverse-proxy-cert-installation-IIS

Fig 10. Adding https bindings in IIS.

lync_reverse-proxy-cert-installation-IIS-binding

Fig 11. Adding https bindings in IIS. Remember to add both the internal and the external interface, with their own certificate.

Now continue following the guides, or use another guide of your choice.

I used the following simple URLs:

  • meet.domain.com
  • dialin.domain.com
  • webext.domain.com
  • lyncdiscover.domain.com

All good, tested and working! 🙂

A note from my own experiences: ARR is VERY STRICT regarding the URL rewrite rules. If something isn’t working, be sure to double check the rules!

 

 

Mobility

I wasn’t quite sure if there was a need for a separate mobility chapter as I’ve covered this area quite well in the DNS and certificate chapters. I guess a couple of lines won’t do no harm however. I’ll once again start by giving you a nice list of homework/reading:

https://technet.microsoft.com/en-us/library/hh690030%28v=ocs.15%29.aspx
http://blog.schertz.name/2013/07/understanding-lync-2013-mobility/
http://www.ucprimer.com/tech-blog/5-essential-tips-for-deploying-lync2013-mobility-service 
http://www.lyncexch.co.uk/lyncdiscover-and-auto-discovery-deeper-dive/
http://blog.schertz.name/2012/12/lync-2013-client-autodiscover/
http://www.ucprimer.com/tech-blog/5-essential-tips-for-deploying-lync2013-mobility-service
https://blogs.technet.microsoft.com/nexthop/2012/02/21/troubleshooting-external-lync-mobility-connectivity-issues-step-by-step/
https://ucgeek.co/2014/02/lync-mobility-dns-records/
http://www.lyncinsider.com/lync-mobile/external-lync-2013-users-need-a-2013-edge-server-to-use-mobility-services/
https://social.technet.microsoft.com/Forums/lync/en-US/daacb3f7-4c09-4d2d-aca4-354cba7a6acc/mobility-without-using-edge-server-2013?forum=lyncdeploy
https://social.technet.microsoft.com/Forums/lync/en-US/b6519fb1-ce56-4977-aaec-3ad23900d8e6/lync-2013-topology-change-now-require-a-reverse-proxy?forum=lyncdeploy
http://windowsitpro.com/lync/decoding-lyncs-endpointconfigurationcache-file (deleting cache)

Much of the mobility stuff has to do with the fact that the mobility service isn’t working properly via lyncdiscoverinternal. Instead you configure the mobile devices to go the external way, via the reverse proxy. See the following picture:

lync-mobility-overview

Fig 12. Lync mobility (source: http://www.ucprimer.com/tech-blog/5-essential-tips-for-deploying-lync2013-mobility-service)

The mobility bit was a big headache, but in the end we got it working in a desirable way. The secret was to remove lyncdiscover.domain.com from the internal DNS (against many recommendations). See the DNS chapter for more information.

 

 

Federation

We’re definitely interested in federation, but we haven’t federated with any partners yet. It’s no harm reading about federation though, and in the end it will be much easier setting it up once you’ve done your homework. I’ve done my homework, so why wouldn’t you 🙂 Here you go:

http://windowsitpro.com/skype-business/understanding-main-types-federation-skype-and-lync-environments
https://technet.microsoft.com/en-us/library/jj204800%28v=ocs.15%29.aspx
https://blogs.technet.microsoft.com/uclobby/2013/09/30/difference-between-open-and-closed-federation-in-lync-20102013/
https://technet.microsoft.com/en-us/library/jj205134.aspx

The Edge server is already enabled for federation, but the front-end is not. This is easily fixed in the Topology builder once we/you decide to federate:

lync-enable-federation-on-FE

Fig 13. Enable federation on the Front-end.

In addition to this, it’s also recommended that you add a DNS SRV record (_sipfederationtls._tcp.domain.com).

 

 

And finally here’s a screenshot I took sometime in the middle of the whole deployment. As I’ve stated before, there were lots and lots of googling and homework to be done 🙂

lync_tab_hell_firefox

Fig 14. Google is your friend. Don’t believe everything you read though…

That’s Firefox with Tab Mix Plus and Multirow Bookmarks Toolbar Plus Extensions, btw.

 

This quite much summarizes the Lync Edge and Reverse Proxy server deployment. Hope you’ve enjoyed reading 🙂

Lync 2013 Persistent Chat problem

I was testing the Lync 2013 Persistent Chat feature the other day and was soon greeted with the error message “1 Warning(s), All the channel servers are down”. I tried googling but didn’t really find any good solutions. Some people suggest removing and then publishing the topology again, but it didn’t work for me.

Here’s an example of a non-working solution:

http://lyncme.co.uk/microsoft-lync-server-2013/all-the-channel-servers-are-down-after-installing-persistent-chat-collocated-on-fe/ 

That said, it could work for someone else. Not saying that the post is useless, just not working in my case.

Some of the errors:

lync_error1

Fig 1. Error in Event Viewer

 

lync_error3

Fig 2. Service says running even though it isn’t.

 

I’ll save you the pain of googling. Here’s the solution (in the comments):

http://windowspbx.blogspot.fi/2012/07/step-by-step-installing-lync-server_19.html

These specific comments helped me:

“thanks for the article i have gone through it and got everything working,
i was getting the error where I was unable to start the persistent chat service and this would then show the error where chat channel unavailable.
i installed the hotfix found at http://support.microsoft.com/kb/2848328
installed the lyncserver update and let this run.
restarted the server and the service started and i was able to set up the persistent chat category.
hope this help out”

“Hi, I just thought I’d post this in case it helps anyone. If you’ve just installed the pChat role into an existing Lync environment that has already been patched, you will get the “all channel servers down” error. You will also notice that when trying to start the pChat services, you will get incompatible database errors as Seyed above.
You need to re-launch the lyncServerUpdateInstaller from the CU level of your Lync Front End server(s). You’ll notice that the Installed Version for Persistent Chat is not up to date. Click Install Updates, then run from Lync Powershell the Install-Database Command against the Persistent Chat Role / SQL Server / DB Instance and the service should now start and you can continue.
Hopefully this helps.”

In other words, just install the hotfix and everything should be running smoothly again. At least it was for me.

lync_error2

Fig 3. Hotfix notices that Persistent Chat is out of date.

 

After the update I gave it a go. In the screenshot I’ve given the user “Kalle” rights to create a chat room. As you can see, the room was successfully created.

lync_chat_room_created

Fig 4. Creating a chat room – working 🙂

 

lync_chat_room_in_action

Fig 5. Chat room in action. Janet and Kalle having a “conversation”.