I was given the task of deploying a full-blown on-premise deployment of Skype for Business Server 2015 for a small company. The deployment also included an Edge Server and a Reverse Proxy server (IIS with ARR). In this blog post I’ll discuss the deployment process in general, and also the problems (and solutions) that were discovered during/after the deployment. I’ll end the post with some check-up/misc. information.
This blog post will be quite massive, so I’ll split it into several chapters.
Planning
General information about planning for Skype for Business Server 2015 can be found at:
https://technet.microsoft.com/en-us/library/dn951427.aspx
However, as I’ve deployed Lync Edge plus Reverse proxy servers before, much of the planning/homework was already done. I’ve also previously written a blog post titled “Test Lab Guide: Windows Server 2016 with Integrated Exchange 2016, SfB Server 2015 and SharePoint 2016”, where I deployed a front-end server for Skype for Business Server 2015. That said, use these two blog posts as a compliment for this guide – they include some deeper information not discussed here. Front-end DNS entries haven’t been discussed earlier however, so here goes:
Front-end DNS
I used the following entries:
A-record:
fe.domain.com
C-names:
lyncdiscoverinternal.domain.com
sip.domain.com
meet.domain.com
dialin.domain.com
webext.domain.com
The next chapter will present the actual front-end installation, including some dilemmas/gotchas. After that I’ll continue with the Edge and Reverse Proxy installation.
Front-end installation
For the front-end installation I followed TechNet’s deployment article with some twists. As usual, it didn’t quite work as planned 🙂 I’ll now try to explain the process in detail, including some problems along the way.
Before starting the actual setup, I double checked that my network and AD infrastructure was setup correctly. I also checked that I had a working ADCS (not setup by me), so I could get internal certificates assigned on the front-end. (This actually turned into quite a mess, see the next chapter about front-end certificate gotchas). Finally I checked that I had created/requested all the DNS aliases needed in the internal DNS zone.
With all these steps done, it was time to move along to the actual front-end installation. My steps:
-
Installed pre-requisites for Windows Server 2016, which are slightly different compared to Windows server 2012 R2. Information can be found at: https://blogs.perficient.com/microsoft/2017/08/skype-for-business-how-to-install-on-windows-server-2016/ for example. (The issue regarding IIS URL Rewrite module wasn’t a problem in my deployment, btw).
-
Add-WindowsFeature RSAT-ADDS, Web-Server, Web-Static-Content, Web-Default-Doc, Web-Http-Errors, Web-Asp-Net, Web-Net-Ext, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Http-Logging, Web-Log-Libraries, Web-Request-Monitor, Web-Http-Tracing, Web-Basic-Auth, Web-Windows-Auth, Web-Client-Auth, Web-Filtering, Web-Stat-Compression, Web-Dyn-Compression, NET-WCF-HTTP-Activation45, Web-Asp-Net45, Web-Mgmt-Tools, Web-Scripting-Tools, Web-Mgmt-Compat, Telnet-Client, Windows-Identity-Foundation
-
-
Installed Silverlight
-
Continued with “official” documentation from https://technet.microsoft.com/en-us/library/dn933889.aspx?f=255&MSPPError=-2147217396
-
Created the file share (local).
-
Important:
“The file share cannot be located on the Enterprise Edition Front End Server, but can be located on a Standard Edition server.
You can define the file share in Topology Builder before you create the file share, but you must create the file share in the defined location you define before you publish the topology”.
-
-
- Installed administrative tools.
- Prepared Active Directory, using an AD account with enterprise/schema admin rights.
- Schema Preparation went fine, but forest preparation failed:
- I hadn’t seen these errors before. Some googling led me to:
- What now? Sigh.
- More googling:
- Success! The -GlobalCatalog option was the key.
- I finally moved over to preparing the domain. The domain preparation worked without problems for a change 🙂
- I then created and published a new topology, again following https://technet.microsoft.com/en-us/library/dn933913.aspx
- Nothing fancy, just the basics with a standard edition front-end and without all the bells and whistles. (Edge server will be configured later):
- Media server is collocated.
- Web services external url is specified as webext.domain.com (in the DNS front-end information earlier).
- Simple urls are also specified according to the DNS information.
- Time to publish the topology…
- But wait. Read carefully: “If this is Standard Edition, you will need to run the Prepare First Standard Edition Server process from the Deployment Wizard before you publish a topology. This prepares for Standard Edition by installing a SQL Server Express Edition instance and creating the Central Management Store”.
- Publishing will actually fail if you forget to prepare first! Been there, done that 🙂
- So, new order is:
- Prepare
- Publish
- Done! …or not 😦 SIGH! New errors:
- Getting sick of all the (never-before-seen) errors. Much googling led me to the following commands:
- I got no source for the above (too much googling), and afaik everything even looked OK in the screenshot.
- Even more googling led me to:
- https://social.technet.microsoft.com/Forums/ie/en-US/b842c139-9cde-4889-9a24-eb09e3cd0064/automatic-collection-of-data-failed-skype-for-business-server-cms-isnt-in-active-directory?forum=sfbfr
http://www.zerohoursleep.com/2013/04/error-when-installing-lync-server-2013-local-configuration-store-cannot-find-any-global-catalogs-in-forest/ - From here on I’m quite clueless what I did. Was it some random PowerShell”magic”, or was it a server reboot that finally did the trick? (I assume the latter…)
- However, by some voodoo magic the publishing wizard completed without errors after the server reboot:
- (Persistent chat and Edge will be enabled later on)
- Now it was finally time to install the system components, following https://technet.microsoft.com/en-us/library/dn951417.aspx?f=255&MSPPError=-2147217396
- Make a note of the following: “Before you follow these steps, make sure you’re logged onto the server with a domain user account that’s both a local administrator and a member of the RTCUniversalServerAdmins group”.
- Ran Step 1, Install Local Configuration Store. No problems.
- Ran Step 2, Setup or Remove Skype for Business Server Components. No problems.
- Ran Step 3, Request, Install or Assign Certificates.
- I checked with the “AD CS guy” that everything was in order for the request. It was.
- I made a custom request with the private key marked as exportable.
- I assigned the certificate (both for default and OAuth).
- (NOTE! Now using a public certificate, read about the gotchas in next chapter)
- It was now time to start all the SfB services.
- This is done from SfB management shell with the command “Start-CsWindowsService”
- All good.
- I finally configured integration between Exchange and Skype for Business (using OAuth). I wrote how to do this in my previous blog post, so have a look there for details.
- I also enabled MAPI for the company’s Exchange server for a smoother integration between SfB and Exchange. I’ve also written about this procedure in a previous blog post.
- As a last step, I wanted to bulk-enable a bunch of users from an AD group.
- I found a nice script from https://gallery.technet.microsoft.com/office/Bulk-Enable-Users-from-an-9a982357
- Ran it:
- Checked the result from SfB Control Panel:
- Success! 🙂
- Finally tested a SfB client, everything seemed OK!
Front-end certificate gotchas
Certificate requirements for internal servers in Lync Server 2013 (same for SfB Server 2015) can be found at:
https://technet.microsoft.com/en-us/library/gg398094%28v=ocs.15%29.aspx?f=255&MSPPError=-2147217396
Some notes:
“Although an internal enterprise certification authority (CA) is recommended for internal servers, you can also use a public CA. For a list of public CAs that provide certificates that comply with specific requirements for unified communications (UC) certificates and have partnered with Microsoft to ensure they work with the Lync Server Certificate Wizard, see article Microsoft Knowledge Base 929395, “Unified Communications Certificate Partners for Exchange Server and for Communications Server,” at https://go.microsoft.com/fwlink/p/?linkId=202834.”
Source: https://technet.microsoft.com/en-us/library/gg398094%28v=ocs.15%29.aspx?f=255&MSPPError=-2147217396
“For internal servers (e.g. Front End Servers and Edge Server internal interface), you can request the certificates from your internal CA. For Reverse Proxy and Edge Server external interface, you should have public certificates installed.You can use SAN certificates, for example.”
“Although this approach technically can work it’s not recommended as (1) it’s not best-practice to use a third party certificate on the internal Front End servers and (2) it’s not advisable to publish your internal server FQDNs and general namespace into an external certificate.”
The above information really begged for internal certificates on the front-end. And yes, I tried the approach. Oh boy, DID I. MANY, MANY times. It always resulted in a mess however. Because of this, we’re now using a public certificate on the front-end. I’ve later realized that I COULD have made it work with internal certificates also. At least to some extent. HOWEVER, you get VERY tired of trying something that works in theory, but not in practice. Let me list some known “problems” (not so easily “found”, at least not in my opinion):
- using internal certificates on the front-end combined with an Edge server requires SSL offloading on the reverse proxy. This is not that well documented afaik, but kind of logical. Not that logical when you have 100 other things to worry/think about though. See: http://www.uclabs.blog/2014/08/lync-2013-reverse-proxy-solution-with.html. I fought with the ARR rules quite a bit so the SSL offloading part wasn’t the hiccup for me. A bigger issue for me was the fact that:
- using internal certificates gave a certificate warning in ALL browsers except Internet Explorer / Microsoft Edge. I don’t know of ANY workaround for this, as I can’t make the internal CA root certificates available for Firefox or Chrome. Short version story: a meeting link (https://meet.domain.com/whatever/XXXX) give certificate error no matter what I do on non-MS browsers. It does not matter if the connection is made internally or via the reverse proxy.
-
The SfB / Lync CLIENT is probably happy with the internal certificates (also through IIS ARR with SSL offloading), HOWEVER I don’t see this as a 100% working solution when an external participant could be using the web client for instance (and receiving a certificate error).
I’ve written more about the certificate dilemma in a later chapter also, see “Problems after the deployment / Certificate problem revisited”.
Edge Server installation
As previously mentioned, I’ve blogged about this before in my blog post Adding Edge and Reverse Proxy Servers to an Existing Lync 2013 Environment. There’s absolutely no need to rewrite this information, as it’s completely usable with Skype for Business also. A little recap is always in place though:
- First, have a look at your network interfaces:
All looks good! Three external interfaces and one internal (all renamed). The third one in the list is the internal NIC and it has no gateway specified. (Never mind the blurs 🙂 )
-
Then add the routes needed for the internal traffic:
route add -p 1.2.3.0 mask 255.255.240.0 1.2.4.40
route add -p 1.2.5.0 mask 255.255.240.0 1.2.4.40
These are just example values. Subnet 1.2.3.0 and subnet 1.2.5.0 should use the internal NIC/interface and the gateway configured on that NIC (1.2.4.40 in the above example).
(You should also edit the hosts-file on the Reverse proxy, NOT on the edge).
- You can read about DNS and certificate requirements in my previous blog post. Please do.
- Check the primary DNS suffix. It should match the suffix from the front-end. This is actually quite an important step so don’t forget to check it!
- Manually install the internal and external certificates via MMC. Your own method may be different.
- Also export (from the front-end) and import the internal CA root certificates. This is needed because the edge server is not domain joined. Google if you don’t know how, or read my previous blog post.
“Activating” the Edge service…
The Edge service is activated on the Front-end. I like the guide I’ve used before, namely https://www.sherweb.com/blog/installing-lync-2013-edge-server/. It’s still usable for SfB Server 2015, but feel free to use whatever guide you like. Some notes from the setup:
- DO make sure that you create the new Edge Pool under the “Skype for Business Server 2015” branch, not under Lync Server 2013.
- This pool has one server
- Did not enable federation
- NOT using a single FQDN and IP for the Edge services
- Using IPv4 and no NAT.
- Specified the external FQDN the same way as in my previous blog post
- …and so forth
- Published the topology
- Exported the configuration
Moving over to the Edge server…
Same prerequisites as on the font-end:
Add-WindowsFeature RSAT-ADDS, Web-Server, Web-Static-Content, Web-Default-Doc, Web-Http-Errors, Web-Asp-Net, Web-Net-Ext, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Http-Logging, Web-Log-Libraries, Web-Request-Monitor, Web-Http-Tracing, Web-Basic-Auth, Web-Windows-Auth, Web-Client-Auth, Web-Filtering, Web-Stat-Compression, Web-Dyn-Compression, NET-WCF-HTTP-Activation45, Web-Asp-Net45, Web-Mgmt-Tools, Web-Scripting-Tools, Web-Mgmt-Compat, Telnet-Client, Windows-Identity-Foundation
- Ran setup.exe from the install media and continued following Step 5 from the guide at https://www.sherweb.com/blog/installing-lync-2013-edge-server/
- Certificates were already installed via MMC so no need to import, just assign.
- Started services (Start-CsWindowsService)
- Created DNS SRV records (Step 8). The first one is needed, so don’t try without it. It won’t work.
- Step 9 in the guide is falling a little bit short. There’s no mention about the Access Edge Configuration – you should also check that tab. See screenshot below:
- While you’re at it, have a look at the conferencing policy at the same time:
Be sure to enable needed rights for the anonymous users.
- Happy days, now moving over to the reverse proxy server installation.
Reverse Proxy server installation
I’ll just do a mini-recap as this installation is almost identical to the existing one in my blog post Adding Edge and Reverse Proxy Servers to an Existing Lync 2013 Environment.
- Checked the network interfaces (renamed and specified correct IP’s).
- Defined some hosts in the host-file. Check my above blog post.
- Installed the certificates via MMC. More detailed information in my previous post.
- Some newer information for Windows Server 2016:
- For IIS, just install the basics:
- Install-WindowsFeature -Name Web-Server, Web-Mgmt-Tools
- For ARR, download the Microsoft Web Platform installer, and from there choose to install ARR 3.0. Example: https://www.youtube.com/watch?v=ILOC91lAyBE
- Add the correct certificates to the https bindings in IIS
- For IIS, just install the basics:
- I previously followed http://jackstromberg.com/2014/11/tutorial-deploying-a-reverse-proxy-for-lync-server-2013/ for the IIS ARR rewrite rules. For some reason they didn’t work this time. Well, I’m not surprised, as IIS ARR is VERY picky about the rules. (It can actually be quite a headache to configure these).
Some alternative links for IIS ARR rewrite rules:
https://lucavitali.wordpress.com/2017/06/30/arr-how-to-setup-and-use-with-multiple-lyncsfb-sip-domains/
http://www.uclabs.blog/2014/08/lync-2013-reverse-proxy-solution-with.html
http://www.ucguys.com/2014/08/using-iis-arr-30-on-windows-server-2012r2-as-a-reverse-proxy-for-lync-server-2013.html
I used the first link as a guidance for my (new) rewrite rules. I tried many different combinations before I found this one, but to no avail. MANY thanks to Luca Vitali for something that actually works 🙂
I love the fact that you get away with using just ONE server farm/rule for all the simple urls. (We’re not using Office Web Apps).
The rule that work:
Pattern: (.*)
{HTTPS} on
{HTTP_HOST} = (meet|dialin|webext|lyncdiscover).domain.com
As a side note I noticed that you can use multiple SIP domains with this same “one liner”. In that case you just change the line to:
{HTTP_HOST} = (meet|dialin|webext|lyncdiscover).domain.com | (meet|dialin|webext|lyncdiscover).domain2.com
“|” stands for “or” and is the key here. This is an alternative approach to Luca’s implementation. See screenshot below:
I tried this with two different SIP domains, and both domains proxied nicely through IIS ARR with the above change 🙂
Problems after the deployment
All of the above would seem like the perfect deployment, right? Wrong. After some testing you’ll probably notice some problems. My deployment was no exception. When I started the SfB client (after finishing the Edge deployment), I was greeted with the following:
A quick look in SfB Control Panel and event viewer told me something was wrong with the replication:
On the Front-end:
On the Edge server:
These error messages were new to me but trusty ol’ Google led me to:
https://social.technet.microsoft.com/wiki/contents/articles/7459.troubleshooting-limited-external-calling-in-lync-client.aspx
http://communicationsknowledge.blogspot.fi/2012/06/limited-external-calling.html
http://www.wavecoreit.com/blog/serverconfig/file-transfer-agent-cannot-send-replication-data-to-replica-replicator-on-edge/
https://support.microsoft.com/en-us/help/2759117/lync-server-central-management-store-replication-failures
It didn’t take me long to realize that this was a firewall problem. I thought I had notified the firewall guy about all needed (open) ports, but this one seemed to have slipped. Well, no problem. We opened up the replication port (4443) in the internal firewall, and the replication started working again almost instantly.
To double check that the problem wasn’t with the replication service, I also used netstat and telnet:
On the Front-end:
On the Edge server:
As seen in the screenshots, port 4443 is listening as it should be. This clearly shows that the service is listening, and it shouldn’t be the cause for the problem.
I could NOT telnet to the port however, which implies a firewall problem. After fixing the (hw) firewall and starting the SfB client again, the error was gone:
Warning gone, happy days! 🙂
Certificate problem revisited
As stated in the chapter “Front-end certificate gotchas”, I had problems with the internal certificate. To illustrate the dilemma, let me show you some screenshots:
From a domain joined client:
To the left we have Firefox. As you can see, the connection is NOT trusted even though I’m testing from a domain joined client.
To the right we have Internet Explorer. The connection IS trusted (root CA is trusted).
Same thing from an external client:
“Since Lync simple URL publishing does not require any SSL Offloading if you have the External Web Site in Lync assigned with a Public Certificate, you do not need a certificate installed on the IIS.
Most likely you have assigned a private certificate from your internal Certificate Authority, and you have to assign the IIS ARR an public certificate and reencrypt the traffic for internal use.
Be aware of two point here:
- this is called SSL Offloading and requires some extra CPU load on your server
- IIS must not be “domain joined” and therefore you need to have the internal Certificate authority Root Certificates assigned as TRUSTED!”
Source: http://www.uclabs.blog/2014/08/lync-2013-reverse-proxy-solution-with.html
Well, I could probably have gotten rid of the Internet Explorer error on the external client (while still using internal certificates) by using SSL offloading on the reverse proxy. It doesn’t solve the problem with other browsers however, afaik. If you have a nice solution to this whole certificate-dilemma, please comment!
All in all I think using a public certificate on the Front-end is easier, even though you’re “going against” best practices to some extent. At least it makes your life a whole lot easier.
Final check-up’s
Skype for Business Configuration Information
You should always check the Skype for Business Configuration Information from the SfB client after a successful deployment. Press the “ctrl” key while right clicking on the Skype for Business icon in the taskbar (down right). Then choose “Configuration Information”. From here, check that everything looks OK:
- MRAS list the connection to the edge server, and it looks OK.
- GAL search tells SfB to look for the address book from the Global Address List (Exchange) instead from the internal SfB server Address Book.
- EWS Internal URL is the connection towards Exchange.
- Contact List Provider UCS (Unified Contact Store) shows that the Address Book is situated on Exchange – we’re not using a local SfB server Address Book.
- UCS (see above) Connectivity State says that the connection to Exchange is active as it should be.
- MAPI Information states that MAPI status is OK (which it should be now that I’ve configured Exchange for MAPI).
- EWS (Exchange Web Services) is also OK.
Get-DnsAndIP PowerShell script
I found an awesome script to get an overview over deployed DNS records in a Skype for Business deployment. This is a lifesaver if your head is about to explode with all the DNS/certificate/firewall/whatever information 🙂 Get the script from:
https://gallery.technet.microsoft.com/office/Get-DnsAndIPPS1-List-your-a4689878
This script (plus Snooper) actually helped me find a missing DNS SRV record. Phew. A screenshot from the script in action:
There are some “false positives” here. Only thing worth mentioning is the missing SRV record at the end. The test was also done with dual SIP domains, which is not the setup anymore. I’ve not added DNS records for federation either, as we’re not federating with any partners (yet).
Lastly, I’m NOT using lyncdiscover.domain.com in the internal DNS (intentionally). I have an explanation for this in my previous blog post Adding Edge and Reverse Proxy Servers to an Existing Lync 2013 Environment.
Snooper
Also worth mentioning is Snooper, which is part of Skype for Business Server 2015 Debugging Tools. It’s downloadable from:
https://www.microsoft.com/en-us/download/details.aspx?id=47263
It is very awesome in debugging client (connection) problems. I used it in combination with the above PowerShell script to find out that my DNS SRV record was missing for example. Here’s a screenshot from that particular “investigation”:
For some quick information on how to use snooper, have a look at http://lynclead.com/?p=148 for example. The log files for the newest SfB client are located in C:\Users\xxxxx\AppData\Local\Microsoft\Office\16.0\Lync\Tracing btw.
Misc.
I’m also using the ABS Configuration Tool in this deployment to customize the Address Book. Detailed information about this can be found in my blog post Disabled (AD) users still searchable in Lync/SfB. The download link is available at: https://technet.microsoft.com/en-us/library/jj945604.aspx
SSL hardening of the SfB servers were done with IIS Crypto, https://www.nartac.com/Products/IISCrypto. I used the “best practices” template and received an “A” grade on Qualys SSL Labs Server Test, https://www.ssllabs.com/ssltest/. If you’re interested in getting an A+, have a look at https://scotthelme.co.uk/getting-an-a-on-the-qualys-ssl-test-windows-edition/ for example.
This quite much summarizes the whole Skype for Business deployment. Be sure to check my other blog post about Edge and reverse proxy deployment also, as it includes some deeper information not mentioned here.
A random blog from a sysadmin 