Lync 2013 Front End Service won't start

After a fresh install of Lync 2013 Standard Edition onto a Windows 2012 server, and its been running for about 2 weeks I may add...I came across the following issue

Issue

After a reboot of the Lync Standard Edition Server I found that the Lync Server Front End Server (RTCSRV) wouldn't start. It just sat there on "Starting"

Further Symptoms

Event ID 32174



Interestingly the Event ID reports that resetting the Quorum is a potential solution, not true in my case.

Event ID 32178



NOTE
There were many other Events triggered by the inability of numerous processes being able to connect etc etc

Cause

A common cause of this issue is if non self-signed certificates are imported into Trusted Root Certification Authorities instead of Intermediate Certification Authorities. Although this was not an issue in Windows Server 2008, Windows 2012 requires a high level of trust dependant on certification authentication.The result is that HTTP communication between Lync servers is broken.

Solution

Check the Trusted Root Certification Authority Store for any NON self-signed certificates. Self signed certificates have the same value for the Issued To and Issued By field. The screen shot below shows the highlighted certificate as not being self-signed (whilst residing in the Trusted Root Certification Authority Store)

Another quick way to check, especially when there are many entries, is to run the following PowerShell command:

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Format-List * | Out-File "c:\computer_filtered.txt"

The output file will contain the details of the culprit.

Now simply move the NON Self-Signed Certificate to the Intermediate Certification Authorities.

and the Self-Signed Certificates to the Trusted Root Certification Authorities

Microsoft KB article http://support.microsoft.com/kb/2795828 - 

Lync Server 2013 Front-End service cannot start in Windows Server 2012

Lync Deployment Wizard fails - Input string was not in the correct format

The Problem
When Running the Lync Server Deployment Wizard - Install oe Update Lync Server System - Step 2: Setup or Remove Lync Server Components I get an error immediately after Disabling unused roles

The Error
Input string was not in the correct format

at System.Text.StringBuilder.AppendFormat(IFormatProvider provider, String format, Object[] args)
at System.String.Format(IFormatProvider provider, String format, Object[] args)
at Microsoft.Rtc.Internal.Tools.Bootstrapper.BootstrapperTask.OutStatusLine(String format, Object[] args)
at Microsoft.Rtc.Internal.Tools.Bootstrapper.BootstrapperTask.DeactivateRemovedRoles()
at Microsoft.Rtc.Management.Internal.Utilities.LogWriter.InvokeAndLog(Action action)

The Culprit
This turned out to be the Lync Admin account, I couldn't find any evidance pointing me to the cause other than a feeling of discomfort with the format of the Lync Admin User account name.
The Lync Admin account was similar to this:
{LyncAdmin}

Couldn't find any documentation saying the curly brackets may be an issue, but tried with an account that had a simpler format and the required permissions and tada...

The Solution
Initially I had the account renamed to something like LyncAdmin but it still misbehaved, recreating the account without curly brackets did the trick

LPE wont sign in to Lync externally

The Problem

Lync Phone Edition (in my case Polycom CX500 and CX600) wont sign in to Lync from externally. Internally working fine.

The phone goes through the steps of finding the time server, then finding the Lync Server - OK. Then gets stuck on Signing in... Connecting to Lync Server
This also happened to the CX600 EVEN when pairing to my desktop over USB for sign in...

The Evidence

Checking the Lync Edge Server logs I found...nothing! Same as the Front End, also nothing!
Since everything else seemed to be working I started to wonder how the LPE's are different from the desktop client.
Going through the motions of validating the DNS using the RUCT.exe tool I found that when I tried to test the port availability on selected records on the SRV record for -sip._tls I got an error stating that the host was not found.

What I found was very interesting. The SRV record for  _sip._tls.sipdoamin.com resloved to sip.sipdomain.com BUT the Public Certificate Common Name was LyncEdge.sipdomain.com with the name sip.sipdomain.com as a Subject Alternate Name.

The Solution

I changed the the SRV record so that the target was the published public name LyncEdge.sipdomain.com. Effectively this meant that the SRV target was also the Common Name on the Public Certificate.

NOTE
The Common Name on the Public Lync Certificate should always be a resolvable target for the SRV records

Sign-in canceled due to internal error - CX500 and CX600

Issue
The Lync phones (CX500 and CX600) no longer sign in to Lync 2013. Error message states "Please verify that the domain entered is correct"

After factory resetting the phone I still got the same error briefly followed by "Sign-in canceled due to internal error.Please try again"

I tested to see if the credentials were working with:

Test-CsPhoneBootstrap -PhoneOrExt "2705" -PIN "12121"

I checked the DNS setup
Lync Phones usually require as a minimum Option 42 and Option 120 (then enable DHCP Options on the Front End with .

set-CsRegistrarConfiguration -EnableDHCPServer $true

I tested the DHCP options with the DHCPUtil.exe -emulateclient
I also ran wireshark to see the client getting the DHCP options, connecting to Lync, negotiating certificates and then just stop as if the credentials were incorrect

Cause
Unknown...
Even though the traces showed that the phone was getting all the required DHCP options and the phone was chatting to the Lync Front End, I still had no success signing in.

Fix
I am not hugely excited about the Phone Logs so before I resorted to digging into them I simply ran through the Lync Deployment Wizard (Step 2)...

That did the trick "sigh"

Other potential causes
http://support.microsoft.com/kb/2464556

Can't call UM - Event ID 44022

Issue

Unable to call UM

Evidence

Every time a call is made to the UM server the following error is logged on the Lync server front end:-

Log Name:      Lync Server

Source:        LS Exchange Unified Messaging Routing
Date:          10/12/2013 09:27:05 PM
Event ID:      44022
Task Category: (1040)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      lyncFE.lynclab.local
Description:
An attempt to route to an Exchange UM server failed.

The attempt failed with response code 504: exchum.lynclab.local.
Request Target: [DefaultUM@exchum.company.local], Call Id: [9f9a3df33d2544a9a22ff2c17feedcb5].
Failure occurrences: 4, since 10/12/2013 08:36:55 PM.
Cause: An attempt to route to an Exchange UM server failed because the UM server was unable to process the request or did not respond within the allotted time.
Resolution:
Check this server is correctly configured to point to the appropriate Exchange UM server. Also check whether the Exchange UM server is up and whether it in turn is also properly configured."

Solution

Make sure the dialplan is configured to use TLS (and not TCP). 

Adding another Front End Server to the Enterprise Pool

Just recently I added another 2013 Front End Server to an Enterprise Pool and was subsequently presented with some interesting issues.

After adding the new Front End to the Topology and running the setup I moved on to migrating some test users and saw the following error:

Distributed Component Object Model (DCOM) operation SetMoveResourceData failed??

I checked the replication and all Lync servers were replicating.
Checking the Lync Topology servicers I found that my new Front End server wasnt in show room condition, see below:

The green check on the 3rd line shows replication OK but status not OK
So now I am wondering which services are not running, off to Front End number 3 to check the Lync services. As supected there are services that arent running.

Turns out both the Call Park Service and the Response Group Service wont start. Perhaps there is a hint as to why in the Event Log..

..and I found this clue. Seems that FE3 is unable to connect to the backend SQL. Also the little Inner exception reference (saw that in the DCOM reference).
Login failed for user was the real gem.

Checking with the SQL admin I got confirmation that FE3 was trying to connect to the Backend SQL database and failing the login about once every second (SQL guy not happy).

OK, so how come FE3 doesn't have the permissions on the Backend SQL server??
Seems that my temporary SQL Admin rights were not in place when adding FE3 to the Topology, this is when the SQL permissions are set. Interestingly there were no errors when I published the Topology.

How to fix these permissions? Either get unhappy SQL admin to re-publish the Topology with his SQL Admin account or get temporary SQL Admin permissions and DIY.

...yes, and voila

OAuth certificate missing

Issue
Whilst deploying Lync Enterprise Edition with 3 Front End Servers I cam across an interesting issue. FE 1 was fine but when I fired up FE 2 and got to the certificate wizard the OAuth Certificate was missing.

One thing you will notice if there is no OAuth certificate is that the Lync Front End Service wont start. OK so where is the cert???

Found a good blog explaining the purpose of OAuth here (thanks Doug)

So fist thing was to see if the Front End Servers were replicating, and indeed they were BUT no OAuth. 

Checking the Cert Manager through MMC shows that the cert isn't in the personal store. Adding it there manually didn't help me much either...

Seems that it needs to be put there by the replication process.

I decided to move along (against my best judgement and the clock) and add the default cert to FE 2 and then come back to OAuth. Re-ran setup Step1 and Step 2, rebooted the server and after that still now OAuth.

Retracing my steps I noticed that the internal DNS records had not been added yet.

Resolution
You must add the Front End Pool FQDN with all the individual Front End Server IP addresses. Ensure that when you run nslookup that all the Front End IP's are found. If an IP is missing from DNS you wont get the OAuth Certificate....

Below is an error I got in the event log.

The replication of certificates from the central management store to ...2013, Replica Replicator Agent will continuously attempt to retry the replication.While this condition persists, the certificates on the local machine will not be updated

Object reference not set to an instance of an object.
at Microsoft.Rtc.Management.Common.Certificates.CertUtils.GetKeyFileName