Troubleshooting: Lync on Prem - UM in Cloud

Got asked to solve an issue where Voicemail for users in a Lync onprem, UM in cloud environment had mysteriously stopped working.

The FE event log had this to say

Running Lync Traces showed the following

Followed by the Error

So as exchange UM is in the O365 cloud and the event says to check that off I went.

Steps to check O365 UM configuration with Lync on premise

I'll start with checking the Lync setup

1. What DNS is required?

         _sipfederationtls._tcp.lynclab.co.nz port 5061 dest Edge FQDN 

         Edge FQDN

2. Ensure Access Edge Configuration is correct

3. Check the Hosting Provider

4. Check the UM Contact Object

5. Make sure that a test user is enabled for Hosted VM

6. Check the Hosted Voicemail Policy 

7. Make sure that the Edge Server is replicating

Now let Check the O365 configuration

1. Check that the UM Dial Plan is setup. Now this is really simple, there is absolutely no trick at all. Just remember that there wont be an IP gateway. Thats it!

2. Check to see the Authoritative domain in O365 matches the Organization configured in CsHostedVoicemailPolicy by running Powershell remotely to connect to O365 deployment.

From Powershell...

$cred = Get-Credential

$s = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell -Credential $cred -Authentication Basic -AllowRedirection

importresults = Import-PSSession $s

Get-AcceptedDomain

Note the Results.

In my case this is where the problem was, somehow ... mysteriously the Authoritative Domain had changed.

Update
Now I have see this else where, when I have the opportunity to investigate why I'll come back and post an update.

When publishing our Lync Topology

When publishing our Lync Topology I'm getting the following error on the Enabling Topology step:

Error: Found multiple objects with identity "lyncFE01.lynclab.local,McxInternal" in Active Directory. ▼ Details └ Type: ActiveDirectoryException └ ▼ Stack Trace └ at Microsoft.Rtc.Management.Deployment.Core.CompatTrustedService.GetTrustedService(ADSession session, ADObjectId containerId, String fqdn, String serviceType) at Microsoft.Rtc.Management.Deployment.Core.CompatTrustedService.Create() at Microsoft.Rtc.Management.Deployment.Roles.WebServices.GlobalActivate(IService service, Computer computer) at Microsoft.Rtc.Management.Deployment.Core.Service.GlobalActivate(Computer computer) at Microsoft.Rtc.Management.Internal.Utilities.LogWriter.InvokeAndLog[T](Action`1 action, T arg)

8/6/2013 2:41:59PM

Error

└

Error: An error occurred: "Microsoft.Rtc.Management.Deployment.ActiveDirectoryException" "Found multiple objects with identity "lyncFE01.lynclab.local.McxInternal" in Active Directory."

Solution
1) Run Test-CsTopology -Report C:\temp\testtopology.html

2) Prepare the appropriate AD for a TXT file:Ldifde -f c:\temp\addif.txt -s DC_FQDN -d "CN=RTC Service, CN=Services, CN=Configuration, DC=lynclab, DC=local"

3) Find the duplicate entries in txt file. Then delete them from AD using ADSIE Edit

IP Change for Gateway\SBC

Issue

Change of IP to SIP SBC causes one way speech for outbound calls. SDP shows internal IP on call setup

Solution

Force the deployment to use fixed addresses in topology, publish. Then remove this (remember to visit the PSTN gateways tab) publish.

You should have no IP's set when running 

get-csnetworkinterfaces for PSTN

Lync cannot verify that the server is trusted

This summary is not available. Please click here to view the post.

Credential Prompt

Problem
Get a second prompt for credentials when logging in with the following text..
"Type your user name and password to connect for retrieving response groups"

Solution

The Lync Share needs to have read\write permissions to itself and containing folders. Corrected in the Advanced Sharing tab as below.



Lync Control Panel

Problem
Can't connect to the Lync Server control Panel directly but https:\\FQDN\cscp works

Solution
Something I didn't realize is that the Control Panel uses the DNS _sipinternal SRV Record. This is the record in the DNS branch matching the SIP Domain and not necessarily the branch that matches the internal DNS naming.

Certificate Authentication Problem

Problem
Lync cannot verify that the server is trusted for your sign-in address. Connect anyway?Cause
Lync Client 2013 has an additional safety check implemented in that the users SIP Domain  is compared with the FQDN of Lync server when the user tries to connect.

In the most environments, the SIP domain is different from the Active Directory domain.

SolutionHKEY_CURRENT_USER\Software\Policies\Microsoft\Office\15.0\Lync

here you need to modify or add the "new String Value" TrustModelData
in this key, you need to add the server listed in the warning.
e.g. lyncpool.lynclab.local

Computer clock

Problem
Communicator can't sign in and and reports:-
Cannot sign in to Communicator because your computer clock in not set correctly...

Solution
This is caused when there is a time difference between the Lync\OCS server and the clients. I think the maximum threshold is around 10 minutes for time difference. Correct this and you should be sorted

problem verifying certificate

Problem
When trying to sign in to Lync get the following error:
There was a problem verifying the certificate

Solution
It's either a certificate trust issue or a DNS name mismatch to the certificate that you have issued.  
The PC or device which you are using to logon to Lync needs to trust the certificate chain from which you generated the Lync certificate(s) and the DNS records used to locate and connect to the Lync server need to match the name(s) on the certificate.

In my case I was using Manual Login and pointed to the IP address which was obviously not in the certificate :p

Web Conferencing: Target Principal Name is incorrect

ProblemWhen accessing the meet url from outside the corporate network you get the error Server error 500 - Target Principal Name is incorrect

Cause
When you tickle the TMG rule the traffic is redirected to the Lync FE, however the requested URL [eg.https://webconf.lynclab.co.nz/meet/john.bravo/9c6gsa] needs to be in the internal FE cert...

Solution
Simply update the cert. So the internal cert will need webconf.abc.com, when youo run through the cert wizard on the FE it will auto populate the cert accordingly

Cant change Meet URL

Problem

Unable to change the default Meet URL. Get a red X and the OK button is greyed out.

Solution

Firstly let me say that I prefer adding a URL/meet than a meet.URL since I don't need to add additional SANs to my cert. This is the reason why this ussue has come up. In any event...
Topology builder will allow you to add a Meet and Dialin URL that actually conflicts with the External Web Services (shouldn't let you...)

 It does however give you an error if you try to make it the default or try to remove another meet url that is different from the External Web
Ultimately the simple URL's and the External URL's need to be different

Meet URL fails

Problem

Meet URL fails

Solution

Ensure that the URL is added in TMG under Published Sites.

TMG test rule will fail as it requires additional switches to be valid.

In my deployment we had multiple Edge Severs and sites - make sure that the meet URL is reachable across all sites, remember that the URL will be directed to the FE based on where the user is homed.

MCX Forbidden

Problem

Can't connect to Lync MCX service. Http Error 403 Forbidden, Lyncdiscover Http Authentication Test failed when testing https://<LyncWebService FQDN>/Mcx/McxService.svc

Also get Authentication Test failed from http://www.testocsconnectivity.com/

Solution

Error was the TMG rule

The error here says that the Credentials for the request to the site were deleted. It also explains how no delegation is set and user authentication isn't enabled. Of course this needs to be enabled!!!

IE Security

Problem
Default install of Windows 2008 internet explorer security blocks just about every page.

Solution
From the Server Manager, Deactivate IE Security as seen below

Frequent invalid SIP requests

Problem
Partners receiving a large number of errors in the Edge Server event log like below

Solution
The cause seems to be Lync still sending discovery packets every 10 minutes.
If federation is allowed, add the SIP domain to the allowed list, if blocked - add the SIP domain to the blocked list.
This will be followed by a final event entry stating that the problem has been resolved

Schema State check has failed.

Problem

Schema State check has failed. 

Solution

Both instances were linked back to DNS.

To prove that AD was healthy I ran the Prepare AD components directly from the DC (that works as usual)..which confirms that a DNS validation issue is present.
So what's going on with DNS?

Fisrtly an NSLOOKUP on the Lync box reveals that the default DNS server is unknown, adding a PTR record for the DNS server solves that.

Secondly, the installer queries the SRV records for contacting the PDC in active directory. This SRV record is: 
_ldap._tcp.pdc._msdcs. DnsDomainName 

UM Badmail

Problem

Actually this is more of a where is it than an issue ;-)

Where is the voicemail stored in UM before sending to Exchange? This includes the bad voicemail folder

Solution

C:\Program Files\Microsoft\Exchange Server\V14\UnifiedMessaging\...

Forcing Join Conference from Browser

Problem
Foreign user is sent a Lync online Meeting Request, if the invited user has Lync installed but doesn't have Federation capabilities the Join Conference request url will fail (since it calls the local Lync client to connect)

Solution
Force the conference invite URL to launch the Web and Lync Attendee options and not local Communicator Client (if present) -Just append this to the url   "?sl=1"

Lync Communicator Mobile wont login

Problem
Lync Communicator Mobile wont login

Error Message
Server unavailable at this time

Solution
On the Sign In page you enter your SIP Login name and password. However you also need to go to More Details (ios and WM7)\Options (Android) and add your user name. 

I have found that the username for WM7 needs to be Domain\User Name, although this format works on Android and ios simpoly adding the user name also works

PSTN Conferencing Error: Sorry, I can't seem to connect you to your meeting..."

Problem
While trying to call in to a conference from an external PSTN connection the error "Sorry, I can't seem to connect you to your meeting..."

Error Message
S4 traces on snooper revealed a "foreign gateway" IP address been called by the Mediation server.

Solution
The default Gateway in Topology Builder was an old (decommisioned) SIP connection (aka "foreign gateway". Changed that to the gateway I was actually using to call out on - solved!

Application Server keeps stopping

Problem
ApplicationServer (includes Call Park Service) Starts and then stops within seconds

Error Message
ErrorCode=-2146893022 
FailureReason=IncorrectNameInRemoteCertificate 
LocalEndpoint=127.0.0.1:62233 
RemoteEndpoint=127.0.0.1:5075 
RemoteCertificate=<null>

Solution
#1 make sure 127.0.0.1  localhost exists in hosts file
#2 For EE Server you need add both the FQDN of pool name and server name as SAN in the default certificate.