12 August 2013

Troubleshooting: Lync on Prem - UM in Cloud

Got asked to solve an issue where Voicemail for users in a Lync onprem, UM in cloud environment had mysteriously stopped working.

The FE event log had this to say

Running Lync Traces showed the following

Followed by the Error

So as exchange UM is in the O365 cloud and the event says to check that off I went.

Steps to check O365 UM configuration with Lync on premise
I'll start with checking the Lync setup
1. What DNS is required?
         _sipfederationtls._tcp.lynclab.co.nz port 5061 dest Edge FQDN 
         Edge FQDN
2. Ensure Access Edge Configuration is correct

3. Check the Hosting Provider

4. Check the UM Contact Object

5. Make sure that a test user is enabled for Hosted VM

6. Check the Hosted Voicemail Policy 

7. Make sure that the Edge Server is replicating

Now let Check the O365 configuration
1. Check that the UM Dial Plan is setup. Now this is really simple, there is absolutely no trick at all. Just remember that there wont be an IP gateway. Thats it!

2. Check to see the Authoritative domain in O365 matches the Organization configured in CsHostedVoicemailPolicy by running Powershell remotely to connect to O365 deployment.
From Powershell...

$cred = Get-Credential
$s = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell -Credential $cred -Authentication Basic -AllowRedirection

importresults = Import-PSSession $s


Note the Results.
In my case this is where the problem was, somehow ... mysteriously the Authoritative Domain had changed.

Now I have see this else where, when I have the opportunity to investigate why I'll come back and post an update.

6 August 2013

When publishing our Lync Topology

When publishing our Lync Topology I'm getting the following error on the Enabling Topology step:
Error: Found multiple objects with identity "lyncFE01.lynclab.local,McxInternal" in Active Directory.
 Type: ActiveDirectoryException
  Stack Trace
at Microsoft.Rtc.Management.Deployment.Core.CompatTrustedService.GetTrustedService(ADSession session, ADObjectId containerId, String fqdn, String serviceType)
at Microsoft.Rtc.Management.Deployment.Core.CompatTrustedService.Create()
at Microsoft.Rtc.Management.Deployment.Roles.WebServices.GlobalActivate(IService service, Computer computer)
at Microsoft.Rtc.Management.Deployment.Core.Service.GlobalActivate(Computer computer)
at Microsoft.Rtc.Management.Internal.Utilities.LogWriter.InvokeAndLog[T](Action`1 action, T arg)
8/6/2013 2:41:59PMError
Error: An error occurred: "Microsoft.Rtc.Management.Deployment.ActiveDirectoryException" "Found multiple objects with identity "lyncFE01.lynclab.local.McxInternal" in Active Directory."

1) Run Test-CsTopology -Report C:\temp\testtopology.html
2) Prepare the appropriate AD for a TXT file:Ldifde -f c:\temp\addif.txt -s DC_FQDN -d "CN=RTC Service, CN=Services, CN=Configuration, DC=lynclab, DC=local"
3) Find the duplicate entries in txt file. Then delete them from AD using ADSIE Edit

IP Change for Gateway\SBC

Change of IP to SIP SBC causes one way speech for outbound calls. SDP shows internal IP on call setup

Force the deployment to use fixed addresses in topology, publish. Then remove this (remember to visit the PSTN gateways tab) publish.

You should have no IP's set when running 

get-csnetworkinterfaces for PSTN

Lync cannot verify that the server is trusted

You get the error message "Lync cannot verify that the server is trusted for your sign-in address"

When Lync Communicator discovers the Lync FE to log on to it uses the SRV Record _sipinternaltls._tcp.SIPDOMAIN.com. If the associated server FQDN is resolved to a server that doesnt match the SIP DOMAIN then this error is presented.EG Below record is for DNS zone xxx.co.nz, Sip Domain is xxx.co.nz but target host is a .local FQDN

Add an A record (xxx.co.nz for the FE Server) with matches the Sip and DNS, then edit the SRV record to point to this record.

Credential Prompt

Get a second prompt for credentials when logging in with the following text..
"Type your user name and password to connect for retrieving response groups"

The Lync Share needs to have read\write permissions to itself and containing folders. Corrected in the Advanced Sharing tab as below.

Lync Control Panel

Can't connect to the Lync Server control Panel directly but https:\\FQDN\cscp works

Something I didn't realize is that the Control Panel uses the DNS _sipinternal SRV Record. This is the record in the DNS branch matching the SIP Domain and not necessarily the branch that matches the internal DNS naming.

Certificate Authentication Problem

Lync cannot verify that the server is trusted for your sign-in address. Connect anyway?
Lync Client 2013 has an additional safety check implemented in that the users SIP Domain  is compared with the FQDN of Lync server when the user tries to connect.

In the most environments, the SIP domain is different from the Active Directory domain.


here you need to modify or add the "new String Value" TrustModelData
in this key, you need to add the server listed in the warning.
e.g. lyncpool.lynclab.local

Computer clock

Communicator can't sign in and and reports:-
Cannot sign in to Communicator because your computer clock in not set correctly...

This is caused when there is a time difference between the Lync\OCS server and the clients. I think the maximum threshold is around 10 minutes for time difference. Correct this and you should be sorted

problem verifying certificate

When trying to sign in to Lync get the following error:
There was a problem verifying the certificate

It's either a certificate trust issue or a DNS name mismatch to the certificate that you have issued.  
The PC or device which you are using to logon to Lync needs to trust the certificate chain from which you generated the Lync certificate(s) and the DNS records used to locate and connect to the Lync server need to match the name(s) on the certificate.

In my case I was using Manual Login and pointed to the IP address which was obviously not in the certificate :p

Web Conferencing: Target Principal Name is incorrect

ProblemWhen accessing the meet url from outside the corporate network you get the error Server error 500 - Target Principal Name is incorrect

When you tickle the TMG rule the traffic is redirected to the Lync FE, however the requested URL [eg.https://webconf.lynclab.co.nz/meet/john.bravo/9c6gsa] needs to be in the internal FE cert...
Simply update the cert. So the internal cert will need webconf.abc.com, when youo run through the cert wizard on the FE it will auto populate the cert accordingly

Cant change Meet URL

Unable to change the default Meet URL. Get a red X and the OK button is greyed out.

Firstly let me say that I prefer adding a URL/meet than a meet.URL since I don't need to add additional SANs to my cert. This is the reason why this ussue has come up. In any event...
Topology builder will allow you to add a Meet and Dialin URL that actually conflicts with the External Web Services (shouldn't let you...)

 It does however give you an error if you try to make it the default or try to remove another meet url that is different from the External Web
Ultimately the simple URL's and the External URL's need to be different

Meet URL fails

Meet URL fails

Ensure that the URL is added in TMG under Published Sites.
TMG test rule will fail as it requires additional switches to be valid.
In my deployment we had multiple Edge Severs and sites - make sure that the meet URL is reachable across all sites, remember that the URL will be directed to the FE based on where the user is homed.

MCX Forbidden

Can't connect to Lync MCX service. Http Error 403 Forbidden, Lyncdiscover Http Authentication Test failed when testing https://<LyncWebService FQDN>/Mcx/McxService.svc

Also get Authentication Test failed from http://www.testocsconnectivity.com/

Error was the TMG rule

The error here says that the Credentials for the request to the site were deleted. It also explains how no delegation is set and user authentication isn't enabled. Of course this needs to be enabled!!!

IE Security

Default install of Windows 2008 internet explorer security blocks just about every page.

From the Server Manager, Deactivate IE Security as seen below

Frequent invalid SIP requests

Partners receiving a large number of errors in the Edge Server event log like below

The cause seems to be Lync still sending discovery packets every 10 minutes.
If federation is allowed, add the SIP domain to the allowed list, if blocked - add the SIP domain to the blocked list.
This will be followed by a final event entry stating that the problem has been resolved

Schema State check has failed.

Schema State check has failed. 

Both instances were linked back to DNS.
To prove that AD was healthy I ran the Prepare AD components directly from the DC (that works as usual)..which confirms that a DNS validation issue is present.
So what's going on with DNS?

Fisrtly an NSLOOKUP on the Lync box reveals that the default DNS server is unknown, adding a PTR record for the DNS server solves that.

Secondly, the installer queries the SRV records for contacting the PDC in active directory. This SRV record is: 
_ldap._tcp.pdc._msdcs. DnsDomainName 

UM Badmail

Actually this is more of a where is it than an issue ;-)
Where is the voicemail stored in UM before sending to Exchange? This includes the bad voicemail folder

C:\Program Files\Microsoft\Exchange Server\V14\UnifiedMessaging\...

Forcing Join Conference from Browser

Foreign user is sent a Lync online Meeting Request, if the invited user has Lync installed but doesn't have Federation capabilities the Join Conference request url will fail (since it calls the local Lync client to connect)

Force the conference invite URL to launch the Web and Lync Attendee options and not local Communicator Client (if present) -Just append this to the url   "?sl=1"

Lync Communicator Mobile wont login

Lync Communicator Mobile wont login

Error Message

Server unavailable at this time


On the Sign In page you enter your SIP Login name and password. However you also need to go to More Details (ios and WM7)\Options (Android) and add your user name. 

I have found that the username for WM7 needs to be Domain\User Name, although this format works on Android and ios simpoly adding the user name also works

PSTN Conferencing Error: Sorry, I can't seem to connect you to your meeting..."

While trying to call in to a conference from an external PSTN connection the error "Sorry, I can't seem to connect you to your meeting..."

Error Message
S4 traces on snooper revealed a "foreign gateway" IP address been called by the Mediation server.

The default Gateway in Topology Builder was an old (decommisioned) SIP connection (aka "foreign gateway". Changed that to the gateway I was actually using to call out on - solved!

Application Server keeps stopping

ApplicationServer (includes Call Park Service) Starts and then stops within seconds

Error Message

#1 make sure  localhost exists in hosts file
#2 For EE Server you need add both the FQDN of pool name and server name as SAN in the default certificate.