Hybrid Configration Wizard fails with error ‘Mail Flow Default Receive Connector cannot be found on server…’

Recently, I got asked to assist with a Hybrid Configuration Wizard which was failing with the following error message:

Updating hybrid configuration failed with error ‎’Subtask NeedsConfiguration execution failed: Configure Mail Flow Default Receive Connector cannot be found on server <server name>. at Microsoft.Exchange.Management.Hybrid.MailFlowTask.DoOnPremisesReceiveConnectorNeedConfiguration‎()‎ at…

Although the message might not reveal much information at first sight, it does contain everything we need to start troubleshooting. Typically, I would suggest you go and have a look into the Hybrid Configuration Wizard log files (located in the logging\Update-HybridConfiguration folder), but the only thing you would find there is the exact same error message.

First, we know that the HCW is trying to configure the hybrid mail flow and that it failed trying to modify the default connector that’s in place. More specifically, it was trying to modify the receive connector on the server that’s specified in the error message.

In this particular case, it wasn’t even able to find the Default Receive Connector. However, when you run the Get-ReceiveConnector -Server <servername>, the receive connector does show up. How is this possible?

The Hybrid Configuration Wizard looks at more specifics than just the existence of the connector. In fact, it will check that the connector’s configuration is valid as well. As such, it will check the bindings on the connector and expect that both bindings for IPv4 and IPv6 are present. So to check whether your existing connector is valid, you should run the following command:

Get-ReceiveConnector -Server <servername> | fl Identity,Bindings

receiveconnector

In this particular case, the IPv6 bindings were missing. This was caused because IPv6 was disabled on the server (which shouldn’t be!). Re-enabling IPv6 and then either manually adding the binding to the connector or re-creating the connector solved the issue.

The morale here is that you shouldn’t disable IPv6 on an Exchange 2013 box. Even more so, it’s not supported if you do. I’ve seen companies that still disable IPv6 by default; maybe a remainder from earlier times where disabling IPv6 would actually solve issues instead of creating them. However, times have changed and the IPv6 implementation in Windows is much better now…

Blog Exchange 2013 Hybrid Exchange

New Hybrid Configuration Wizard features in Exchange 2013 CU5

As posted here, Microsoft today released Cumulative Update 5 for Exchange 2013. At first sight, this update doesn’t appear to make lots of changes – at least not visibly. However, it does contain a lot of fixes and, as you will find out, there have been some changes to the Hybrid Configuration Wizard as well.

New options in the Hybrid Configuration Wizard

Whenever you enable an organization for a hybrid deployment in CU5, you will find the following new option:

21Vianet is Microsoft’s partner which offers Office 365 in China. You could say that they “host” Office 365 for Chinese customers as outlined in this Press Release

MRS Proxy now configured automatically

This is one of my personal asks for quite a long time now. Although the HCW already did an excellent job configuring all the components for a hybrid deployment, it did not enable the MRS Proxy on the Exchange Web Services Virtual Directory. Even though you could do it yourself with only a single command, I’m a big fan of having the HCW take care of this. It’s one less thing you can forget yourself!

OAuth now configured automatically

You’ll also notice that towards the end, the Hybrid Configuration Wizard will now prompt you to configure oAuth automatically:

The wizard will then automatically redirect you to a webpage where you’ll be asked to start the configuration (again):

Once you click configure, you will be asked to download an application which will automatically configure oAuth for you. Because it seems to be browser-integrated, you cannot run this step from a computer other than your Exchange Server and then copy over the executable. Beware and make sure that you run the HCW from the Exchange server itself instead from a remote workstation, like I tried the first time…

Once the first application was downloaded, you’ll be asked to run it:

Note: make sure that *.configure.office.com is added to your trusted sites or that you at least allow content to be downloaded from that website.

Then, after this first application ran, you’ll be prompted for an identical, second, application. Only this time the application (or assistant, if you will) will be a bit bigger: 22.2 MB instead of 18MB.

Once the second assistant completed successfully, you’ll see the following:

In fact, all that these “applications” do, is configure oAuth as outlined in the following article: http://technet.microsoft.com/en-us/library/dn594521(v=exchg.150).aspx

Note The configuration of the Intra-Organization Connector is the only thing that’s already handled by the Hybrid Configuration Wizard itself.

It’s definitely a good thing this is now done automatically. However, I would love to see it be more integrated with the HCW. At the moment, these changes don’t show up in the Hybrid Configuration Wizard logs.

Conclusion

It was already clear that Microsoft is moving forward with oAuth; potentially to replace other technologies currently used in Hybrid deployments. Personally, I wouldn’t be too surprised to see oAuth take over the duties from Microsoft’s Federation Gateway in the future. Not sure if this will actually happen, but it seems like a good thing. If you have ever been in a discussion with a pesky security administrator you would understand why… But don’t expect that to happen in a few months’ time though – as long as Exchange 2010 is officially supported, I reckon Microsoft will have to keep the MFG around.

It’s surely a good thing to move forward with oAuth as it has the potential to solve some long-standing issues regarding the handling of authentication and security in a cross-premises scenario like a hybrid deployment.

Blog Exchange 2013 Hybrid Exchange Office 365

Help! Where do I put my Hybrid server?

As part of a hybrid Exchange server deployment, you also deploy the so-called Hybrid Server(s). The name itself might be a little misleading though. After all it’s not some sort of new Exchange server role, nor is it an Exchange server that you deploy specifically to be able to configure a hybrid environment – at least not if you’re already running Exchange 2010 or Exchange 2013 on-premises.

In fact, once you configure a hybrid environment, every Exchange Server in your environment becomes part of that hybrid deployment and will perform one, or more, functions in that regard. However, when referring to Hybrid Exchange servers, we actually mean the Exchange servers which are directly involved in hybrid functions. More specifically these will be the servers that you select during the Hybrid Configuration Wizard.

Exchange 2003 / 2007

If you have still Exchange 2003 on-premises (shame on you!), than your only option is to deploy at least one Exchange 2010 SP3 server and use that one to setup a hybrid deployment. The reason why you have to use an Exchange 2010 server is because Exchange 2013 cannot coexist with Exchange 2003.

Once you installed the Exchange 2010 server, it is the only server capable of understanding the hybrid logic; and therefore considered to be the Hybrid Server. There’s also another reason why a server would be referred to as your Hybrid Server, but more about that later when we’ll talk about the free Hybrid Server license key.

Hybrid Server License Key

Microsoft offers eligible customers free Hybrid Edition/Server licenses. Yes, indeed: multiple licenses if needed. In fact, you’ll get a single license key which you are allowed to deploy on multiple Exchange servers, for as long as you abide to the license requirements. This allows you to maintain high availability – also for hybrid functionality.

The license requirements tell you that you cannot use these ‘dedicated’ Hybrid Servers for anything else but that: you should not host any mailboxes on them. If you do, you are required to purchase a proper Exchange Server license. Once you assigned a Hybrid License to an Exchange server, that server also becomes a Hybrid Server in the pure sense of the word.

Hybrid Server Placement

When you are doing things by the book, introducing a new Exchange Server version could be a rather disruptive action. First, you have to prepare your environment for it (Active Directory schema updates etc) and then, once you have deployed the server, you are expected to point all client access traffic to it. This means that you will have to consider all the things involved with setting up coexistence. In smaller environments this might be a trivial task, but the larger the environment gets, the bigger the implications might be.

Although I prefer this approach (“by the book”), there are times where this isn’t appropriate. Even more, doing this might cause all sorts of issues which you might want to avoid – especially if you’re just looking for a quick way to move to the cloud. If so, the placement of the Hybrid Exchange can become a game changer.

One approach that I have used in the past is to install the new server into the Exchange organization and provide it with its own hybrid namespace. This hybrid namespace is nothing more than a dedicated namespace for hybrid functionality. By doing so, I prevent having to point client access traffic to the new servers and possibly disrupt my existing environment. I can then use the Hybrid Server(s) only     for mailbox moves, hybrid mail flow etc.

Multiple Internet-Connected sites

One of the tasks of hybrid servers is to facilitate mailbox moves to and from Exchange Online. The endpoint that you use for mailbox moves is normally discovered automatically using AutoDiscover. However, sometimes you might want to use Exchange Servers in a different location to perform the mailbox move. One of the reasons why you would want to do this is because that other server is maybe closer to the mailbox or it might have more bandwidth available.

When you want to use other internet-facing Exchange servers for mailbox moves, you must make sure that the MRS Proxy is enabled on those internet-facing servers. You can enable the MRS Proxy on each of these servers by executing the following command:

Set-WebServicesVirtualDirectory <identity> –MRSProxyEnabled:$true

Secondly, you could specify a new migration endpoint using PowerShell. This will allow you to pick your desired endpoint from the Mailbox Migration wizard as well (see image below). You can create new migration endpoints through PowerShell, using New-MigrationEdpoint cmdlet.

Once you have defined multiple migration endpoints, this is how it looks like in the GUI:

One thing to note here is that – regardless of the amount of migration endpoints you create – the sum of value of the “MaxConcurrentMigrations” attribute for all endpoints cannot exceed 100. The default endpoint (created automatically) will already have that set to 100. So make sure that you modify that first before creating additional endpoints.

The following image depicts the primary endpoint (outlook.domain.com) and the new secondary (and manually created) endpoint “migrationendpoint2.domain.com”:

Alternatively – if you don’t want to create additional endpoints or you plan on using that endpoint only once – you can create the move requests with PowerShell and specify the –RemoteHostname parameter manually.

Conclusion

Either approach outlined above should work just fine. Which one you choose greatly depends on your current deployment and the effort that goes with introducing a newer Exchange version into your environment. Whenever possible, try to take the by-the-book approach as it might save you some headaches further down the road.

Blog Exchange 2013 Hybrid Exchange Office 365

Windows Server 2012 R2 ADFS ‘alternative login ID’, removes the need to have an internet-routable UPN

Recently, Microsoft released an update to Windows Server 2012 R2 which – next to a bunch of bug fixes – also includes new features to some of the Operating System’s components. Amongst these new features there’s one that I found particularly interesting, more specifically the update to the AD FS 3.0 component which enables customers to use a different attribute to identify federated uses in Windows Azure AD. The feature itself is better known as “Alternate Login ID”.

As the TechNet documentation on this topic describes, it would now be possible to use a different attributed from the User Principal Name to identify federated users in Office 365. This helps customers who aren’t able to change their UPNs from the current value (like e.g. domain.local or domain.corp) to an internet-routable domain (like domain.com). Even though that in many situations changing the UPN isn’t a big of a deal, some customers leverage the existing UPN in third party applications and therefore might not be able to make this change easily.

If you want to deploy this feature, you’ll have to figure some things out by yourself. The documentation that is currently available doesn’t explain all the steps. At least, that is if you want to implement it right away. I expect the documentation to become available shortly. Also mind that I haven’t seen any official statement that the use of “Alternate Login ID” is already supported by Office 365 today, but the documentation certainly hints to it and if I recall correctly, it was also announced at the Microsoft Exchange Conference, last week.

The configuration itself requires you to jump through a few hoops, including modifying DirSync to refer to the new attribute you’ve selected as being the Alternate Login ID instead of the UPN. Personally, I would still recommend changing the UPN – if possible. But there’s an alternative now and having alternative is always good thing, isn’t it?

I’ll definitely have a go at this later this week and will post my findings here.

-Michael

[Update 04/14/2014] Here’s the KB article describing the update I reference in this article: http://support.microsoft.com/kb/2927690

 

ADFS Blog Exchange Exchange 2013 Hybrid Exchange News Office 365

The limitations of calendar federation in a hybrid deployment

Recently, Loryan Strant (Office 365 MVP) and myself joined forces to create an article for the Microsoft MVP blog regarding some of the limitations of calendar federation in a hybrid Exchange deployment. In this article we discuss how running a hybrid deployment might affect calendar sharing with other organizations and what your options are to work around this limitation.

To read the full article, please click here.

Enjoy!

Michael

Blog Exchange 2013 Hybrid Exchange Office 365

You get an error “the connection to the server <servername> could not be completed” when trying to start a hybrid mailbox move in Exchange 2013.

As part of running through the “New Migration Batch”-wizard, the remote endpoint (the on-premises Exchange server) is tested for its availability. After running this step, the following error is displayed:

image

By itself, this error message does not reveal much information as to what might be causing the connection issues. In the background, the wizard actually leverages the “Test-MigrationServerAvailability” cmdlet. If you run this cmdlet yourself, you will get a lot more information:

image

In this particular case, you’ll see that the issue is caused by 501 response from the on-premises server. The question is of course: why? We recently moved a number of mailboxes and then we did not encounter the issue. The only thing that had changed between then and now is that we reconfigured our load balancers in front of Exchange to use Layer 7 instead of Layer 4. So that is why I shifted my attention to the load balancers.

While reproducing the error, I took a look at the “System Message File” log in the KEMP load balancer. This log can be found under Logging Options, System Log Files. Although I didn’t expect to see much here, I saw the following message which drew my attention:

kernel: L7: badrequest-client_read [157.56.251.92:61541->192.168.2.130:443] (-501): <s:Envelope ? , 0 [hlen 1270, nhdrs 8]

A quick lookup learned that the 157.56.251.92 address was indeed coming from Microsoft. So now I knew for sure that something was wrong here. A quick search on the internet brought me to the following article which suggested to change the 100-Continue Handling in the Layer 7 configuration of the Load Master: http://blog.masteringmsuc.com/2013/10/kemp-load-balancer-and-lync-unified.html

After changing the value from its default (RFC Conformant), I could now successfully complete the wizard and start a hybrid mailbox move. So the “workaround” was found. But I was wondering, why does the Load Master *think* that the request coming from Microsoft is non-RFC compliant?

The first thing I did is ask Microsoft if they could clarify a bit on what was happening. I soon got a reply that – from Microsoft’s point of view – they were respecting the RFC documentation regarding the 100 (Continue) Status. No surprise here.

After reading the RFC specifications I decided to take some network traces to find out what was happening and maybe understand how the 501 response was triggered. The first trace I took, was one from the Load Master itself. In that trace, I could actually see the following:

image

Effectively, Office 365 was making a call to the Exchange Web Services and using the 100-continue status. As described per the RFC documentation, the Exchange on-premises server should now respond appropriately to the 100-continue status. Instead, we can see that in the entire SSL conversation, exactly 5 seconds go by after which Office 365 makes another call to the EWS virtual directory without having received a response to the 100-continue status. At the point, the KEMP Load Master generated the “501 Invalid Request”.

I turned back to the (by the way, excellent) support guys from KEMP and explained them my findings. Furthermore, when I tested without Layer 7 or even without a Load Master in between, there wasn’t a delay and everything was working as expected. So I knew for sure that the Exchange 2013 on-premises was actually replying correctly to the 100-continue status. As a matter of fact, without the KEMP LM in between, the entire ‘conversation’ between Office 365 and Exchange 2013 on-premises was perfectly following the RFC rules.

So, changing the 100-continue settings from “RFC Conformant” to “Ignore Continue-100” made sense as now KEMP would just ignore the 100-continue “rules”. But I was still interested in finding out why the LM thought the conversation was not RFC conformant in the first place. And this is where it gets interesting. There is this particular statement in the RFC documentation:

“Because of the presence of older implementations, the protocol allows ambiguous situations in which a client may send “Expect: 100- continue” without receiving either a 417 (Expectation Failed) status or a 100 (Continue) status. Therefore, when a client sends this header field to an origin server (possibly via a proxy) from which it has never seen a 100 (Continue) status, the client SHOULD NOT wait for an indefinite period before sending the request body.”

In fact, that was exactly what is happening here. Office 365 (the client) sent an initial 100-continue status and waited for a response to that request. In fact, it waits for exactly 5 seconds and sends the payload, regardless of it having received a response. In my opinion, this falls within the boundaries of the scenario described above. However, talking to the KEMP guys there seems to be a slightly different interpretation of the RFC which caused this mismatch and therefore the KEMP issuing the 501.

In the end, there is still something we haven’t worked out entirely: why the LM doesn’t send back the Continue-100 status back to Office 365 even though it receives it back almost instantaneously from the Exchange 2013 server.

All in all, the issue was resolved rather quickly and we know that changing the L7 configuration settings in the Load Master solves the issue (and this workaround was also confirmed as being the final solution by KEMP support, btw). Again, changing the 100-continue handling setting too “Ignore” doesn’t render the configuration (or the communication between Office 365 or Exchange on-premises) non-RFC compliant. So there’s no harm in changing it.

I hope you found this useful!

-Michael

Blog Exchange 2013 Hybrid Exchange Office 365

Free/Busy in a hybrid environment fail and Test-Federationtrust returns error “Failed to validate delegation token”

Following an issue with Free/Busy in Exchange online, earlier this week, I was troubleshooting the exchange of Free/Busy information in some of my hybrid deployments as Free/Busy information was still not working.
After having checked some (obvious things) like the Organization Relationships and whether or not Autodiscover was working properly, I discovered an issue when running the Test-FederationTrust cmdlet.

In fact, the cmdlet completed almost entirely successful, except for the very last step in the process:

Id         : TokenValidation
Type       : Error
Message    : Failed to validate delegation token.

This also explained why I was seeing 401 Unauthorized messages when running the Test-OrganizationRelationship command.

I then checked the same in some of my other deployments and found out the all had the same issue. At least, there was some common ground to start working from.
I turned to co-MVP Steve Goodman and asked him to run the same command in one of his labs in order to have a point of reference. At the same time, he asked me to run a command which might help:

Get-FederationTrust | Set-Federationtrust –RefreshMetaData

After running the command, I re-ran the Test-FederationTrust command which now completed successfully.

Conclusion

Although the Free/Busy issues in Office 365 should be solved, some customers might still experience problems exchanging Free/Busy information. In this case, the problem manifests itself by e.g. online users not being able to request on-premises user’s availability information.

Blog Exchange Hybrid Exchange Office 365