Hybrid Configration Wizard fails with error ‘Mail Flow Default Receive Connector cannot be found on server…’

Recently, I got asked to assist with a Hybrid Configuration Wizard which was failing with the following error message:

Updating hybrid configuration failed with error ‎’Subtask NeedsConfiguration execution failed: Configure Mail Flow Default Receive Connector cannot be found on server <server name>. at Microsoft.Exchange.Management.Hybrid.MailFlowTask.DoOnPremisesReceiveConnectorNeedConfiguration‎()‎ at…

Although the message might not reveal much information at first sight, it does contain everything we need to start troubleshooting. Typically, I would suggest you go and have a look into the Hybrid Configuration Wizard log files (located in the logging\Update-HybridConfiguration folder), but the only thing you would find there is the exact same error message.

First, we know that the HCW is trying to configure the hybrid mail flow and that it failed trying to modify the default connector that’s in place. More specifically, it was trying to modify the receive connector on the server that’s specified in the error message.

In this particular case, it wasn’t even able to find the Default Receive Connector. However, when you run the Get-ReceiveConnector -Server <servername>, the receive connector does show up. How is this possible?

The Hybrid Configuration Wizard looks at more specifics than just the existence of the connector. In fact, it will check that the connector’s configuration is valid as well. As such, it will check the bindings on the connector and expect that both bindings for IPv4 and IPv6 are present. So to check whether your existing connector is valid, you should run the following command:

Get-ReceiveConnector -Server <servername> | fl Identity,Bindings

receiveconnector

In this particular case, the IPv6 bindings were missing. This was caused because IPv6 was disabled on the server (which shouldn’t be!). Re-enabling IPv6 and then either manually adding the binding to the connector or re-creating the connector solved the issue.

The morale here is that you shouldn’t disable IPv6 on an Exchange 2013 box. Even more so, it’s not supported if you do. I’ve seen companies that still disable IPv6 by default; maybe a remainder from earlier times where disabling IPv6 would actually solve issues instead of creating them. However, times have changed and the IPv6 implementation in Windows is much better now…

Blog Exchange 2013 Hybrid Exchange

New Hybrid Configuration Wizard features in Exchange 2013 CU5

As posted here, Microsoft today released Cumulative Update 5 for Exchange 2013. At first sight, this update doesn’t appear to make lots of changes – at least not visibly. However, it does contain a lot of fixes and, as you will find out, there have been some changes to the Hybrid Configuration Wizard as well.

New options in the Hybrid Configuration Wizard

Whenever you enable an organization for a hybrid deployment in CU5, you will find the following new option:

21Vianet is Microsoft’s partner which offers Office 365 in China. You could say that they “host” Office 365 for Chinese customers as outlined in this Press Release

MRS Proxy now configured automatically

This is one of my personal asks for quite a long time now. Although the HCW already did an excellent job configuring all the components for a hybrid deployment, it did not enable the MRS Proxy on the Exchange Web Services Virtual Directory. Even though you could do it yourself with only a single command, I’m a big fan of having the HCW take care of this. It’s one less thing you can forget yourself!

OAuth now configured automatically

You’ll also notice that towards the end, the Hybrid Configuration Wizard will now prompt you to configure oAuth automatically:

The wizard will then automatically redirect you to a webpage where you’ll be asked to start the configuration (again):

Once you click configure, you will be asked to download an application which will automatically configure oAuth for you. Because it seems to be browser-integrated, you cannot run this step from a computer other than your Exchange Server and then copy over the executable. Beware and make sure that you run the HCW from the Exchange server itself instead from a remote workstation, like I tried the first time…

Once the first application was downloaded, you’ll be asked to run it:

Note: make sure that *.configure.office.com is added to your trusted sites or that you at least allow content to be downloaded from that website.

Then, after this first application ran, you’ll be prompted for an identical, second, application. Only this time the application (or assistant, if you will) will be a bit bigger: 22.2 MB instead of 18MB.

Once the second assistant completed successfully, you’ll see the following:

In fact, all that these “applications” do, is configure oAuth as outlined in the following article: http://technet.microsoft.com/en-us/library/dn594521(v=exchg.150).aspx

Note The configuration of the Intra-Organization Connector is the only thing that’s already handled by the Hybrid Configuration Wizard itself.

It’s definitely a good thing this is now done automatically. However, I would love to see it be more integrated with the HCW. At the moment, these changes don’t show up in the Hybrid Configuration Wizard logs.

Conclusion

It was already clear that Microsoft is moving forward with oAuth; potentially to replace other technologies currently used in Hybrid deployments. Personally, I wouldn’t be too surprised to see oAuth take over the duties from Microsoft’s Federation Gateway in the future. Not sure if this will actually happen, but it seems like a good thing. If you have ever been in a discussion with a pesky security administrator you would understand why… But don’t expect that to happen in a few months’ time though – as long as Exchange 2010 is officially supported, I reckon Microsoft will have to keep the MFG around.

It’s surely a good thing to move forward with oAuth as it has the potential to solve some long-standing issues regarding the handling of authentication and security in a cross-premises scenario like a hybrid deployment.

Blog Exchange 2013 Hybrid Exchange Office 365

Microsoft releases Exchange 2013 Cumulative Update 5 and Exchange 2010 Update Rollup 6

Today, Microsoft released Cumulative Update 5 for Exchange 2013 and Update Rollup 6 for Exchange 2010.

Exchange 2013 Cumulative Update 5

Next to a ton of bug fixes, Microsoft made changes to a few components including:

  • Offline Address Book generation
  • Hybrid Configuration Wizard

Except for the above changes, it looks like CU5 will mostly consist of fixes. By the looks of it and as Tony Redmond already pointed out CU5 promises to be a stable release. Whether it will stay that way is something only time will tell…

Installing Cumulative Update 5

Installing CU5 is no different from older versions. You can also immediately upgrade from any previous version of Exchange 2013 to CU5. There is no requirement to install SP1 (a.k.a. CU4) first.

After installation, Microsoft warns there might be a Managed Availability probe which went into overdrive and repeatedly restarts a newly added service called the Microsoft Exchange Shared Cache Service. However, this service isn’t used in CU5 (planned for the future?) and as such there is no impact at all.

However, if you are worried about your application log filling up with events from Managed Availability, you can disable the probe. More information can be found here.

This update also includes Active Directory changes, so you will be required to extend the AD schema. Given that you’re used to it by now, this shouldn’t present much of a problem. For more information on how to deploy a Cumulative Update, I suggest you have a look at the following article by ExchangeServerPro: 

Installing Cumulative Updates and Service Packs for Exchange Server 2013

You can download Cumulative Update 5 from here. The original release announcement is here.

Exchange 2010 Update Rollup 6

This update seems mainly to be a routine update to Exchange 2010. As expected, there are no major revelations except for a bunch of updates and fixes:

  • 2960652 Organizer name and meeting status field can be changed by EAS clients in an Exchange Server 2010 environment
  • 2957762 “A folder with same name already exists” error when you rename an Outlook folder in an Exchange Server 2010 environment
  • 2952799 Event ID 2084 occurs and Exchange server loses connection to the domain controllers in an Exchange Server 2010 environment
  • 2934091 Event ID 1000 and 7031 when users cannot connect to mailboxes in an Exchange Server 2010 environment
  • 2932402 Cannot move a mailbox after you install Exchange Server 2010 SP3 RU3 (KB2891587)
  • 2931842 EWS cannot identify the attachment in an Exchange Server 2010 environment
  • 2928703 Retention policy is applied unexpectedly to a folder when Outlook rule moves a copy in Exchange Server 2010
  • 2927265 Get-Message cmdlet does not respect the defined write scope in Exchange Server 2010
  • 2925273 Folder views are not updated when you arrange by categories in Outlook after you apply Exchange Server 2010 Service Pack 3 Update Rollup 3 or Update Rollup 4
  • 2924592 Exchange RPC Client Access service freezes when you open an attached file in Outlook Online mode in Exchange Server 2010
  • 2923865 Cannot connect to Exchange Server 2010 when the RPC Client Access service crashes

You can download Rollop Update 6 from here.

Microsoft’s original release announcements can be found here.

Blog Exchange Exchange 2013 News

Help! Where do I put my Hybrid server?

As part of a hybrid Exchange server deployment, you also deploy the so-called Hybrid Server(s). The name itself might be a little misleading though. After all it’s not some sort of new Exchange server role, nor is it an Exchange server that you deploy specifically to be able to configure a hybrid environment – at least not if you’re already running Exchange 2010 or Exchange 2013 on-premises.

In fact, once you configure a hybrid environment, every Exchange Server in your environment becomes part of that hybrid deployment and will perform one, or more, functions in that regard. However, when referring to Hybrid Exchange servers, we actually mean the Exchange servers which are directly involved in hybrid functions. More specifically these will be the servers that you select during the Hybrid Configuration Wizard.

Exchange 2003 / 2007

If you have still Exchange 2003 on-premises (shame on you!), than your only option is to deploy at least one Exchange 2010 SP3 server and use that one to setup a hybrid deployment. The reason why you have to use an Exchange 2010 server is because Exchange 2013 cannot coexist with Exchange 2003.

Once you installed the Exchange 2010 server, it is the only server capable of understanding the hybrid logic; and therefore considered to be the Hybrid Server. There’s also another reason why a server would be referred to as your Hybrid Server, but more about that later when we’ll talk about the free Hybrid Server license key.

Hybrid Server License Key

Microsoft offers eligible customers free Hybrid Edition/Server licenses. Yes, indeed: multiple licenses if needed. In fact, you’ll get a single license key which you are allowed to deploy on multiple Exchange servers, for as long as you abide to the license requirements. This allows you to maintain high availability – also for hybrid functionality.

The license requirements tell you that you cannot use these ‘dedicated’ Hybrid Servers for anything else but that: you should not host any mailboxes on them. If you do, you are required to purchase a proper Exchange Server license. Once you assigned a Hybrid License to an Exchange server, that server also becomes a Hybrid Server in the pure sense of the word.

Hybrid Server Placement

When you are doing things by the book, introducing a new Exchange Server version could be a rather disruptive action. First, you have to prepare your environment for it (Active Directory schema updates etc) and then, once you have deployed the server, you are expected to point all client access traffic to it. This means that you will have to consider all the things involved with setting up coexistence. In smaller environments this might be a trivial task, but the larger the environment gets, the bigger the implications might be.

Although I prefer this approach (“by the book”), there are times where this isn’t appropriate. Even more, doing this might cause all sorts of issues which you might want to avoid – especially if you’re just looking for a quick way to move to the cloud. If so, the placement of the Hybrid Exchange can become a game changer.

One approach that I have used in the past is to install the new server into the Exchange organization and provide it with its own hybrid namespace. This hybrid namespace is nothing more than a dedicated namespace for hybrid functionality. By doing so, I prevent having to point client access traffic to the new servers and possibly disrupt my existing environment. I can then use the Hybrid Server(s) only     for mailbox moves, hybrid mail flow etc.

Multiple Internet-Connected sites

One of the tasks of hybrid servers is to facilitate mailbox moves to and from Exchange Online. The endpoint that you use for mailbox moves is normally discovered automatically using AutoDiscover. However, sometimes you might want to use Exchange Servers in a different location to perform the mailbox move. One of the reasons why you would want to do this is because that other server is maybe closer to the mailbox or it might have more bandwidth available.

When you want to use other internet-facing Exchange servers for mailbox moves, you must make sure that the MRS Proxy is enabled on those internet-facing servers. You can enable the MRS Proxy on each of these servers by executing the following command:

Set-WebServicesVirtualDirectory <identity> –MRSProxyEnabled:$true

Secondly, you could specify a new migration endpoint using PowerShell. This will allow you to pick your desired endpoint from the Mailbox Migration wizard as well (see image below). You can create new migration endpoints through PowerShell, using New-MigrationEdpoint cmdlet.

Once you have defined multiple migration endpoints, this is how it looks like in the GUI:

One thing to note here is that – regardless of the amount of migration endpoints you create – the sum of value of the “MaxConcurrentMigrations” attribute for all endpoints cannot exceed 100. The default endpoint (created automatically) will already have that set to 100. So make sure that you modify that first before creating additional endpoints.

The following image depicts the primary endpoint (outlook.domain.com) and the new secondary (and manually created) endpoint “migrationendpoint2.domain.com”:

Alternatively – if you don’t want to create additional endpoints or you plan on using that endpoint only once – you can create the move requests with PowerShell and specify the –RemoteHostname parameter manually.

Conclusion

Either approach outlined above should work just fine. Which one you choose greatly depends on your current deployment and the effort that goes with introducing a newer Exchange version into your environment. Whenever possible, try to take the by-the-book approach as it might save you some headaches further down the road.

Blog Exchange 2013 Hybrid Exchange Office 365

Upcoming speaking engagements

2014 promises to be a busy year, just like 2013. So far, I’ve had the opportunity to speak at the Microsoft Exchange Conference in Austin and soon I’ll be speaking at TechEd in Houston as well. Below are my recent and upcoming speaking engagements. If you’re attending any of these conferences, feel free to hit me up and have a chat!

-Michael

Pro-Exchange, Brussels, BE

Just last week, Pro-Exchange held another in-person event at Xylos in Brussels. The topic of the night was “best of MEC” where I presented the full 2.5 hours about all and everything that was interesting at the Microsoft Exchange Conference in Austin.

TechEd North America – Houston, TX

This year, I’ve got the opportunity to speak at TechEd in Houston. I’ll be presenting my “Configure a hybrid Exchange deployment in (less than) 75 minutes”.
The session takes place on Monday May 12 from 4:45 – 6:00 PM

More information: OFC-B312 Building a Hybrid Microsoft Exchange Server 2013 Deployment in Less than 75 Minutes

ITPROceed – Antwerp, BE

Given that Microsoft isn’t organizing any TechDays in Belgium, this year, the Belgian IT PRO community took matters in their own hands and created this free one-day event. It will take place on June 12th in Antwerp at ALM.
The conference consists of multiple tracks, amongst which also is “Office Server & Services” for which I will be presenting a session on “Exchange 2013 in the real world, from deployment to management”.

For more information, have a look at the official website here.

Blog Events TechEd NA 2014

The limitations of calendar federation in a hybrid deployment

Recently, Loryan Strant (Office 365 MVP) and myself joined forces to create an article for the Microsoft MVP blog regarding some of the limitations of calendar federation in a hybrid Exchange deployment. In this article we discuss how running a hybrid deployment might affect calendar sharing with other organizations and what your options are to work around this limitation.

To read the full article, please click here.

Enjoy!

Michael

Blog Exchange 2013 Hybrid Exchange Office 365

You get an error “the connection to the server <servername> could not be completed” when trying to start a hybrid mailbox move in Exchange 2013.

As part of running through the “New Migration Batch”-wizard, the remote endpoint (the on-premises Exchange server) is tested for its availability. After running this step, the following error is displayed:

image

By itself, this error message does not reveal much information as to what might be causing the connection issues. In the background, the wizard actually leverages the “Test-MigrationServerAvailability” cmdlet. If you run this cmdlet yourself, you will get a lot more information:

image

In this particular case, you’ll see that the issue is caused by 501 response from the on-premises server. The question is of course: why? We recently moved a number of mailboxes and then we did not encounter the issue. The only thing that had changed between then and now is that we reconfigured our load balancers in front of Exchange to use Layer 7 instead of Layer 4. So that is why I shifted my attention to the load balancers.

While reproducing the error, I took a look at the “System Message File” log in the KEMP load balancer. This log can be found under Logging Options, System Log Files. Although I didn’t expect to see much here, I saw the following message which drew my attention:

kernel: L7: badrequest-client_read [157.56.251.92:61541->192.168.2.130:443] (-501): <s:Envelope ? , 0 [hlen 1270, nhdrs 8]

A quick lookup learned that the 157.56.251.92 address was indeed coming from Microsoft. So now I knew for sure that something was wrong here. A quick search on the internet brought me to the following article which suggested to change the 100-Continue Handling in the Layer 7 configuration of the Load Master: http://blog.masteringmsuc.com/2013/10/kemp-load-balancer-and-lync-unified.html

After changing the value from its default (RFC Conformant), I could now successfully complete the wizard and start a hybrid mailbox move. So the “workaround” was found. But I was wondering, why does the Load Master *think* that the request coming from Microsoft is non-RFC compliant?

The first thing I did is ask Microsoft if they could clarify a bit on what was happening. I soon got a reply that – from Microsoft’s point of view – they were respecting the RFC documentation regarding the 100 (Continue) Status. No surprise here.

After reading the RFC specifications I decided to take some network traces to find out what was happening and maybe understand how the 501 response was triggered. The first trace I took, was one from the Load Master itself. In that trace, I could actually see the following:

image

Effectively, Office 365 was making a call to the Exchange Web Services and using the 100-continue status. As described per the RFC documentation, the Exchange on-premises server should now respond appropriately to the 100-continue status. Instead, we can see that in the entire SSL conversation, exactly 5 seconds go by after which Office 365 makes another call to the EWS virtual directory without having received a response to the 100-continue status. At the point, the KEMP Load Master generated the “501 Invalid Request”.

I turned back to the (by the way, excellent) support guys from KEMP and explained them my findings. Furthermore, when I tested without Layer 7 or even without a Load Master in between, there wasn’t a delay and everything was working as expected. So I knew for sure that the Exchange 2013 on-premises was actually replying correctly to the 100-continue status. As a matter of fact, without the KEMP LM in between, the entire ‘conversation’ between Office 365 and Exchange 2013 on-premises was perfectly following the RFC rules.

So, changing the 100-continue settings from “RFC Conformant” to “Ignore Continue-100” made sense as now KEMP would just ignore the 100-continue “rules”. But I was still interested in finding out why the LM thought the conversation was not RFC conformant in the first place. And this is where it gets interesting. There is this particular statement in the RFC documentation:

“Because of the presence of older implementations, the protocol allows ambiguous situations in which a client may send “Expect: 100- continue” without receiving either a 417 (Expectation Failed) status or a 100 (Continue) status. Therefore, when a client sends this header field to an origin server (possibly via a proxy) from which it has never seen a 100 (Continue) status, the client SHOULD NOT wait for an indefinite period before sending the request body.”

In fact, that was exactly what is happening here. Office 365 (the client) sent an initial 100-continue status and waited for a response to that request. In fact, it waits for exactly 5 seconds and sends the payload, regardless of it having received a response. In my opinion, this falls within the boundaries of the scenario described above. However, talking to the KEMP guys there seems to be a slightly different interpretation of the RFC which caused this mismatch and therefore the KEMP issuing the 501.

In the end, there is still something we haven’t worked out entirely: why the LM doesn’t send back the Continue-100 status back to Office 365 even though it receives it back almost instantaneously from the Exchange 2013 server.

All in all, the issue was resolved rather quickly and we know that changing the L7 configuration settings in the Load Master solves the issue (and this workaround was also confirmed as being the final solution by KEMP support, btw). Again, changing the 100-continue handling setting too “Ignore” doesn’t render the configuration (or the communication between Office 365 or Exchange on-premises) non-RFC compliant. So there’s no harm in changing it.

I hope you found this useful!

-Michael

Blog Exchange 2013 Hybrid Exchange Office 365