Help! Where do I put my Hybrid server?

As part of a hybrid Exchange server deployment, you also deploy the so-called Hybrid Server(s). The name itself might be a little misleading though. After all it’s not some sort of new Exchange server role, nor is it an Exchange server that you deploy specifically to be able to configure a hybrid environment – at least not if you’re already running Exchange 2010 or Exchange 2013 on-premises.

In fact, once you configure a hybrid environment, every Exchange Server in your environment becomes part of that hybrid deployment and will perform one, or more, functions in that regard. However, when referring to Hybrid Exchange servers, we actually mean the Exchange servers which are directly involved in hybrid functions. More specifically these will be the servers that you select during the Hybrid Configuration Wizard.

Exchange 2003 / 2007

If you have still Exchange 2003 on-premises (shame on you!), than your only option is to deploy at least one Exchange 2010 SP3 server and use that one to setup a hybrid deployment. The reason why you have to use an Exchange 2010 server is because Exchange 2013 cannot coexist with Exchange 2003.

Once you installed the Exchange 2010 server, it is the only server capable of understanding the hybrid logic; and therefore considered to be the Hybrid Server. There’s also another reason why a server would be referred to as your Hybrid Server, but more about that later when we’ll talk about the free Hybrid Server license key.

Hybrid Server License Key

Microsoft offers eligible customers free Hybrid Edition/Server licenses. Yes, indeed: multiple licenses if needed. In fact, you’ll get a single license key which you are allowed to deploy on multiple Exchange servers, for as long as you abide to the license requirements. This allows you to maintain high availability – also for hybrid functionality.

The license requirements tell you that you cannot use these ‘dedicated’ Hybrid Servers for anything else but that: you should not host any mailboxes on them. If you do, you are required to purchase a proper Exchange Server license. Once you assigned a Hybrid License to an Exchange server, that server also becomes a Hybrid Server in the pure sense of the word.

Hybrid Server Placement

When you are doing things by the book, introducing a new Exchange Server version could be a rather disruptive action. First, you have to prepare your environment for it (Active Directory schema updates etc) and then, once you have deployed the server, you are expected to point all client access traffic to it. This means that you will have to consider all the things involved with setting up coexistence. In smaller environments this might be a trivial task, but the larger the environment gets, the bigger the implications might be.

Although I prefer this approach (“by the book”), there are times where this isn’t appropriate. Even more, doing this might cause all sorts of issues which you might want to avoid – especially if you’re just looking for a quick way to move to the cloud. If so, the placement of the Hybrid Exchange can become a game changer.

One approach that I have used in the past is to install the new server into the Exchange organization and provide it with its own hybrid namespace. This hybrid namespace is nothing more than a dedicated namespace for hybrid functionality. By doing so, I prevent having to point client access traffic to the new servers and possibly disrupt my existing environment. I can then use the Hybrid Server(s) only     for mailbox moves, hybrid mail flow etc.

Multiple Internet-Connected sites

One of the tasks of hybrid servers is to facilitate mailbox moves to and from Exchange Online. The endpoint that you use for mailbox moves is normally discovered automatically using AutoDiscover. However, sometimes you might want to use Exchange Servers in a different location to perform the mailbox move. One of the reasons why you would want to do this is because that other server is maybe closer to the mailbox or it might have more bandwidth available.

When you want to use other internet-facing Exchange servers for mailbox moves, you must make sure that the MRS Proxy is enabled on those internet-facing servers. You can enable the MRS Proxy on each of these servers by executing the following command:

Set-WebServicesVirtualDirectory <identity> –MRSProxyEnabled:$true

Secondly, you could specify a new migration endpoint using PowerShell. This will allow you to pick your desired endpoint from the Mailbox Migration wizard as well (see image below). You can create new migration endpoints through PowerShell, using New-MigrationEdpoint cmdlet.

Once you have defined multiple migration endpoints, this is how it looks like in the GUI:

One thing to note here is that – regardless of the amount of migration endpoints you create – the sum of value of the “MaxConcurrentMigrations” attribute for all endpoints cannot exceed 100. The default endpoint (created automatically) will already have that set to 100. So make sure that you modify that first before creating additional endpoints.

The following image depicts the primary endpoint (outlook.domain.com) and the new secondary (and manually created) endpoint “migrationendpoint2.domain.com”:

Alternatively – if you don’t want to create additional endpoints or you plan on using that endpoint only once – you can create the move requests with PowerShell and specify the –RemoteHostname parameter manually.

Conclusion

Either approach outlined above should work just fine. Which one you choose greatly depends on your current deployment and the effort that goes with introducing a newer Exchange version into your environment. Whenever possible, try to take the by-the-book approach as it might save you some headaches further down the road.

Blog Exchange 2013 Hybrid Exchange Office 365

Upcoming speaking engagements

2014 promises to be a busy year, just like 2013. So far, I’ve had the opportunity to speak at the Microsoft Exchange Conference in Austin and soon I’ll be speaking at TechEd in Houston as well. Below are my recent and upcoming speaking engagements. If you’re attending any of these conferences, feel free to hit me up and have a chat!

-Michael

Pro-Exchange, Brussels, BE

Just last week, Pro-Exchange held another in-person event at Xylos in Brussels. The topic of the night was “best of MEC” where I presented the full 2.5 hours about all and everything that was interesting at the Microsoft Exchange Conference in Austin.

TechEd North America – Houston, TX

This year, I’ve got the opportunity to speak at TechEd in Houston. I’ll be presenting my “Configure a hybrid Exchange deployment in (less than) 75 minutes”.
The session takes place on Monday May 12 from 4:45 – 6:00 PM

More information: OFC-B312 Building a Hybrid Microsoft Exchange Server 2013 Deployment in Less than 75 Minutes

ITPROceed – Antwerp, BE

Given that Microsoft isn’t organizing any TechDays in Belgium, this year, the Belgian IT PRO community took matters in their own hands and created this free one-day event. It will take place on June 12th in Antwerp at ALM.
The conference consists of multiple tracks, amongst which also is “Office Server & Services” for which I will be presenting a session on “Exchange 2013 in the real world, from deployment to management”.

For more information, have a look at the official website here.

Blog Events TechEd NA 2014

The limitations of calendar federation in a hybrid deployment

Recently, Loryan Strant (Office 365 MVP) and myself joined forces to create an article for the Microsoft MVP blog regarding some of the limitations of calendar federation in a hybrid Exchange deployment. In this article we discuss how running a hybrid deployment might affect calendar sharing with other organizations and what your options are to work around this limitation.

To read the full article, please click here.

Enjoy!

Michael

Blog Exchange 2013 Hybrid Exchange Office 365

You get an error “the connection to the server <servername> could not be completed” when trying to start a hybrid mailbox move in Exchange 2013.

As part of running through the “New Migration Batch”-wizard, the remote endpoint (the on-premises Exchange server) is tested for its availability. After running this step, the following error is displayed:

image

By itself, this error message does not reveal much information as to what might be causing the connection issues. In the background, the wizard actually leverages the “Test-MigrationServerAvailability” cmdlet. If you run this cmdlet yourself, you will get a lot more information:

image

In this particular case, you’ll see that the issue is caused by 501 response from the on-premises server. The question is of course: why? We recently moved a number of mailboxes and then we did not encounter the issue. The only thing that had changed between then and now is that we reconfigured our load balancers in front of Exchange to use Layer 7 instead of Layer 4. So that is why I shifted my attention to the load balancers.

While reproducing the error, I took a look at the “System Message File” log in the KEMP load balancer. This log can be found under Logging Options, System Log Files. Although I didn’t expect to see much here, I saw the following message which drew my attention:

kernel: L7: badrequest-client_read [157.56.251.92:61541->192.168.2.130:443] (-501): <s:Envelope ? , 0 [hlen 1270, nhdrs 8]

A quick lookup learned that the 157.56.251.92 address was indeed coming from Microsoft. So now I knew for sure that something was wrong here. A quick search on the internet brought me to the following article which suggested to change the 100-Continue Handling in the Layer 7 configuration of the Load Master: http://blog.masteringmsuc.com/2013/10/kemp-load-balancer-and-lync-unified.html

After changing the value from its default (RFC Conformant), I could now successfully complete the wizard and start a hybrid mailbox move. So the “workaround” was found. But I was wondering, why does the Load Master *think* that the request coming from Microsoft is non-RFC compliant?

The first thing I did is ask Microsoft if they could clarify a bit on what was happening. I soon got a reply that – from Microsoft’s point of view – they were respecting the RFC documentation regarding the 100 (Continue) Status. No surprise here.

After reading the RFC specifications I decided to take some network traces to find out what was happening and maybe understand how the 501 response was triggered. The first trace I took, was one from the Load Master itself. In that trace, I could actually see the following:

image

Effectively, Office 365 was making a call to the Exchange Web Services and using the 100-continue status. As described per the RFC documentation, the Exchange on-premises server should now respond appropriately to the 100-continue status. Instead, we can see that in the entire SSL conversation, exactly 5 seconds go by after which Office 365 makes another call to the EWS virtual directory without having received a response to the 100-continue status. At the point, the KEMP Load Master generated the “501 Invalid Request”.

I turned back to the (by the way, excellent) support guys from KEMP and explained them my findings. Furthermore, when I tested without Layer 7 or even without a Load Master in between, there wasn’t a delay and everything was working as expected. So I knew for sure that the Exchange 2013 on-premises was actually replying correctly to the 100-continue status. As a matter of fact, without the KEMP LM in between, the entire ‘conversation’ between Office 365 and Exchange 2013 on-premises was perfectly following the RFC rules.

So, changing the 100-continue settings from “RFC Conformant” to “Ignore Continue-100” made sense as now KEMP would just ignore the 100-continue “rules”. But I was still interested in finding out why the LM thought the conversation was not RFC conformant in the first place. And this is where it gets interesting. There is this particular statement in the RFC documentation:

“Because of the presence of older implementations, the protocol allows ambiguous situations in which a client may send “Expect: 100- continue” without receiving either a 417 (Expectation Failed) status or a 100 (Continue) status. Therefore, when a client sends this header field to an origin server (possibly via a proxy) from which it has never seen a 100 (Continue) status, the client SHOULD NOT wait for an indefinite period before sending the request body.”

In fact, that was exactly what is happening here. Office 365 (the client) sent an initial 100-continue status and waited for a response to that request. In fact, it waits for exactly 5 seconds and sends the payload, regardless of it having received a response. In my opinion, this falls within the boundaries of the scenario described above. However, talking to the KEMP guys there seems to be a slightly different interpretation of the RFC which caused this mismatch and therefore the KEMP issuing the 501.

In the end, there is still something we haven’t worked out entirely: why the LM doesn’t send back the Continue-100 status back to Office 365 even though it receives it back almost instantaneously from the Exchange 2013 server.

All in all, the issue was resolved rather quickly and we know that changing the L7 configuration settings in the Load Master solves the issue (and this workaround was also confirmed as being the final solution by KEMP support, btw). Again, changing the 100-continue handling setting too “Ignore” doesn’t render the configuration (or the communication between Office 365 or Exchange on-premises) non-RFC compliant. So there’s no harm in changing it.

I hope you found this useful!

-Michael

Blog Exchange 2013 Hybrid Exchange Office 365

You get an error “StalledDueToMailboxLock” when moving mailboxes to Office 365 in a hybrid configuration.

Recently, a customer of ours experienced an issue while moving mailboxes to Office 365 in a Hybrid Configuration. After the move request was generated, they would end up with a status “StalledDueToMailboxLock” after a short while. The move request would then stay in that state for an indefinite amount of time.

Although I wasn’t able to test everything myself, first hand, a certain mailbox move was reported to proceed after been in the “StalledDueToMailboxLock” for about 2 days. Obviously, this isn’t the pace you’d want your mailboxes to be moved to Office 365 and some further investigation is needed.

Before any “deep” troubleshooting was done, we checked if there wasn’t something that could cause a mailbox lock, maybe a process that was blocking access to the mailbox like a misconfigured Anti-Virus. However, we soon found out that everything seemed normal…

Troubleshooting

Whenever a move requests fails or seems to be hanging in a specific state for a while, it’s always a good idea to take a look at the Move Request Statistics using “Get-MoveRequestStatistics”:

Get-MoveRequest <Identity> | Get-MoveRequestStatistics –IncludeReport | fl

The command above will immediately output the results on-screen. However, for requests that have been running for a while, the amount of information from the command is too much to handle through the console. As an alternative, you can either output it to e.g. a text file using Out-File or – and in my opinion this is much easier to handle – export it to an XML file using the Export-Clixml command:

Get-MoveRequest <Identity> | Get-MoveRequestStatistics –IncludeReport | Export-Clixml c:\moverequeststats.xml

Now, you can open that XML file in a more specialized XML-reader (or just Internet Explorer if you want) and go through its content.

Cause

In this particular case, the problem was caused by Exchange Web Services. Somewhere in the XML file (and multiple times), you’d find the following reference:

<S N=”Message”>Communication with remote service ‘https://autodiscover.company.be/EWS/mrsproxy.svc CAS1.fqdn (14.3.123.2 caps:05FFFF)’ has failed. –&gt; The call to ‘https://autodiscover.company.be/EWS/mrsproxy.svc CAS1.fqdn (14.3.123.2 caps:05FFFF)’ failed. Error details: The remote endpoint no longer recognizes this sequence. This is most likely due to an abort on the remote endpoint. The value of wsrm:Identifier is not a known Sequence identifier. The reliable session was faulted..

Similarly, we also found the exact same reference, but for another CAS:

<S N=”Message”>Communication with remote service ‘https://autodiscover.company.be/EWS/mrsproxy.svc CAS2.fqdn (14.3.123.2 caps:05FFFF)’ has failed. –&gt; The call to ‘https://autodiscover.company.be/EWS/mrsproxy.svc CAS2.fqdn (14.3.123.2 caps:05FFFF)’ failed. Error details: The remote endpoint no longer recognizes this sequence. This is most likely due to an abort on the remote endpoint. The value of wsrm:Identifier is not a known Sequence identifier. The reliable session was faulted..

This points out that the move request was trying to connect over different Client Access Servers. However, when keeping in mind the load balancing requirements for Exchange 2010, it’s clearly stated that using EWS without affinity is not supported:

Exchange Web Services   Only a subset of Exchange Web Services requires affinity. Availability Service requests don’t require affinity, but subscriptions do. All aspects of Exchange Web Services experience performance enhancements from affinity. An affinity timeout value of 45 minutes is recommended for Exchange Web Services clients to ensure that periodic polling for events associated with a subscription are not directed to a new Client Access server resulting in inefficient new subscriptions for each request. We don’t support the use of Exchange Web Services without affinity.

Edit note: as you might’ve guessed, this problem happened because on-premises an Exchange 2010 hybrid server was used. If that would’ve been an Exchange 2013 environment, this wouldn’t have happened as there’s no need to keep affinity for EWS.

Solution

So, because of this, we took another look at the load balancer configuration. To confirm whether or not it was the load balancer causing the issue, we temporarily bypassed them and connected directly to one of the CAS servers after which we found out that the move requests processed successfully.

The efforts now turned back to the Load Balancer and it was found out that the affinity for EWS, which was configured in the first place, wasn’t applied correctly. After correcting the settings in the Load Balancers everything worked as expected.

Acknowledgements

I would like to thank William Rall (MSFT) for his efforts and time. He quickly pointed out the root cause, which helped us determine what caused the problem. His feedback also allowed me to dive deeper into the XML and find what I was looking for. Admitted: diving into the XML without actually knowing what to look for, isn’t always easy!

Blog Exchange Hybrid Exchange

How to check what the version of your tenant is in the cloud (Office 365)

I sometimes get the question how one can verify what the version of Exchange they’re running in the cloud. Although it should be pretty obvious based on the GUI (Exchange 2010 vs. Exchange 2013) and the fact that the latter isn’t generally available yet, it could come in handy once it does. According to some sources, the release might be sooner than later.

Additionally, when you’re planning on going hybrid with the new version of Exchange in Office 365, you’ll have to make sure your tenant isn’t in the process of being upgraded and is running version 15.

To check the version, open up a PowerShell session to your Office 365 tenant and run the following command:
[sourcecode language=”powershell”]Get-OrganizationConfig | ft AdminDisplayVersion,IsUpgradingOrganization[/sourcecode]
With the command for connecting to Office 365 via PowerShell, that would look something like this:
[sourcecode language=”powershell”]$session = New-PSSession –ConnectionUri https://ps.outlook.com/powershell –AllowRedirection –Authentication Basic –Credential (Get-Credential) –ConfigurationName Microsoft.Exchange

Import-PSSession $session

Get-OrganizationConfig | ft AdminDisplayVersion,IsUpgradingOrganization[/sourcecode]

Running these commands would then lead up to a result similar to the following:

Get-OrganizationConfig | ft AdminDisplayVersion,IsUpgradingOrganization -Autosize

AdminDisplayVersion IsUpgradingOrganization
------------------- -----------------------
0.10 (14.16.190.8)                    False
How-To's Office 365

You get an error: “The <name> connector cycle has stopped. Object with DN <GUID> failed…”

As part of setting up a hybrid configuration between Exchange on-premise and Exchange online (or when configuring Exchange Online Archiving), you also need to setup DirSync.

In these scenarios DirSync fulfills an important role as it will also configure the write-back of some attributes in your local Active Directory. This “write-back” is required for Hybrid/EOA to work. For a list of attributes that are sync to/from Office 365, have a look at the following article: http://support.microsoft.com/kb/2256198

As part of the best-practices when installing DirSync, you should always run the Office 365 Deployment Readiness tool which will scan your local Active Directory and search for incompatible objects. The tool will create a report in which incompatible objects are mentioned. This will allow you to modify these object before configuring DirSync.

However, sometimes object can still contain incompatible object attributes, which might cause issues for DirSync. In such case, you’ll likely be presented with the following error in the application event log. Please note that this example mentions an issue with the “TargetWebService” Management Agent. It could very well be that you’ll encounter an issue in the SourceAD Management Agent.

The TargetWebService Connector export cycle has stopped.  Object with DN CN=<guid> failed validation for the following attributes: proxyAddresses. Please refer to documentation for information on object attribute validation.

This error contains 2 important items:

  1. The Distinguished name (CN=<guid>)
  2. The attribute that is causing issues

However, matching the guid to a user-account isn’t very easy. The best way to go about is to open the MIIS Management Interface and work from there. Usually, the client can be found in the following directory:

C:\Program Files\Microsoft Online Directory Sync\SYNCBUS\Synchronization Service\UIShell

image

After opening the client, navigate to Management Agents, right-click the management agent mentioned in the error message and select Search Connector Space:

image

In the “Search Connector Space” window, select DN or Anchor from the drop-down list under Scope and specify the Distinguished Name from the error message. Afterwards, click Search:

image

The search should return a single object. Double-click it to view additional information. Search for the attribute that was mentioned in the event log entry to review its value(s):

image

In this particular case, one of the proxy addresses contained an illegal character which caused the Management Agent to fail. Once you determined what the issue was, correct the value in AD and re-start synchronization. Normally, synchronization should happen successfully now.

How-To's Office 365