Help! Where do I put my Hybrid server?

As part of a hybrid Exchange server deployment, you also deploy the so-called Hybrid Server(s). The name itself might be a little misleading though. After all it’s not some sort of new Exchange server role, nor is it an Exchange server that you deploy specifically to be able to configure a hybrid environment – at least not if you’re already running Exchange 2010 or Exchange 2013 on-premises.

In fact, once you configure a hybrid environment, every Exchange Server in your environment becomes part of that hybrid deployment and will perform one, or more, functions in that regard. However, when referring to Hybrid Exchange servers, we actually mean the Exchange servers which are directly involved in hybrid functions. More specifically these will be the servers that you select during the Hybrid Configuration Wizard.

Exchange 2003 / 2007

If you have still Exchange 2003 on-premises (shame on you!), than your only option is to deploy at least one Exchange 2010 SP3 server and use that one to setup a hybrid deployment. The reason why you have to use an Exchange 2010 server is because Exchange 2013 cannot coexist with Exchange 2003.

Once you installed the Exchange 2010 server, it is the only server capable of understanding the hybrid logic; and therefore considered to be the Hybrid Server. There’s also another reason why a server would be referred to as your Hybrid Server, but more about that later when we’ll talk about the free Hybrid Server license key.

Hybrid Server License Key

Microsoft offers eligible customers free Hybrid Edition/Server licenses. Yes, indeed: multiple licenses if needed. In fact, you’ll get a single license key which you are allowed to deploy on multiple Exchange servers, for as long as you abide to the license requirements. This allows you to maintain high availability – also for hybrid functionality.

The license requirements tell you that you cannot use these ‘dedicated’ Hybrid Servers for anything else but that: you should not host any mailboxes on them. If you do, you are required to purchase a proper Exchange Server license. Once you assigned a Hybrid License to an Exchange server, that server also becomes a Hybrid Server in the pure sense of the word.

Hybrid Server Placement

When you are doing things by the book, introducing a new Exchange Server version could be a rather disruptive action. First, you have to prepare your environment for it (Active Directory schema updates etc) and then, once you have deployed the server, you are expected to point all client access traffic to it. This means that you will have to consider all the things involved with setting up coexistence. In smaller environments this might be a trivial task, but the larger the environment gets, the bigger the implications might be.

Although I prefer this approach (“by the book”), there are times where this isn’t appropriate. Even more, doing this might cause all sorts of issues which you might want to avoid – especially if you’re just looking for a quick way to move to the cloud. If so, the placement of the Hybrid Exchange can become a game changer.

One approach that I have used in the past is to install the new server into the Exchange organization and provide it with its own hybrid namespace. This hybrid namespace is nothing more than a dedicated namespace for hybrid functionality. By doing so, I prevent having to point client access traffic to the new servers and possibly disrupt my existing environment. I can then use the Hybrid Server(s) only     for mailbox moves, hybrid mail flow etc.

Multiple Internet-Connected sites

One of the tasks of hybrid servers is to facilitate mailbox moves to and from Exchange Online. The endpoint that you use for mailbox moves is normally discovered automatically using AutoDiscover. However, sometimes you might want to use Exchange Servers in a different location to perform the mailbox move. One of the reasons why you would want to do this is because that other server is maybe closer to the mailbox or it might have more bandwidth available.

When you want to use other internet-facing Exchange servers for mailbox moves, you must make sure that the MRS Proxy is enabled on those internet-facing servers. You can enable the MRS Proxy on each of these servers by executing the following command:

Set-WebServicesVirtualDirectory <identity> –MRSProxyEnabled:$true

Secondly, you could specify a new migration endpoint using PowerShell. This will allow you to pick your desired endpoint from the Mailbox Migration wizard as well (see image below). You can create new migration endpoints through PowerShell, using New-MigrationEdpoint cmdlet.

Once you have defined multiple migration endpoints, this is how it looks like in the GUI:

One thing to note here is that – regardless of the amount of migration endpoints you create – the sum of value of the “MaxConcurrentMigrations” attribute for all endpoints cannot exceed 100. The default endpoint (created automatically) will already have that set to 100. So make sure that you modify that first before creating additional endpoints.

The following image depicts the primary endpoint (outlook.domain.com) and the new secondary (and manually created) endpoint “migrationendpoint2.domain.com”:

Alternatively – if you don’t want to create additional endpoints or you plan on using that endpoint only once – you can create the move requests with PowerShell and specify the –RemoteHostname parameter manually.

Conclusion

Either approach outlined above should work just fine. Which one you choose greatly depends on your current deployment and the effort that goes with introducing a newer Exchange version into your environment. Whenever possible, try to take the by-the-book approach as it might save you some headaches further down the road.

Blog Exchange 2013 Hybrid Exchange Office 365

Windows Server 2012 R2 ADFS ‘alternative login ID’, removes the need to have an internet-routable UPN

Recently, Microsoft released an update to Windows Server 2012 R2 which – next to a bunch of bug fixes – also includes new features to some of the Operating System’s components. Amongst these new features there’s one that I found particularly interesting, more specifically the update to the AD FS 3.0 component which enables customers to use a different attribute to identify federated uses in Windows Azure AD. The feature itself is better known as “Alternate Login ID”.

As the TechNet documentation on this topic describes, it would now be possible to use a different attributed from the User Principal Name to identify federated users in Office 365. This helps customers who aren’t able to change their UPNs from the current value (like e.g. domain.local or domain.corp) to an internet-routable domain (like domain.com). Even though that in many situations changing the UPN isn’t a big of a deal, some customers leverage the existing UPN in third party applications and therefore might not be able to make this change easily.

If you want to deploy this feature, you’ll have to figure some things out by yourself. The documentation that is currently available doesn’t explain all the steps. At least, that is if you want to implement it right away. I expect the documentation to become available shortly. Also mind that I haven’t seen any official statement that the use of “Alternate Login ID” is already supported by Office 365 today, but the documentation certainly hints to it and if I recall correctly, it was also announced at the Microsoft Exchange Conference, last week.

The configuration itself requires you to jump through a few hoops, including modifying DirSync to refer to the new attribute you’ve selected as being the Alternate Login ID instead of the UPN. Personally, I would still recommend changing the UPN – if possible. But there’s an alternative now and having alternative is always good thing, isn’t it?

I’ll definitely have a go at this later this week and will post my findings here.

-Michael

[Update 04/14/2014] Here’s the KB article describing the update I reference in this article: http://support.microsoft.com/kb/2927690

 

ADFS Blog Exchange Exchange 2013 Hybrid Exchange News Office 365

This was MEC 2014 (in a nutshell)

As things wind down after a week full of excitement and – yes, in some cases – emotion, MEC 2014 is coming to an end. Lots of attendees have already left Austin and those who stayed behind are sharing a few last drinks before making their way back home as well. As good as MEC 2012 in Orlando was, MEC 2014 was E-P-I-C. Although some might state that the conference had missed its start – despite the great Dell Venue Pro 8 tablet giveaway – you cannot ignore the success of the rest of the week.

With over 100 unique sessions, MEC was packed with tons and tons of quality information. To see that amount of content being delivered by the industry’s top speakers is truly an unique experience. After all, at how many conferences is the PM or lead developer presenting the content on a specific topic? Also, Microsoft did a fairly good job of keeping a balance between the different types of sessions by having a mix of Microsoft-employees presenting sessions that reflected their view on things (“How things should work / How it’s designed to be”) and MVPs and Masters presenting a more practical approach (“How it really works”).

I also like the format of the “unplugged” sessions where you could interact with members of the Product Team to discuss a variety of topics. I believe that these sessions are not only very interesting (tons of great information), but they are also an excellent way for Microsoft to connect with the audience and receive immediate feedback on what is going out “out there”. For example, I’m sure that the need for some better guidance or maybe a GUI for Managed Availability is a message that was well conveyed and that Microsoft should use this feedback to maybe prioritize some of the efforts going into development. Whether that will happen, only time will tell..

This edition wasn’t only a success because of the content, but also because of the interactions. It was good to see some old friends and make many new ones. To  me, conferences like this aren’t only about learning but also about connecting with other people and networking. There were tons of great talks – some of which have given me food for thought and blog posts.

Although none of them might seem earth-shattering, MEC had a few announcements and key messages; some of which I’m very happy to see:

  • Multi-Factor Authentication and SSO are coming to Outlook before the end of the year. On-premises deployments can expect support for it next calendar year.
  • Exchange Sizing Guidance has been updated to reflect some of the new features in Exchange 2013 SP1:
    • The recommended page file size is now 32778 MB if your Exchange server has more than 32GB of memory. It should still be a fixed size and not managed by the OS.
    • CAS CPU requirements have increased with 50% to accommodate for MAPI/HTTP. It’s still lower than Exchange 2010
  • If you didn’t know it before, you will now: NFS is not supported for hosting Exchange data.
  • The recommended Exchange deployment uses 4 database copies, 3 regular 1 lagged. FSW preferably in a 3rd datacenter.
  • Increased emphasis on using a lagged copy.
  • OWA app for Android is coming
  • OWA in Office 365 will get a few new features including Clutter, People-view and Groups. No word if and when this will be made available for on-premises customers.

By now, it’s clear that Microsoft’s development cycle is based on a cloud-first model which – depending on what your take on things is – makes a lot of sense. This topic was also discussed during the Live recording of The UC Architects, I recommend you have a listen at it (as soon as it’s available) to hear how The UC Architects, Microsoft and the audience feels about this. Great stuff!

It’s also interesting to see some trends developing/happening. “Enterprise Social” is probably one of the biggest trends at the moment. With Office Graph being recently announced, I am curious to see how Exchange will evolve to embrace the so-called “Social Enterprise”. Features like Clutter, People View and Groups are already good examples of this.

Of course, MEC wasn’t all about work. There’s also time for fun. Lots of it. The format of the attendee party was a little atypical for a conference. Usually all attendees gather at a fairly large location. This time, however, the crowd was shattered across several bars in Rainey Street which Microsoft had rented off. Although I was a little skeptical at first, it rather worked really well and had tons of fun.

Then there was the UC Architects party which ENow graciously offered to host for us. The Speakeasy rooftop was really amazing and the turnout even more so. The party was a real success and I’m pretty confident there will be more in the future!

I’m sure that in the course of the next few weeks, more information will become available through the various blogs and websites as MVPs, Masters and other enthusiasts have digested the vast amount of information distributed at MEC.

I look forward to returning home, get some rest and start over again!

Au revoir, Microsoft Exchange Conference. I hope to see you soon!

Blog Events Exchange Exchange 2013 Microsoft Exchange Conference 2014 Office 365

Why MEC is the place to be for Exchange admins/consultants/enthusiasts!

In less than a month, the 2014 edition of the Microsoft Exchange Conference will kick off in Austin, Texas. For those who haven’t decided if they will be going yet, here’s some reasons why you should.

The Value of Conferences

Being someone who frequently attends conferences, I *think* I’m in a position I can say that conferences provide great value. Typically, you can get up-to-date with the latest (and greatest) technology in IT.

Often, the cost for attending a conference are estimated higher than a traditional 5-day course. However, I find this not to be true – at least not all the time. It is true that – depending on where you fly in from – Travel & Expenses might add up to the cost. However, I think it is a good thing to be ‘away’ from your daily work environment. That typically leaves one less tempted to be pre-occupied with work rather than soaking in the knowledge shared throughout the conference. The experience is quite different from a training course. Conferences might not provide you the exact same information as in a training, but you’ll definitely be able to learn more (different) things. Especially if your skills in a particular product are already well-developed, conferences are the place to widen your knowledge.

On top of that, classroom trainings don’t offer you the same networking capabilities. In case of MEC, for instance, there will be a bunch of Exchange MVPs and Masters who you can talk to. All of them very knowledgeable and I’m sure they won’t mind a good discussion on Exchange! This could be your opportunity to ask some really difficult questions or just hear what their opinion is on a specific issue. Sometimes the insights of a 3rd person can make a difference…!

It is also the place where all the industry experts will meet. Like I mentioned earlier, there will be Masters and MVPs, but also a lot of people from within Microsoft’s Exchange Product Group will be there. What better people are there to ask your questions to?

Great Content

Without any doubt, the Exchange Conference will be the place in 2014 to learn about what’s happening with Exchange. Service Pack 1 – or Cumulative Update 4, if you will – has just been released and as you might’ve read there are many new things to discover.

At the same time, it’s been almost 1.5 years since Exchange 2013 has been released and there are quite some sessions that focus on deployment and migration. If you’re looking to migrate shortly, or if you’re a consultant migrating other companies, I’m sure you’ll get a lot of value from these sessions as they will be able to provide you with first-hand information. When MEC 2012 was held – shortly before the launch of Exchange 2013 – this wasn’t really possible as there weren’t many deployments out there.

Sure, one might argue that the install base for Exchange 2013 is still low. However, if you look back at it, deployments for Exchange 2010 only really kicked of once it was past the SP1 era. And I expect nothing else to happen for Exchange 2013.

As a reference: here’s a list of sessions I definitely look forward to:

And of course the “Experts unplugged” sessions:

I realize that’s way too many sessions already and I will probably have to make a choice which ones I will be able to attend…
But the fact that I have so many only proves that there’s so much valuable information at MEC…

Great speakers

I’ve had a look through who is speaking at MEC and I can only conclude that there is a TON of great speakers. All of which I am sure they will make it worth the wile. While Microsoft-speakers will most likely give you an overview of how things are supposed to work, many of the MVPs have sessions scheduled which might give you a slight less biased view of things. The combination of both makes for a good mix to get you started on the new stuff and broaden your knowledge of what was already there.

Location

Austin, Texas. I haven’t been there myself. But based on what Exchange Master Andrew Higginbotham blogged a few days ago; it looks promising!

Microsoft has big shoes to fill. MEC 2012 was a huge success and people are expecting the same – if not better – things from MEC 2014. Additionally, for those who were lucky enough to attend the Lync Conference in Vegas earlier this month, that is quite something MEC has to compete with. Knowing the community and the people behind MEC, I’m pretty confident this edition will be EPIC.

See you there!

Michael

Blog Exchange 2013 Microsoft Exchange Conference 2014 News Office 365 Uncategorized

The limitations of calendar federation in a hybrid deployment

Recently, Loryan Strant (Office 365 MVP) and myself joined forces to create an article for the Microsoft MVP blog regarding some of the limitations of calendar federation in a hybrid Exchange deployment. In this article we discuss how running a hybrid deployment might affect calendar sharing with other organizations and what your options are to work around this limitation.

To read the full article, please click here.

Enjoy!

Michael

Blog Exchange 2013 Hybrid Exchange Office 365

You get an error “the connection to the server <servername> could not be completed” when trying to start a hybrid mailbox move in Exchange 2013.

As part of running through the “New Migration Batch”-wizard, the remote endpoint (the on-premises Exchange server) is tested for its availability. After running this step, the following error is displayed:

image

By itself, this error message does not reveal much information as to what might be causing the connection issues. In the background, the wizard actually leverages the “Test-MigrationServerAvailability” cmdlet. If you run this cmdlet yourself, you will get a lot more information:

image

In this particular case, you’ll see that the issue is caused by 501 response from the on-premises server. The question is of course: why? We recently moved a number of mailboxes and then we did not encounter the issue. The only thing that had changed between then and now is that we reconfigured our load balancers in front of Exchange to use Layer 7 instead of Layer 4. So that is why I shifted my attention to the load balancers.

While reproducing the error, I took a look at the “System Message File” log in the KEMP load balancer. This log can be found under Logging Options, System Log Files. Although I didn’t expect to see much here, I saw the following message which drew my attention:

kernel: L7: badrequest-client_read [157.56.251.92:61541->192.168.2.130:443] (-501): <s:Envelope ? , 0 [hlen 1270, nhdrs 8]

A quick lookup learned that the 157.56.251.92 address was indeed coming from Microsoft. So now I knew for sure that something was wrong here. A quick search on the internet brought me to the following article which suggested to change the 100-Continue Handling in the Layer 7 configuration of the Load Master: http://blog.masteringmsuc.com/2013/10/kemp-load-balancer-and-lync-unified.html

After changing the value from its default (RFC Conformant), I could now successfully complete the wizard and start a hybrid mailbox move. So the “workaround” was found. But I was wondering, why does the Load Master *think* that the request coming from Microsoft is non-RFC compliant?

The first thing I did is ask Microsoft if they could clarify a bit on what was happening. I soon got a reply that – from Microsoft’s point of view – they were respecting the RFC documentation regarding the 100 (Continue) Status. No surprise here.

After reading the RFC specifications I decided to take some network traces to find out what was happening and maybe understand how the 501 response was triggered. The first trace I took, was one from the Load Master itself. In that trace, I could actually see the following:

image

Effectively, Office 365 was making a call to the Exchange Web Services and using the 100-continue status. As described per the RFC documentation, the Exchange on-premises server should now respond appropriately to the 100-continue status. Instead, we can see that in the entire SSL conversation, exactly 5 seconds go by after which Office 365 makes another call to the EWS virtual directory without having received a response to the 100-continue status. At the point, the KEMP Load Master generated the “501 Invalid Request”.

I turned back to the (by the way, excellent) support guys from KEMP and explained them my findings. Furthermore, when I tested without Layer 7 or even without a Load Master in between, there wasn’t a delay and everything was working as expected. So I knew for sure that the Exchange 2013 on-premises was actually replying correctly to the 100-continue status. As a matter of fact, without the KEMP LM in between, the entire ‘conversation’ between Office 365 and Exchange 2013 on-premises was perfectly following the RFC rules.

So, changing the 100-continue settings from “RFC Conformant” to “Ignore Continue-100” made sense as now KEMP would just ignore the 100-continue “rules”. But I was still interested in finding out why the LM thought the conversation was not RFC conformant in the first place. And this is where it gets interesting. There is this particular statement in the RFC documentation:

“Because of the presence of older implementations, the protocol allows ambiguous situations in which a client may send “Expect: 100- continue” without receiving either a 417 (Expectation Failed) status or a 100 (Continue) status. Therefore, when a client sends this header field to an origin server (possibly via a proxy) from which it has never seen a 100 (Continue) status, the client SHOULD NOT wait for an indefinite period before sending the request body.”

In fact, that was exactly what is happening here. Office 365 (the client) sent an initial 100-continue status and waited for a response to that request. In fact, it waits for exactly 5 seconds and sends the payload, regardless of it having received a response. In my opinion, this falls within the boundaries of the scenario described above. However, talking to the KEMP guys there seems to be a slightly different interpretation of the RFC which caused this mismatch and therefore the KEMP issuing the 501.

In the end, there is still something we haven’t worked out entirely: why the LM doesn’t send back the Continue-100 status back to Office 365 even though it receives it back almost instantaneously from the Exchange 2013 server.

All in all, the issue was resolved rather quickly and we know that changing the L7 configuration settings in the Load Master solves the issue (and this workaround was also confirmed as being the final solution by KEMP support, btw). Again, changing the 100-continue handling setting too “Ignore” doesn’t render the configuration (or the communication between Office 365 or Exchange on-premises) non-RFC compliant. So there’s no harm in changing it.

I hope you found this useful!

-Michael

Blog Exchange 2013 Hybrid Exchange Office 365

Free/Busy in a hybrid environment fail and Test-Federationtrust returns error “Failed to validate delegation token”

Following an issue with Free/Busy in Exchange online, earlier this week, I was troubleshooting the exchange of Free/Busy information in some of my hybrid deployments as Free/Busy information was still not working.
After having checked some (obvious things) like the Organization Relationships and whether or not Autodiscover was working properly, I discovered an issue when running the Test-FederationTrust cmdlet.

In fact, the cmdlet completed almost entirely successful, except for the very last step in the process:

Id         : TokenValidation
Type       : Error
Message    : Failed to validate delegation token.

This also explained why I was seeing 401 Unauthorized messages when running the Test-OrganizationRelationship command.

I then checked the same in some of my other deployments and found out the all had the same issue. At least, there was some common ground to start working from.
I turned to co-MVP Steve Goodman and asked him to run the same command in one of his labs in order to have a point of reference. At the same time, he asked me to run a command which might help:

Get-FederationTrust | Set-Federationtrust –RefreshMetaData

After running the command, I re-ran the Test-FederationTrust command which now completed successfully.

Conclusion

Although the Free/Busy issues in Office 365 should be solved, some customers might still experience problems exchanging Free/Busy information. In this case, the problem manifests itself by e.g. online users not being able to request on-premises user’s availability information.

Blog Exchange Hybrid Exchange Office 365