A few days ago, Microsoft released Cumulative Update 6 for Exchange 2013 to the world. There used to be a time where Exchange server updates were fairly safe. However, pretty much like in every other Cumulative Update for Exchange 2013, this one also includes some bugs which break functionality in one way or another. While one would say that it starts to become painful for Microsoft, I’m starting to believe it’s more of a joke.
Exchange Server MVP Jeff Guillet was the one to first report the issue. As it turns out, the Hybrid Configuration Wizard in CU6 runs just fine, but some of the features (like initiating a mailbox move from the on-premises EAC or the ability to switch between the on-prem/cloud EAC) no longer work. Although the scope of the break is somewhat limited (it only applies to customers in a hybrid deployment), one could argue it’s an important focus area for Microsoft – especially given that it’s cloud-related. Microsoft has been trying really hard (with success, may I add) to promote Office 365 and get customers to onboard to “the service”. As such, I find it really surprising that it’s the n-th issue related to hybrid deployments in such a short time. In Cumulative Update 5, the Hybrid Configuration Wizard is broken and now there’s this.
Needless to say, you are warned about deploying Cumulative Updates into production. Pretty much every MVP which announced the Cumulative Update made the remark that you should better test the update before deploying it. I would say this is a general best-practice, but given the history of recent Exchange Server updates, I wouldn’t dare to deploy one without thoroughly testing it.
This brings me to another point: what happened to testing, Microsoft? I understand that it’s impossible to test every customer scenario that you can find out there, but how come that pretty obvious functionalities like these manage to slip through the cracks? If it were a one-time event, I could understand. But there’s a clear trend developing here.
Running a service like Office 365 is not easy. More so, the cadence at which the service evolves can be really scathing. On-premises customers have been struggling to keep up with the updates that are being released in the cloud, but it seems that Microsoft itself is having a hard time to keep up too.
On a final note, I’m wondering what customers with a hybrid deployment should do. According to Microsoft support guidelines, hybrid customers are requested to stay current with Exchange Server updates. But given that this is now two consecutive update that are causing problems, one might start to wonder if it’s not better to stay at CU4 as it was the last CU which did not have any hybrid issues…
I imagine that Microsoft is working hard on a fix for this issue, even during a holiday weekend… Let’s wait and see what happens early next week!
Until then, I would hold off on deploying CU6 and revert to using CU5 with the interim update which fixes the HCW bug or – if you don’t like IUs – stick to CU4/SP1.
Today, Microsoft released its latest updates for Exchange 2007, 2010 and 2013.
The updates for Exchange 2007 and 2010 mostly evolve around the Daylight Saving Time changes and a bunch of fixes for the latter version.
Cumulative Update 6 for Exchange 2013 doesn’t introduce any new feature or feature changes, but I’m happy to see that the Hybrid Configuration Wizard bug – which caused the HCW to fail – is now included by default. An Interim Update was already available, but it’s nice to see it included into the full build.
Along with a bunch of other fixes, Cumulative Update 6 now also closes the gap with Office 365 when it comes to Public Folder performance and scalability: you can now also deploy up to 100,000 public folders on-premises. Along with this change, there are some other (minor) behavioral changes which Microsoft outlined beautifully here.
For more information on these updates, have a look at the following announcements for Microsoft:
As posted here, Microsoft today released Cumulative Update 5 for Exchange 2013. At first sight, this update doesn’t appear to make lots of changes – at least not visibly. However, it does contain a lot of fixes and, as you will find out, there have been some changes to the Hybrid Configuration Wizard as well.
New options in the Hybrid Configuration Wizard
Whenever you enable an organization for a hybrid deployment in CU5, you will find the following new option:
21Vianet is Microsoft’s partner which offers Office 365 in China. You could say that they “host” Office 365 for Chinese customers as outlined in this Press Release
MRS Proxy now configured automatically
This is one of my personal asks for quite a long time now. Although the HCW already did an excellent job configuring all the components for a hybrid deployment, it did not enable the MRS Proxy on the Exchange Web Services Virtual Directory. Even though you could do it yourself with only a single command, I’m a big fan of having the HCW take care of this. It’s one less thing you can forget yourself!
OAuth now configured automatically
You’ll also notice that towards the end, the Hybrid Configuration Wizard will now prompt you to configure oAuth automatically:
The wizard will then automatically redirect you to a webpage where you’ll be asked to start the configuration (again):
Once you click configure, you will be asked to download an application which will automatically configure oAuth for you. Because it seems to be browser-integrated, you cannot run this step from a computer other than your Exchange Server and then copy over the executable. Beware and make sure that you run the HCW from the Exchange server itself instead from a remote workstation, like I tried the first time…
Once the first application was downloaded, you’ll be asked to run it:
Note: make sure that *.configure.office.com is added to your trusted sites or that you at least allow content to be downloaded from that website.
Then, after this first application ran, you’ll be prompted for an identical, second, application. Only this time the application (or assistant, if you will) will be a bit bigger: 22.2 MB instead of 18MB.
Once the second assistant completed successfully, you’ll see the following:
Note The configuration of the Intra-Organization Connector is the only thing that’s already handled by the Hybrid Configuration Wizard itself.
It’s definitely a good thing this is now done automatically. However, I would love to see it be more integrated with the HCW. At the moment, these changes don’t show up in the Hybrid Configuration Wizard logs.
It was already clear that Microsoft is moving forward with oAuth; potentially to replace other technologies currently used in Hybrid deployments. Personally, I wouldn’t be too surprised to see oAuth take over the duties from Microsoft’s Federation Gateway in the future. Not sure if this will actually happen, but it seems like a good thing. If you have ever been in a discussion with a pesky security administrator you would understand why… But don’t expect that to happen in a few months’ time though – as long as Exchange 2010 is officially supported, I reckon Microsoft will have to keep the MFG around.
It’s surely a good thing to move forward with oAuth as it has the potential to solve some long-standing issues regarding the handling of authentication and security in a cross-premises scenario like a hybrid deployment.
Today, Microsoft released Cumulative Update 5 for Exchange 2013 and Update Rollup 6 for Exchange 2010.
Exchange 2013 Cumulative Update 5
Next to a ton of bug fixes, Microsoft made changes to a few components including:
Offline Address Book generation
Hybrid Configuration Wizard
Except for the above changes, it looks like CU5 will mostly consist of fixes. By the looks of it and as Tony Redmond already pointed out CU5 promises to be a stable release. Whether it will stay that way is something only time will tell…
Installing Cumulative Update 5
Installing CU5 is no different from older versions. You can also immediately upgrade from any previous version of Exchange 2013 to CU5. There is no requirement to install SP1 (a.k.a. CU4) first.
After installation, Microsoft warns there might be a Managed Availability probe which went into overdrive and repeatedly restarts a newly added service called the Microsoft Exchange Shared Cache Service. However, this service isn’t used in CU5 (planned for the future?) and as such there is no impact at all.
However, if you are worried about your application log filling up with events from Managed Availability, you can disable the probe. More information can be found here.
This update also includes Active Directory changes, so you will be required to extend the AD schema. Given that you’re used to it by now, this shouldn’t present much of a problem. For more information on how to deploy a Cumulative Update, I suggest you have a look at the following article by ExchangeServerPro:
With Service Pack 1, the Exchange team introduced a new connectivity model for Exchange 2013. Instead of using RPC/HTTP (which has been around for quite a while), they have now introduced MAPI/HTTP. The big difference between both is that RPC is now cut away and therefore allow for a more resilient / lenient way to connect to Exchange. HTTP is still used for transport, but instead of ‘encapsulating’ MAPI in RPC packets, it’s now transported directly with the HTTP stream.
To enable MAPI/HTTP, run the following command:
Set-OrganizationConfig –MapiHttpEnabled $true
As you can see from the cmdlet, deploying MAPI/HTTP is an “all-or-nothing” approach. This means that you have to plan the deployment carefully. Switching from ‘traditional’ RPC/HTTP to MAPI/HTTP involves users restarting their Outlook (yes, the dreadful “Your Administrator has made a changed…”-dialog box is back). Luckily, the feature will – for now? – only work on Office 2013 Service Pack 1. Anyone who isn’t using this version will continue to use RPC/HTTP and will not be required to restart. Just keep it in mind when you upgrade your clients so that you don’t create a storm of calls to your helpdesk…
Anyway, because the feature is disabled by default – and because it traditionally takes a while before new software gets deployed – I don’t expect this feature to be widely used any time soon though.
Exchange Admin Center Command Logging
This is one of the most-wanted features ever since Exchange 2013 was released. Previously the Exchange 2010 logged all the cmdlets that it executed when you performed a task through the Management Console. However, because of the move from the EMC to the new web-based Exchange Admin Center (EAC), this feature disappeared which caused a lot of protest.
Now, in SP1, the feature – somewhat – returns and gives you the ability to capture the cmdlets the EAC executes whenever you’re using it. The feature itself can be found in the top-right corner of the EAC, when clicking the question mark button:
Support for Windows Server 2012 R2
Another long-awaited and much-asked-for feature is the support for Windows Server 2012 R2. This means that you will be able to deploy Exchange 2013 SP1/CU4 on a server running Microsoft’s latest OS. At the same time, the support for Domain Controllers running Windows Server 2012 R2 was also announced. This effectively means that you no longer have to wait to upgrade your Domain Controllers!
S/MIME support for OWA
Another feature that existing in Exchange 2010, but didn’t make the bar for the RTM release of Exchange 2013 is S/MIME support for OWA. Now, however, it’s available again.
The return of the Edge Transport Server Role
It looks like the long lost son made its way back into the product. The Edge Transport Server role, that is. Although – honestly – the Edge Transport Server isn’t a much deployed server role – at least not in the deployments I come across, it is a features which is used quite a bit in hybrid deployments. This is mainly because it’s the only supported filtering solutions in a hybrid deployment. Any other type of filtering device/service/appliance [in a hybrid deployment] will cause you to do more work and inevitably cause more headaches as well.
This is definitely good news. However, there are some things to keep in mind. First of all, the Edge Transport server doesn’t have a GUI. While this is not much of an issue for seasoned admins, people who are new to Exchange might find the learning curve (PowerShell-only) a little steep.
General Fixes and Improvements
As with every Cumulative Update, this one probably also contains a bunch of improvements and fixes. More information to the download and the updates can be found here.
Support for SSL Offloading
Now, there’s also support again for SSL Offloading. This means that you are no longer required to re-encrypt traffic coming from e.g. a load-balancer after it decrypted it first. Although many customers like to decrypt/re-encrypt, there are deployments where SSL Offloading makes sense. Additionally, by offloading SSL traffic you spare some resources on the Exchange Server as it no longer has to decrypt traffic. The downside – however – is that traffic flows unencrypted between the load balancer and the Exchange Servers.
DLP Policy Tips in OWA
Data Loss Protection was one of the new features in Exchange 2013 RTM and was very well received in the market. It allows you to detect whenever sensitive data is being sent and take appropriate actions if so. Although DLP policies worked just fine in OWA, you wouldn’t get the Policy Tips (Warnings) as they were displayed in Outlook 2013. These tips are – in my opinion – one of the more useful parts of the DLP feature and that’s why I find it great they’ve finally added it into OWA. Now, you’re no longer required to stick to Outlook to get the same experience!
As mentioned above, DLP allows you to detect whenever sensitive information is sent via email. However, detecting sensitive information isn’t always easy. Until now, you had to build (complex) Regular Expressions which would then be evaluated against the content being sent through Exchange. With the DLP Fingerprinting feature, you can now upload a document to Exchange which will then use that document as a template to evaluate content against. It is a great and easy way to make Exchange recognize certain files / type of files without having to code everything yourself in RegEx!
The DLP Fingerprinting feature can be found under Compliance Management > Data losse preventsion > Manage Document Fingerprints
Outlook Web App is already one of the best web-based email clients available. In search of brining more features to OWA to make it even better, the Exchange team now added also some – maybe less visible – but very welcome improvements to OWA. The rich text editing features is one of them.
For example, you have now more editing capabilities and you can easily add items like tables or embedding images:
Database Availability Group without IP (Administrative Access Point)
Leveraging the new capabilities in Windows Server 2012 R2 (Failover Clustering), you can now deploy a DAG without an administrative Access Point (or IP Address). This should somehow simplify the deployment of a Database Availability Group.
Deploying Service Pack 1
The process for deploying Service Pack 1 isn’t different from any other Cumulative Update. In fact, Service Pack 1 is just another name for Cumulative Update 4. Basically, upgrading a server will do a back-to-back upgrade of the build which means that any customizations you have made to configuration files will most likely to be lost. Make sure to backup those changes and don’t forget to re-apply them. This is especially important if you have integrated Lync with Exchange 2013 as this (still) requires you to make changes to one of the web.config files!
After you have upgraded the servers, I would suggest that you reboot them. Because the way Managed Availability works, you might sometimes find the Frontend Transport Service not to work as expected for a while. Typically a reboot solves the ‘issue’ right away.
By the time I published this overview, some of the other MVPs already put some thoughts out there. Make sure to check them out:
After some issues with Cumulative Update 2, which had to be pulled and re-released, Microsoft put more effort into testing and validating CU3 before releasing it to the public. That is one of the reasons why it took a little longer than expected for CU3 to be available. A good thing which hopefully pays of in a stable update without any (major) issues! CU3 introduces a bunch of new features to Exchange 2013, amongst which are:
Improved experience for Group Management in EAC
Integration with Online RMS for on-premises-only deployments
Improved Admin Audit Logging
As you can see, there’s quite some new – and interesting – stuff in CU3, which makes it definitely worth taking a closer look at. I’m particularly interested in finding out more about the RMS Online integration (which is a good thing!). Next to a bunch of new features, there are also some important bug fixes in CU3:
KB2888315 Event 2112 or 2180 is logged when you try to back up a database in an Exchange Server 2013 environment
KB2874216 Security issue that is described in Security Bulletin MS13-061 is resolved by an Exchange Server update
KB2902929 You cannot forward an external meeting request in an Exchange Server 2013 environment
KB2890814 No redirection to the Outlook Web App URL for Exchange Online users in an Exchange hybrid deployment
KB2883203 Exchange Server 2013 restarts frequently after Cumulative Update 2 is installed
A complete list of the most important bug fixes can be found here.
Deploying CU3 is similar to deploying previous CUs. Just like these previous CUs, CU3 also includes Active Directory schema updates. For more information on how to deploy a Cumulative Update, have a look at Paul Cunningham’s blog here.
How about Exchange 2013 Service Pack 1?
As a side-note to the release is that Microsoft previously announced that Exchange Server 2013 Cumulative Update 4 would be released as Service Pack 1. Taking the three-month cadence in which Cumulative Updates are expected to be released, puts Service Pack 1 to be released around the February/March 2014 timeframe – that is assuming the CU release cadence is respected. This is a little earlier than I anticipated, to be honest. I expected SP1 not to be released until at the Microsoft Exchange Conference in April (which – now I come to think of it is merely a month later). I, for one, am looking forward to “SP1”, usually this is a milestone that many companies wait for before deploying a new server product like Exchange. Traditionally, Service Packs were used to introduce a bucket of new features to the product along with some other improvements. Given that each Cumulative Update so far has added functionality, I wonder if SP1 (Cumulative Update 4) will generate the same impact as it has done with previous releases…
Exchange 2010 SP3 Update Rollup 3
This latest Update Rollup for Exchange 2010 Service Pack 3 contains a rather long list of bug fixes. Amongst these fixes, I found the following ones to stand out, mainly because I faced them a few times, myself:
KB2839533 RPC Client Access service freezes in an Exchange Server 2010 environment
KB2887609 Hybrid Configuration wizard does not display the Domain Proof of Ownership list in an Exchange Server 2010 SP3 environment
A complete list of the most important fixes, can be found here. (note: content of this link may not yet be available) Have fun!
Yesterday, Microsoft released an updated version of its latest Cumulative Update for Exchange 2013; CU2. This decision follows the discovery of quite an important bug in the original release.
The problem was that when Public Folder Mailboxes were moved between databases, the public folder permissions would get lost. Obviously, this not only represents quite a bug, it’s also a risk for data loss/leakage.
Despite earlier statements they would release an interim update, the Product Team decided to incorporate the fix in a new build of CU2.
As a result, this “version 2” will require you to do a full upgrade just like any other Cumulative Update. See the section below for some more information.
Click here for the original announcement by the Product Team.
Do you have to upgrade?
Yes. Even if you’re not impacted, you’re required to upgrade (at some point). If Microsoft releases security update for Exchange 2013 in the future, these security update will require the new CU2 (v2) to be installed, as Ross described in the original post:
“Important: Regardless of whether you are using modern public folders, we strongly recommend upgrading to this build of Exchange 2013 RTM CU2. Any security updates released for CU2 will be dependent on this build.”
Honestly, I would wait a few more days before deploying this re-released CU. Just to see what feedback comes from the first deployments. I don’t expect any major issues, but you never know…
How to Upgrade?
To install the new CU, run the following command from either a command prompt or PowerShell. Make sure to browse to the location of the binaries first:
The new build number of this ‘new’ version of CU2 would be 712.24 instead of 712.22 (CU2 v1)3
I think it was only a few weeks ago that Hyper-V MVP Aidan Finn blogged about how Microsoft had a serious quality problem after having released a few buggy Windows Server patches. No matter how hard I would like to state otherwise, I cannot but join him (and many others) in expressing my concerns about the lack of quality (testing?) of recent (and some not so recent) updates.
It’s not the first time something like this has happened with Exchange. I clearly remember Update Rollups for Exchange 2010 to be re-released; sometimes even multiple times like UR4 which had to be re-released twice!. I am well aware that Microsoft does conduct a number of tests before releasing update, but I also have no doubt their testing is primarily focused on scenarios related to Office 365. Might it be that certain typical on-prem scenarios are either disregarded or (much) lower on the priority list, it definitely looks like it!
I fully understand the priority is with Office 365 and in some way that’s a good thing; making sure that ‘the service’ remains fully operational should be Microsoft’s primary concern, especially given its size and popularity. However, Microsoft shouldn’t forget they have a huge customer base that have Exchange running on-premises… And they do expect – and deserve! – the same quality.
On a side note: the overall (world-wide) impact of this “bad” update might be rather limited, yet significant enough. Although no official numbers have been released, I suspect that the amount of Exchange 2013 deployments might not be that high at the moment. I have no doubt that number will grow over the following few months, but Microsoft needs to do something about the overall quality of their updates, not only for us, but for themselves as well.
As announced earlier this year at TechEd, CU2 contains a number of fixes as well as new/improved features, including:
Per-server database support increased back to 100
OWA Silent Redirection
Addition of the DAG Management service
Changes to Managed Availability
Improved PowerShell Help Updating
OWA Search improvements
Malware Filter Rules
What didn’t make it (yet), is the official support for running a File Share Witness in Azure as well as support for IIS Application Request Routing as a reverse-proxy solution for Exchange. However, both have been announced and will probably be availably any time soon – no committed dates yet:
The Exchange Product Group is in the final validation stages to support Windows Azure for Witness Server placement. Specific guidance on using Windows Azure for the Witness Server placement will be available via TechNet at a later date. Support for this scenario will occur once the guidance has been released.
Personally, I love to see some of the features “return”, like OWA Silent Redirect. But to be honest, I’m most curious about the DAG Management service. Although at the moment it seems to only exist to take some of the load off the Replication service, I wouldn’t be surprised to see some more changes related to this feature in future releases.
Please keep in mind that Cumulative Updates are full builds of Exchange, as a result they might take a while to install. Also keep in mind that they might wipe any of the customization you have made to e.g. web.config files.
As for the preparation, CU2 also requires an AD schema update. Make sure to read the deployment notes from the original article for more information. When I find some time, I’ll try to figure out what changes are made and report them here.