Estimating the size of an Exchange (online) Archive

As part of some of the (archiving-) projects I have worked on, I frequently get asked if there is an easy way to determine what the size of the archive will be once it’s been activated. Although a bit odd at first, there are actually many good reasons why you’d want to know how big an archive will be.

First of all, determining the archive size allows to better size (or schedule for) the storage required for the archives. While there are also other ways to do this, knowing how big an archive will be when enabled is very helpful.

Secondly, if you’re using Exchange Online Archiving (EOA), it allows you to determine the amount of data that will pass through your internet connection for a specific mailbox. If the amount of data is large enough (compared to the available bandwidth), I personally prefer to provision an archiving on-premises, after which I can move it to Office 365 using MRS. But that’s another discussion. Especially for this scenario it can be useful to know how much archive you can (temporarily) host on-premises before sending them off to Office 365 and freeing up disk space again.

In order to calculate how big an archive would be, I’ve created a script which will go through all the items in one (or more) mailbox(es) and calculate the total size of all the items that will expire. When an item expires (and thus is eligible to be moved to the archive) depends on the Retention Policy you assign to a mailbox and what retention policy tags are included in that policy.

As the name of the script depicts, it’s important to understand that it’s an estimation of the archive size. There are situations in which the results of the script will be different from the real world. This could be the case when you enabled the archive and a user assigned personal tags to items before the Managed Folder assistant has processed the mailbox. In such a scenario, items with a retention tag that are different from the AgeLimit defined in the script will be calculated wrongfully. Then again, the script is meant to be ran before an archive is created.

Secondly, the script will go through all the folders in a mailbox. If you disabled archiving of calendar items, these items will be wrongfully included in the calculation as well. I will try to built this into the script in future releases, but this has a lower priority as the script was built to provide a pretty good estimation, not a 100% correct number.

The script, which you can download here, accepts multiple parameters:

UserPrimarySMTPAddresses the Primary SMTP Address of the mailbox for which you want to estimate the archive size
Report full file path to a txt file which contains the archive sizes
AgeLimit The retention time (in days) against which items should be probed. If you have a 60 day retention before items get moved to the archive, enter 60.
Server Used for connecting with EWS. Optional. Can be used if autodiscover is unable to determine the connection URI.
Credentials The credentials of an account that has the ApplicationImpersonation Management Role assigned to it.

 

The output of the script will be an object that contains the user’s Primary SMTP Address and the size of the archive in MB (TotalPRMessageSize).

Credit where credit is due! I would like to thank Michel de Rooij for his immensely insane PowerShell scripting skills and for helping me with cleaning up this script to its current form. Before I sent it off to Michel, the code was pretty inefficient [but hey! it was working], what you’ll download has been cleaned up and greatly enhanced. Now you have a clean code, additional error handling and some more parameters than in my original script [see parameters above].

I hope you’ll enjoy the script and find it useful. I’ve used it in multiple projects so far and it really helped me with planning of provisioning the archives.

Note:  To run the script, you’ll need to have Exchange Web Services installed and run it with an account that has the Application Impersonation Management Role assigned to it.

Cheers,

Michael

Blog Exchange

You get an error: “The <name> connector cycle has stopped. Object with DN <GUID> failed…”

As part of setting up a hybrid configuration between Exchange on-premise and Exchange online (or when configuring Exchange Online Archiving), you also need to setup DirSync.

In these scenarios DirSync fulfills an important role as it will also configure the write-back of some attributes in your local Active Directory. This “write-back” is required for Hybrid/EOA to work. For a list of attributes that are sync to/from Office 365, have a look at the following article: http://support.microsoft.com/kb/2256198

As part of the best-practices when installing DirSync, you should always run the Office 365 Deployment Readiness tool which will scan your local Active Directory and search for incompatible objects. The tool will create a report in which incompatible objects are mentioned. This will allow you to modify these object before configuring DirSync.

However, sometimes object can still contain incompatible object attributes, which might cause issues for DirSync. In such case, you’ll likely be presented with the following error in the application event log. Please note that this example mentions an issue with the “TargetWebService” Management Agent. It could very well be that you’ll encounter an issue in the SourceAD Management Agent.

The TargetWebService Connector export cycle has stopped.  Object with DN CN=<guid> failed validation for the following attributes: proxyAddresses. Please refer to documentation for information on object attribute validation.

This error contains 2 important items:

  1. The Distinguished name (CN=<guid>)
  2. The attribute that is causing issues

However, matching the guid to a user-account isn’t very easy. The best way to go about is to open the MIIS Management Interface and work from there. Usually, the client can be found in the following directory:

C:\Program Files\Microsoft Online Directory Sync\SYNCBUS\Synchronization Service\UIShell

image

After opening the client, navigate to Management Agents, right-click the management agent mentioned in the error message and select Search Connector Space:

image

In the “Search Connector Space” window, select DN or Anchor from the drop-down list under Scope and specify the Distinguished Name from the error message. Afterwards, click Search:

image

The search should return a single object. Double-click it to view additional information. Search for the attribute that was mentioned in the event log entry to review its value(s):

image

In this particular case, one of the proxy addresses contained an illegal character which caused the Management Agent to fail. Once you determined what the issue was, correct the value in AD and re-start synchronization. Normally, synchronization should happen successfully now.

How-To's Office 365

Update: Disappearing (online) archives after moving your mailbox to Office 365

Update

After a few weeks of mailing back and forth with Microsoft’s support, I was today (finally) able to confirm that the issue which I described below is now solved.

It seems that Microsoft rolled out a hotfix/code change for their Exchange Online service. Although, at first, I thought the issue was related to a bug in EMC for not correctly issuing all parameters when initiating a remote mailbox move, it seems the issue had more to it than that. Basically, what happened is that when MRS moved the mailbox from on-premises environment to Office 365, it wouldn’t keep the link to the already-existing archive. This caused a new (empty) archive to be created and could possibly cause data loss.

I’m happy to see what time and effort Microsoft has put into solving this issue. It proves that Microsoft is concerned about the quality of their product / service. In fact, it would surprise me if they weren’t. A bug that could cause data-loss is not really something you’d want to carry around for a long time!

Thanks to everyone involved and kudo’s to Philippe Phan Cao Bach (Sr. Escalation Engineer) who was working with me on this case.

Original Post

Office 365 offers great ways to enhance the functionalities of your on-premises deployment. By running the Hybrid Configuration Wizard (which Steve Goodman explains in this article) you can configure both environments to act as one; allowing you to make use of features such as e.g. Online Archives (EOA).

With Exchange Online Archives, your primary mailbox stays in your on-premises Exchange server, whereas the archive will – as the name might have given away – be hosted in Office 365. If you’re interested in finding out more about Online Archives, I suggest that you take a look at Bharat Suneja’s session at TechEd this year: “Archiving in the cloud with Exchange Online Archiving

The problem

To me, one of the most interesting things about a hybrid deployment is the flexibility it offers. You can put a few mailboxes in Office 365, try them out and move more to the service if you like it.

If you are looking to take that approach, this information might be interesting for you!

Imagine the following: you are trying out Office 365 and decide to use Online Archives to start with. You provision the archives and life is great! After a while you decide you want to use more and you decide to move some mailboxes to Office 365. However, after your users have been moved to Office 365 they start complaining that their archive is empty.

It seems that – although this scenario is supported – there are some issues with the provisioning process when you move a user to Office 365 that previously already had an Online Archive: it get’s “wiped”. At least, that’s how it looks like.

At first, I though the data would reappear after a while, so I made sure that I waited long enough. Unfortunately even after a few days, the archives was still empty.

I decided to do some tests, to make sure this wasn’t a standalone case. Perhaps something went wrong during the move. To my surprise, tests confirmed what was going on: although the archive contained items prior to the move, they are now empty.

To explain what happens, let me describe the process I used to reproduce this issue.

This first screenshot show the details of the on-premises mailbox that has a cloud-based archive (EOA) enabled. This archive contains 4 (test) items:

image

Afterwards, I moved the mailbox through the Exchange Management Console using the “New Remote Move Request”-wizard.

Because on-premises only a mailbox exists, you don’t have the option to move an archive (which is normal):
image

The move completed successfully, and after having waited long enough (DirSync etc.). I verified the mailbox’s settings:

image

The interesting part here is that the Archive, although having the same GUID, appears to have been moved to the same database as the mailbox. Before the archive resided in database “EURPRD04DG032-db055” whereas now it’s in “EURPRD04DG030-db041”.

To ascertain myself that this wasn’t causing problems, I decided to do another test. When executing the MoveRequest, I specified to what database the archive should be moved to. I made sure that the target database of the Online Archive was set to the database it was already residing in before moving the mailbox:

New-MoveRequest “Testmivh5” –RemoteHostName “hostname.company.com” –targetdeliverydomain “tenant.mail.onmicrosoft.com” –ArchiveTargetDatabase “EURPRD04DC032-db055” –RemoteCredential (Get-Credential)

Note   this cmdlet was executed from PowerShell connected to Exchange Online.

After the move completed (successfully btw), I – again – waited long enough for DirSync/replication/provisioning to occur. I deliberately didn’t force DirSync to ensure that wasn’t causing any issues either. But alas, none of that helped: the archive was again empty.

A quick look at the object’s attributes revealed that – although a target database parameter was provided – the archive still got moved to the same database as the user:

image

Then, I was thinking that the ‘old’ archive perhaps got disabled and that a new one was created. Although this would be strange since the GUID of the archive remains the same, I thought it was worth a try. Again: no joy! No disconnected mailboxes were to be found.

After all this testing, I had reasons enough to call Microsoft Support. After a few calls back and forth, they recently came back to me confirming that this is a known issue and that they’re currently working on it.

Until today I’m still not sure what the cause of the problem is. I haven’t received any feedback yet either. Of course, I will keep you posted as soon as I find out more!

Temporary workaround

It might sound too obvious, but the workaround is simple: either create both archive and mailbox in the cloud or create the (both!) on-premises first and move them together to the cloud. Both cases work just fine!

Conclusion

Although the last thing you’d want to experience is data-loss, I’m well aware that only a few customers, world-wide, would try this scenario. Nevertheless, it’s an issue that should be addressed quickly.

In our case, we have lost only a single archive worth a few hundred megabytes of emails. I can imagine that losing the wrong kind of emails might be a real big issues for some companies. I haven’t asked, but I’m pretty confident that – even though the emails seem lost – Microsoft can somehow recover the data so that you don’t really “lose” anything. I honestly cannot imagine otherwise.

Does this mean that I discourage using features like EOA? Absolutely not. I still have my hosted archive and I am pretty happy with it. Apart from some inconveniences which I will write about another time, it provides me with everything I need. Furthermore, it allows us to give everyone a relatively large archive without having to bear the costs of additional storage.

Until later!

Blog Exchange Hybrid Exchange Office 365

Disappearing (online) archives after moving your mailbox to Office 365…

Office 365 offers great ways to enhance the functionalities of your on-premises deployment. By running the Hybrid Configuration Wizard (which Steve Goodman explains in this article) you can configure both environments to act as one; allowing you to make use of features such as e.g. Online Archives (EOA).

With Exchange Online Archives, your primary mailbox stays in your on-premises Exchange server, whereas the archive will – as the name might have given away – be hosted in Office 365. If you’re interested in finding out more about Online Archives, I suggest that you take a look at Bharat Suneja’s session at TechEd this year: “Archiving in the cloud with Exchange Online Archiving

The problem

To me, one of the most interesting things about a hybrid deployment is the flexibility it offers. You can put a few mailboxes in Office 365, try them out and move more to the service if you like it.

If you are looking to take that approach, this information might be interesting for you!

Imagine the following: you are trying out Office 365 and decide to use Online Archives to start with. You provision the archives and life is great! After a while you decide you want to use more and you decide to move some mailboxes to Office 365. However, after your users have been moved to Office 365 they start complaining that their archive is empty.

It seems that – although this scenario is supported – there are some issues with the provisioning process when you move a user to Office 365 that previously already had an Online Archive: it get’s “wiped”. At least, that’s how it looks like.

At first, I though the data would reappear after a while, so I made sure that I waited long enough. Unfortunately even after a few days, the archives was still empty.

I decided to do some tests, to make sure this wasn’t a standalone case. Perhaps something went wrong during the move. To my surprise, tests confirmed what was going on: although the archive contained items prior to the move, they are now empty.

To explain what happens, let me describe the process I used to reproduce this issue.

This first screenshot show the details of the on-premises mailbox that has a cloud-based archive (EOA) enabled. This archive contains 4 (test) items:

image

Afterwards, I moved the mailbox through the Exchange Management Console using the “New Remote Move Request”-wizard.

Because on-premises only a mailbox exists, you don’t have the option to move an archive (which is normal):
image

The move completed successfully, and after having waited long enough (DirSync etc.). I verified the mailbox’s settings:

image

The interesting part here is that the Archive, although having the same GUID, appears to have been moved to the same database as the mailbox. Before the archive resided in database “EURPRD04DG032-db055” whereas now it’s in “EURPRD04DG030-db041”.

To ascertain myself that this wasn’t causing problems, I decided to do another test. When executing the MoveRequest, I specified to what database the archive should be moved to. I made sure that the target database of the Online Archive was set to the database it was already residing in before moving the mailbox:

New-MoveRequest “Testmivh5” –RemoteHostName “hostname.company.com” –targetdeliverydomain “tenant.mail.onmicrosoft.com” –&lt;strong&gt;ArchiveTargetDatabase &lt;/strong&gt;“EURPRD04DC032-db055” –RemoteCredential Get-Credential)

Note   this cmdlet was executed from a remote PowerShell connection to Exchange Online.

After the move completed (successfully btw), I – again – waited long enough for DirSync/replication/provisioning to occur. I deliberately didn’t force DirSync to ensure that wasn’t causing any issues either. But alas, none of that helped: the archive was again empty.

A quick look at the object’s attributes revealed that – although a target database parameter was provided – the archive still got moved to the same database as the user:

image

Then, I was thinking that the ‘old’ archive perhaps got disabled and that a new one was created. Although this would be strange since the GUID of the archive remains the same, I thought it was worth a try. Again: no joy! No disconnected mailboxes were to be found.

After all this testing, I had reasons enough to call Microsoft Support. After a few calls back and forth, they recently came back to me confirming that this is a known issue and that they’re currently working on it.

Until today I’m still not sure what the cause of the problem is. I haven’t received any feedback yet either. Of course, I will keep you posted as soon as I find out more!

Temporary workaround

It might sound too obvious, but the workaround is simple: either create both archive and mailbox in the cloud or create the (both!) on-premises first and move them together to the cloud. Both cases work just fine!

Conclusion

Although the last thing you’d want to experience is data-loss, I’m well aware that only a few customers, world-wide, would try this scenario. Nevertheless, it’s an issue that should be addressed quickly.

In our case, we have lost only a single archive worth a few hundred megabytes of emails. I can imagine that losing the wrong kind of emails might be a real big issues for some companies. I haven’t asked, but I’m pretty confident that – even though the emails seem lost – Microsoft can somehow recover the data so that you don’t really “lose” anything. I honestly cannot imagine otherwise.

Does this mean that I discourage using features like EOA? Absolutely not. I still have my hosted archive and I am pretty happy with it. Apart from some inconveniences which I will write about another time, it provides me with everything I need. Furthermore, it allows us to give everyone a relatively large archive without having to bear the costs of additional storage.

Until later!

Blog Exchange Hybrid Exchange Office 365