Spooky! The curious case of the ‘ghost’ File Share Witness…

Recently, I was doing some research for a book that I’m working on together with Exchange MVP’s Paul Cunningham and Steve Goodman. It involved recovering a “failed” server using the /m:recoverserver switch. The process itself is straightforward, but depending on what server role(s) you are recovering, you might have to perform some additional post-recovery steps.

In this particular case, I was recovering a Client Access Server (single role) which also happened to be the File Share Witness for one of my Database Availability Groups.
As such, you need to ‘reconfirm’ the recovered server as a File Share Witness. One way of doing so, is to run the following command:

Get-DatabaseAvailabilityGroup <DAG name> | Set-DatabaseAvailabilityGroup

However, upon executing the command, I was presented with the following error message:

clip_image002

Given that I didn’t use a System State Backup, I was surprised to read that a File Share Witness already existed.
The first thing I did was to check the restored server itself to see if the share existed. As expected, there was nothing to see.

By default, a DAG uses the File Share Witness + Node Majority Cluster Quorum model. This prevents you from removing the File Share Witness from the cluster because it is a critical resource. So, my next thought was to temporary ‘move’ the File Share Witness to another server and then move it back. First, I executed the following command to move the FSW to another server:

Get-DatabaseAvailabilityGroup <DAG name> | Set-DatabaseAvailabilityGroup –WitnessServer <server name>

The command completed successfully, after which I decided to move the FSW back to the recovered server using the following command:

Get-DatabaseAvailabilityGroup <DAG name> | Set-DatabaseAvailabilityGroup –WitnessServer <recovered server name>

I was surprised to see that the command failed with the same error message as before:

clip_image002[5]

I then took a peak at the cluster resources and found the following:

clip_image002[7]

It seemed there were now TWO File Share Witnesses for the same DAG, where the failed one is the one that used to live on the recovered server.

At this point, I decided to clear house and remove both resources. Before being able to do so, I had to switch the Quorum Model to “Node Majority Only”:

Set-ClusterQuorum –NodeMajority

I then re-ran the command to configure the recovered server as the File Share Witness:

Get-DatabaseAvailabilityGroup <DAG name>| Set-DatabaseAvailabilityGroup –WitnessServer <recovered server name>

Note:  when configuring the File Share Witness, the cluster’s quorum model is automatically changed back into NodeAndFileShareMajorty

After this series of steps, everything was back to the way it was and working as it should. I decided to double-check with Microsoft whether they had seen this before. That’s also where I got the [unofficial] naming for this “issue”: ghost file share witness (Thanks to Tim McMichael). If you ever land in this situation, I suggest that you contact Microsoft Support to figure out how you got into that situation. From personal testing, however, I can tell that this behaviour seems consistent when recovering a File Share Witness using the /m:recoverserver switch.

Blog Exchange Exchange 2013

A closer look at the “new” Public Folders in Exchange Server 2013

Disclaimer:

All the information in this blog post is subject to change as Exchange Server 2013 is still under construction. Although it’s not very likely that big changes will occur between now an RTM, it might.

In addition to some of the existing blogs and information out there, I wanted to take the time and explain the “new” Public Folders in Exchange Server 2013 a bit more in detail.

History

Ever since Exchange Server 2007 was released, Microsoft proclaimed Public Folders would eventually disappear. At that time, it was said Public Folders would stop to exist in “future versions” and consultants were given the advice to evangelize SharePoint as being thé replacement for Public Folders. It seemed, however, that these statements provoked quite some comments from the field: companies world-wide would still be using Public Folders and were not planning on giving that up so easily. Knowing what a migration to SharePoint could potentially cost, it’s quite understandable that some companies were rather reluctant to immediately jump on the SharePoint-train.

In fact, I cannot blame them. Even through today, there’s no real solution that offers the same functionality and ease of use as Public Folders.

Over time, Microsoft seems to have recognized that killing Public Folders might not be such a great idea after all and changed it’s attitude little at a time. Nevertheless, they’ve always deemphasized using Public Folders.

…At least until NOW.

To better understand at what the changes in Exchange Server 2013 are, let’s first have a look at how things were in Exchange Server 2010.

Public Folders in Exchange 2010 (and before)

In Exchange Server 2010 (and before), Public Folders were stored in their own database(s).

Basically speaking, Public Folders consists of two elements:

    • Hierarchy which contains the properties of the public folders and also  includes the tree-structure in which the Public Folders are organized. This tree structure contains Public- and System Public Folders.
    • Content which is the actual data (e.g. messages) in a Public Folder. Each Public Folder database contains a copy of the hierarchy, however not every database contained the same content:

image
image: a simplified graphical view of a public folder database

It was up to the administrator to define what content would be replicated across one (or more) servers. When a public folder was setup to have multiple copies, a process called “Public Folder replication” ensured that data was replicated across these “replicas” (= other public folder databases).

This replication is in no way comparable to how a DAG replicates data though: SMTP messages are sent between the different Mailbox Servers that hosted a Public Folder database:

image
image:a simplified overview of the PF replication model

Although from a data point-of-view having multiple writeable copies of Public Folders highly available: it is not. At least not when it comes to the end-user experience during a failover. Each Mailbox database is configured with a default (“preferred”) Public Folder database. Mailboxes within that mailbox database would automatically connect to their preconfigured “preferred” Public Folder database.

    • If – for some reason – the server hosting the Public Folder database would be unavailable entirely (e.g. offline), a timeout (+/- 60 seconds) would occur and clients would be redirected to another Public Folder database.
    • If, however, the server would still be online but the database would e.g. be dismounted, there would be no redirection and clients would end up with no Public Folders.

As you can imagine, not really a nice user experience, is it?

From a design point of view, Public Folders allowed you to easily spread data across multiple locations and have people make changes to the data in different locations. These changes are then replicated amongst the different replicas. In fact, Public Folders (in Exchange 2010 and before) are implemented in a sort of multi-master model:  all public folder replicas in a PF database are writeable.

Changes in Exchange Server 2013

Exchange Server 2013 entirely changes how Public Folders operate (from a technical point-of-view). From an end user’s perspective, everything remains as it used to be.

The main architectural changes that are introduced are:

    • Public folders are now stored in a mailbox database
    • Public folders now leverage a DAG for high-availability


Public Folder Mailboxes

Public folders are now also mailboxes, but their type is “Public Folder” (just like a Room Mailbox is a Mailbox with type “Room”). In a way, Public Folders still exist of two main elements mentioned above: the hierarchy and contents.

  • The hierarchy is represented by what is called the Master Hierarchy Public Folder Mailbox. This PF Mailbox contains a writeable copy of your public folder hierarchy. There is only a single Master Hierarchy PF mailbox in the organization.
  • Contents (in Public Folders) are now stored in one or more Public Folder mailboxes. These Public Folder mailboxes usually contain one or more Public Folders. Next to the contents, each PF Mailbox also contains a read-only copy of the hierarchy.

image

Note   because Public Folders are now stored in mailboxes, mailbox quota’s apply to them. For instance, this means that if a PF Mailbox would grow too large, you’d have to move Public Folders to another PF Mailbox (or increase the quota).

PF Mailboxes can grow quite large without really suffering from performance issues, just like regular mailboxes. Although real-world experience will have to tell us what the performance of Public Folders would be with a large amount of numbers, I’m confident that most companies won’t have a problem fitting a Public Folders in one (or more) PF Mailboxes. Nonetheless, you should carefully plan your Public Folder deployment, especially If you’re company heavily relies on them.

How do these “new” Public Folders work?

The process of connecting to and performing actions on a Public Folder could be summarized as follows:

    1. A user connects to a Public Folder (Mailbox).
    2. Any operation to the Public Folder is recorded in the PF mailbox
    3. If the data is located in another Public Folder Mailbox, the operation is redirected to the appropriate PF mailbox (which contains the Public Folder against which the action was performed)
    4. Hierarchy changes (e.g. adding or removing folders) are redirected to the Master Hierarchy PF Mailbox from where they are in turn replicated to all other PF Mailboxes (5).

Note   Hierarchy changes are never recorded through a regular PF Mailbox

image

High Availability.

Because Public Folders now reside in regular mailbox databases, they can benefit from the High Availability model that a Database Availability Group offers. Since you can store Public Folder mailboxes in any database, there’s no difference as to how they are treated in case of a failover.

The implications of changes

Because of these changes, the way we will implement Public Folders will also change. Foremost, placement for Public Folders will require a bit more planning from now on.There can only be a single active copy of a mailbox database at any given time, so we can no longer “benefit” from the multi-master model that existed previously.

This means that you should – preferably – place your Public Folder Mailboxes as close to your users as possible. If, however, to different groups need to work on the same set of folders but at opposite sites of the globe; you might be in for a challenge… Of course, one might start to wonder if Public Folders would be the right choice in such a scenario after all.

It’s not only good news

Unfortunately, every upside also has a downside: due to all the changes that were introduced, it seems that Microsoft isn’t able to get everything finished in time. At RTM we won’t have Public Folders in OWA. Perhaps this is something we’ll see added in SP1 (just like with Exchange 2010)…

Migration and coexistence

I will keep the migration and coexistence part for a future article. At the moment, Exchange Server 2013 Preview only support greenfield installs… On the other hand, if you’re interested in reading more on the migration stuff, you might want to take a look at a fellow-UC Architect Mahmoud Magdy’s blog about the new Public Folders.

Blog Exchange 2013