A Peculiar Bug in Server 2012/R2 Essentials and 2016 Essentials that is not exactly a "bug"

[BonzaiDuck]

So I had my new (home) server built, setup and running with 2012 R2 Essentials. AV software from AVAST installed; StableBit DrivePool installed; Serviio Media Server installed; SyncBack Se installed; backups for all client machines started and occurring successfully; users and machines added -- all checking out as "Kosher."

My backup solution in a nutshell involved using Windows Backup exclusively for the server OS disk, planning another backup of important data with SyncBack SE to a different disk and volume.

I decided to change out smaller drives in the drive pool with larger enterprise-class disks. (See the Newegg sale on "refurbished" 3K7000 Hitachi 3TB drives). Of course, I went through the process in Stablebit to drop a drive at a time from the pool and successively add back the replacement drives. Shouldn't be a problem -- should it?

Soon thereafter, I booted up the server and attempted to open the dashboard. It took about ten minutes for the dashboard to run! I clicked on the "Users" tab. Suddenly, all my users are just -- gone! I clicked on "Devices." All the client computers had disappeared!

Controlling my panic and inclination to reinstall the OS all over again with the software, the hard-to-find drivers -- I started poking around the web to see who else had been having this sort of trouble. Easy to find -- there were lots of folks.

Among the things they mentioned doing just before the symptoms occurred was new disk installation. What could it be? Why hasn't MS fixed it? How can I get back my dashboard?

Event log symptoms included failure to start WSSMgmtEng service. It would be in a "stopped" state, and attempting to start it would show "Starting" and then quickly flip back to "Stopped."

MS Techies tried proposing complex solutions, which many attempted, and few if any reporting success. In one thread, the posts just kept coming after one poster noted a simple way of fixing the problem. It was as though nobody noticed his comment.

What did he do? He executed the command " wbadmin delete catalog" from an elevated command window or Powershell. This deleted what apparently was corrupted catalog files for the server's own backup.
One could make some vague but sensible guesses as to why this had anything to do with the dashboard display of "users" and "devices." But people were wringing their hands and pulling their hair.

Apparently, adding a new disk can confuse the backup service in identifying the previously-configured backup drive.

Some advice for the server OSes suggests using a detachable or external USB drive for the server's own backup. I use a hot-swap drive in a 5.25" bay, which I leave connected most of the time, since backups are scheduled and automated.

But we all need to swap disks in and out.

What sort of procedure would guarantee this doesn't happen again?

I'd think one should probably remove the server backup configuration and disconnect the disk before adding new hardware, and then re-create the backup.

Certainly, someone who works in a data-center managing enterprise servers might have a clear idea of what is done routinely in those situations.

But I just had to post this thread with its questions. I might have completely missed the shortcut fix for it, and then reinstalled the OS and software from scratch.

Do people who work in data-centers managing servers ever thump their chests?

 

[XavierMace]

It's not terribly clear to me what dashboard you are talking about and I'm unclear about your backup configuration as well?

1) For the dashboard, are you talking about Server Manager, or does Essentials have some sort of custom dashboard thing like the old Home Servers did? A quick Google search would seem to indicate yes, but I want to be sure.

2) For the backups, if I'm reading it correctly, you have a single server and you're using the built in Windows Backup to backup the boot drive to the 3rd party drive pool on the same system? I could easily see pool changes causing problems with Windows backup.

This isn't a situation you'd run into in an Enterprise environment. Enterprise environments don't use local accounts, don't use a 3rd party apps for their storage, and generally have a dedicated backup server/device or a replication system. If they can't afford or choose not to have a dedicated server for backups, they generally use USB drives for backup at that point. I've literally never seen a server in a business environment running 3rd party software for storage. There's far too much that can go wrong and when something goes wrong with your storage it usually goes really wrong.

 

[BonzaiDuck]

It's not terribly clear to me what dashboard you are talking about and I'm unclear about your backup configuration as well?

1) For the dashboard, are you talking about Server Manager, or does Essentials have some sort of custom dashboard thing like the old Home Servers did? A quick Google search would seem to indicate yes, but I want to be sure.

2) For the backups, if I'm reading it correctly, you have a single server and you're using the built in Windows Backup to backup the boot drive to the 3rd party drive pool on the same system? I could easily see pool changes causing problems with Windows backup.

This isn't a situation you'd run into in an Enterprise environment. Enterprise environments don't use local accounts, don't use a 3rd party apps for their storage, and generally have a dedicated backup server/device or a replication system. If they can't afford or choose not to have a dedicated server for backups, they generally use USB drives for backup at that point. I've literally never seen a server in a business environment running 3rd party software for storage. There's far too much that can go wrong and when something goes wrong with your storage it usually goes really wrong.

That's probably pretty good judgment about these OS versions and add-on software. But I've never had a problem with Stablebit Drivepool. All of the add-ons I use were meant to run on these Essentials versions of the OS.

What I gleaned from the discussions I'd found pertaining to this problem was as follows. The trouble occurred when backups were scheduled for Windows own backup service. There was nothing of any commonality among folks stymied by the problem that indicated drive-pool software. The problem simply occurred after adding a disk, whether or not it would be used in storage spaces or another type of drive-pool. Also, to eliminate misunderstanding -- no -- I'm not trying to use Windows Backup with a target drive that is a drive-pool. That won't work.

As for the dashboard, yes -- the 2012 R2 Essentials dashboard is similar in concept to the WHS-2011 dashboard. One doesn't need to use it, but then using traditional features to keep track of users and the machines connected under the domain controller can be tedious. I had noticed that the users and devices database had not been lost, but it was absent from the dashboard. One person who added their troubled experience to the forum discussions noted that "I don't use the dashboard, so I don't care." Not my view and philosophy about this: if a component of the OS isn't working properly, even some unnecessary component like Dashboard, it needs to be fixed.

If I didn't say so, my plan -- followed with the old server -- was to use Windows backup for the server OS-boot-system disk exclusively. The data on the other disks is above all duplicated at the level of chosen files and folders -- a feature of stablebit that offers protection for even more than 1 simultaneous drive failure as desired. Similarly, the folders are distinguished as to data that needs to be backed up, and data for which loss of some of it could be acceptable (DVR'd movies, for instance.) I had originally used RoboCopy and RichCopy to do periodic backups of the important data to a hot-swap drive. I finally settled on Syncback SE -- a $40 lifetime license which can be reinstalled on new hardware if needed.

Because I'd been impressed with use of Macrium backup on my best Skylake system, I started poking around to see "what's out there." Macrium Free, HOme or Workstation won't work on server OSes. Their server-version is priced at $275. AOMEI Backupper server version costs $169. I see how I can automate unattended incremental backups with Syncback. So I'm in no hurry about spending more money.

I understand what you are saying about the "Enterprise environment." But at this point, I don't have either a desire or an obligation to replicate some enterprise microcosm in my household. I thought about having a backup server, if I could do it with the old box. But it's just unnecessary.

Also, I think you were referring to the issue of workstations joining the domain. I was happy with that idea, until I realized I needed to migrate user profiles from "local" to the domain profiles established when each user is added. There are migration tools available for free. In the meantime, what M$ meant to work as a "workaround" for joining the domain is -- well -- Microsoft's idea. Until I can feel comfortable about migrating those profiles, it will do just fine. They weren't about to make it impossible for a Windows Home OS installed on a workstation to access a server with a domain controller.

 

[XavierMace]

I'm not saying those choices aren't reasonable for a home setup. I'm simply saying that most enterprise people may not have much experience that would fit your setup. This is basically a home enthusiast setup that just happens to be running Server 2016 instead of Windows 10. I have a fairly unusual setup at home as well, so I generally have trouble finding people with similar experiences.

I'm also not saying you can't run things like StableBit (or Primo ) without having issues. I'm just saying that IF you start having issues, the more pieces there are in your puzzle, the harder it is to narrow down the problem. That's why people are generally much stricter about adhering to QVL's on the server side.

Also, sometimes developers make odd decisions that don't work with all environments. For example the monitoring system we use at work uses drive labels to identify the drives. This unfortunately makes it impossible to monitor the external USB drives clients rotate for backups because the labels are different so every time they rotate the drive, it breaks the monitoring. Why they decided to use the label instead of the letter, I couldn't tell you.

 

[BonzaiDuck]

Sure -- different environments and different aspects to configurations or priorities -- issues about economies-of-scale, as just one example.

On the one hand, a discipline of minimizing time and expense and imposing various requirements and restrictions on company employee computer usage. On the other side, how much time and trouble do you want to spend managing a network for a family of five or less? Maybe the home user could have more security vulnerabilities, or more malfunctions per user. You'd wonder what prompted MS to offer the public a special OS bundled with OEM compact-server configurations. And then, why they chose to terminate their offerings below a small-business option where a simple server was deemed . . . . uh . . . Essential. Not enough of a market, but a way of offering it to those willing to pay.

And it becomes a real customization when a home network has ambitions, like hosting its own web-page. Over all of that, both ways, is the security requirement.

But I like running my LAN, and giving the others in the house certain opportunities and advantages. I don't think a basement electric-train with scenery and little buildings on a table would do it for me . . .

 

[Ketchup]

I'll just put this here - I spent a few years running Server 2008 on my file/media server in the house. It was fine, did everything I needed, but there are things you run into that you just don't want to mess with at home - "this software doesn't run on a server OS", have to change this setting so my password doesn't expire, have to go through twice the processes to share files the way I want, "this driver isn't compatible with this version of Windows", and the list goes on. I finally bit the bullet and put Windows 10 on the machine earlier this year - and couldn't be happier!

 

[BonzaiDuck]

I'll just put this here - I spent a few years running Server 2008 on my file/media server in the house. It was fine, did everything I needed, but there are things you run into that you just don't want to mess with at home - "this software doesn't run on a server OS", have to change this setting so my password doesn't expire, have to go through twice the processes to share files the way I want, "this driver isn't compatible with this version of Windows", and the list goes on. I finally bit the bullet and put Windows 10 on the machine earlier this year - and couldn't be happier!

So, the way I understand you -- you chose to use a desktop OS for essential server purposes? Or you just eliminated a server-system from your household equation? If you opted to make a Win 10 system perform a lot of server chores for you, it reminds me of an article in Maximum PC magazine about configuring Windows 8/8.1 for that purpose. I had done that with Windows NT Workstation, when a system began to fail with a "training-evaluation" release of NT Server. Somehow, I made that configuration last until WHS 1.1 came along, then moved up to WHS-2011, and with the WHS I was headed in a server-OS direction.

It is an annoying aspect of getting older, but once I commit to a way of doing things, if it becomes a routine -- it's hard for me to conceive of changing.

I don't think I've come up to speed as to what components I'll need to provide as add-ons to Server Essentials. Take for instance backup.

I had been contemplating the task of taking apart the old WHS server and harvesting drives and the SSD. Then, I thought of the possibility to make one of the server boxes a backup server. And I looked into the capabilities of Windows (Server) Backup. The full-blown version of Server 2008 R2 and 2012/R2 (or later) provide the capability to backup a server using Windows Backup over a network. But I find this feature missing in WHS-2011 (2008 R2) and 2012 R2 Essentials as well.

Then, you look at other options with advantages over Windows Backup, and you find that the regular desktop versions of Macrium, AOMEI and Acronis won't even work for local backups on 2012 R2 Essentials. You have to buy their "Server" versions -- $275 for Macrium and $169 for AOMEI, and something in that range for Acronis.

And I finally find that I can get decent backup capability as supplement to any local backup made with Windows Server Backup with some $40 program like SyncBack SE. But, like Speedfan compared to ASUS AI Suite, you have to take charge and do a lot of tweaking for configuration. I'm still in the learning curve with SyncBack's new version, which probably after all will do network backup. It's more tedious than the other programs.

The WHS-2011 server demonstrates your point about hardware. You CAN use more mainstream desktop hardware, but it is always a scavenger hunt for this or that driver. Motherboard LAN Ethernet controller, for instance. One frets over proper installation of chipsets and USB drivers, but this has been only a minor hurdle for me with either WHS and 2012 R2 E. Is it worth it to you to buy that SuperMicro server board, the Xeon processor(s) and ECC RAM? I don't think so. But to some people, it's an easy inclination and choice.

I just think it's time to retire an NVidia 680i Q6600 system with 8GB DDR2 RAM and an nForce chipset that was short of AHCI 1.0 compliance. And there's also the PCIE v.1.0 limitations. That technology is maybe 12 years old, even though my components have been running for only 4 or 5 years. It was "leftover surplus cyber-stuff," but I made it work to my purposes well.

So with this box, I have a motherboard released 8 years ago purchased 4 years ago as corporate IT asset surplus for $80. The processor was a "pull" from an OEM system -- maybe $150. The RAM was part of a bundle I purchased 4 years ago, still sold new for $200 but I estimate it cost me $90 for 16GB. And the disks are "refurb" Hitachi 7K3000 enterprise drives that are essentially brand-new surplus IT assets. Without the original 5-year warranty, I got them at $50 each.

I can say it was cheap enough, but I can see that I will torture myself with it for the next few years.

 

[XavierMace]

So, the way I understand you -- you chose to use a desktop OS for essential server purposes? Or you just eliminated a server-system from your household equation? If you opted to make a Win 10 system perform a lot of server chores for you, it reminds me of an article in Maximum PC magazine about configuring Windows 8/8.1 for that purpose. I had done that with Windows NT Workstation, when a system began to fail with a "training-evaluation" release of NT Server. Somehow, I made that configuration last until WHS 1.1 came along, then moved up to WHS-2011, and with the WHS I was headed in a server-OS direction.

Unless I'm missing something, all you're using it for is file/media storage and potentially network backups. You're not really running any normal "essential" server stuff (AD, DNS, DHCP, Print Server, etc). You could just as easily do that with the consumer versions of Windows or better yet a Synology NAS. I convinced my parents a year ago to buy a Synology instead of adding a bigger drive to their main PC as well as to consolidate/get rid of an old PC. Best decision I've ever done. It backs up both of their computers, and I haven't had a single "help" call from them since the week after I set it up.

The WHS-2011 server demonstrates your point about hardware. You CAN use more mainstream desktop hardware, but it is always a scavenger hunt for this or that driver. Motherboard LAN Ethernet controller, for instance. One frets over proper installation of chipsets and USB drivers, but this has been only a minor hurdle for me with either WHS and 2012 R2 E. Is it worth it to you to buy that SuperMicro server board, the Xeon processor(s) and ECC RAM? I don't think so. But to some people, it's an easy inclination and choice.

People don't buy SuperMicro just to avoid installing drivers.

 

[Ketchup]

I also believe that the "standard" version of Windows has a much better backup system for home use. And the media and driver pool programs mentioned in your first post will run just as well or better on good ol' Windows 10.

 

[BonzaiDuck]

Unless I'm missing something, all you're using it for is file/media storage and potentially network backups. You're not really running any normal "essential" server stuff (AD, DNS, DHCP, Print Server, etc). You could just as easily do that with the consumer versions of Windows or better yet a Synology NAS. I convinced my parents a year ago to buy a Synology instead of adding a bigger drive to their main PC as well as to consolidate/get rid of an old PC. Best decision I've ever done. It backs up both of their computers, and I haven't had a single "help" call from them since the week after I set it up.



People don't buy SuperMicro just to avoid installing drivers.

I've always had "open" plans on this aspect of our home machines here -- in this regard of the server. When it's a matter of choosing Xeon-enabled hardware and ECC RAM, I'd always been intrigued with kloodging together my server system (whatever OS flavor it had) from spare desktop parts. It's just a common experience with people who do that: they will always encounter a surmountable obstacle about drivers.

What I have at this point is running great. It's backing up the other systems. I'm tweaking the server-backup configuration now.

This all started years ago when I was teaching ORACLE courses and insisted that I had a home-office setup in which the DBMS was installed in a client-server context as opposed to just using the "personal" version.

My needs now are flexible, and as I said -- open-ended with a serious wish-list. But I shelled out approximately $280 in 2015 for this version of 2012 R2 E -- probably the last academic version before 2016/R2 that can be used in a not-for-profit home pursuit without meeting an edu-e-mail or photo-ID purchase requirement. I'm "activated." 2016/R2 has those more stringent prerequisites for an "academic" license. It's money spent -- water under the bridge. It wasn't a tremendous saving over the regular retail license -- maybe $100 to $150. If I upgrade, I'll probably look at shelling out the bucks for the 2016 retail version. So I'm not just going to jettison success-thus-far so I can do the same thing with a desktop OS.

It's "up and running."

Here's the skinny on DrivePool, though. In order to use Windows Server Backup, you have to backup the "Pool Part" folders of the underlying disks, as opposed to the virtual drive of the pool. This means that you will have duplicates in the backup if there is file/folder duplication in the pool. That's why I'd initially settled on SyncBack SE. The expensive Macrium and AOMEI options can be made to work, but totally unnecessary. At the same time, the degree of duplication is a minor factor in the volume of backup data. So that works too -- with the Windows Server Backup.