How an OpsMgr 2012 upgrade with multiple servers is handled, and how it can go horribly wrong

I am currently assisting a customer in upgrading their current Operations Manager 2007 R2 environment to 2012. There are several management groups, ranging from a development group towards a group with all monitored production systems.

We decided to test the upgrade on the development management group first, so we could produce customized documentation and discover potential issues we could tackle. This would reduce the overhead and risk when performing the upgrade on the more critical management groups.

The development-environment consists of one RMS and one MS where all agents (3) where reporting to. By following the documentation provided by Microsoft, we drew out the following action plan:

– Upgrade the MS
– Upgrade all agents
– Upgrade the RMS

All prerequisites regarding OS and SQL version were ok, so no additional migration work was necessary.

So we went on our merry way and upgraded the MS, which went fairly smooth. We noticed that the following thing occurs when you are upgrading a MS, but not yet the RMS:

– The database is partially upgraded, but it is kept compatible with Operations Manager 2007. We noticed that the version in the MOMManagementGroupInfo-table still was that of 2007 R2. However, the APM and other tables are already added, so the database resides in some sort of ‘twilight-zone’ until all management servers are upgraded.

– This implicated that we could not yet use the Operations Manager 2012 console, as it couldn’t communicate with the database at this point.

So when performing the next step, upgrading the agents, you have to use the Operations Manager 2007 console, which will still work momentarily. The management server we upgraded previously still shows up in the management server view, including its new version number. The management server however cannot communicate with the database (the SDK service will log events about it), for the same reason we cannot use the 2012-console: the database isn’t fully upgraded yet.

The next step would now be: upgrade the RMS, this will trigger the full upgrade of the database, enabling all management servers and so on.

However, when we performed this upgrade, our upgrade crashed horribly with the following message in the log: ‘unable to start the SQL browser service’.  A quick check on the SQL server did indeed point out that this service wasn’t running, and even more, that it was disabled. This was very weird, as it is a rather important service, and that Operations Manager 2007 actually didn’t complain about it.

The worst part is, while this was an error during the internal pre-upgrade check, the upgrade did uninstall the RMS first! So the server was basically empty, leaving us with a 2012-MS and a partially upgraded database… As the database wasn’t compatible yet with 2012, both items were useless to each other. We did attempt some repairs, but in the end the conclusion was ‘back to the drawing board’. The management group will be restored to its previous state, and the upgrade will be re-attempted (with enabled SQL browser service) at a later time.

My gripe about this is that we weren’t warned about the issue. There is an exquisite prerequisite-checker which really helps out, but it bailed on this tiny ‘misconfiguration’. I am happy however that we encountered this issue at this stage and not later on on a production management group, so we learned a lot at a relatively low cost.

I am confident that our next attempt will succeed, but should I encounter anymore of this “small oversight, bug consequences” issues, I’ll let you know!

Advertisements
This entry was posted in #scom, #sysctr, Uncategorized and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s