A Nice Welcome Back

Back in the middle of April my wife and I took a trip for our 4th anniversary. We saved up over the past couple of anniversaries to go somewhere big. We went to the Dominican Republic. It was wonderful.

The night before we left to come home I got a call from work to let me know that our exchange server had crashed and burned. My stomach immediately turned and my mind started to race. I was out of the country where my mobile phone didn't work, I had no computer access, so there was absolutely nothing I could do. I kept asking myself if I did have some kind of connectivity could I actually be doing something to fix the problem.

I was able to contact the church from our hotel phone and got some more details. None of them were good. After a $75 phone call I found out...

1. We lost a disk in our array for Exchange
2. The were able to call in a friend to come and help
3. A new disk was ordered and installed in the array
4. As it turns out when installing a replacement disk into an array on Dell PowerEdge servers there is a possibility that the array will become corrupted
5. Our array became corrupted when the replacement disk was installed
6. The most recent complete exchange backup was a month old (a whole other story)
7. The friend we had help us was able to purchase a program to restore the data from the corrupted array (Arax Disk Doctor - totally saved my bacon)
8. After the restoring the exchange database files we found out the they were corrupt

Needless to say I didn't sleep very much the next couple of days.

As I find out more information I learn that the server crashed the night that I left for vacation and had essentially been down the entire week I was gone. At the Sunday service they announced that the we weren't mad at the congregation and we weren't dodging their emails but our server had crashed. I my heart sank as did my body in the pew.

Over the next few days I worked nonstop to do a hard repair and a defrag of our database (which wasn't easy since the db was 40GB and we had almost no servers with enough room/power to deal with files that large). We were able to restore Exchange by Wednesday morning at 3:30 am with no noticeable data loss. The outage was about a week which is far beyond unacceptable. The only good thing is that it brings DR to forefront of management's mind which is something I've been trying to do this entire year.

Needless to say I needed another vacation to recover from the welcome back I received from my last one. The unfortunate part is that I may never be able to take a vacation again given what happened this past one.

I don't know what it is about servers but somehow they know when you leave and the worst possible things happen when you are gone. I think that it has something to do with separation anxiety.

Blackberry Storm

A few years ago Blackberry swept over Crossings like a raging storm. It was in high demand and a large portion of our staff decided that this was the answer to all of their problems.

Through these years I've seen the iterations of phones roll through like spring, summer, and fall. It is not uncommon for me to swap out a phone once a month or more. (We only have 15 Blackberry users). I am proud to say I'm still rocking and rolling with the archaic 7920. Don't be jealous, but it does what I need it to and it is as durable as a brick. I should know; I've dropped my phone down the concrete stairs more times than I'd like to admit.

The thing that interests me is how many people are jumping the blackberry ship. Has the SS BES lost its luster? I can't speak for every organization but I think that a few of our staffers finally realized that receiving and responding to email isn't as cool as it first seemed.

Don't get me wrong I don't see CCC going BB free anytime soon. We have several die hards that you couldn't pry their blackberry out of their hands with a rusty crowbar. These people also don't mind the expectations that comes with a Blackberry so it is a perfect fit. I just never thought that I see someone go cold turkey from a crackberry and not go through any withdraws. So far of the people that jumped ship have even mentioned going back to BES.
To me, this is neither a good thing nor a bad thing, it's just an observation.

Exchange 2007: Beginnings

Just to update you on what is keeping me busy. We have our new server and I am working at getting Exchange 2007 installed.

So far it has been pretty straightforward. In theory Exchange 2007 implementation is anticlimactic because there are wizards that do all of the installation tasks for you. So if you can read and install software in a windows environment it should be a snap getting MS Exchange 2007 loaded.

I did have a few issues getting the domain prepared for Exchange. Currently we have a Windows Server 2000 box running Exchange 2003. That machine use be the "everything server" here at CCC which really means he used to do everything.
Here is a list that this server did at one time or another...

• Exchange
• File share
• The only Domain Controller (for 100+ users)
  (which would include all of the FSMO roles)
• DHCP
• DNS
  (inluding WINS)
• Certificate Server
• IIS for Outlook Web Access

Call me crazy or "old school" but I think that is a lot to expect from a server so we have slowly been moving roles off of this workhorse to other hardware.

Here is where thing went a little rough for us.
Exchange 2007 updates the Active Directory Schema and it cannot have an Domain Controllers that are older than Server 2003. This forced me to demote our current server that had exchange on it. Scary! As it turns out she also was the primary DNS Server. After the demotion this server had nothing in DNS and on one could get any where. It took me quite a while to figure out why mail wasn't routing and people could neither get on the internet nor get to their mailboxes. It took a while to get everyone's DHCP changes updated and then there were the static addresses.

My motto is...
All in the name of progress.

The best part is that if everything else goes right no one should notice.

I will, however, notice many benefits for the upgrade.

• Most importantly: hardware that is under warranty and in good working condition
• The hardware is wicked fast (2 - Dual Core Opteron x64 Processors w/ 6GB RAM)
  
• the Exchange 2007 management has simplified many of the views for administration tasks
• All this server will do is be a DC and run Exchange.

It is all simply magic. Now all I have to do is get everyone's mailboxes over to the new server and make sure that BES 4.0 is working with Ex07 since it is not officially recommended.

Email Server

In my post "New Email Server" I revealed a dilemma that I face every time I upgrade software that we have had for awhile. I want to go and look out at other solutions that would be better/cheaper before I do my upgrade. Well I have finally made a decision on the Email Server and we are sticking with MS Exchange.
Reasons:

1. We already have Software licensing for exchange
2. We desperately need to replace hardware and enterprise is enterprise as far as I'm concerned.
3. People do not like change and training is costly
4. We are currently an all-production MS shop and everything fits very well together now
5. There is the possibility of implementing Share Point and I would really like it to fit in with our mail solution.

I like to talk a big game but I really have to see an extreme value difference to actually move.

New Email Server

We plan to replace our email server this year and I feel like we are at a crossroads. We currently use Microsoft Exchange for our email server and it is on version 2003 sp2. We also have the licensing to upgrade to 2007 with software assurance. While we are completely satisfied with Exchange I still feel compelled to take a look at some other solutions.
Some of the available solutions that have beckoned me...

1. Zimbra - Seems to be a fairly flexible message and collaboration suite that was just recently purchased by Yahoo!. It was written with open source but there is a version available that has a support option. The native interface is a cross browser compatible ajax enabled web client but you can use Outlook and most other common email clients including those for Apple or linux. It can run on Unix or Linux. It integrates with Active Direcory or other LDAP enable directories.
2. Go completely open source with mail and server. I am no where daring or knowledgeable enough to implement this but for some reason I'm always drawn to the idea. But with the importance of email these days it needs to be a guaranteed.
3. Go totally with a hosted solution. The options for hosted solutions are almost endless. You can have a hosted Exchange, Zimbra, Sendmail, or anything in between. This gives the responsibility to someone else for disaster recovery. Google mail looks the most cost effective and the integration of other google apps it increases the functionality. Hmmm, very tempting.

Sad to say that after it is all said and done I'll most likely stay with MS Exchange. Why you ask? Well, it is what everyone is used to. We are almost exclusively a MS shop so why stop now. The price point is good. I'll keep you updated on the new install of my exchange server. I plan to do a migration from 2003 to 2007.