Updates - Thursday, February 9th, 2012 11:14:14am
December 23, 11:45 AM - Happy Holidays!
Posted by: Jenn
Crosswinds has a new place for "CW" business, and a blog!
Check it out here: http://cwcrew.com/
Happy Holidays to you and yours!
--Jenn
December 11, 10:10 AM - Upgrade Complete!
Posted by: tony
The upgrade this morning but did take longer than expected. With computers this is normal...
Have a great weekend!
Tony
December 7, 2:00 PM - Major Upgrade
Posted by: tony
There is a major upgrade as we get ready to support IPV6. We will be having a maintenance window from 1-3 AM on Sunday Dec 11th.
There will be some hard downtime as the switches are physically replaced with new ones that support IPV6 and other functions.
This is a heads up.
Tony
August 29, 9:51 AM - The Summer of Hell - And It's Silver Lining
Posted by: tony
Hello All,
To say that things have gone badly over the past couple of months is a major understatement. The odd part is that almost every single issue is weather related.
How is that possible since we are in a data center with redundant power/cooling/network?
It's a long complicated story but I will summarize as much as I can.
This year has been the hottest most violent-weather (ie. severe thunderstorms) we've ever seen. The data-center is located in Montreal - known for cold winters but not for overly hot summers with thunderstorms. This summer saw many days on end with 35C and higher temperatures (40C+ with humidity) and with those high temperatures, lots of severe thunderstorms were born - in some cases one every 2 days. I would see them here in Toronto first, then up to Montreal they would hit the next day since the prevailing direction is West to East.
The first issue was a storm that affected power in Montreal - but it caused surges and brownouts that actively damaged surge suppression gear (they are rated to suppress a certain point but these were massive spikes). These caused some hidden damage to UPS-es and AC units. UPS units started to fail and were quickly replaced. In some cases, the secondary UPS-es failed.
Secondly, some of the AC units that were damage started to fail. Repairs were done... and some units still failed again (different types of damage).
Third, a series of very intense thunderstorms ended up in a direct lightning strike to the data-center last weekend - killing generators, circuits and suppressors again, with power going down as UPS-es drained.
Fourth, last week a series of very large internet providers had issues with routing and were down - 1 at a time is not all that unusual, but having 4 in a row is highly suspicious, until you consider all of the issues were localized to Montreal - weather would be the culprit.
What this meant is, we had servers that lost clean power over and over, got baked by failing UPS-es and ended up throwing their own errors due to hardware damage. Even when they were up, you couldn't reach them due to backbone issues!
This was finally resolved last Friday with the last (and most critical) server having hardware replaced and it has been stable since.
So, what is the silver lining in all this?
We have been planning since February to do a complete hardware refresh of ALL our servers - every one of them. It was just unfortunate (and fortunate) that all these power/heat issues happened before we got the new gear in. It is actually fortunate because if we had gotten all the new gear in and then suffered these issues, we would have had to replace all the gear for a second time. As it was, we identified and replaced what was known to be broken and are watching things carefully to make sure everything else is stable.
The second part of the silver lining is that the data-center has replaced, and upgraded, many parts of its infrastructure with a focus on power - bigger suppression units, newer generators with larger circuits, and newer, more efficient ACs with additional fault tolerance.
I am in the process of ordering the new hardware (this is a saga in it's own right). As a result the servers will be faster and more reliable.
For sticking with us, we are going to be giving *EVERYONE* upgrades when the new gear is put in place. Upgrades will vary with what you already have.
We're working hard to make Crosswinds better for you and thank you for sticking with us - even through these last months of utterly horrible conditions.
Thank you.
Tony
August 25, 10:58 PM - Back
Posted by: tony
cwpro1 is back up and running. I am not touching it until tomorrow!
August 25, 10:51 PM - Move Aborted for Tonight
Posted by: tony
A new issue appeared in the middle of the data copy - so we have to abort for this evening.
We identified a heat issue in the original system that caused it to reboot in the middle of data copy. The system is currently fscking and coming up.
We have a plan that shortens any continued downtime with this system and speeds up the actual final account moves when we do the final move - whether onto a temporary system or onto the new final systems that are being ordered.
In any case, it's aborted for this evening.
Tony
August 25, 9:11 PM - Argh!
Posted by: tony
In the middle of data copy, cwpro1 hangs...
Techs dispatched to bring it up so we can continue. Will keep you updated. This gear is getting flakier and flakier by the moment.
Tony
August 25, 8:43 PM - Account Copying is in Progress
Posted by: tony
There is a lot of data to move.
As your account gets moved over, your database might stop working. This will be temporary until we can reassign the identity of the box to take over the identity of the old box.
Do not panic! We are working to keep your data safe and sound.
Watch here for status updates.
Tony
August 25, 7:31 PM - Server Move
Posted by: tony
I am in the process of moving the accounts to the new server!
Things might get a little wonky but in general it should be smooth.
I will let you know when the process is complete.
Tony
August 25, 3:55 PM - Back
Posted by: tony
cwpro1 is back up - we will be moving accounts off tonight!!
Tony
August 25, 3:34 PM - It Knows...
Posted by: tony
So I couldn't get the move of accounts started yesterday due to thunderstorms and loss of internet. However I did state I was planning to remove accounts to another box until the full replacements in...
WELL it appears cwpro1 heard and is throwing a fit - it spontaneously rebooted with no warning.
We have gotten the process down to a science and it won't take too long to reboot, but something has damaged the disks (they've been transplanted to a new server) so I cannot rely on them.
Account moves will start tonight and continue until everyone is off, onto a solid server.
Tony
August 24, 6:24 PM - cwpro1 Back
Posted by: tony
cwpro1 is back up and running.
Something is wrong with the hardware - we are looking at the resources we have and will be moving accounts off that box onto others that are already there.
This will occur over the next few days and I will give notice when each batch of move occurs.
This is a temporary measure until the new servers come in but I can't risk loss of sites/data.
Tony
August 24, 5:31 PM - cwpro1 Rebooting
Posted by: tony
cwpro1 spontaneously rebooted - no error messages. The system is fscking and we'll get it back up asap.
It has been acting more odd since the first A/C issue last month and we suspect hardware damage. Replacement server has been ordered but is delayed due to backordered parts out of Japan.
We are investigating other solutions in the mean time. Will keep you posted.
Tony
August 21, 9:54 PM - Server is Back Up
Posted by: tony
The server is back up and the datacenter is investigating what other damage has been done. For the moment we are up and working. If any work/maintenance needs to be done we will be giving a heads up.
Tony
August 21, 9:20 PM - Lightning Strike
Posted by: tony
Got a series of alerts - there was a lightning strike that hit the datacenter directly. The UPS had a couple circuits blow out so some gear eventually went down after the UPS drained and the rest stayed up okay.
This has been a crazy summer for storms and the second direct hit we've seen. The first did no apparent damage.
Systems are just FSCK-ing (verifying the data) and coming up.
Tony
August 14, 4:10 AM - Finally Done!
Posted by: tony
A last minute issue made the maintenance go longer than it should have, but we are back up and everything is running great.
Tony
August 14, 12:34 AM - Maintenance Nearly Finished
Posted by: tony
I just realized that the message I put to indicate the start of maintenance did not go up!
In any case, the maintenance on the servers and A/C in our area is nearly complete. Service will be restored shortly.
Some work was done to prepare for new hardware that will be installed within a month.
It shouldn't be too long until this is done and I will post to let you know when everything is back up!
Tony
August 11, 8:48 AM - Weekend Maintenance
Posted by: tony
We will be conducting maintenance this weekend on the pro servers. There will be a brief outtage as the servers are rebooted after the maintenance, update and upgrade cycle is completed.
The reboot is expected to take approximately 15mins - no more.
The maintenance will begin at approximately 9pm ET on Saturday and completing at approximately 1am ET Sunday morning.
Tony
August 6, 3:54 AM - Back
Posted by: tony
The system took a while to verify but it's back up. It just hung - no reason, no errors, nothing.
We have a scheduled replacement of that server in the coming weeks - many upgrades are due!
Tony
August 6, 1:14 AM - cwpro1 Acting Odd
Posted by: tony
The cwpro1 pro host is acting odd - it's not responding but it's doing it in a way that the monitors think it's still alive... I've seen behaviour like this before.
Tech has been dispatched to investigate.
Tony
Powered by Coranto
|
