Some of you might have experienced some network latency during the day and some minor outages. Here is what has been going on today.
At 5:00AM MST A diagnostic alert showed a problem with our link to XEEX/SAVVIS. Problem was causing minor service outage that we had a temporary fix on until issue is resolved.
At 9:00AM MST Problem was found and a solution was found.
At 11:00AM MST XEEX/SAVVIS was temporary removed from BGP announcement pending solution implementation.
At 11:00AM MST Solution was implemented.
At 11:30AM MST XEEX/SAVVIS link went out of service.
At 12:00PM MST We suspected it might be due to recent solution provided however the communication line with XEEX/SAVVIS was severed at Layer2 level which is not the problem we were having.
At 01:00PM MST Nlayer our second provider started reporting latency issues with Global Crossing impacting about 20% of our global traffic.
At 02:00PM MST After exhausting all attempts of a resolution on problem Edge switch was power cycled. This cleared the issue with XEEX/SAVVIS. Our Edge switch has not been rebooted in over years which we suspect might have been the reason since it maybe on it's MTBF (Mean-Time-Before-Failure) threshold.
At 02:30PM NLayer reported that Global Crossing has been able to decrease the impact of this network latency from 30-40% packetloss to 5-10% packet loss to the 20% of our global traffic impacted.
At 2;40PM MST XEEX/SAVVIS Link was re-activated all reported OK
At 3:00PM we are currently working on the 5-10% packetloss problem.
We apologize for the inconvenience this may have caused and we are doing our best to make sure all problems are resolved in a timely manner.
发表于 2010-5-20 10:38:34