PDA

View Full Version : A story from inside the datacenter


datacentercop
10-28-04, 04:48 PM
I have waited awhile to tell my story to try to gain some perspective to the entire situation before I went public. I recently quit working for a popular data center that receives posts regularly on this site due to several reasons. The main reason was the complete lack of professionalism that I witnessed on a daily basis. This provider's claims of no tier-1 technicians, on-site certified experts, fully N+2 redundant UPS, ultra-secure flagship datacenter, multihomed BGP network, A/C Power fed from our outstanding UPS System, just isn't true!!

While I was there, I witnessed customers being lied to on a daily basis. When I was told to lie directly to a customer, and refused, I was repremanded for it.

So to everyone out there researching companies for a good data center, my only advise to you is: do your homework. If you are with a company that has several "outages" or a lot of "downtime" for "reasons beyond their control", please press a little further. Just because a facility is in an area where a hurricane went through, that doesn't excuse any loss of power IF they claim that they have redundant systems. When they say that they "double and tripple" their staff, it really is only one or two extra people. For a good indicator of strength, don't just look how a company performs during an adverse event, look at how they handle the aftermath. They may have all of their staff working during an emergency, but what about after that emergency is over?

If they claim that they have several carriers (not just level 3), then why does the network go down while level 3 does planned maintenance. Couldn't they just switch over to another carrier?

Also, why in the world would you build a data center that is in a level 2 flood area, right next to the water? They may claim that the datacenter is a room within a room, but what about the roof? If the roof comes off the building, wouldn't this particular datacenter be exposed?

Also, they may be able to keep the datacenter up and running during an adverse event, but what about the NOC? If the NOC has no lights and no A/C how can that be an effective operations center?

But the most important (at least to me) factor when considering a service company (any company that provides a service) is to observe how they treat their employees. How can a company provide "...top-notch service to our customers" but yet treat their employees the total opposite?

Haven't you noticed some key employees just "gone"?

Also, what if you (the custoemr) are not an expert in managing a server? Shouldn't the company that you hire to provide you a server, at least try to answer some basic questions? I saw, time after time, one particular tech support person specifically put customers down, not answer their questions and totally belittle them in a service ticket, all because he was their only "expert" on a certain OS. If you are a new user, or you have questions, don't let the tech support person treat you with disrespect; take it up with their supervisor.

And speaking of supervisors, I was totally shocked to find out that this particular company's tech support manager was completely ignorant of the most basic tech support issues. I watched as she hired someone with NO experience, put this person on third shift ALONE, and expected this person to run the shift with no help. How can this possibly make any since?

So, in summary, when choosing a datacenter, do so carefully. Always, I mean ALWAYS, backup your data. (I saw a tech wipe a hd clean, on purpose, b/c the customer pissed him off, and then claimed that it was "an accident").

One final thought, always pay with a credit card or another method of payment where you can dispute the charges. I saw many, many customers taken advantage of, left with no recourse b/c they paid by wire transfer, money order, etc. Credit card (or paypal) customers were treated with much more respect, b/c the company knew that the customer could dispute charges on their bill.

Buyer beware and good luck!

Dixiesys
10-31-04, 12:28 AM
Haven't you noticed some key employees just "gone"?

This is the BIGGEST warning sign - if every time you visit (or call) the data center you talk to people you've never met before, EVERY TIME then it's a pretty sure sign something's up.

I know for LARGE data centers this isn't necessarily a good indicator but I prefer to stick with smaller/medium sized data centers. The local DC I used for a while every time I went there were different people there, even the "vice president" who talked me into signing on wasn't there for 2 months after I started, the only person I recognized last time I was there (to pickup my billing server that I wanted to leave there since it was 10 minutes away) was the net admin and I'm not convinced he COULD find another job but that's another story :D

gaiahost
01-01-05, 01:22 AM
let me guess... managed.com #1 pick... burst.net #2 pick... or our wonderfull favorite theplanet.. which is really starting to turn into managed and burst combined

NSCNAP Jeff
01-03-05, 01:21 AM
gaiahost,

None of those are in flood zones, but i'm relatively sure who he's talking about.



Also, why in the world would you build a data center that is in a level 2 flood area, right next to the water?


Thank you, i've been preaching this for months. :idea:

tranz
01-03-05, 08:04 AM
WOW. is all I have to say. Not to turn it into an ad thread but we havent had any issues with Atlantic.net Well all except that they charge $225/MB for BGP pipe.

All of us should learn from this post as what to look for and the level of service we should all be providing.

Adam228
01-03-05, 02:17 PM
Well the only place I know that has been hit by hurricanes like mad is Florida....I should know because I had to help clean all of them up lol..

My Advice if your colo to get a local datacenter, that you can tour....

Sometimes you have to rely on other peoples expiernces with that datacenter, or go with a reputable one :)

GH_mike
01-03-05, 04:28 PM
As someone else put it, not to make this an ad thread, but I physically go to this data center almost daily (our offices are across the street). They are the most professional and courteous people I have ever met and they are wonderfully friendly. http://data393.com/

NSCNAP Jeff
01-05-05, 06:14 AM
All of us should learn from this post as what to look for and the level of service we should all be providing.

Not building datacenters in flood zones *check* :D

Adam228
01-05-05, 02:18 PM
"Sop pumps are not something you want to have in your datacenter"

datacentercop
01-15-05, 09:05 AM
let me guess... managed.com #1 pick... burst.net #2 pick... or our wonderfull favorite theplanet.. which is really starting to turn into managed and burst combined


Nope, but good guesses though! This shows that you are doing your homework, and that is the most important point!

maxhest
01-26-05, 09:41 AM
I was thinking Maybe Sagonet psb dedicatednow.. but maybe not.

JeremyJ
02-01-05, 02:34 PM
Perhaps they are?...
http://cheetahsolutions.com/noc.html

There are reference to "top-notch service to our customers" at the main page, claim "...built as a "building within a building"" which similar to "...the datacenter is a room within a room..." in the post.

There is "...full N+2..." mentioned as well...

Have I catched them?

latefreight
02-02-05, 01:15 PM
Perhaps they are?...
http://cheetahsolutions.com/noc.html

There are reference to "top-notch service to our customers" at the main page, claim "...built as a "building within a building"" which similar to "...the datacenter is a room within a room..." in the post.

There is "...full N+2..." mentioned as well...

Have I catched them?
Ding Ding Ding! I think I see a winner!

JeremyJ
02-02-05, 02:27 PM
Ding Ding Ding! I think I see a winner!

Actually no matter who is the "winner" - the fact that well-known company provide a false information to the clients - it is a very sad fact...

RiderMayBail
02-08-05, 09:48 AM
This is the first time I have even looked at this provider, and just reading their descriptions on their NOC page (linked above), I would go elsewhere. The wording, like most, is prepared as a sales statement. While this is generally expected, they left out some key factors, and worded some things in a way that they really don't commit to anything, and just convey a message to pull in people that don't know any better.


Besides the staff having access to 590 Tons of CRAC(k) (sorry, couldn't resist that one)

Our Engineers have designed a Full Cisco Powered Network that optimizes wire speed technology
What was designed by the engineers, does not necessarily mean that it was implemented. Sure, they may have Cisco routers and such running the network, but perhaps some of the redundency was left out of original deisng to save on build costs.

In addition to redundant dedicated connections to Tampa's largest Carrier Hotel, our facility also has extensive availability of dark fiber and several Internet Carriers onsite.
So, the building has multiple connections to a carrier hotel, but once inside that building, how many contracts/agreements/connections are actually in place for protection. Dark fiber? The entire point of fiber is to have light on it, turning up dark fiber isn't a 15 minute emergency restoration type job, most of the time, it take days, if not weeks or months to get the equipment in place to actually use dark fiber. In my professional experience, I would imagine that this dark fiber is contained within the same cable as the existing live fiber, in which a cut would affect them all anyway.


Of course, this is all just my opinion, and goes along with what has been stated previously in the thread. This is not based on actual experience with this provider, nor any personal references.

Cow
02-08-05, 04:34 PM
DARK FIBER? :rofl: The idea of dark fiber in an area is not because it's there and full of bandwidth and huge capacity which what I'm sure he's trying to get at. To the average man, "Dark fiber" may sound like a fancy term which is again I'm sure they are playing on. Of course, I don't think they know what dark fiber really is.

Most Tier-1 carriers have most of their "dark fiber" in areas such as the ocean, lakes or remote places that will cost more to light. Secondly, it's there for future use if necessary.

Please, give me some dark fiber for lunch!

RiderMayBail
02-09-05, 05:28 AM
Actually, most dark fiber is on your normal fiber routes, being that they generally install a larger cable than they initially require, and thus just leave most of the pair dark. Dark fiber in the ocean is not overly abundant (decommissioned cables excluded), as the cable is expensive (especially for the double armored shore ends), and so expensive to lay, that most submarine networks will only put in a cable with 2,4,6,8 pairs, and light as many as they can. In most cases, if the submarine network has any dark fiber on the transoceanic portions of the network, it would only be a couple of pairs.

Compare that with throwing in a 30-50 mile cable along a highway, the cost isn't any more for installation for a 72 pair cable than it is for a 24 pair cable, only the intial purchase cost, so they move up a few pairs for future growth.

I'll stop there as I don't want to hijack the thread and take it off topic ... ooops

latefreight
02-10-05, 05:20 PM
Actually it is quite interesting. It is fascinating to hear about those ocean cables. Nobody talks about fiber optic ocean cables. It should be a new thread, I mean I think most average people are like: :confused: There are fiber optic cables in the ocean???? Its just annoying explaining this over, and over.

RiderMayBail
02-11-05, 10:44 AM
No kidding, I have to explain it every time someone asks me what I do, when I tell them submarine fiber networks, they almost always think that I am talking about actual submarines. :bonkhead:

Ross
02-13-05, 08:59 PM
I was thinking sagonet but perhaps the author should just post who he is talking about...

mojoServers
02-22-05, 09:10 PM
I'm sure it bites being with a colo provider who lies to you, but its worse to pay for services and have the owner run off with your cash.

:eek2:

tcgwebs
04-13-05, 02:44 PM
I'm sure it bites being with a colo provider who lies to you, but its worse to pay for services and have the owner run off with your cash.

:eek2:

You're one to talk! My server had maybe a 50% uptime, you don't answer the phone or IMs, and I'm doing a chargeback to get my $199 back for my dedicated server that I've since migrated from. And I can't get to your website either. It really bites being with a hosting company who lies to me, but its worse to pay for services and have the owner run off with my cash.. *rolleyes*

Chrysalis
04-21-05, 02:23 PM
to me looks like sagonet, hurricanes and level3 been the biggest giveaways.

Francisco
04-22-05, 08:46 AM
I was thinking sagonet but perhaps the author should just post who he is talking about...

It's sagonet.

NyteOwl
04-22-05, 10:21 AM
Not building datacenters in flood zones *check*


The same could be said of areas subject to frequent tornadoes, or in seismically active areas subject to earthquake shock. :D

tuxman
05-09-05, 02:19 PM
That's sagonet for you...