• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Server Farm Details

T

TechBoyJK

Lifer
Hello,

As I promised, I am posting the current info about a server farm. It hasn't been built yet, and many parts of the plan are still undecided. Some members showed some interest in what I have so far, and said they would be willing to give a thumbs up/down for anything I post. Constructive criticism is welcomed.

Location - The farm will be hosted in St. Louis at Cybercon, a web host that is still making money and has an impressive setup. They have connections to four different provider's backbones. I have a 42U rack to work with. I will be provided with an ethernet cable for the internet connection. It is burstable up to the full gigE speeds. Cybercon.com

I plan to get a 12-18 port GigE Switch, a 12-18 port KVM switch, and a 1U LCD Monitor/Keyboard sliding tray with trackball, all of which should be good enough to manage the farm. I am undecided of which switches to get yet, and I think the LCD Monitor/Keyboard combo I want is from APC. I will use Cat6 cable to make all connections.


Servers >


Firewall - OpenBSD + Packet Filter


Intel SR2300 2U with redundant 500 watt power supplies

Intel Westville SE7500WV2 Dual Xeon Motherboard

One (1) Intel 1.8GHz Xeon processor

1Gb Total (2x 512mb) PC2100 ECC Registered DDR Memory

Dual Intel 10/100/1000 Server Adapters On Board

4 additional 10/100/1000 Server Adapters (for direct crossover connections)

Onboard Video Controller

Dual Channel Ultra 160 integrated

Intel SRCMRU Raid Controller

Two (2) 9gb Ultra 160 SCSI Hot Swap Hard Drives

CD-ROM/Floppy Combo

price - $4,200



WebServer - FreeBSD + Apache


Intel SR2300 2U Rackmount Chassis with redundant 500 watt power supplies

Intel Westville SE7500WV2 Dual Xeon Motherboard

One (1) Intel P4 1.8GHz Xeon processor

1Gb Total (2x 512mb) PC2100 ECC Registered DDR Memory.

Dual Intel 10/100/1000 Server Adapters

Onboard Video Controller

Dual Channel Ultra 160 integrated

Intel SRCMRU Raid Controller

Two (2) 36gb Ultra 160 SCSI Hot Swap Hard Drives

CD-ROM /Floppy Combo

Price - $3,900



Coldfusion Server - Microsoft 2000 Advanced Server + Coldfusion Server 5.0 Enterprise


Intel SR2300 2U Rackmount Chassis with redundant 500 watt power supplies

Intel Westville SE7500WV2 Dual Xeon Motherboard

One (1) Intel P4 1.8GHz Xeon processor

1Gb Total (2x 512mb) PC2100 ECC Registered DDR Memory.

Dual Intel 10/100/1000 Server Adapters

Onboard Video Controller

Dual Channel Ultra 160 integrated

Intel SRCMRU Raid Controller

Two (2) 36gb Ultra 160 SCSI Hot Swap Hard Drives

CD-ROM /Floppy Combo

Price - $3,900



Database Server - Microsoft 2000 Advanced Server + MS SQL


Intel SR2300 2U Rackmount Chassis with redundant 500 watt power supplies

Intel Westville SE7500WV2 Dual Xeon Motherboard

One (1) Intel P4 1.8GHz Xeon processor

1Gb Total (2x 512mb) PC2100 ECC Registered DDR Memory.

Dual Intel 10/100/1000 Server Adapters

Onboard Video Controller

Dual Channel Ultra 160 integrated

Intel SRCMRU Raid Controller

Two (2) 36gb Ultra 160 SCSI Hot Swap Hard Drives

CD-ROM /Floppy Combo

Price - $3,900



Community Server - Forums/Chat - Microsoft 2000 Advanced Server + Coldfusion


Intel SR2300 2U with redundant 500 watt power supplies

Intel Westville SE7500WV2 Dual Xeon Motherboard

One (1) Intel 1.8GHz Xeon processor

1Gb Total (2x 512mb) PC2100 ECC Registered DDR Memory

Dual Intel 10/100/1000 Server Adapters

Onboard Video Controller

Dual Channel Ultra 160 integrated

Intel SRCMRU Raid Controller

Two (2) 18gb Ultra 160 SCSI Hot Swap Hard Drives

CD-ROM/Floppy Combo

Price - $3,600



Email Server - FreeBSD + qMail


Intel SR2300 2U with redundant 500 watt power supplies

Intel Westville SE7500WV2 Dual Xeon Motherboard

One (1) Intel 1.8GHz Xeon processor

1Gb Total (2x 512mb) PC2100 ECC Registered DDR Memory

Dual Intel 10/100/1000 Server Adapters

Onboard Video Controller

Dual Channel Ultra 160 integrated

Intel SRCMRU Raid Controller

Two (2) 18gb Ultra 160 SCSI Hot Swap Hard Drives

CD-ROM/Floppy Combo

Price - $3,600



Primary Photo Server - FreeBSD + Apache + ProFtpd


Intel SR2300 2U Rackmount Chassis with redundant 500 watt power supplies

Intel Westville SE7500WV2 Dual Xeon Motherboard

One (1) Intel P4 1.8GHz Xeon processor

1Gb Total (2x 512mb) PC2100 ECC Registered DDR Memory.

Dual Intel 10/100/1000 Server Adapters

Onboard Video Controller

Dual Channel Ultra 160 integrated

Intel SRCMRU Raid Controller

Two (2) 36gb Ultra 160 SCSI Hot Swap Hard Drives

CD-ROM /Floppy Combo

Price - $3,900


Secondary Photo Server - FreeBSD + Apache + ProFtpd


4U with redundant 500 watt power supplies

Intel Woodruff S845WD1-E Motherboard

One (1) Intel P4 2.0A-Ghz processor

1gb Total (2x 512mb) PC2100 ECC Registered DDR Memory.

Dual Intel 10/100 Server Adapters

Onboard Video Controller

3Ware 8500-8 Raid Controller

Eight (8) 120gb 8mb Buffer Hard Drives

CD-ROM/Floppy Combo

Price - $5,200



Backup Server - Backup for databases/email/primary photoserver


Promise Technology Ultratrak RM8000 3U ATA Raid Subsystem

Eight (8) Western Digital 120gb 7200rpm Hard Drives with 8mb cache

Price -$4,800



*note - all raid systems will be setup as mirrored arrays for failover.
 
Not sure if openbsd or freebsd support that 'Intel SRCMRU' raid controller, I don't see it on their hardware compat. pages. Most people don't need to waste space with a kb/monitor in a colocation environment, if your provider is worth anything they will have some there for you to use. for an all copper gigabit environment, I'd recommend the cisco 3508 or a 355012G w/ copper gbics. You'll be paying more for the copper gbics compared to the normal SX fiber ones though. A second CPU in the sql server would be a good idea too. On a side note, how do you know cybercon makes money? 98% of internet companies don't see any profit, ever.
 
about the boards and the raid support, i can use something else. i was also thinking about going with the Athlon MP systems.
 
June 03, 2002: Trio of St. Louis survivors drives past the tech wreck
Cybercon was one of three technology companies featured in the Business section of the St. Louis Post-Dispatch as having been able to successfully withstand the sharp downturn in the technology sector. The other two companies featured were GlobalStreams and Asynchrony Solutions.

May 27, 2002: Cybercon named in top 10 of St. Louis' fastest-growing private companies
Cybercon, a privately funded company, was ranked 9 out of 150 fastest-growing private companies in St. Louis for the year 2002 by the St. Louis Business Journal. Cybercon is headquartered in St. Louis, MO where it operates a 20,000 square foot data center in the downtown area.

They just opened a similar location in Newark New jersey and are planning one on the west coast.
 
The package we are looking to get from Cybercon>

One dedicated 10mbps Ethernet connection to your own network router, switch or hub;
One 8-port 10/100mb network hub;
One full height rack space for hosting multiple servers and network equipment;
1mbps bandwidth usage measured by the 95th percentile bandwidth basis;
128 IPs;
24 x 7 on-site live tech support;
20GB backup space on a RAID 5 system
 
What do you guys think of netscreen firewalls?

Firewall, VPN, and traffic management technology
4 auto-sensing 10/100 Base-T Ethernet ports
50 VPN connections or tunnels
8000 concurrent sessions
170 Mbps of firewall and 50 Mbps of 3DES VPN
 
1megabit/s on a 10mbit ethernet pipe is a far cry from a bursting gigabit connection. Do they let you remotely power cycle your servers, most colo places have that. If i were you, I'd go with a dedicated solution instead of colo. There are lots of places with much larger setups than what you got and dedicated is fine for them. Most competent dedicated server places will sell you a managed firewall as well.
 
I talked to the account rep at Cybercon. The line will be burstable to gigE speeds.

They do have a managed firewall service with the netscreen firewalls. They also do unix type firewalls.
 
1mbps on the 95% rule is about ~250GB of transfer per month.

Not sure I understand the 10 meg ethernet connectivity and then the burstable to gigE scenario. I don't understand the 10 meg specification if the bursting is specified as greater and your stream allotment average is 1mbps.
 
I will have to pay extra for the burstable, no question about that, i am only trying to point out that it is available. If we have special events and such, they will activate it.
 
One thing to look into, if possible is to negotiate your bandwidth by actual transfer, rather than by the 95% rule.

When we were on the 95% rule, we would hit it mid day, and usually around 5.5mbps, conversely though, in the middle of the night, we'd be running more like 1-1.5.

If you average that out over a month it's obviously way less transfer than 5.5 sustained would be. Had we run 4-4.5 most of the time and bursted to 5.5, that wouldn't have been such a big deal, but as it was we were paying for a lot of bandwidth we weren't using. Then when we were doing some big marketing one month we bursted to around 8 just enough times for that to be what we paid for. Painful.

The contract we have now is for 1mbps paid for no matter what, and then we pay by the GB for anything after that, no matter what we burst to. So I could be pushing 15mbps constant for two weeks, then 0 for two more weeks, and I'd end up paying for 7.5 in the end because of how it works out.

Definately something to look into if you can get it, I know some places don't have that structure until you're at 5mbit + guaranteed, but it's worth asking about for sure.
 
For any given service you're offering, each server it lives on is a single point of failure.

OK it sounds like you've got the following

1.) "regular" webserver
2.) Webserver w/coldfusion
3.) another webserver w/coldfusion
4.) database server
5.) mail server
6.) primary photo server (webserver)
7.) secondary photo server (webserver)
8.) firewall

I definatly think you could consolidate a bit of the webserver for cheaper, and impliment some high availability. I'll come up with something better tommorow morning. There's beer pong to be played.

bart

 
Anybody have any ideas on additional "physical" security? I would be happy to have retinal scanning. Is there any security devices I can get for the monitor and keyboard level? A keyboard that requires a thumbprint or something?

Cybercon also has a location in Newark, New Jersey, so setting up a mirror there is definately part of phase two. I have to make do with a single location farm for now.
 
retinal scans and all that crap is useless. physical security is provided by the colo company, not you. You should have a key to your cabinet, that's all you need. It's very rare that a whole datacenter will go down, so only staying in 1 location would be wise. buy a load balancer, a few 1u webservers and LB them. I've worked as a tech for a while with people like you who are new to the data center thing. A few dedicated servers is exactly what you need, pretty sure you'll be unhappy with colo.
 
A few quick thoughts. You may want to look at a KVM switch that will do KVM-over-IP, unless you really really want to drive to the datacenter a lot. Combined with either a VPN or a private admin circuit will be helpful. A KVM switch that uses twisted pair is a real headache saver. Not that you have a whole lot of servers, but the sheer bulk of KVM cables is a nightmare.

What is your plan for data backup? Your colo facility likely has a managed service that you will subscribe to, or will you be driving down to swap out tapes on a regular basis?

Physical security at the customer level should be handled by your datacenter. Some sort of biometric access, plus lockdown at the customer level.

I see that you have redundant power supplies, which of course is good. Just be sure that you are getting split power to your rack. It sounds like a basic thing, but (especially with smaller local and regional facilities) I've seen the incoming power traced back to a single PDU or UPS.

-jw
 
KVM over IP sounds interesting, but I don't have any experience with the implementation of it. Does anybody have any links so I can see if this is something I can setup myself? How secure is this?

The datacenter isn't that far away, fifteen minute drive at the most. Like I said earlier in the post, I am getting an apartment two blocks from the datacenter.

With the server farm package, we get 128 IP's, A full 42U rack (40 usable), a one meg ethernet connection (for starters) and we also get up to 20 gigs of data backup space on their raid 5 servers. We will have our own backup server, but I will use the 20 gigs of space as a third level backup.

About onsite security, you have to pass through three gateways (locked with key/and or palm scan) in order to get to the datacenter.

I also thought about getting something like a P3 800 with 256 megs of ram and a few 120 gig drives that I could locate at their New Jersey center to act as an additional (not mirrored) backup source. I think their rep said they can offer me a discount on data transfer between datacenters if we setup a vpn or something of that sort. I'll ask about that.

About the redundant power sources, I guess you are referring to "not" hooking both redundant power supplies on a single server to the same upc, powerstrip, etc. This way if the whole power source gives out, the other power supplies are hooked to another source. Good Idea! Thanks! Little things like this are incredibly important.

 
I dont understand why you'd have two 1.8GHz in the firewall and one in the SQL server. Nor does the firewall need a gig of ram. Why does the the primary photo server only need 36GB while the secondary needs about a TB ? I think you should consider SCSI so you can atleat share the storage, no reason for a duplicate 8 IDE setup then if you have it setup with raid 5 and a couple of hot spares in scsi and share it among the three servers.

Also, why MS win2k AS ? AS is more expensive then server and I dont see why you need it. Also, on the freebsd boxes, why dont you combine the apache and qmail server? Then run a second identical one as a failover?


I dont think you have very much redundancy here at all, and I agree with everyone else saying you should have dedicated hosting
 
when you say dedicated, are you referring to leasing the servers from the host and letting them manage the actual server?

With all of the servers, we will start with a single processor in the server and upgrade if we need to. The firewall should only need one processor, as it will be running openbsd and can only use one processor, but I was trying to keep the MB's the same across the system, for the sake of redundancy. Yea, if you read closely, its a dual processor setup, but we only put one processor in for starters.

You are right the firewall doesn't need a gig of ram, I will bring it down to 512.

I am trying to keep the servers seperated by application. The servers do have redundant PS and hot swap mirrored drives.

Mistake - I didn't list that I am going to have a real backup machine (not running but plugged up) so that if a machine fails, we can drop the drives into the backup machine and reboot. By having the mb across the farm, this should be easier and somewhat transparant to the end user. I also plan to have spare parts on site, so if one machine fails and we start using another one, we can also fix the failed one.

Yes, in a perfect world we could afford to buy duplicate servers, but for this phase, I need to get the basic foundation set first. Once I have the farm setup the way it should be, I will then focus on replicating it. Money is a huge object in this project.

Windows 2000 Advanced server is selected for the clustering service. Is there another way to get this w/o paying extra for advanced server? I don't need it for anything else that I know of.
 
oops, I thought it said two cpus in the firewall

nope, AS is good for the clustering service, You didn't state anywhere you were going to use it.

Having an identical machine to put drives in works, but requires your driving to the site and manually doing it, I would suiggest consolidating your hardware a bit and clustering everything from the get go.
 
Nope...if you want to do clustering with Windows then you have to buy Windows 2000 Advanced Server.
Something to remember as well.....your applications may or may not require a seperate version that can handle clustering.

Also, did you ever think about the fact that the hosting comapny probably has some sort of Local director or layer-7 switch availabel to you?
You could drop your 2k AS boxes down to regular old 2k server and do hardware load balancing,
 
Don't get me wrong, I want as much replication as possible. I'm a paranoid person by design, so I would be happy if everything* was replicated ten times over with every fail safe measure possible ready to go. But that isn't possible.

I was thinking about keeping the application servers seperated as listed, but also designating each server as a basic "secondary" server for another application. And from there I guess I would just set a rule that if "primary" server A does not respond look to secondary server "b" If I am wrong please let me know.



*Everything!
 
Also, I have four extra gigE nics set for the firewall. I was going to use a crossover ethernet cable and directly connect the two most important servers (yet to be decided) to the firewall. The other four ethernet ports on the firewall would be for two incoming (from two different routers, kinda like the power source thing) and two outgoing to two switches (probably twelve ports each). Each server behind a switch will have at least two gigE ethernet ports, so each server will be connected to both switches. I'd like the network setup to be a mesh.

As part of the setup process, I am creating a text file for each object in the farm, from the server as a whole, all the way down to the individual parts. Each item will have a full description of how it precisely relates to every other object on the farm. This should make troubleshooting the network much easier.

ex gigE Nic #1 in server #12 IP Address accepts/denies

oe PS Model #xcdftg fits servers #1-8 not #9-12

I think i am going to create another post about the subject of creating a server farm object index.
 
SaigonK, thanks for the headsup. Cybercon actually offers hardware load balancing, so I will talk to them about it.

Doesn't Linux have a load balancing project out there? Is this something we could implement on our own w/o having to pay a hard fee to the host? Or could we do this in tandom with the hosting company as an additional layer of load balancing? or will this just make things more bloated?
 
KVM over IP: you can get one that works just like a regular KVM switch (ie has a local keyboard and monitor), but also has available client software to access over the network. Avocent is a popular brand. I know you said you'll be close the datacenter at all times, but it's really handy. Especially when VNC hangs, or you need to get into a machine's BIOS or something to that effect. It's not a secure solution in itself, so should be used with a VPN connection or something similar. (Which is another point - do you have a VPN planned for regular machine maintenance and administration?).

I'm not sure exactly what you were getting at with the text file for each object thing, but obviously documentation is important. Also, when doing your cabling, I'd recommend labeling all of the cables. You can probably borrow a specialized cabel label printer (prints smaller than regular label printers) from the datacenter folks. Labeling the cables can save hassle down the road, especially if you end up having to run through wire guides and that sort of thing, or you're doing some maintenance and see a cable dangling that is unplugged from God-knows-where. So, for example, on both ends of an ethernet cable, print out what's on each end, both the device and the port or interface.

STLWEBSRV01 : ETH0
STLSWITCH01 : 1/12

On both ends of a cable. You can easily see that this particular cable is into your first web server, ethernet interface 0 on one end, and your first switch, port 12 on module 1 on the other.

I know some of my tips are more geared to large setups, but I figure they can scale down easily. Plus, it's good habit, right?

-jw
 
Yea, little things like proper cable marking are really important. Any fact about the farm that could have any relevance during any troubleshooting session or scheduled maintenance should be documented and/or marked out.

About the vpn connection, I really wanted to try and get by w/o one, for security reasons. I was hoping to avoid any remote connections.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top