Bandwidth and server requirements for an internet TV channel

[Atheus]

So my boss wants to bid on a contract for an internet TV site - basically a youtube clone that shows some official content channels as well as user created stuff. We would be responsible not only for writing the software (my department), but also for providing servers and bandwidth, so we need to come up with some estimates. If we get anywhere with the bid I'm sure we'll get the relevant experts to look at this, but for now it's just me, so...

Say we start off with an OC-3 line and serve videos at 250kbps that's 520 concurrent users - not all that many considering the videos would last quite a while. In fact if they lasted half an hour you would only need one request every 3.5 seconds to fill the pipe.

If we wanted to serve 5000 concurrent streams we'd be looking at 1.25Gb per second, or 400TB per month every month.

"what?!" he said, staring at his calculator...

Can you even buy that kind of bandwidth? Would it involve having fiber installed somewhere or could we get it from a datacenter? How do the youtubes of this world manage to pay for this stuff and still make money?

In a related question, how many actual (UDP?) connections to the streaming server would we be looking at in the above scenarios, and how much load would it be putting on the CPUs? I understand it's not simply a matter of making a request and then receiving the video - the client and server would actually be in regular contact...


Network guru responses appreciated :thumbsup:

 

[spidey07]

Of course you can buy that kind of bandwidth. What you should probably do is get multiple OC connections to multiple tier 1 providers and run full Internet routing tables.

Secondly - what you are looking for is multicast. You send out a single stream and then viewers "join" the stream.

General accepted bandwidth for broadcast standard definition video is 1.5 Mbs.

 

[azev]

with this much bandwidth, i think a colo is a much better option. Cheaper, and extra bandwidth can be added much faster.

 

[Atheus]

Thanks spidey.

Originally posted by: spidey07
Of course you can buy that kind of bandwidth. What you should probably do is get multiple OC connections to multiple tier 1 providers and run full Internet routing tables.

That's for redundancy/reliability is it? OK I'll pass that on.

Secondly - what you are looking for is multicast. You send out a single stream and then viewers "join" the stream.

Actual IP multicast to the whole world? Do providers even allow this?

I don't think we will be showing much actual live stuff anyway - it's a youtube clone as I said, where a user can choose what he/she watches and then the stream starts. That bit will have to be unicast regardless.

General accepted bandwidth for broadcast standard definition video is 1.5 Mbs.

There's no way we'd be sending that kind of bitrate - youtube/peekvid quality is what we're assuming for now. I know youtube uses a maximum of 260kbps, so that's the number going into my calculations until we get more info from the prospective client.

What I'd really like to know is how much CPU power this is going to take on the streaming server. Assuming we don't encode in real time I reckon it won't be that bad, but having never used a streaming server before, I really have no idea...

 

[spidey07]

The streaming won't be a big deal if you're not encoding. The multiple OC connections isn't just for redundancy, it's for capacity as well.

As far as server load - I'd put in a content switch to do the application layer load balancing front ending a server farm for content, that way you can scale it to however big you like.

That's how you do it right, that's how it's done in the U S of A.

 

[Atheus]

Originally posted by: spidey07
The multiple OC connections isn't just for redundancy, it's for capacity as well.

Looks like these colocation places (cheers azev) have many links to different providers so I'm not going to worry about that right now.

As far as server load - I'd put in a content switch to do the application layer load balancing front ending a server farm for content, that way you can scale it to however big you like.

Like a load balanced group you might use for a bigass SQL database? Or sorta like this?

I honestly can't see us needing a server farm or cluster for sending this stuff - what's going to use the CPUs?

We might be better off with one (or two redundant) servers hooked up to a huge SAN... what do you think?

That's how you do it right, that's how it's done in the U S of A.

[insert nationalistic British comment, carefully calculated to be both funny and insulting, but not so insulting that spidey stops helping me...]

 

[spidey07]

I still like the load balancer idea - just makes your life easier and it can scale.

This can be done in two layers - web and database. Both can be clusters if you like. Depending on storage/reduduancy needs a small SAN (just a disk array enclosure with some host ports) would be a good idea.

Also on your bandwidth calculations - double your anticipated numbers. 50% utilizaiton is still pretty hefty. For the speed you need and any support work it's your call to colocate it or not, both have pros/cons.

I have no idea of your scalablity needs - web layer is asking for blades (this is where the content switch comes into play, as many web servers as you like), database layer is asking for a powerful server where IO is the key.

 

[Atheus]

double your anticipated numbers

Yea gotta allow for bursts of activity. I bet I'm overestimating already though - nobody but nobody gets 5000 concurrent users at launch. Plus if we have really good scalability like you suggest then we can easily just add more capacity as we go.

web layer is asking for blades

Yea we might do blades or something for the web layer, but it's not too hard to spec out something like that - what I'm really thinking about is the streaming side.

Thanks again man, you've given me a lot to go on. I don't need anything more than 'ballpark' estimates at this point anyway.

 

[p0lar]

Originally posted by: Atheus
"what?!" he said, staring at his calculator...

How do the youtubes of this world manage to pay for this stuff and still make money?

Short answer: They're not.