What happens when you download something from a website?

[chrstrbrts]

Hello,

We all download tons of things from websites all the time: PDFs, installers, images, etc.

I'm assuming that because this data was originally found on websites on the internet the original protocol involved is HTTP.

But what happens when the download is initiated?

Does the protocol switch over to FTP?

Is it possible to run two streams simultaneously using different protocols through the same NIC?

Thank you.

 

[Merad]

No, it is possible to download via HTTP. A download that goes via FTP will have a url that begins with FTP://

For the latter question... depends on exactly what you're asking. Can you (from a user perspective) have simultaneous transfers occurring via HTTP and FTP? Yes. Both are just application protocols that sit on top of TCP. As far as how the NIC actually works, I don't know offhand if it is typically capable of simultaneous TCP connections, or if it just multiplexes between connections very quickly. I have a feeling that it's either/or, depending on the operating system and NIC hardware/drivers.

 

[Ken g6]

The simple answer is that, over HTTP, the server sends your file with MIME encoding. In most cases, this is as simple as:

1. Sending headers. The most important is probably "Content-Type", which may be something like "application/octet-stream" for a generic file. Many other headers, including "Content-Transfer-Encoding" may also be used.

2. Sending the file. Assuming "Content-Transfer-Encoding" is "binary", which I think it usually is for HTTP, this is as simple as dumping the file over the HTTP connection.

 

[chrstrbrts]

I've never seen the 'FTP' prefix on a URL. Is FTP common nowadays?

Can you download a file over HTTPS?

That is, can you download a file over an encrypted connection?

Which do you think is better, HTTP or FTP?

 

[Ken g6]

I've never seen the 'FTP' prefix on a URL. Is FTP common nowadays?

Nope. FTP requires a login and used to require a socket on the client machine. The fixed the former with a standard "anonymous" login, and the latter with "passive FTP". But HTTP is probably better overall. Edit: Better as a URL. For uploading and filesystem manipulation, I much prefer FTP.

Edit: Here's an FTP URL.

Can you download a file over HTTPS?

That is, can you download a file over an encrypted connection?

Sure. It's basically a security wrapper on TCP, with HTTP on top of it. Some browsers now won't accept inline images on an HTTPS site unless they're also served over HTTPS.

 

[Cogman]

No, it is possible to download via HTTP. A download that goes via FTP will have a url that begins with FTP://

For the latter question... depends on exactly what you're asking. Can you (from a user perspective) have simultaneous transfers occurring via HTTP and FTP? Yes. Both are just application protocols that sit on top of TCP. As far as how the NIC actually works, I don't know offhand if it is typically capable of simultaneous TCP connections, or if it just multiplexes between connections very quickly. I have a feeling that it's either/or, depending on the operating system and NIC hardware/drivers.

All devices between the NIC and the server send 1 packet at a time. A packet may be from any TCP/UDP/whatever stream. As those packets arrive, they are recombined at the driver/OS level.

Simultaneous streams work because often a single stream isn't enough to saturate the pipeline. For example, between you and your router is probably anywhere from a 100mbps to 1gbps connection. Yet your inbound internet connection may only allow for 10mbps. Further, whoever you are talking to may only be able to supply you with 1mbps worth of data.

The routers as they receive data will forward it on to the next link in the chain as fast as they can forward. If there is some backup going on, the routers will do some buffering or signal to the upstream data provider "Hey, slow this stuff down!". Often (but not always) routers will route packets on a first come first server basis. So if a signal router has packets going to the same host it will push down the first packet it received. Even if this packet is for a different TCP stream.

All of this gets more complicated when you start talking about QOS or the crazy crap routers have to do because of TCP's anti-congestion algorithm. Interestingly the anti-congestion algorithm is the entire reason for QUIC, SPDY, and HTTP2.

 

[chrstrbrts]

Edit: Here's an FTP URL.

Whoa, whoa. You can serve HTML/CSS/Javascript with other protocols besides HTTP?

I thought that FTP was used exclusively for the downloading itself, but you can use it to serve full websites as well?

As those packets arrive, they are recombined at the driver/OS level.

Are you saying that the protocol is understood at the software level, not the hardware level?

So, the NIC has no native physical understanding of the protocol?

 

[Cogman]

Are you saying that the protocol is understood at the software level, not the hardware level?

So, the NIC has no native physical understanding of the protocol?

There are several layers of protocols, it isn't just "the" protocol. Does a nic understand HTTP, nope. Does it understand TCP, maybe but probably not (but not definitely). Does it understand IP? Most likely. Does it understand frames? Absolutely.

Further, there are link protocols that the NICs understand for physically interacting with the environment to send messages. They understand how to create the electrical wave on the wire that translates to the 1s and 0s at the receivers end. They also know how to read those waves. (Think of something like DOCSIS, 802.11ac, CDMA, GSM, or whatever the standard is for fiber communication).

 

[Ken g6]

Whoa, whoa. You can serve HTML/CSS/Javascript with other protocols besides HTTP?

I thought that FTP was used exclusively for the downloading itself, but you can use it to serve full websites as well?

Well, you can certainly serve the files. Whether they'd work like a website I'm not sure, but I see no obvious reason why they wouldn't. What you saw there was a directory listing.

There are several layers of protocols

Here they are.

 

[chrstrbrts]

There are several layers of protocols, it isn't just "the" protocol. Does a nic understand HTTP, nope. Does it understand TCP, maybe but probably not (but not definitely). Does it understand IP? Most likely. Does it understand frames? Absolutely.

Further, there are link protocols that the NICs understand for physically interacting with the environment to send messages. They understand how to create the electrical wave on the wire that translates to the 1s and 0s at the receivers end. They also know how to read those waves. (Think of something like DOCSIS, 802.11ac, CDMA, GSM, or whatever the standard is for fiber communication).

Of course, of course.....the protocol stack.

So, the NIC understands the low level protocols: the physical layer, the link layer, even maybe the network layer.

But, beyond that, the protocols are understood by software.

Can you explain more about how the OS puts all packets in order and passes them to the right process?

 

[chrstrbrts]

Well, you can certainly serve the files. Whether they'd work like a website I'm not sure, but I see no obvious reason why they wouldn't. What you saw there was a directory listing.

I viewed the source code with Chrome, and I saw CSS and Javascript.

That's why I posted as I did.

Incidentally, what's a GIMP?

Do those files have something to do with the graphics program GIMP?

 

[Ken g6]

Incidentally, what's a GIMP?

Do those files have something to do with the graphics program GIMP?

GIMPS is the Great Internet Mersenne Prime Search. They're the group who use Prime95 to find Mersenne primes.

 

[chrstrbrts]

GIMPS is the Great Internet Mersenne Prime Search. They're the group who use Prime95 to find Mersenne primes.

Mersenne primes are represented as all 1's in binary, yes?

 

[Cogman]

Of course, of course.....the protocol stack.

So, the NIC understands the low level protocols: the physical layer, the link layer, even maybe the network layer.

But, beyond that, the protocols are understood by software.

Can you explain more about how the OS puts all packets in order and passes them to the right process?

Not much too it really. What happens sort of depends.

Likely the OS will say to the NIC "Hey, this memory location is yours to scribble over." As the NIC gets packets, it will write directly to that memory location the full packet (full frame?). This is called DMA. Once it is done writing, it notifies the OS via a CPU interrupt "Hey, I just wrote out some data". From there the OS will do what it does, whether it hands control over to the driver to do some extra processing on the data or something else depends somewhat on the particular OS. Generally, though, when it comes time to do the TCP stuff, the OS will do it all, it will inspect each packet, figure out the ordering, port, and application (the port is basically way of controlling what app gets what data). As part of the TCP protocol, the OS will check and see if it has all the information needed to start forwarding packets, if a packet comes in out of order the OS will hold off on forwarding one or more packets until the missing packet comes in. After a timeout, there will be a re-request for a numbered packet assuming that it simply died in the ether.

The driver is for a NIC is mostly for establishing a memory location that the NIC can write to as it comes in. Though, it may do more if the NIC has memory on it (like the Killer or whatever NICs that have 256mb of ram).

Once it gets into the application, that is where the Http protocol is handled. The application will read the http headers and then interpret the data as needed.