how to detect unplugged cable or other i/o error in C++ sockets?

[Red Squirrel]

I've been working on a socket class for writing server apps. I've had it for a while but been adding more to it and making it more bullet proof.

I ran into an issue, and that's when a client is disconnected abnormally (ex: unplug network cable) my class can't detect it. I tried with both send and recv then checking errno, nothing changes, at all. This is TCP, so to me there should be some kind of lower level signal of some sort that indicates error, but aparantly not.

This leaves any server app I write susceptible to a sort of DDoS attack where one could simply keep connecting but killing the connection abnormally. (I'm sure there must be a way to simulate an unplugged cable at client's end, and do this very fast). Over time the server would have so many clients "open" it would run out of memory. I need a way to scan though client connections and detect if it's lost contact with the client.

Now, some protocols have their own system for this such as sending keep alives, but I want to detect this at a lower level, so I don't have to depend on the application protocol to have this. Not all protocols may have this so if I write a server app for such protocol, I want it to work properly.

 

[degibson]

Originally posted by: RedSquirrel
I've been working on a socket class for writing server apps. I've had it for a while but been adding more to it and making it more bullet proof.

The only any server can know is that a client has stopped sending packets.

I ran into an issue, and that's when a client is disconnected abnormally (ex: unplug network cable) my class can't detect it. I tried with both send and recv then checking errno, nothing changes, at all. This is TCP, so to me there should be some kind of lower level signal of some sort that indicates error, but aparantly not.

The sender can find out if the local ethernet is down, but not though sockets API. You'd have to query the state of the ethernet link, via ifconfig for instance. Obviously there's no way to query a non-local ethernet link.

This leaves any server app I write susceptible to a sort of DDoS attack where one could simply keep connecting but killing the connection abnormally. (I'm sure there must be a way to simulate an unplugged cable at client's end, and do this very fast).

Thats basically a SYN flood.

Over time the server would have so many clients "open" it would run out of memory. I need a way to scan though client connections and detect if it's lost contact with the client.

1) You need to limit the number of connections to keep your memory manageable.
2) TCP connections will timeout. You can make your own timeouts, too.
3) You can build a heartbeat into your socket class if you want, though that means your class will only connect properly to other members of your class (maybe you want that feature anyway).

Now, some protocols have their own system for this such as sending keep alives, but I want to detect this at a lower level, so I don't have to depend on the application protocol to have this. Not all protocols may have this so if I write a server app for such protocol, I want it to work properly.

See 1-3 above.

 

[Nothinman]

I ran into an issue, and that's when a client is disconnected abnormally (ex: unplug network cable) my class can't detect it. I tried with both send and recv then checking errno, nothing changes, at all. This is TCP, so to me there should be some kind of lower level signal of some sort that indicates error, but aparantly not.

Eventually it should time out and you should get an EOF from recv but I think that's about it. The man page for send lists the possible errors but again you won't get them until the connection times out.

This leaves any server app I write susceptible to a sort of DDoS attack where one could simply keep connecting but killing the connection abnormally. (I'm sure there must be a way to simulate an unplugged cable at client's end, and do this very fast)

You don't need to simulate anything, just open a raw socket and send lots of SYN packets.

 

[Red Squirrel]

Hmm so no way to do this then other then code it in the application protocol? Guess I could add time outs and max connection. I just thought there might of been a way to do it right at the socket level as I'm sure server apps like apache have something like this otherwise it would be very easy to knock them offline.

 

[Nothinman]

Hmm so no way to do this then other then code it in the application protocol? Guess I could add time outs and max connection. I just thought there might of been a way to do it right at the socket level as I'm sure server apps like apache have something like this otherwise it would be very easy to knock them offline.

I'm sure daemons like Apache have some checks and the kernel has it's own precautions like syncookies in Linux.

Why don't you just grab the source for something like Apache, vsftp, etc and see what they do?

 

[IHateMyJob2004]

I had to do code like this 10 years ago.

First thing is first, someone unplugging a network cable is only abniormal in a lab environment. If you are deploying to the real world, you need to test things like this because they are normal.

You should be able to get various error codes for various networking issues. I know we were able to detect an unplugged cable 10 years ago. Well, I hsould say that we got the same error over adnd over for an unplugged network cable when a TCP/IP session was in existance.

Depending on your requirements, UDP might be a better solution. We use UDP regularly for embedded applications. It's alot easier just using a protocal that is expected to fail and doing the retry logic on our own. If 3 retries fail, the machine on the other end is assumed to be gone.

 

[Nothinman]

Depending on your requirements, UDP might be a better solution. We use UDP regularly for embedded applications. It's alot easier just using a protocal that is expected to fail and doing the retry logic on our own. If 3 retries fail, the machine on the other end is assumed to be gone.

So you honestly believe that reimplementing packet ordering, retry, etc that already exists in TCP is better than just using TCP?

 

[IHateMyJob2004]

Originally posted by: Nothinman

Depending on your requirements, UDP might be a better solution. We use UDP regularly for embedded applications. It's alot easier just using a protocal that is expected to fail and doing the retry logic on our own. If 3 retries fail, the machine on the other end is assumed to be gone.

So you honestly believe that reimplementing packet ordering, retry, etc that already exists in TCP is better than just using TCP?

Like I said, for certain applications, it makes 100% sense. I should expand. I'm talking about embedded systems that are stand alone systems with maybe 30 nodes all connected with dedicated switches/hubs. Point is, you need to consider the system you are designing for.

Typicall,y if you do UDP, you need to code in your own handshaking. And trust me, it is alot easier than dealing with the error conditions one must test with TCP/IP.

 

[degibson]

Originally posted by: IHateMyJob2004
Typicall,y if you do UDP, you need to code in your own handshaking. And trust me, it is alot easier than dealing with the error conditions one must test with TCP/IP.

Agreed. TCP isn't a silver bullet. Its just really really popular. Most importantly: it doesn't save you from having to deal with failures: it just makes nastier failures that are harder to deal with.

 

[IHateMyJob2004]

Originally posted by: degibson

Originally posted by: IHateMyJob2004
Typicall,y if you do UDP, you need to code in your own handshaking. And trust me, it is alot easier than dealing with the error conditions one must test with TCP/IP.

Agreed. TCP isn't a silver bullet. Its just really really popular. Most importantly: it doesn't save you from having to deal with failures: it just makes nastier failures that are harder to deal with.

I think I missed something.

With UDP, you need to code your own handshaking into the network interface definition. This is easy though. You basically need to acknowledge all message that are sent.

The second thing you need to do is code the resend logic. Not a big deal. Simply put, you have ti listen for hte acknowledgments and remove the associated messages from a global queue. Then have a thread that checks the sent messages are acknowledged within X seconds (5 seconds is normal). If not, resend it.

And the third thing you have to do is to add the logic to determine that a network device is missing. Whcih is part of the thread mentioned in the previous paragraph.

The other thing one might need to do is throttling, but I have never had to do that. At most, you might need to code in a delay between messages (10 ms maybe).

 

[dighn]

Originally posted by: IHateMyJob2004

Originally posted by: Nothinman

Depending on your requirements, UDP might be a better solution. We use UDP regularly for embedded applications. It's alot easier just using a protocal that is expected to fail and doing the retry logic on our own. If 3 retries fail, the machine on the other end is assumed to be gone.

So you honestly believe that reimplementing packet ordering, retry, etc that already exists in TCP is better than just using TCP?

Like I said, for certain applications, it makes 100% sense. I should expand. I'm talking about embedded systems that are stand alone systems with maybe 30 nodes all connected with dedicated switches/hubs. Point is, you need to consider the system you are designing for.

Typicall,y if you do UDP, you need to code in your own handshaking. And trust me, it is alot easier than dealing with the error conditions one must test with TCP/IP.

Definitely depends on the application. For our application we actually had to go below UDP (i.e. raw ethernet) for performance reasons. We are talking about embedded processors here so even UDP was using too much CPU.

 

[IHateMyJob2004]

dighn,

You had to what? All I have to say is WOW!

Was this recently or many years ago?

If recently, is it because yo uhad some sort of price point you were trying to reach?

I guess I am lucky. I work on DoD stuff. I'm be lucky if 100 of a product are made. We just buy the hardware we need.

 

[Red Squirrel]

Well keep in mind I'm not coding a specific app at this point. I'm just coding a TCP class so I can use it to write apps, so I was hoping to make this detection built in to the class itself. Later on I might also write a UDP class, but most of my coding needs are TCP based apps anyway. UDP is more complex to work with.

I just find it odd, as TCP has hand shaking built in at it's level so figured there was a way to check this without having to code it at the application level.


Though, if I did this at the raw socket level, would I get more control? Does TCP send keep alives through syn/acks or other lower level packets? (Ex: not actual data). There is a flag to set keep alive, but would it also have to be set on the application at the other end?

 

[dighn]

Originally posted by: IHateMyJob2004
dighn,

You had to what? All I have to say is WOW!

Was this recently or many years ago?

If recently, is it because yo uhad some sort of price point you were trying to reach?

I guess I am lucky. I work on DoD stuff. I'm be lucky if 100 of a product are made. We just buy the hardware we need.

This is recently. These are data collection+processing systems (images) for industrial control, and we basically want to squeeze as much performance out of them as possible. It's all about higher and higher collection rates.

 

[dighn]

Originally posted by: RedSquirrel
Well keep in mind I'm not coding a specific app at this point. I'm just coding a TCP class so I can use it to write apps, so I was hoping to make this detection built in to the class itself. Later on I might also write a UDP class, but most of my coding needs are TCP based apps anyway. UDP is more complex to work with.

I just find it odd, as TCP has hand shaking built in at it's level so figured there was a way to check this without having to code it at the application level.


Though, if I did this at the raw socket level, would I get more control? Does TCP send keep alives through syn/acks or other lower level packets? (Ex: not actual data). There is a flag to set keep alive, but would it also have to be set on the application at the other end?

I believe that there is a "keep alive" feature that you can optionally enable, but it may not be standard across different platforms.

Check this out (from the winsock FAQ - a good read for winsock beginners): http://tangentsoft.net/wskfaq/newbie.html

Using a ping/pong mechanism is probably the most portable method.

 

[degibson]

Originally posted by: IHateMyJob2004

Originally posted by: degibson

Originally posted by: IHateMyJob2004
Typicall,y if you do UDP, you need to code in your own handshaking. And trust me, it is alot easier than dealing with the error conditions one must test with TCP/IP.

Agreed. TCP isn't a silver bullet. Its just really really popular. Most importantly: it doesn't save you from having to deal with failures: it just makes nastier failures that are harder to deal with.

I think I missed something.

With UDP, you need to code your own handshaking into the network interface definition. This is easy though. You basically need to acknowledge all message that are sent.

The second thing you need to do is code the resend logic. Not a big deal. Simply put, you have ti listen for hte acknowledgments and remove the associated messages from a global queue. Then have a thread that checks the sent messages are acknowledged within X seconds (5 seconds is normal). If not, resend it.

And the third thing you have to do is to add the logic to determine that a network device is missing. Whcih is part of the thread mentioned in the previous paragraph.

The other thing one might need to do is throttling, but I have never had to do that. At most, you might need to code in a delay between messages (10 ms maybe).

If the needs of your project are:
1) Ordered message delivery
2) Tolerance for dropped packets
3) Some (lousy) tolerance for congestion via backoff
... then TCP fits the bill nicely and DO NOT go and re-implement anything on UDP.

The problem is, when your only tool is a hammer, every problem looks like a nail. For instance, if your project needs:
1) Tolerance for dropped packets
2) Handling of link-level failure between trusted peers (e.g. in RedSquirrel's case)

... then TCP is only a 50% solution. Getting the other 50% is HARD and requires serious architectural considerations. Hard enough that most folks' automatic rejection of UDP is ill-advised.

To summarize: Not every problem needs all the features of TCP. And to get the most important ones (failure-free guaranteed delivery, in-order receipt), its pretty easy with just UDP, especially at low bandwidth. Should everyone switch to UDP? No, of course not. But it should be on the table more than it is.

I am now stepping off my soapbox.