Architecture/Design question - reliable and efficient messging

[WannaFly]

I'm considering writing an application to replace a legacy app that we have, but it has a few requirements that I'm not sure how to go about addressing. The basis will be a messaging application, to send and recieve messages between server/client.

1) Can not maintain open connection (UDP?)
2) Multiple programs that I write will need to communicate with the server (COM?) Each program should not have to authenticate (once authenticated, all comm from that client is allowed, no matter what program it originates from)
3) Needs to be reliable - client MUST recieve message and server MUST recieve message
4) Can push or pull messages to client, but must be very efficient with bandwith (we use aircards, slow connections)

I'm working within .NET and have looked a lot at remoting, i think it might be the way to go. Have also considered MSMQ - but not sure if that is too complicated for my needs. I've seen a remoting example that uses UDP to query to see if there is a message waiting, which seems pretty good.

Any suggestions? I'll give more info if need.

 

[Markbnj]

The thing about message queuing sysems like MSMQ is that you get he guarantee of delivery even in the presence of network outages, etc. Remoting would probably require you to roll your own. If this is a business critical system then from what you describe I would think MSMQ or IBM's version would be a good fit.

 

[imported_Dhaval00]

IBM's Websphere is a maintenance nightmare. Plus why spend extra on licensing when you can get MSMQ for free. If you use the MSMQ API via .NET 2.0, you'll run into the 4MB message limit. This however can be addressed by using .NET 3.0 or greater or a BizTalk hack that MS implemented internally (the DLL can be obtained on the Internet, and there are no licensing caveats). Refer to John's blog for more details on the 4MB limit and how to get "big message" DLL: http://blogs.msdn.com/johnbreakwell/.

UDP is unreliable! It will drop packets, and is not the same as TCP, which is what you would want. UDP is used for things like video/audio streaming

 

[trexpesto]

Managers love to spend money on licensed software because they get the illusion of control. AKA tech support.

 

[degibson]

If you cannot maintain connection state (1, above), and you're talking about an IP-layer protocol, then you're going to have to use UDP, as bad as it is. I'd highly encourage you relax requirement 1 and use something TCP-based. Otherwise you'll end up doing TCP-like things in your UDP stream (i.e. retransmits and acknowledgements) in order to meet requirement 3.

 

[WannaFly]

Markbnj/Dhaval00: Thanks, MSMQ sounds like it might be ideal for what I want to do, the 4MB limit shouldn't be a problem for me...Thanks for the info. I do want to avoid licensing fees, so i'll look at it more in depth.

I know that UDP is unreliable, which is they there would have to be a check in the if no UPD has been recieved for say 10 sec, to query automatically and see if there are messages waiting. It's all about making the most efficient use of the very limited bandwith the aircards supply.

degibson: I agree with you a bit, but again because we are on aircards, and are typically driving to places they might momentarily lose the connection i'm not sure I can use TCP. Our VPN client supposedly maintains connections even though the connection is momentarily lost but just with general use it doesnt seem to do a very good job.

Can anyone comment on recommendations for Req #2?

All: Thanks for the discussion.

 

[Markbnj]

On (2), what is the server running? We do exactly this all the time on Windows and Linux clients connecting to Linux or AIX servers, using SSH. You set up public key authentication on the server and an ssh-agent on the clients. The first time they try to drop a connection to the remote the agent pops a dialog looking for a passphrase. Once entered it unlocks the local key and that key is reused for all subsequent connections.

Must be a way to do something similar on Windows. I know that in code most APIs that offer connections (named pipes, for example) offer a way to impersonate the current logged-in user. Perhaps MSMQ supports something similar.

 

[degibson]

#2 could also be accomplished fairly easily with a daemon -- have a single point of communication on each client, thus a single point of authentication, too. Programs that use the communication channel just hand their messages to the daemon.

Another point about TCP -- TCP is pretty good at dealing with brief periods of disconnection (thats one of the things its designed to do), and just about anything built on UDP that does the same thing will need similar bandwidth requirements. BUT if the periods of disconnection last longer than the maximum TCP timeout (somewhere in the vicinity of 240 seconds, if I remember correctly), THEN you'll start having problems. I think there's probably a way to change the timeout time to something absurdly high, though I don't know how to do so off the top of my head. This page might tell you how: http://support.microsoft.com/kb/170359

 

[WannaFly]

Markbnj: the server will be running Windows Server 2003, so SSH isn'y really a viable option. I really think COM is probably the way to go, unfortunately I dont have experience with it - And it might be obsolete...not sure.

Degibson: A Daemon is what I need, but HOW the programs will communicate with the daemon efficiently is more of my question - how to transfer the messages to it.

I ran a network trace with our current program and it does indeed use TCP, which helps answer my question a bit - so I guess that shouldn't be a problem. Just gotta see if MSMQ is the way to go....Thanks for tall the input.

 

[SearchMaster]

MSMQ ended up being a nightmare for us for several reasons so we ended up rolling our own db-based solution. We are a very high transaction website though and it is very possible, like many of MS's 'free' solutions, it just doesn't scale that well. When it worked, it worked fine, but when there were problems, it didn't recover well, and it would eat messages from time to time with no apparent cause and only a reboot would solve the problem.