How to implement command line executable with central "manager"?

[avhokie]

Hey folks,

I am trying to do a little project out of general interest (and it may be useful in my job). It will be based on some existing Java code (I am not really a Java programmer, but I can pick up that part myself).

What I want to do is the following:
1) Have the ability to run commands specific to my application on the command line. There may be multiple commands executed at any given time. Everything happens on one machine.
2) Have the ability for some kind of central "manager" pick up those commands and act on them. The reason for the central "manager" is that it may be responsible for some things, such as throttling, retries, etc.

Is there a good model to use as a reference for this project? Not sure I'm evening asking the right questions to get me on the right path, but any help/advice is appreciated!

Thanks,
avhokie

 

[DaveSimmons]

It sounds like the manager app should be the one starting the command line processes, so it knows when they are running and has their process IDs.

That could be done in C/C++, VB, etc. using the UI builder of your choice. The UI could be something as simple as one launch button per app and a scrolling edit control for status messages.

 

[Cogman]

The manager app seems to be stepping on the toes of the OS. I don't know if you can specifically throttle running applications that aren't built to be throttled. (IE, they get a message that says "slow down" and so they slow down.)

You may find this http://stackoverflow.com/questions/287633/java-c-like-fork useful.

 

[avhokie]

Let me give you some more info/analogies that may help.

First, the actions have to be started at the command line. And they are going to be kicked off by other applications (what I am designing is something to be used in integrating different applications, with many of them only being able to execute something on the command line).

Second, this will not step on the OS.

Let me give you an analogy. You can FTP from the command line (comes with Windows). Any application that has the ability to run something on the command line can make that happen. But the Windows command line FTP app is very simple and each instance runs independently. So if two different applications try to FTP at the same time, each one is independent and there is no awareness that multiple FTPs are happening at the same time. Also, there are no advanced features, like retires, throttling, etc.

What I would like to do is build a more advanced app (like many other FTP applications, such as CuteFTP) that has the advanced features, but instead of being instructed through a UI (which another application you are trying to integrate to cannot typically do), I want to instruct the app via command line. Does this help?

 

[DaveSimmons]

> First, the actions have to be started at the command line.

You are saying there is a technical or policy reason why someone has to manually type commands in a shell window to run applications, you can't have an already-running application type them for you? That is, automate the starting of the command-line application?

That seems odd, but if so:

Option a: you could perhaps have batch files or wrapper exes that start the command line app, but also write a flag file into a special folder, and your central manager looks for those flag files.

For example, you type "FtpApWrapper site1 dl foofile.txt" to run FtpApp that downloads foofile.txt from site1, but also writes a status text file to /appsfolder/

Option b: Use the process explorer functions to create something similar to Task Manager, but that only shows processes for the EXE files that you care about. You won't be able to do much with this though since it only sees processes while they run and can't tell if they succeeded or failed.

 

[avhokie]

Dave, thanks for the reply. I'm either incapable of explaining my need or my need is way out in left field

I work with enterprise software (things that typically cost $100k+). Most of these applications don't integrate very well, so it's up to users to figure out how to integrate them. One thing that most applications can do is to run something on the command line when some activity happens. For example, if you open a ticket in a ticketing system, chances are that it can run something on the command line in response to the ticket being opened. I want my application to be thing called from the command line by the ticketing system (and various other systems).

And instead of each one of these commands being processed in a vacuum, I want some kind of central manager to be responsible for picking up the requests and processing them with some additional intelligence built into that manager.

In the past, I've written small client-server apps that do something like this, but since this is all on one machine, I was thinking that the networking component may be overkill here. But that's my fallback...

 

[DaveSimmons]

That's clearer. You aren't running command-line apps, a program is running them itself as the equivalent of events / notifications.

If you can control what apps are run (telling the enterprise app to run MyEventHandler.exe instead of the EventHandler.exe it runs now) then your MyEventHandlerExe can do almost anything to notify a second, always-running management application about the event.

It can post messages to the management app, and/or it can write files to a special folder. It can skip doing any processing itself, relying on the management app to do (whatever).

If you give a more concrete example of a workflow, with more detail than just (something happens when a ticket is created), then we can probably offer more detailed suggestions.

Also list the platform (Windows? linux?). It might even make sense to use an existing transaction framework.

 

[Markbnj]

Sounds like a message processing application to me, i.e. something that could be built on MQ, or whatever Microsoft's equivalent is, or perhaps Windows Workflow Foundation. I think you're focusing too much on the means of sending the message (a command line app) in your description. You've got a bunch of disparate apps running in various places, and when certain things happen you want them to respond by sending a message to a central processor that will act on it, log the outcome, whatever. The means of sending the message might vary with circumstances, and you have lots of options to handle different scenarios. You could have the event trigger a command line program as mentioned. You could run an agent local to the process that generates the event and have them use a named pipe, memory-mapped file, raw socket, etc., to communicate. You could run an http server that receives messages from some set of apps and then forwards the message to the processor. Or you could give the processor itself an http interface. There are literally thousands of possible architectures.

 

[avhokie]

That's clearer. You aren't running command-line apps, a program is running them itself as the equivalent of events / notifications.

If you can control what apps are run (telling the enterprise app to run MyEventHandler.exe instead of the EventHandler.exe it runs now) then your MyEventHandlerExe can do almost anything to notify a second, always-running management application about the event.

It can post messages to the management app, and/or it can write files to a special folder. It can skip doing any processing itself, relying on the management app to do (whatever).

If you give a more concrete example of a workflow, with more detail than just (something happens when a ticket is created), then we can probably offer more detailed suggestions.

Also list the platform (Windows? linux?). It might even make sense to use an existing transaction framework.

Sweet, now we're on the same page

I'm looking for a solution that works on Windows/Unix/Linux.

A more concrete example of what I am doing: I have various enterprise apps that need to make a call to another application using a SOAP interface. These various enterprise apps (such as ticketing, event management) can make command line calls in response to activities that happen in those apps (such as the opening of a ticket or the receipt of an event). They then need to make a command line which will be converted and sent over as a SOAP request (this part I have today).

There are a couple of problems with the basic approach today. First, there is no retry mechanism. So if you can't get to the web service interface (app is down, network issues, etc), then the request never goes through. Second, in the case that there is an app down or network issue situation, I expect that a large number of requests will get queued up. Once the issue is resolved, I don't want all those requests to be launched at the web service interface all at one time. It will choke that app. So I want to build a throttling mechanism to gradually process the buffer of requests.

Sounds like a message processing application to me, i.e. something that could be built on MQ, or whatever Microsoft's equivalent is, or perhaps Windows Workflow Foundation. I think you're focusing too much on the means of sending the message (a command line app) in your description. You've got a bunch of disparate apps running in various places, and when certain things happen you want them to respond by sending a message to a central processor that will act on it, log the outcome, whatever. The means of sending the message might vary with circumstances, and you have lots of options to handle different scenarios. You could have the event trigger a command line program as mentioned. You could run an agent local to the process that generates the event and have them use a named pipe, memory-mapped file, raw socket, etc., to communicate. You could run an http server that receives messages from some set of apps and then forwards the message to the processor. Or you could give the processor itself an http interface. There are literally thousands of possible architectures.

I think you have some experience on this front

This is pretty much new to me. In a past life, I was a embedded software engineer who also lightly dabbled in client/server apps (emphasis on lightly ). And I've been away from that for years. So now getting into Java based message processing application is like learning from scratch. I've done raw sockets before and would use that as a fallback, but I figured there may be a better way and possible an existing framework to build on.

Thanks guys

 

[Markbnj]

So now getting into Java based message processing application is like learning from scratch. I've done raw sockets before and would use that as a fallback, but I figured there may be a better way and possible an existing framework to build on.

In the broadest sense this field is so huge it would be difficult to make any concrete recommendations without knowing a lot more about your architecture, timeline, budget/resource constraints, etc. I would do some research on message queue and workflow solutions and sort of dig in from there. There are many different approaches and tools available. A message queuing system like MQ has the benefit of guaranteeing message delivery and processing (though not necessarily message order) an being tailored to a wide-area enterprise environment. You have nearly complete freedom to determine how the messages get into the queue, and what happens after they do.

Edit: it's also worth noting that many enterprise level applications generate, or are capable of generating, SNMP (Simple Network Management Protocol) traps, which is another approach you could use to integrate event reporting and response.

 

[trexpesto]

That's enterprise workflow.
It is fun to use JNI and runtime.exec but it can be awkward integrating third party tools if all they do is output files. I would try to use existing workflow software with high abstraction level, and a decent UI. If you do have an "enterprise app", good chance they would have a nice one. All the app-servers seem to have something, may not be appropriate.

Saw that Taverna conspicuously mentions command line.

Here is one of the newer workflow editors, could be overkill.
http://www.together.at/prod/workflow/twe
Edit: here is the open source project page for it:
http://sourceforge.net/projects/jawe/

 

[trexpesto]

From a JaWE reviewer:

It is the leading open source editing solution with xpdl support. Together with Shark workflow engine it is just an amazingly capable solution.

 

[degibson]

I'm looking for a solution that works on Windows/Unix/Linux.

Have your command line tools write commands into a file.
Have your central manager read the file occasionally and look for new commands.

 

[trexpesto]

This process would make a great blog!
Seen modisco? I think a new version just came out.
http://www.eclipse.org/MoDisco/

 

[avhokie]

trexpesto...thanks for the links. Interesting material

degibson...that could be a nice quick and easy way to do it!

Thanks!

 

[trexpesto]

If you had to do more than that retry queue, like decision points or need to roll back some transaction, or want adapters like for a database, scheduling, that would more justify the higher level of workflow.
And the higher level language to go with it, in a way.
Aannnd at some point, getting down to the nitty gritty is necessary in any scenario.

Could be a security hazard to basically have a service that executes arbitrary command line complete statements. Like so someone doesn't put
' --ignore errors; disable input; format drive; '
in there somehow. Some kind of validation.


I have used cron and task scheduler to run some scripts before, if that is all you need. Python is supposedly a good script language, all I did with it was a standalone app, but fun to learn with the interpreter. It's like casual java.
What you are allowed to install where is sometimes a big consideration, like Python or Cygwin. If you can use Cygwin, you can use the same bash scripts on Windows and Linux-Unix.
Do you get a return code / exit code for success/failure or is there a log you can monitor? It's a longshot that such return receipt is a configurable property on what would be receiving your message.
If operating on files, it can be worth using a copy to avoid locks and avoid corruption due to errors or power problems.
Well have fun with it!

 

[degibson]

degibson...that could be a nice quick and easy way to do it!
Thanks!

You're welcome. Many inter-process communication mechanisms are built on file systems, so this approach simply cuts the fat. But feel free to use whatever mechanism seems most intuitive to you -- this isn't performance-critical.