How to control/interface with closed source programs?

[Red Squirrel]

I'd like to learn on how to write programs that can interface with other programs that are closed source, so, without having to recompile it. For example, some programs such as cheat engine can take control of a certain program and change certain parameters and make the program act differently. I've also seen programs that can even add features such as add an extra button in outlook, or something.

I'm just wondering where do I go about learning these techniques, as well as reverse engineering a program to see what it's doing. I most likely have to learn and understand assembly I'm guessing. I'm not planning to do anything malicious or illegal or anything, it's just a subject I've always wanted to learn and become good at. It can even be very useful to try to solve complex problems with a piece of software, such as why a certain app keeps crashing.

 

[Ken g6]

It really varies from program to program what you can do, and how easily you can do it.

First, there's the user-level interface you can use. I like AutoIt; there are also some very expensive programs to do the same thing, like HP Quicktest Pro.

Second, some programs provide an API (Application Procedural Interface) for you to use. These programs want you to modify them, at least in some ways, so they make it relatively easy.

Third, if you're using a .NET program (or possibly a Java program), you can use reflection to modify an application's classes. This is not easy to do; I'm not entirely sure of all the steps involved. But IronPython is a good tool to help you do it, because it lets you write .NET code on-the-fly.

 

[Modelworks]

Go ahead and learn ASM . You may not have to use every bit of it you learn but when working with programs you don't have the source to , it sure makes it easier if you can look at the dis-assembly and understand what is going on.
One easy way to learn it, that comes with a simulator and easy examples:
http://www.emu8086.com/

It really isn't as bad to learn as some make it out to be.

 

[degibson]

It helps if your target app has an API, as Ken suggested. If it doesn't, there are lots of ways to manipulate the input layer (as Ken suggested... see the pattern yet?). I prefer AutoHotkey myself.

 

[JasonCoder]

got to be as many ways to work with a program as there are ways to write a program... plain old c APIs, *nix signals, COM, RMI, JNI, .NET Remoting, WCF, etc, etc. And that doesn't cover the test suites already mentioned.

 

[Red Squirrel]

Ahh I see so really it depends on how the program was written, then I can use whatever API's features to my advantage. Though with assembly, it would probably be the same concept across the board right? Just that APIs would make it way easier.

 

[Markbnj]

You used to be able to subclass an application Window, intercept its message queue, and add or change features without touching the affected program at all. As everyone else has said, there are literally dozens of different techniques and approaches to this stuff.