Command-line interaction with web pages?

[avhokie]

I know this sounds odd, but I'm looking for a way to help automate some interaction I do with websites. I know there are programs (like macro programs) that can move a mouse around a page and type characters and such, but the problem with those is that if you run on multiple machines, things such as screen resolution can affect your scripts.

So I got to thinking that there may be a program out there that lets you interact with web pages in a command line fashion? So it would allow me to log in to a web page using a login and password, and then I can grab information I need by just examining the html or something like that or I could then navigate to another page.

Is this crazy or has someone thought of a solution to my problem?

 

[kingtas]

Why would you want to revert back to the days of DOS.

I think you might get a better response if you post in a Linux community. They're all over that command line stuff. LOL.

 

[avhokie]

Ha...good old DOS! Honestly, it's because automating tasks is infinitely easier on the command line as opposed to a GUI.

 

[Atheus]

What do you mean? Just grab html from the command line? You can do that using programs like wget, or you can even interact directly with the webserver using telnet - typing the HTTP requests manually!

Or do you mean you want a command line window within your browser? You could have a textarea object with a javascript function to catch the 'enter' key - every time you press it the last line is submitted via an AJAX request to a php or jsp page which processes the command. Is that the sort of thing you're looking for?

Originally posted by: kingtas
Why would you want to revert back to the days of DOS.

I think you might get a better response if you post in a Linux community. They're all over that command line stuff. LOL.

The command line is very powerful. It is used for doing actual work, rather than watching videos and playing games, which is why windows users are generally not familiar with it.

 

[Looney]

Like Atheus said, why not just telnet in?

 

[avhokie]

Originally posted by: Atheus
What do you mean? Just grab html from the command line? You can do that using programs like wget, or you can even interact directly with the webserver using telnet - typing the HTTP requests manually!

I think that this is all I need. I'm not sure if wget would work, because I need to be able to pass data to the website (like login information) and keep a session active from that point.

I've never tried telneting to a website, so I'll look into that...

Thanks for the help so far!

 

[avhokie]

Oh, btw, does the solution change if I need to access to dynamic pages on websites? Sorry if these questions are dumb, I'm a novice at this stuff

 

[MrChad]

It's certainly possible, but it's going to be tedious. Load testing applications like LoadRunner will do exactly what you're describing (automate a series of repetitive web tasks and preserve cookies and other stateful information). The problem you're going to run into is:

1. Constructing the HTTP GET/POST calls with the correct header information so that your session state is maintained.
2. Parsing the HTTP response to extract not only the information you want, but also the state information you need for #1. This becomes more difficult if the pages/layouts are dynamic.

 

[WannaFly]

It's probably not what you are looking for, but if you know .NET this can be done in a windows application w/ VB. You add a activex web brwosing control, and you can programatically control it and access the HTML DOM to get the info you need. I've done very simlar to what you are asking, but it did require a bit of programming, but works great!

 

[kingtas]

Originally posted by: Atheus
The command line is very powerful. It is used for doing actual work, rather than watching videos and playing games, which is why windows users are generally not familiar with it.

I am familiar with the command line. I program some and work with VI.

I thought he wanted to read web page text with a command line by his use of "interacting" since there was no mention of a database or anything. So , as he put it, it sounded weird to me. My bad.

 

[jiggahertz]

I'm interested in doing the same thing from a *nix environment. For example, something along the lines of logging into gmail, downloading/deleting old chat logs, etc. But I'm not sure as to where a good starting point is for this.

 

[Atheus]

Originally posted by: kingtas

Originally posted by: Atheus
The command line is very powerful. It is used for doing actual work, rather than watching videos and playing games, which is why windows users are generally not familiar with it.

I am familiar with the command line. I program some and work with VI.

I thought he wanted to read web page text with a command line by his use of "interacting" since there was no mention of a database or anything. So , as he put it, it sounded weird to me. My bad.

Yea sorry, that came out more condescending than I intended.

Originally posted by: jiggahertz
I'm interested in doing the same thing from a *nix environment. For example, something along the lines of logging into gmail, downloading/deleting old chat logs, etc. But I'm not sure as to where a good starting point is for this.

Should all be much easier from *nix than windows - what are you using exactly? Linux?

I know lynx can be automated, wget comes standard, the telnet program is far superior to windows, etc etc.

 

[avhokie]

Quick follow up, I'm not necessarily constrained to Windows. I'm interested in looking at *nix alternatives too. And jiggahertz gave a good example of something I would like to do as well. I'm looking at lynx and wget now, but I'm not sure where to start with telnet. I know what telnet is and does, but I don't know how to use it to interact with, say, yahoo.com.

Thanks!

 

[Atheus]

Originally posted by: avhokie
Quick follow up, I'm not necessarily constrained to Windows. I'm interested in looking at *nix alternatives too. And jiggahertz gave a good example of something I would like to do as well. I'm looking at lynx and wget now, but I'm not sure where to start with telnet. I know what telnet is and does, but I don't know how to use it to interact with, say, yahoo.com.

Thanks!

Well, just to get you started:

$ telnet yahoo.com 80 [enter]
$ GET / HTTP/1.0 [enter] [enter]

If you're using windows you won't be able to see the commands you enter after the 'telnet' command itself. This is because windows is sh1t.

 

[avhokie]

Son of a.....

Thanks! I swear I tried the same thing yesterday and it would not connect, but now it's working

 

[Nothinman]

If you want to learn perl the WWW::Mechanize module should be able to do whatever you want.

 

[avhokie]

Originally posted by: Nothinman
If you want to learn perl the WWW::Mechanize module should be able to do whatever you want.

Thanks! I just installed this package a couple of hours ago and it looks like it's what I need. I just came back here to post and let jiggahertz know, but you beat me to it

 

[jiggahertz]

Originally posted by: avhokie

Originally posted by: Nothinman
If you want to learn perl the WWW::Mechanize module should be able to do whatever you want.

Thanks! I just installed this package a couple of hours ago and it looks like it's what I need. I just came back here to post and let jiggahertz know, but you beat me to it

Ah cool, I will check this out and look into automating lynx.

 

[kingtas]

Originally posted by: Atheus

Yea sorry, that came out more condescending than I intended.

No sweat. But I have a question about the power of the command line. I know you can open and edit files, transfer files, run scripts and executables - but these can all be done from a GUI. Am I missing the real power of the command line?

 

[Nothinman]

No sweat. But I have a question about the power of the command line. I know you can open and edit files, transfer files, run scripts and executables - but these can all be done from a GUI. Am I missing the real power of the command line?

How do you automate those with a GUI? Sure it's possible to write apps to click on the right things but it's a huge PITA. And you if you want you can run vi over ssh and it'll run virtually as fast as if it was local which can't be said for a GUI. Then there are other things like the shell's history that aren't replicatable able via a GUI either.

 

[Atheus]

Originally posted by: kingtas
No sweat. But I have a question about the power of the command line. I know you can open and edit files, transfer files, run scripts and executables - but these can all be done from a GUI. Am I missing the real power of the command line?

Well, there is plenty more you can do than the things you listed there (not least automation), but even basic operations are actually a lot quicker in the command line. Example - say I want to delete all the .php files in this directory, plus all the other directories under this one. In the Windows (or Linux) GUI I would have to physically look through all those folders, sort the files by type, and select all the php ones. With the Linux command line I would just do:

$ rm -rf *.php

Simple. Or if I want to know if a certain program is running in the background - on Windows I would open the task manager and the services list and visually scan through looking for it. In the command line I would do this:

$ ps -aux | grep process_name

That would instantly tell me if a program with a name like 'process_name' is running. I could take a network connection down with:

$ ifdown eth0

...and bring another one up with:

$ ifup eth1

You get the idea.

 

[kingtas]

Ah, I see now. Thanks.

 

[avhokie]

This Mechanize module (and there's cool IE-Mechanize version) rocks! It's doing pretty much everything I wanted to. It's not great with javascript, so I'm trying to find some ways to work with that, but the rest is great!