What programming language does this

[sswingle]

Basically what I want to do, is to create a page where you can type in information about a shipment, ie, class, weight, number of pallets, and then the site goes out to 4 different freight companies, puts the info into their web forms, submits, and then takes the results and puts them all on the same page.

Here is an example which is doing this with search engine results:

http://blindsearch.fejus.com/

I am pretty good at learning how to do new things, but I don't even know what its called to search for a tutorial.

Thanks!

 

[Markbnj]

You can do that with any language that can submit an HTTP forms POST, and parse the resulting document, and that covers a lot of languages. What is your technical background?

 

[Snapster]

You either will need to use web services, or you can create a web request, post data to each site and parse the returned documents.

 

[VinylxScratches]

If the freight company offers a API, you can use that.

 

[ahurtt]

The web services approach as mentioned by Snapster would be the best way to approach this but it comes with the assumption that the websites you want to post your data to have made said services available as web services. If that is the case they are service providers and you would be authoring clients to consume their services. Web Services are usually defined in XML based definition files called WSDL's. (Web Service Description Language) If the parties from whom you want to scrape your data have not made their services available as Web Services, then you would have to take the second approach of programatically sending POST's or GET's to the desired URL's sending your query data the same way the web browser would do when acting as the client. This approach works IF there is no security implemented around the data that you're after. If the web sites from which you are trying to collect your data require login and/or SSL to access the desired content, your job becomes more difficult.

As for what languages could do this. . .any number of languages. Java, C#/.NET, Perl, C++, you name it... It's not so much a matter of knowing which language to use to do it. It's knowing the technologies and standards that are involved and understanding how they work and how to use them to implement what you want to do.

 

[sswingle]

Ok, most of my experience is ASP and HTML. I was hoping to just run the page locally though, and not through a server. I guess we could put them out on the webserver if we HAD to though.

These sites do require a login, I figured I could hard code them into the page. They aren't a big secret or anything. I think the web service method is out, I don't see any mention of that anywhere on their sites.

Anyone know of a tutorial I could read that shows an example of doing the posts and gets? I did a search, but I haven't found anything specifically dealing with a 3rd party form.

 

[ahurtt]

I suppose you could cobble something together using DHTML/Javascript/AJAX that would run totally client side within the browser and without need for any deployment of anything to a server but it would be pretty complex and kludgy in my opinion. You could just use javascript to dynamically populate hidden inputs in "hidden" forms embedded in the page or in hidden frames or something like that. So something like:

Make a visible form with inputs for the data you want to collect,
Have x number of hidden forms, one for each other site you want to post to.
Have javascript copy appropriate data from user input form to hidden form fields.
Have javascript submit each form.
Use DOM to parse data out of responses and populate it into the result display using DHTML.

You'd have to figure out how to handle the login which is probably a separate prior request. I really don't endorse this way but if you REALLY want to do it I guess it might be theoretically possible.

 

[sswingle]

Maybe it will be more effort than its worth then. I just thought it would be really cool to have a page to enter the form once and sit back and watch the quotes populate, rather than manually going to each of the 4 sites.

 

[ahurtt]

I agree with you on that, it would be a cool thing definitely worthy of doing. But it is a task that is really more suited to server side processing logic with an application server of some sort than browser scripting. You mentioned you were familiar with ASP and that is one choice that you could use to implement your solution which should work fine. It'll just be up to you to figure out the right libraries and API's and such you'd need to use. A couple more popular web-centric programming language alternatives popular these days are Ruby on Rails and PHP. They are not as heavy-weight as some of the other alternatives I mentioned such as .NET or Java / J2EE but should be able to do what you are trying to do just fine and are pretty easy to set up and get up and running with.

 

[nakedfrog]

Originally posted by: sswingle
Ok, most of my experience is ASP and HTML. I was hoping to just run the page locally though, and not through a server. I guess we could put them out on the webserver if we HAD to though.

These sites do require a login, I figured I could hard code them into the page. They aren't a big secret or anything. I think the web service method is out, I don't see any mention of that anywhere on their sites.

Anyone know of a tutorial I could read that shows an example of doing the posts and gets? I did a search, but I haven't found anything specifically dealing with a 3rd party form.

In Classic ASP, I think Server.CreateObject("MSXML2.ServerXMLHTTP") may do what you're looking for, with a bit of work. Probably easier to write in ASP.Net if you're familiar enough with it.

 

[Ken g6]

Javascript apparently doesn't allow fetching other sites' web pages for security reasons. Personally, I think it's a real pain, as there are quite a few things I'd like to do with Javascript parsing other web pages. But I guess if I could make your computer post nasty messages somewhere, or DDOS a web server, just by having you visit my web site, that would be a security problem.

 

[PhatoseAlpha]

I imagine it's more another layer of security to prevent things like bank account or cc theft.

 

[ahurtt]

Originally posted by: Ken g6
Javascript apparently doesn't allow fetching other sites' web pages for security reasons. Personally, I think it's a real pain, as there are quite a few things I'd like to do with Javascript parsing other web pages. But I guess if I could make your computer post nasty messages somewhere, or DDOS a web server, just by having you visit my web site, that would be a security problem.

I believe you can do it but it might require modification of the default security settings to allow cross-site scripting or something like that.