C# HttpWebRequest problems

[MGMorden]

Hey guys. I'm running into an issue here and figured I'd ask for a bit of advice. I've got an app that needs to interface with a third party utility. Specifically, it needs to submit an address to the utility and have it process it for CASS certification.

The only "server mode" that this utility allows though, is it has a web interface that allows you to type in the criteria and submit them in. It can then respond back with either an HTML or XML file (your choice) which you can then parse.

So, my plan was simple: have my C# program simply hit the HTML page, get the XML response back, and then parse my data out of that.

The following function is one that I borrowed and modified a bit from a quick web example on doing this.

        private string GetResponseWithPost(string StrURL, string strPostData)
        {
            string strReturn = "";
            HttpWebRequest objRequest = null;
            ASCIIEncoding objEncoding = new ASCIIEncoding();
            Stream reqStream = null;
            HttpWebResponse objResponse = null;
            StreamReader objReader = null;
            try
            {
                objRequest = (HttpWebRequest)WebRequest.Create(StrURL);

                objRequest.Method = "POST";
                byte[] objBytes = objEncoding.GetBytes(strPostData);
                objRequest.ContentLength = objBytes.Length;
                objRequest.ContentType = "application/x-www-form-urlencoded";
                reqStream = objRequest.GetRequestStream();
                reqStream.Write(objBytes, 0, objBytes.Length);

                objResponse = (HttpWebResponse)objRequest.GetResponse();
                objReader = new StreamReader(objResponse.GetResponseStream());
                strReturn = objReader.ReadToEnd();

            }
            catch (Exception exp)
            {
                throw exp;
            }
            finally
            {
                objRequest = null;
                objEncoding = null;
                reqStream = null;
                if (objResponse != null)
                    objResponse.Close();
                objResponse = null;
                objReader = null;
            }
            return strReturn;
        }

Using this, the program works just fine. Here is the code used to actually create the call the function and put the XML into a string. As you can tell, the original address that I'm sending is contained in a series of text boxes.

string response;
string postData = "company=" + textBoxOrigAddress1.Text + "&address=" + textBoxOrigAddress1.Text + "&address2=" + textBoxOrigAddress2.Text + "&city=" + textBoxOrigCity.Text + "&state=" + comboBoxOrigState.Text + "&zip=" + textBoxOrigZip.Text;

All comes back great. The problem however, is the speed. It's highly variable - taking anywhere from 5 to 15 seconds to get a response back from the web server. This really, really adds up when doing submissions. Thing is, submitting the web form from a browser results in a nearly instant response, leading me to believe that there is some issue there.

Any ideas on this? Any obvious reason why the function above would behave so slowly?

Thanks.

 

[Markbnj]

I don't see any obvious reason why it should take longer. Try using a proxy like Fiddler to capture the requests in both cases and compare them.

 

[nickbits]

The web server is probably slow. Can you test a request in a browser?

Maybe try closing the request stream.

Personally I'd use the WebClient class instead but I doubt that is your problem.

 

[DaveSimmons]

Download Wireshark (it's free) and compare what your app sends and receives against the traffic using IE.

I'd suggest IE over Firefox since then you're using MS internet libraries for both tests.

Some possible differences:
- Errors or missing variables in your POST data
- Missing cookies (could be set by JavaScript in web pages rather than passed as SetCookie headers)
- Missing headers (sometimes the server might care about the Referer: header)
- User-Agent string

 

[Markbnj]

Another option is Paros. It's a good proxy tool for checking request/response chains between a web services client and the service entry point, and it will work great for comparing http messages.

http://www.parosproxy.org/

 

[Oyster]

Have you tried "chunking" your streams?

http://stackoverflow.com/questions/16998/reading-chunked-response-with-httpwebresponse

Depending on the load on your server, it is possible that the application is unable to cache the data in the RAM, going straight to virtual memory and slowing things down. Notice that the link uses an 8 meg chunk - that's pretty big, so I would reduce it to something like 2 megs.

If this seems to improve performance, I'd probably implement the same solution for your HttpWebRequest portion, too.

 

[MGMorden]

Update: Thanks for the response guys. I managed to track down the issue. Due to the way the WebRequest object was being created, it was trying to perform automatic proxy detection each time the object was created, which was slowing down the app. I disabled auto-proxy detection in the application's config file and it is now working MUCH better.

 

[MrChad]

Update: Thanks for the response guys. I managed to track down the issue. Due to the way the WebRequest object was being created, it was trying to perform automatic proxy detection each time the object was created, which was slowing down the app. I disabled auto-proxy detection in the application's config file and it is now working MUCH better.

Thanks for updating this thread with the solution.