Asynchronous HttpWebRequest

I'm working on a web crawler and I want to use HttpWebRequest. it allows asynchronous operations such as BeginGetResponse, but connecing using HttpWebRequest.Create isn't asynchronous - and I want to make about 1,000 connections simultaneously, so using this method (with an extra thread for asynchronous) I can't even get 2 connections because until the second one connects the first connection already finished downloading content, and it's almost as if I connected to the web page after page instead of simultaneously.

I was wondering if I there's a good way to connect about 1,000 times using HttpWebRequest without creating tons of threads or anything...

Thanks in advance.

Edit: Eventually it wasn't the HttpWebRequest that was slow and blocking, it was the BeginGetResponse - it's blocking until the request headers are sent? how can I bypass this, use asynchronous send as well with BeginGetRequestStream?

-------------Problems Reply------------

Are all these connections going to the same domain?

Try adding this to your app/web.config

<> <connectionManagement> <add address="*" maxconnection="1000" /> </connectionManagement> </>

I don't think you can make multiple connections on the same thread. You need one thread per connection. But you can modify your design to make it more scalable.

You can make one control thread which does all the heavy lifting (or maybe several of these) and every such control thread spaws several child threads which go out and get the data and put them in some kind of array inside the parent class. Then the control class can recycle the child threads. Once a child thread is finished, it gets another "task". The main idea, IMHO, is to seperate the crawling from the processing of the retrieved data. Get it, store it and process it later.

Hope this helps in some way :)

There is no reason that this should be blocking. There are some oddities about how asynchronous web requests work which could force your supposed asynchronous requests to be synchronous. For starters, if you are actually posting data, you must use BeginGetRequestStream (you cannot mix asynch and synch) see:

If I recall correctly nothing actually happens with WebRequest.Create, it just sets up the object, the request doesn't start until either BeginGetRequestStream or BeginGetResponse (depending if it's a post or get).

Another big note, in my findings, there is a lot more delay with reading the stream which comes from EndGetResponse than there is from the request. You should also use the asynchrnous version of read on the stream.

Category:c# Views:0 Time:2011-07-14

Related post

  • How to Create Asynchronous HttpWebRequest in Silverlight( F#) 2010-06-21

    As I mentioned, because Silverlight HttpWebRequest.Create hangs inside async block, I just created a bundle of callback functions to implement the same async block. The login process requires two steps : 1) Get request to a page that returns a cookie

  • How can I access the ui thread in an asynchronous httpwebrequest callback in a bho 2011-07-26

    I'm struggling to get access to the ui thread in an asynchronous httpwebrequest callback in a browser helper object. The current document and windows don't reflect those witnessed on the calling thread, so I can't update the UI as required. Can anyon

  • Asynchronous HttpWebRequests and a null HttpContext.Current.Response/Request object 2009-12-21

    I have a web application the relies heavily on web services. Everything with the services is done asynchronously and with AddOnPreRequestHandlerExecuteAsync. Anyhow, most of my calls work just fine, but some are returning from their asynchronous serv

  • parts of an asynchronous httpwebrequest 2010-08-23

    There are many pieces related to an httpwebrequest that can go asynchronous. I recall reading a question here about this very topic, but I can't seem to find it any more. So I'll re-ask the question. Which of the following get the most bang for the b

  • (C#) Asynchronous HTTPWebRequest Failing To Contact Server 2010-12-09

    I'm trying to develop a Windows Phone 7 application. Part of this application includes contacting an API to retrieve some data. To do so, I am utilizing HTTPWebRequest and a small helper class I've written; I call HTTPHelper's sendAPIRequest and then

  • Locking with asynchronous httpwebrequest 2011-03-14

    I have an object that downloads a file from a server, saves it into Isolated Storage asynchronously and provides a GetData method to retrieve the data. Would I use a IsolatedStorageFile storageObj; //initialized in the constructor lock(storageObj) {

  • What's the best way to handle asynchronous HttpWebRequest exceptions in C#? 2011-05-25

    I am working on some code to use HttpWebRequest asynchronously. If any of you have ever done this before, then you know that error handling can be a bit of a pain because if an exception is thrown in one of the callback methods, it can't be passed ba

  • NullReferenceException reading from an asynchronous HttpWebRequest stream 2011-12-29

    I'm programming an application for Windows Phone 7. This application firstly sends, and then receives data from a server via HttpWebRequest. Most times it works fine, but sometimes, after receiving a portion of the data properly, I get a NullReferenc

  • Asynchronous Webrequest best practices 2009-02-23

    What is the best practice for getting a webrequest asynchronously? I want to download a page from the internet (doesn't matter what) and avoid blocking a thread as much as possible. Previously I believed that it was enough to just use the 'BeginGetRe

  • Need to prioritize asynchronous socket reads in C# 2011-05-13

    My application pre-fetches a large number of video frames using asynchronous HttpWebRequest requests. So, if there are 100 frames, the prefetcher will request all 100 frames asynchronously, all at once, and process when received back. i.e. it makes 1

  • How many concurrent connections can I have open at one time when using HttpWebRequest? 2011-05-19

    I am using asynchronous HttpWebRequest BeginGetResponse calls; the docs say that not closing the stream in the callback may result in running out of connections. How many connections can I have before "running out"? I am using .NET 3.5. Also, is ther

  • HttpWebRequest AsyncCallback parameter problem 2011-07-29

    When I attempt to click to button in the first method,it's creating asynchronous http request inside for loop.But,I can't pass the parameter to my asynchronous callback function. I wanna do thing,I want to send IDs inside for loop by using POST metho

  • HttpWebRequest Grinding to a halt, possibly just due to page size 2012-01-17

    I have WPF app that processes a lot of urls (thousands), each it sends off to it's own thread, does some processing and stores a result in the database. The urls can be anything, but some seem to be massively big pages, this seems to shoot the memory

  • Threading Web requests handled in Main? 2009-11-24

    I'm writing an application in C#, and I am creating multiple BackgroundWorker threads to grab information from webpages. Despite them being BackgroundWorkers, my GUI Form is becoming unresponsive. When I am debugging, I pause when the program goes un

  • Help threading HttpWebRquest in c# 2009-12-23

    Hi guys just wondering if somebody could help me try and correctly thread my application, I am constantly hitting an hurdle after another, I have never been to clued up on threading in applications. I have tryed following this http://www.developerfus

  • What's the easiest way of making several web requests in a row? 2010-02-01

    I have a given list of URL's, and i make an HTTP web request object, and try to connect with it, i have an 'array' of url's, and i try to connect with each one. the objective is seeing which ones are out. It already works, but one request only starts

  • Optimizing HttpWebResponse - GetResponse 2010-02-09

    I'm using the following lines of code to read the response of an asynchronous HttpWebRequest. This seems to be the largest amount of time spent in a particular operation. Is there anything I can optimize here? System.Net.HttpWebResponse oResp =(Syste

  • ASP.NET Parallel Web Service Calls 2010-02-16

    I have an aspx page which reads items from an SQL DB and displays them on a GridView. For every item located in DB there are several metadata associated with it which can be retrieved by making a Web Service call per item in the GridView. What I want

  • cannot create event of type Action 2011-03-26

    Following discuss Problem to pass the result to the caller for asynchronous httpWebRequest I tried to created an event of type Action public event Action<string> ResponseResult; but Silverlight refuses event keyword: it says expect class, deleg

Copyright (C), All Rights Reserved.

processed in 0.077 (s). 11 q(s)