Non-blocking/NIO HTTP requests in Java with Jetty's HttpClient
Posted by Kelvin on 05 Mar 2012 at 12:45 am | Tagged as: programming, crawling
Jetty 6/7 contain a HttpClient class that make it uber-easy to issue non-blocking HTTP requests in Java. Here is a code snippet to get you started.
Initialize the HttpClient object.
HttpClient client = new HttpClient(); client.setConnectorType(HttpClient.CONNECTOR_SELECT_CHANNEL); client.setMaxConnectionsPerAddress(200); // max 200 concurrent connections to every address client.setTimeout(30000); // 30 seconds timeout; if no server reply, the request expires client.start();
Create a ContentExchange object which encapsulates the HTTP request/response interaction.
ContentExchange exchange = new ContentExchange() { @Override protected void onResponseComplete() throws IOException { System.out.println(getResponseContent()); } }; exchange.setAddress(new Address("supermind.org", 80)); exchange.setURL("http://www.supermind.org/index.html"); client.send(exchange);
We override the onResponseComplete() method to print the response body to console.
By default, an asynchronous request is performed. To run the request synchronously, all you need to do is add the following line:
exchange.waitForDone();