Many times during your life as a java developer, you will face the situation of retrieving some resources using an HTTP connection. At first, it will seem easy, but probably some problems will arise such as:

  1. Needing to use an HTTP Proxy (maybe authentication would be required)
  2. Establishing an HTTP authenticated connection
  3. Connecting to a server that uses SSL self-signed certificates

I’m sure you will quickly find the Apache Commons solution: commons-httpclient. In this article I will show you some code that, I hope, will ease you the  resolution of the previous obstacles using this great API from Apache.

First of all I will show you the basic code needed to establish an HTTP GET connection using commons-httpclient and then we will add continuous enhancements step by step:

GetMethod httpget = null;
String result = null;
try {
    httpget = new GetMethod(url);

    HttpClient client = new HttpClient();
    int status = client.executeMethod(httpget);
    if (status == 200) {
        result = httpget.getResponseBodyAsString();
    } else {
        System.err.println("Error accessing to URL " + status + ": " +  httpget.getStatusLine());
    }

} catch (MalformedURLException e) {
    System.err.println("Malformed URL: " + e.getMessage());
} catch (IOException e) {
    System.err.println("I/O problems: " + e.getMessage());
} catch (Exception e) {
    System.err.println("URL not found: " + e.getMessage());
} finally {
    if (httpget != null) {
        httpget.releaseConnection();
    }
}

1. Use an HTTP Proxy (maybe authentication would be required)

Now lets add the necessary code to establish the HTTP connection using a Proxy. I will use a static method to configure proxy settings in our HttpClient instance:

private static void configureProxy(HttpClient client, HttpMethodBase method, String proxyHost, int proxyPort, String proxyUsername, String proxyPassword, String proxyNTDomain) {

    client.getHostConfiguration().setProxy(proxyHost, proxyPort);

    if (proxyUsername != null && proxyUsername.length() > 0) {

        if (proxyNTDomain != null && proxyNTDomain.length() > 0)  {
            // use NT Domain authentication
            NTCredentials credentials = new NTCredentials(proxyUsername, proxyPassword, proxyHost, proxyNTDomain);
            HttpState state = client.getState();
            state.setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), credentials);
        } else {
            // use plain user/password authentication
            Credentials defaultcreds = new UsernamePasswordCredentials(proxyUsername, proxyPassword);
            client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), defaultcreds);
        }
        method.setDoAuthentication(true);
    }

}

and then we will have to insert this line of code in our previous example (I have inserted it between lines 6-7):

configureProxy(client, httpget, proxyHost, proxyPort, proxyUsername, proxyPassword, proxyNTDomain);

At this moment, we have point 1 solved, so let’s go for point 2: Basic HTTP Authentication.

2. Establish an HTTP authenticated connection

In order to keep code as clean as possible, I will create another static method to set HTTP authentication:

private static void configureHTTPAuthentication(HttpClient client, String host, int port, String httpUsername, String httpPassword) {
    client.getState().setCredentials(new AuthScope(host, port), new UsernamePasswordCredentials(httpUsername, httpPassword));
}

and the required call to this method, just after the previous one:

configureHTTPAuthentication(client, httpget.getURI().getHost(), httpget.getURI().getPort(), httpUsername, httpPassword);

This one was easy 😉

3. Connect to a server that uses SSL self-signed certificates

Now lets solve the last point, probably the more difficult of all of them. We will need two implementations of ProtocolSocketFactory and X509TrustManager that accept self-signed certificates: EasySSLProtocolSocketFactory and EasyX509TrustManager, which you can find in httpclient-contrib (I have grabbed them from Adrian Sutton and Oleg Kalnichevski and you can also find them attached at the end of this article). Once in our classpath, we have to insert the following code in our first example, just below the call to configureHTTPAuthentication:

if (url.startsWith("https")) {
    Protocol protocol = new Protocol("https", new EasySSLProtocolSocketFactory(), 443);
    client.getHostConfiguration().setHost(httpget.getURI().getHost(), 443, protocol);
    // we also can set this socketfactory as global for all the connections
    //Protocol.registerProtocol("https", protocol);
}

Finally, we should be able to connect to a HTTP-Authenticaded SSL URL, using an http (authenticated-?)proxy. Isn’t it cool? This is how our final code looks like:

private static void configureProxy(HttpClient client, HttpMethodBase method, String proxyHost, int proxyPort, String proxyUsername, String proxyPassword, String proxyNTDomain) {

    client.getHostConfiguration().setProxy(proxyHost, proxyPort);

    if (proxyUsername != null && proxyUsername.length() > 0) {

        if (proxyNTDomain != null && proxyNTDomain.length() > 0)  {
            // use NT Domain authentication
            NTCredentials credentials = new NTCredentials(proxyUsername, proxyPassword, proxyHost, proxyNTDomain);
            HttpState state = client.getState();
            state.setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), credentials);
        } else {
            // use plain user/password authentication
            Credentials defaultcreds = new UsernamePasswordCredentials(proxyUsername, proxyPassword);
            client.getState().setProxyCredentials(new AuthScope(proxyHost, proxyPort, AuthScope.ANY_REALM), defaultcreds);
        }
        method.setDoAuthentication(true);
    }

}

private static void configureHTTPAuthentication(HttpClient client, String host, int port, String httpUsername, String httpPassword) {
    client.getState().setCredentials(new AuthScope(host, port), new UsernamePasswordCredentials(httpUsername, httpPassword));
}

public static String doGETConnection(String url, String proxyHost, int proxyPort, String proxyUsername, String proxyPassword, String proxyNTDomain, String httpUsername, String httpPassword) {

    GetMethod httpget = null;
    String result = null;
    try {
        httpget = new GetMethod(url);

        HttpClient client = new HttpClient();
        configureProxy(client, httpget, proxyHost, proxyPort, proxyUsername, proxyPassword, proxyNTDomain);
        configureHTTPAuthentication(client, httpget.getURI().getHost(), httpget.getURI().getPort(), httpUsername, httpPassword);

        if (url.startsWith("https")) {
            Protocol protocol = new Protocol("https", new EasySSLProtocolSocketFactory(), 443);
            client.getHostConfiguration().setHost(httpget.getURI().getHost(), 443, protocol);
            // we also can set this socketfactory as global for all the connections
            //Protocol.registerProtocol("https", protocol);
        }

        int status = client.executeMethod(httpget);
        if (status == 200) {
            result = httpget.getResponseBodyAsString();
        } else {
            System.err.println("Error accessing to URL " + status + ": " +  httpget.getStatusLine());
        }

    } catch (MalformedURLException e) {
        System.err.println("Malformed URL: " + e.getMessage());
    } catch (IOException e) {
        System.err.println("I/O problems: " + e.getMessage());
    } catch (Exception e) {
        System.err.println("URL not found: " + e.getMessage());
    } finally {
        if (httpget != null) {
            httpget.releaseConnection();
        }
    }

    return result;

}

Possible exceptions

But, if we haven’t been lucky, we can find one of this two exceptions (if not both) when trying to establish connection:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This one is due to the JDK can not verify the self-signed certificate. We have to add it to a trusted keystore and make it available to the JDK, using InstallCert java standalone application.

Another different problem that I got while developing this code, was the next one:

javax.net.ssl.SSLKeyException: RSA premaster secret error

It’s related to this size of the certificate and the JDK cryptographic capabilities. To resolve this issue, download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files version 5.0. Extracted from its README file:

Due to import control restrictions, the version of JCE policy files that are bundled in the JDK(TM) 5.0 environment allow “strong” but limited cryptography to be used. This download bundle (the one including this README file) provides “unlimited strength” policy files which contain no restrictions on cryptographic strengths.
Please note that this download file does NOT contain any encryption functionality since such functionality is supported in Sun’s JDK 5.0.Thus, this installation applies only to Sun’s JDK 5.0, and assumes that the JDK 5.0 is already installed.

And finally…

I hope this article would help you to resolve your HTTP connection issues. If not, please post a comment and I will try to reply you ASAP. You can also find me at twitter @danielpecos

Feedback is welcomed!

Files used for this demo:

  • Full sources of the example developed here: httpclient-example.
  • Standalone java application to create a keystor__e: InstallCert.
  • I also provide a mirror for the needed policy files in case of RSA premaster secret error.