Oxylabs’ Residential Proxies integration with Selenium using Java

• Introduction
• Prerequisites
• Requirements
• Running the Code
  • Compiling the Source Code
  • Running the Jar
• Proxy Authentication
• Testing Proxy Connection
• Getting Country Specific Proxy
• SSL Support
  • Installation Certificate for macOS
  • Installation Certificate for Windows
• Understanding the Code

Introduction

Integrating proxies that need authorization using the Selenium framework and Java programming language can be challenging.

This tutorial contains complete code demonstrating how Oxylabs’ Residential Proxies can be easily integrated with Selenium using Java.

Prerequisites

• Download and install Maven.
• Download and install Java SE Development Kit.
• Download and install Google Chrome.

Requirements

To make this integration easier, we used BrowserMob Proxy as a middle layer. It runs proxies locally in JVM and allows chaining of Oxylabs' authenticated proxies. If you’re using Maven, add this dependency to the pom.xml file:

<dependency>
    <groupId>net.lightbody.bmp</groupId>
    <artifactId>browsermob-core</artifactId>
    <version>2.1.5</version>
</dependency>

The other library used in this project – WebDriverManager, is optional. It just makes downloading and setting up Chrome Driver easier. To use this library, include the following dependency in pom.xml file:

<dependency>
    <groupId>io.github.bonigarcia</groupId>
    <artifactId>webdrivermanager</artifactId>
    <version>5.0.2</version>
</dependency>

If you don’t want to use WebDriverManager, download the Chrome Driver and set the system property as follows:

System.setProperty("webdriver.chrome.driver","/path/to/chromedriver");

Running the Code

Compiling the Source Code

This is a maven project. To compile this project, run the following command from terminal:

mvn clean package

This will create the file oxylabs.io-jar-with-dependencies.jar in the target folder.

Running the Jar

To run the jar, execute the following command from the terminal:

java -cp target/oxylabs.io-jar-with-dependencies.jar ProxyDemo

Proxy Authentication

Open ProxySetup.java file and update your username, password, and endpoint.

static final String ENDPOINT="pr.oxylabs.io:7777";
static final String USERNAME="yourUsername";
static final String PASSWORD="yourPassword";

You shouldn’t include the prefix customer- in the USERNAME. This will be added in the code for country-specific proxies.

Testing Proxy Connection

Open this project in IDE, open the ProxySetup.java file, and run the main() function. This will print two IP addresses.

• The first IP address will be completely random;
• The second IP address will be a country-specific IP address in Germany.

Getting Country Specific Proxy

Open ProxyDemo.java file and send a two-letter country code to the function CountrySpecficIPDemo.

countrySpecificIPDemo("DE");

The value of this parameter is a case-insensitive country code in two-letter 3166-1 alpha-2 format. For example, DE for Germany, GB for the United Kingdom, etc. For more details, see Oxylabs’ documentation.

SSL Support

This code uses BrowserMob Proxy, which supports full MITM.

You may still see invalid certificates warnings. To solve this, install the ca-certificate-rsa.cer file in your browser or HTTP client. Alternatively you can generate your own private key rather than using the .cer files distributed with repository.

Installation Certificate for macOS

• Open Keychain Access
• Selct System on left bar
• Click the padlock icon next to System and enter your password when prompted
• Select Certificates tab on the top
• Drag and drop the ca-certificate-rsa.cer file into the Certificates tab. You will notice a new certificate appears in the Certificates tab with the name LittleProxy MITM.

• Right click the certificate and select Get Info
• Select Always Trust, close the dialog, and enter password again when promoted

Installation Certificate for Windows

• Open the ca-certificate-rsa.cer file in Windows Explorer.
• Right click the file, and select Install.
• In the Certificate Import Wizard window, click Browse and select Trusted Publishers. Click OK to continue.

• If you see a Security Warning, select Yes.

• Follow the wizard to complete the installation.

Understanding the Code

All the complexity of setting up BrowserMob Proxy and Chrome Options is hidden in the ProxyHelper class.

In most cases, you should be able to use this file directly without any change.

To create a Chrome Driver instance, go through a two-step process as follows:

First, create an instance of BrowserMobProxyServer. This is where you need to provide the proxy endpoint, username, and password.

The fourth parameter is a two-letter country code. If you don’t need a country-specific proxy, set it to null:

BrowserMobProxyServer proxy=ProxyHelper.getProxy(
        ProxySetup.ENDPOINT,
        ProxySetup.USERNAME,
        ProxySetup.PASSWORD,
        countryCode)

Next, call the ProxyHelper.getDriver() function.

This function takes up two parameters -BrowserMobProxyServer and a boolean headless. To run the browser in headless mode, send true:

WebDriver driver=ProxyHelper.getDriver(proxy,true);

driver is an instance of Chrome Driver.

Now, you should write your code to use the Chrome Driver.

Before exiting, remember to close the driver and stop the proxy:

driver.quit();
proxy.stop(); 

If you're having any trouble integrating Oxylabs’ Residential Proxies with Selenium and this guide didn't help you – feel free to contact our customer support at [email protected].