v1.2.0

README.md

@@ -1,94 +1,144 @@
 # puppeteer-page-proxy <img src="https://i.ibb.co/kQrN9QJ/puppeteer-page-proxy-logo.png" align="right" width=150 height=150/>
-Additional Node.js module to use with '[puppeteer](https://www.npmjs.com/package/puppeteer)' for setting proxies per page basis.
+Additional Node.js module to use with **[puppeteer](https://www.npmjs.com/package/puppeteer)** for setting proxies per page basis.
 
-Uses **(http | https | socks)-proxy-agent** to achieve the desired results.
+Forwards intercepted requests from the browser to Node.js where it handles the request then returns the response to the browser, changing the proxy as a result.
 
-This small module consists of a class and a function taken from [ppspider](https://github.com/xiyuan-fengyu/ppspider).
+## Features
 
-All credit for the original code goes to the author [@xiyuan-fengyu](https://github.com/xiyuan-fengyu).
+- Proxy per page **and** per request
+- Supports **(** http, https, socks4, socks5 **)** proxies
+- Authentication
+- Cookie handling internally
 
 ## Installation
 ```
 npm i puppeteer-page-proxy
 ```
+## API
+#### PageProxy(pageOrReq, proxy)
 
-## Methods
-#### PageProxy(page, proxy[, enableCache])
-
-* `page` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> Page object to set a proxy for.
-* `proxy` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Proxy to use in the current page. Must begin with a protocol e.g. **http://**, **https://**, **socks://**.
-* `enableCache` <[boolean](https://developer.mozilla.org/en-US/docs/Glossary/Boolean)> Whether to enable caching. Defaults to `true`.
+- `pageOrReq` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> 'Page' or 'Request' object to set a proxy for.
+- `proxy` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Proxy to use in the current page.
+ * Begins with a protocol e.g. **http://**, **https://**, **socks://**
 
 #### PageProxy.lookup(page[, lookupService, isJSON, timeout])
 
-* `page` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> Page object to execute the request on.
-* `lookupService` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> External lookup service to request data from. Fetches data from `api.ipify.org` by default.
-* `isJSON` <[boolean](https://developer.mozilla.org/en-US/docs/Glossary/Boolean)> Whether to [JSON.parse](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse) the received response. Defaults to `true`.
-* `timeout` <[number](https://developer.mozilla.org/en-US/docs/Glossary/Number)|[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Time in milliseconds after which the request times out. Defaults to `30000` ms.
-* returns: <[Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)> Promise which resolves to the response of the lookup request.
+- `page` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> 'Page' object to execute the request on.
+- `lookupService` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> External lookup service to request data from.
+ * Fetches data from `api.ipify.org` by default.
+- `isJSON` <[boolean](https://developer.mozilla.org/en-US/docs/Glossary/Boolean)> Whether to [JSON.parse](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse) the received response.
+ * Defaults to `true`.
+- `timeout` <[number](https://developer.mozilla.org/en-US/docs/Glossary/Number)|[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Time in milliseconds after which the request times out.
+ * Defaults to `30000` ms.
+- returns: <[Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)> Promise which resolves to the response of the lookup request.
 
 **NOTE:** By default this method expects a response in [JSON](https://en.wikipedia.org/wiki/JSON#Example) format and [JSON.parse](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse)'s it to a usable javascript object. To disable this functionality, set `isJSON` to `false`.
 
 ## Examples
-#### General usage:
+### Proxy per page:
 ```js
 const puppeteer = require('puppeteer');
 const useProxy = require('puppeteer-page-proxy');
 
 (async () => {
- let site = 'https://example.com';
- let proxy1 = 'http://host:port';
- let proxy2 = 'https://host:port';
- let proxy3 = 'socks://host:port';
-
- // For proxies that require authentication
- // pass in the username and password in the format shown below
- let proxy4 = 'https://login:pass@host:port';
+ const site = 'https://example.com';
+ const proxy = 'http://host:port';
+ const proxy2 = 'https://host:port';
 
  const browser = await puppeteer.launch({headless: false});
 
- const page1 = await browser.newPage();
- await useProxy(page1, proxy1);
- await page1.goto(site);
+ const page = await browser.newPage();
+ await useProxy(page, proxy);
+ await page.goto(site);
 
  const page2 = await browser.newPage();
  await useProxy(page2, proxy2);
  await page2.goto(site);
+})();
+```
+#### Changing proxy if a page is already using one:
+```js
+await useProxy(page, proxy);
+await page.goto(site, {waitUntil: 'networkidle2'});
 
- const page3 = await browser.newPage();
- await useProxy(page3, proxy3);
- await page3.goto(site);
-
- const page4 = await browser.newPage();
- await useProxy(page4, proxy4);
- await page4.goto(site);
+await useProxy(page, proxy2);
+await page.reload();
+```
+#
+### Proxy per request:
+```js
+const puppeteer = require('puppeteer');
+const useProxy = require('puppeteer-page-proxy');
+
+(async () => {
+ const site = 'https://example.com';
+ const proxy = 'socks://host:port';
+
+ const browser = await puppeteer.launch({headless: false});
+ const page = await browser.newPage();
+
+ await page.setRequestInterception(true);
+ page.on('request', async req => {
+ await useProxy(req, proxy); // 'req' as argument
+ });
+ await page.goto(site);
 })();
 ```
+When changing proxies this way, the request object itself is passed as the first argument. Now 'proxy' can be changed every request.
+Leaving it as is will have the same effect as `useProxy(page, proxy)`, meaning that the same proxy will be used for all requests within the page.
+
+**NOTE:** It is necessary [page.setRequestInterception()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#pagesetrequestinterceptionvalue) to true, otherwise the function will fail.
+#### Using with other request listeners:
+```js
+await page.setRequestInterception(true);
+page.on('request', async req => {
+ if (req.resourceType() === 'image') {
+ req.abort();
+ } else {
+ await useProxy(req, proxy);
+ }
+});
+```
+Internally `puppeteer-page-proxy` handles the request which was passed in, then responds back to the browser using the interception method [request.respond()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestrespondresponse). Calling this method will fulfill the request with a given response and set the proxy.
+
+**NOTE:** Since all requests can be handled exactly once, it's not possible to call other interception methods e.g. [request.abort()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestaborterrorcode), [request.continue()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestcontinueoverrides) after calling `useProxy()`, without getting a *'Request is already handled!'* error message.
+
+#
+#### Authentication:
+```js
+const proxy = 'https://login:pass@host:port';
+```
 #### Lookup IP used by proxy -> Useful in headless environment:
 ```js
 const puppeteer = require('puppeteer');
 const useProxy = require('puppeteer-page-proxy');
 
 (async () => {
- let site = 'https://example.com';
- let proxy1 = 'http://host:port';
- let proxy2 = 'https://host:port';
+ const site = 'https://example.com';
+ const proxy1 = 'http://host:port';
+ const proxy2 = 'https://host:port';
 
  const browser = await puppeteer.launch({headless: false});
 
  /**1*/
  const page1 = await browser.newPage();
  await useProxy(page1, proxy1);
- let data = await useProxy.lookup(page1); // Waits until done, "then" continues
+ let data = await useProxy.lookup(page1); // Waits until done, 'then' continues
  console.log(data.ip);
  await page1.goto(site);
 
  /**2*/
  const page2 = await browser.newPage();
  await useProxy(page2, proxy2);
- useProxy.lookup(page2).then(data => { // Executes and "comes back" once done
+ useProxy.lookup(page2).then(data => { // Executes and 'comes back' once done
  console.log(data.ip);
  });
  await page2.goto(site);
 })();
 ```
+## Dependencies
+- [Got](https://github.com/sindresorhus/got)
+- [http-proxy-agent](https://github.com/TooTallNate/node-http-proxy-agent)
+- [https-proxy-agent](https://github.com/TooTallNate/node-https-proxy-agent)
+- [socks-proxy-agent](https://github.com/TooTallNate/node-socks-proxy-agent)
+- [tough-cookie](https://github.com/salesforce/tough-cookie)

changelog.md

@@ -0,0 +1,12 @@
+# Change log
+### [1.2.0] - 2020-02-09
+#### Major changes
+- Added capability to change proxy per request.
+- Added capability of changing the proxy if a page is already using one.
+- Replaced [Request](https://github.com/request/request) with [Got](https://github.com/sindresorhus/got) for forwarding requests.
+- Added [tough-cookie](https://github.com/salesforce/tough-cookie) for handling cookies
+- Now communicates directly with Chrome DevTools Protocol when getting cookies
+#### Minor changes
+- Added a simple type enforcer and proxy validator.
+- Removed some redundant code.
+- Removed obsolete 'cache' parameter

package.json

@@ -1,11 +1,11 @@
 {
  "name": "puppeteer-page-proxy",
- "version": "1.0.9",
  "description": "Additional Node.js module to use with 'puppeteer' for setting proxies per page basis.",
+ "version": "1.2.0",
  "author": "Cuadrix <[email protected]> (https://github.com/Cuadrix)",
  "homepage": "https://github.com/Cuadrix/puppeteer-page-proxy",
- "main": "puppeteer-page-proxy.js",
- "types": "puppeteer-page-proxy.d.ts",
+ "main": "./src/index.js",
+ "types": "./src/index.d.ts",
  "scripts": {
  "test": "echo \"Error: no test specified\" && exit 1"
  },
@@ -15,17 +15,17 @@
  },
  "keywords": [
  "puppeteer",
- "page",
- "different",
  "proxy",
+ "page",
  "headless",
  "chromium"
  ],
  "license": "MIT",
  "dependencies": {
- "request": "^2.88.0",
- "http-proxy-agent": "^2.1.0",
- "https-proxy-agent": "^2.2.1",
- "socks-proxy-agent": "^4.0.2"
+ "got": "^10.5.2",
+ "http-proxy-agent": "^4.0.0",
+ "https-proxy-agent": "^5.0.0",
+ "socks-proxy-agent": "^5.0.0",
+ "tough-cookie": "^3.0.1"
  }
 }