[ProxyPoolPlugin] Trouble on https request through http upstream proxy

[Samll-Kosmos]

Hi, thank you so much in building this awesome proxy.

I'm trying to upstream from proxy.py to a remote http proxy and and do an https request but I'm having no success. For the upstreaming I'm using the ProxyPoolPlugin. I'm able to do http requests the problem are https requests. Here is an example trying to connect to https://ipinfo.io/

CURL:
curl -v -x localhost:8899 https://ipinfo.io/

* Uses proxy env variable no_proxy == 'localhost,127.0.0.0/8,::1'
*   Trying 127.0.0.1:8899...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to ipinfo.io:443
> CONNECT ipinfo.io:443 HTTP/1.1
> Host: ipinfo.io:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< server: squid
< mime-version: 1.0
< date: Tue, 05 Jan 2021 16:54:35 GMT
< content-type: text/html;charset=utf-8
< content-length: 3667
< x-squid-error: ERR_DNS_FAIL 0
< vary: Accept-Language
< content-language: en
< 
* Received HTTP code 503 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 503 from proxy after CONNECT

Proxy.py

I'm using
Python 3.8.0
proxy.py 2.3.0

HTTP request are working without problem e.g. curl -v -x localhost:8899 http://ipinfo.io/
Also, direct https connections through external proxy are working without problem. e.g. curl -x <external_proxy> https://ipinfo.io/

I'm really lost and any help would be highly appreciated, thank you in advance!

[abhinavsingh]

Hi @Samll-Kosmos , Thanks for opening this discussion.

I see code is broken as rebuilt request doesn't match the original CONNECT request. Example here https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/proxy_pool.py#L77-L83 must be changed to return ( request.host + COLON + str(request.port).encode() ) to match the original CONNECT request.

But perhaps even after this fix, https proxy pool won't work as expected. Due to None being returned here https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/proxy_pool.py#L50-L52.

I currently have little time on hands to address this. @paul-alkhimov in the past has given this a shot and recommended exposing internal event loop to plugins, which seems to be at the core of all issues (note that pool plugin makes synchronous call within asynchronous proxy core).

[abhinavsingh]

jFYI, #638 landed support for allowing plugins to register custom descriptors for read/write. This will now allow plugins to asynchronously work with upstream socket objects.

[abhinavsingh]

Marking this as answer now. Best to use v2.4.0rc1 releasing in a few days. Ref #777