[TlsInterception] Responding to GET/POST method in plugin

[softhub-software-development]

When I access proxy.py, the browser always sends the request as a CONNECT method. There is an example plugin ModifyPostDataPlugin which seems to do what I am looking for but I never get it to call the proxy with the POST method. That's odd, I suppose I am doing something wrong. Could someone please help me getting on track? I'd like to write a plugin, which reacts on GET or POST requests but the parsers method always indicates being called by CONNECT.

[abhinavsingh]

CONNECT method is used to open a tunnel with upstream server and are made for all https requests. You cannot inspect or modify what's happening within this (possibly) encrypted connection. But you can enable TLS interception, which will then allow proxy.py to decrypt the content and plugins to handle them as they wish.

[softhub-software-development]

It's basically the following, but this works only for small files. How should one handle large files?

def handle_client_request(self, request: HttpParser) -> Optional[HttpParser]:
	if request.path.decode() == ‘/the_download.zip’:
		self.handle_request_locally(‘/the_download.zip’)
	return request

def handle_request_locally(self, path) -> None:
	f = open(path, ‘rb’)
	file_data = f.read()
	file_data_size = len(file_data)
	f.close()
	self.client.queue(memoryview(build_http_response(
		httpStatusCodes.OK,
		reason=b’OK’
		headers={
			b’Content-Type’: b’application/octet-stream’
			b’Content-Length: str(file_data_size).encode(‘ascii’)
		},
		body=bytes_(file_data)
	)))

[abhinavsingh]

I'll recommend you to use okResponse utility, see https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/http/responses.py#L98

self.client.queue(
    okResponse(
        file_data,
        {b'Content-Type': b'application/octet-stream'},
        conn_close=True,
    )
)

1. It will automatically take care of Content-Length header generation
2. Further, it will also compress (gzip) your content, when its size exceeds --min-compression-length flag value
3. conn_close will add a Connection: close header. You may or may not want it and is optional. Working should not get effected by presence of this flag.

Let me know.

[abhinavsingh]

Also, how large is your file?

We do have integration tests to guarantee file downloads work as expected , see https://github.com/abhinavsingh/proxy.py/blob/develop/tests/integration/test_integration.sh#L98-L118

Though files being downloaded by our integration tests are in the range of 30-70 Mb.

[softhub-software-development]

I am using debian-11.xxx.iso which is about 3.8 GB for testing. The browser shows 14 days as an approximate time for download (locally).

[abhinavsingh]

I am using debian-11.xxx.iso which is about 3.8 GB for testing. The browser shows 14 days as an approximate time for download (locally).

Thank you for this detail. We can add a large file test too.

But I notice another issue in your code. Please read documentation of

proxy.py/proxy/http/proxy/plugin.py

Lines 104 to 119 in 372a9ed

	def handle_client_request(
	self, request: HttpParser,
	) -> Optional[HttpParser]:
	"""Handler called before dispatching client request to upstream.
	Note: For pipelined (keep-alive) connections, this handler can be
	called multiple times, for each request sent to upstream.
	Note: If TLS interception is enabled, this handler can
	be called multiple times if client exchanges multiple
	requests over same SSL session.
	Return optionally modified request object to dispatch to upstream.
	Return None to drop the request data, e.g. in case a response has already been queued.
	Raise HttpRequestRejected or HttpProtocolException directly to
	tear down the connection with client.

Your plugin must return None if it doesn't want request to go to upstream server. But looks like your plugin never does that. Which in-turn means, request is going to upstream server even after you have queued the data locally.

That seems like a reason why you have a 14 days ETA. Browser is likely getting corrupt data?

You must replace your code to return None e.g.

return self.handle_request_locally(‘/the_download.zip’) will work just fine.