HTTP20Adapter sends extra headers ??

[qwerty32123]

import hashlib
import requests
import json
from tls_version import MyAdapter
import collections
from userdata import  UserData
import time
from random import randrange
from hyper.contrib import HTTP20Adapter


headers2 = [('Upgrade-Insecure-Requests', '1'),
        ('User-Agent', 'Mozilla/5.0 (Linux; Android 5.1.1; google Pixel 2 Build/LMY47I; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36'),
        ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'),
        ('Accept-Encoding', 'gzip, deflate'),
        ('Accept-Language', 'es-ES,es;q=0.9,en-US;q=0.8,en;q=0.7'),

        ]
class post_calls():
        def start(self,headers_pass,body_pass,params,url,method):
            proxies = {
                'http': ip,
                'https': ip
            }
            body = str(body_pass)


            #send the POST request
            session = requests.session()
            session.mount('https://', MyAdapter())
            session.headers =  collections.OrderedDict(headers_pass)
            if method == 'get':
                q = 'https://' + server + '.' + host
                q = q.replace('.www.', '.')
                session.mount('https://', HTTP20Adapter())
                print('q='+q)
                response = session.get(url, proxies=proxies, params=params, verify=charlesproxy)





def login_world2(sid): 

    a = post_calls()
    q ='https://'+server+'.'+ host+'/login.php?mobile&sid='+sid+'&2'
    q = q.replace('.www.','.')
    params = {}
    url = q
    body = '0'
    login = a.start(headers2,body,params,url,'get')
    return login

if __name__ == "__main__":
    login_get = login_world(sid)
    print(login_get)

and it sends those headers(im sniffing my code http/s traffic with charlesproxy)

:method: GET
:scheme: https
:authority: server.url.com
:path: /login.php?mobile&sid=577f0967545d6acec94716d265dd4867fa4db4a446326ecde7486a97feede14702f4911438f4a4cd097677f0dd962786ef14b3f16f1184ee63a506155d522f53&2
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 5.1.1; google Pixel 2 Build/LMY47I; wv) AppleWebKit/537.36 (KHTML
user-agent: like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36
accept: text/html
accept: application/xhtml+xml
accept: application/xml;q=0.9
accept: image/webp
accept: image/apng
accept: */*;q=0.8
accept: application/signed-exchange;v=b3
accept-encoding: gzip
accept-encoding: deflate
accept-language: es-ES
accept-language: es;q=0.9
accept-language: en-US;q=0.8
accept-language: en;q=0.7

i need to send them with orderectdict like these(with only ) requests goes good problem is thath server is http 2.0 so i need pseudo headers so i need hyper

:method: GET
:authority: server.url.com
:scheme: https
:path: /login.php?mobile&sid=2ea530a62cb63af6c14be116b7df86ad85cd77c9a11aa3c881b3a460e6c14fbd1fd8b79bd66c9782073705cdff25e890e65b5aeb852fde24c2d54a6e4ee49890&2
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Linux; Android 5.1.1; google Pixel 2 Build/LMY47I; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate
accept-language: es-ES,es;q=0.9,en-US;q=0.8,en;q=0.7

[VeNoMouS]

Yea im seeing something sort of simular wit the accept-range and age

cache-control:           max-age=0, must-revalidate, no-cache, no-store
content-type:            application/json
expires:                 Sat, 28 Jul 2018 10:26:42 GMT
pragma:                  no-cache
set-cookie:              PHPSESSID=45511e9a5f378584c2d90e5494251808; expires=Sun, 28-Jul-2019 11:26:42 GMT; Max-Age=3600; path=/; domain=xxxxxxxx.com; secure; HttpOnly
set-cookie:              X-Magento-Vary=a74fedc5efc1d1867a421578922f820cb12e0565; expires=Sun, 28-Jul-2019 11:26:42 GMT; Max-Age=3600; path=/; secure; HttpOnly
x-content-type-options:  nosniff
x-debug-info:            eyJyZXRyaWVzIjowfQ==
x-frame-options:         SAMEORIGIN
x-platform-server:       i-087261e92df9f2aed
x-request-id:            hr74djxfx23r74mjhlizn6df
x-xss-protection:        1; mode=block
accept-ranges:           bytes
age:                     0
accept-ranges:           bytes
age:                     0
accept-ranges:           bytes
age:                     0
accept-ranges:           bytes
date:                    Sun, 28 Jul 2019 10:26:43 GMT
age:                     0
set-cookie:              _pxhd=518bb81822ce9d7eb4bcf241dbe060d923c5abf748f6f70a5fedc5ad70900f52:315a59c0-b122-11e9-b5de-590cfe45bdb4; Expires=Fri, 01 Jan 2021 00:00:00 GMT; path=/;
x-served-by:             cache-lhr6326-LHR, cache-akl1426-AKL
x-cache:                 MISS, MISS
x-cache-hits:            0, 0
vary:                    Accept-Encoding
content-length:          79

[qwerty32123]

dunno why sends headers in very very strange way, for each "," in headers dict it sends in new value insttead of in the same, and my requests gets rejected cause of these, if sends all in same field like normal requests would be much better

[VeNoMouS]

@GnubiBORED not to disrespect this project, but i've moved onto httpx it seems more active and it aligns with requests model which is what i like and prefer, while alpha, i think it'll go places imho

[qwerty32123]

@VeNoMouS i can speak with you somewhere? telegram? discord?

btw does httpx has support for proxy ? i cannot find it in their docs

[VeNoMouS]

@GnubiBORED it does not as of yet, but their dev's are responding to tickets, my discord is VeNoMouSNZ#5979

[zb14]

I had the same problem.
`

user-agent: | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML
user-agent: | like Gecko) Chrome/76.0.3809.100 Safari/537.36
accept-encoding: | gzip
accept-encoding: | deflate
accept-encoding: | br

`

[qwerty32123]

I had the same problem.
`

user-agent: | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML
user-agent: | like Gecko) Chrome/76.0.3809.100 Safari/537.36
accept-encoding: | gzip
accept-encoding: | deflate
accept-encoding: | br

`

move to httpx but it dosnt have proxy support

[zb14]

I had the same problem.
user-agent: | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML user-agent: | like Gecko) Chrome/76.0.3809.100 Safari/537.36 accept-encoding: | gzip accept-encoding: | deflate accept-encoding: | br

move to httpx but it dosnt have proxy support

hyper version 0.7.0 is ok, but it also doesn't have proxy support

[sethmlarson]

Just a BTW since this thread mentions proxy support for HTTPX a lot: We're working on proxy support currently. :) Would love to have some users to test it once PRs are made, can I cc everyone in this thread when that happens?

[VeNoMouS]

@sethmlarson whats a rough eta of like proxies and ssl context's being implemented into the current release? we all have tickets... but we dont know the time schedule?

I've been waiting 3 weeks for ssl contexts for example.

[sethmlarson]

Yeah the timing is unfortunate because the last two weeks I've changed job, moved to a new apartment, and traveled out of town. Not a lot of time for open source!

@tomchristie has also been on vacation for two weeks so larger features don't have reviewers and we haven't had a release recently. I've done a good amount of work on HTTP tunnel proxies. I can get the SSLContext change in quick if that is an issue for you.

Just keep in mind that the more contributors we have the faster features can be implemented. :)

[qwerty32123]

Just a BTW since this thread mentions proxy support for HTTPX a lot: We're working on proxy support currently. :) Would love to have some users to test it once PRs are made, can I cc everyone in this thread when that happens?

when it happens you can for sure send me or send me a message in telegram if you want(@gnubie) im more active there anyway, i would like to test i have a big project that needs http/2 so yes :)

[sethmlarson]

BTW the verify=SSLContext() interface for SSLConfig is functional in master.

[qwerty32123]

BTW the verify=SSLContext() interface for SSLConfig is functional in master.

In what way you do it? Im currently managing tls and ssl like these with requests, but these will not work in httpx i think

from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager
import ssl


class MyAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       ssl_version=ssl.PROTOCOL_TLSv1_2)`

then i just mount these into request session

session = requests.session()

session.mount('https://', MyAdapter())

[sethmlarson]

Actually in master TLSv1.2 and TLSv1.3 are already the only available protocols. :)

But let's say you only want ECDHE+AESGCM for ciphers you can do:

import httpx, ssl

ctx = ssl.SSLContext()
ctx.set_ciphers("ECDHE+AESGCM")
client = httpx.Client(verify=ctx)

...

[qwerty32123]

i just need 1.2 so its fine.
thanks for fast reply :)

[sethmlarson]

Just an FYI to everyone in this thread, HTTPX now ships with HTTP proxy support. :)

[qwerty32123]

Just an FYI to everyone in this thread, HTTPX now ships with HTTP proxy support. :)

Its already testeable? or will come up in some days

[sethmlarson]

HTTPX 0.7.3 supports the proxies parameter and HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY environment variables. It should be testable so any defects you find should be reported as bugs. :)

See documentation for more info: https://encode.io/httpx/advanced

[txtsd]

This is happening at this line

hyper/hyper/common/headers.py

Line 252 in 18b629b

for sub_val in v.split(b','):