[Bug]: NO_PROXY does not seem to be respected #535

zprebosnyak-lm · 2024-12-16T17:36:43Z

What steps did you take and what happened:

New Witness version 0.7.0 seems to not respect NO_PROXY value. Witness 0.6.0 works as intended with the same options. When running witness run ... the HTTPS_PROXY is read in but the NO_PROXY does not seem to be. If I am missing proxy settings please let me know, I checked the --help output as well as this doc but do not see anything about proxies.

Pulled down the witness source code and debugging shows the proxy trying to be used for the internal Fulcio instance. NO_PROXY is set to .mycompany.com

WARNING: [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "<IP>:443", ServerName: "fulcio.url.mycompany.com", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup <proxyURL>: i/o timeout"
INFO    Failed creating signing certificate          
ERROR   failed to create fulcio signer: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded" 
ERROR   failed to load signers: failed to load any signers

Removing the proxy environment variables results in below output. The external time stamp server requires the proxy

$WITNESS_RUN_CLI_OPTS is set to:

-s build -a environment -o attestation.json \
--signer-fulcio-url fulcio.url.mycompany.com \
--signer-fulcio-oidc-client-id <clientID> \
--signer-fulcio-oidc-issuer <authURL> \
--timestamp-servers <timeStampServerURL>"

witness -l debug run $WITNESS_RUN_CLI_OPTS --signer-fulcio-token $IDENTITY_TOKEN -- echo test > test.txt
DEBUG   .witness.yaml does not exist, using command line arguments 
INFO    Starting prematerial attestors stage...      
INFO    Starting environment attestor...             
INFO    Finished environment attestor... (3.8208e-05s) 
INFO    Completed prematerial attestors stage...     
INFO    Starting material attestors stage...         
INFO    Starting material attestor...                
INFO    Finished material attestor... (0.016600833s) 
INFO    Completed material attestors stage...        
INFO    Starting execute attestors stage...          
INFO    Starting command-run attestor...             
INFO    Finished command-run attestor... (0.004563583s) 
INFO    Completed execute attestors stage...         
INFO    Starting product attestors stage...          
INFO    Starting product attestor...                 
INFO    Finished product attestor... (0.014448292s)  
INFO    Completed product attestors stage...         
INFO    Starting postproduct attestors stage...      
INFO    Completed postproduct attestors stage...     
ERROR   failed to sign collection: Post "<timeStampServerURL>": dial tcp <IP>:443: i/o timeout

What did you expect to happen:

Witness run to use the proxy for external time stamp server and not use it for internal fulcio instance

witness v0.6.0 output

witness -l debug run $WITNESS_RUN_CLI_OPTS --signer-fulcio-token $IDENTITY_TOKEN -- echo test > test.txt
DEBUG   .witness.yaml does not exist, using command line arguments 
INFO    Starting prematerial attestors stage...      
INFO    Starting environment attestor...             
INFO    Finished environment attestor... (0.000871083s) 
INFO    Completed prematerial attestors stage...     
INFO    Starting material attestors stage...         
INFO    Starting material attestor...                
INFO    Finished material attestor... (0.033799083s) 
INFO    Completed material attestors stage...        
INFO    Starting execute attestors stage...          
INFO    Starting command-run attestor...             
INFO    Finished command-run attestor... (0.007654666s) 
INFO    Completed execute attestors stage...         
INFO    Starting product attestors stage...          
INFO    Starting product attestor...                 
INFO    Finished product attestor... (0.015729625s)  
INFO    Completed product attestors stage...         
INFO    Starting postproduct attestors stage...      
INFO    Completed postproduct attestors stage...

Environment:

Witness version: 0.7.0
Architecture: darwin-arm64 and linux-amd64
Attestors used: environment locally and in pipeline git,gitlab
Archivista version:

The text was updated successfully, but these errors were encountered:

mikhailswift · 2024-12-16T18:24:18Z

We rely on golang's http client to handle any proxy settings for us -- Witness itself doesn't do anything special regarding HTTPS_PROXY or NO_PROXY, see here in the docs describing DefaultTransport.

I don't see any changes to how we instantiate HTTP clients in the diffs between 0.7.0 and 0.6.0, so I wouldn't expect any changes as to how Witness regards proxy variables.

I'll try this out locally and see if I can reproduce.

jkjell · 2024-12-16T20:40:18Z

@zprebosnyak-lm I think this will solve the issue but, I don't have a quick way to test it. In the last release, we moved from a deprecated GRPC function to a new one and probably got the connection options wrong. in-toto/go-witness#402

If you're comfortable pulling this down and testing the code, that would be great! If not, I can create a "beta" release with the binary to pull down directly.

zprebosnyak-lm · 2024-12-16T23:35:34Z

@jkjell Happy to help test! Unfortunately I am not familiar with golang. If you could provide the commands to build go-witness from the source code or create the "beta" release binary that'd be great

ChaosInTheCRD · 2024-12-17T12:26:57Z

Hi @zprebosnyak-lm! If you are familiar with git, you can do the following to run witness from a branch with go:

Install Go (brew, apt or whichever)
Run git clone https://github.com/in-toto/witness (if you haven't already)
Run git clone https://github.com/in-toto/go-witness (this is our client library repository, which is where @jkjell's changes are).
cd go-witness
git fetch
git checkout grpc_new_client_opts
cd ../witness
Add a replace line to the top of the go.mod file in the root of the witness repository like so:

module github.com/in-toto/witness

go 1.23.2

+ replace github.com/in-toto/go-witness => ../go-witness // <- this is the relative path of `go-witness` repository

go mod tidy
You will now be using witness with the version of go-witness with @jkjell's changes.
go run . will compile witness from source and run it on each invocation. You should get the witness help menu printed into your console when you run it.
go run . run... (to use witness run)
go run . verify... (to use witness verify)
Reply here if there are any issues!

Thanks for the help @zprebosnyak-lm 🙏

zprebosnyak-lm · 2024-12-17T17:27:55Z

@ChaosInTheCRD thank you for that!

So I cloned fresh copies of both repos, switched to the go-witness grpc_new_client_opts branch, and in the witness repo my go.mod looks like this now

module github.com/in-toto/witness

go 1.23.2

replace github.com/in-toto/go-witness => ../go-witness

require (
	github.com/gobwas/glob v0.2.3
	github.com/in-toto/go-witness v0.7.0
	...

ran go mod tidy
did a test with the witness 0.6.0 binary which worked, but the update yielded the same result. Added two debugging vars I found that helped show the proxy issue prior. Here is the full output

export GRPC_GO_LOG_VERBOSITY_LEVEL=99
export GRPC_GO_LOG_SEVERITY_LEVEL=info

go run . run -l debug run $WITNESS_RUN_CLI_OPTS --signer-fulcio-token $IDENTITY_TOKEN -- $WITNESS_RUN_TARGET_COMMANDS

DEBUG   .witness.yaml does not exist, using command line arguments 
2024/12/17 10:04:46 INFO: [core] original dial target is: "fulcio.url.mycompany.com:443"
2024/12/17 10:04:46 INFO: [core] [Channel #1]Channel created
2024/12/17 10:04:46 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"dns", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/fulcio.url.mycompany.com:443", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2024/12/17 10:04:46 INFO: [core] [Channel #1]Channel authority set to "fulcio.url.mycompany.com:443"
2024/12/17 10:04:46 INFO: [core] [Channel #1]Channel exiting idle mode
2024/12/17 10:04:46 INFO: [core] [Channel #1]Resolver state updated: {
  "Addresses": [
    {
      "Addr": "<ip>:443",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    },
    {
      "Addr": "<ip>:443",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    },
    {
      "Addr": "<ip>:443",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "<ip>:443",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "<ip>:443",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "<ip>:443",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2024/12/17 10:04:46 INFO: [core] [Channel #1]Channel switches to new LB policy "pick_first"
2024/12/17 10:04:46 INFO: [pick-first-lb] [pick-first-lb 0x140006ba7e0] Received new config {
  "shuffleAddressList": false
}, resolver state {
  "Addresses": [
    {
      "Addr": "<ip>:443",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    },
    {
      "Addr": "<ip>:443",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    },
    {
      "Addr": "<ip>:443",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "<ip>:443",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "<ip>:443",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    },
    {
      "Addresses": [
        {
          "Addr": "<ip>:443",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
}
2024/12/17 10:04:46 INFO: [core] [Channel #1 SubChannel #2]Subchannel created
2024/12/17 10:04:46 INFO: [core] [Channel #1]Channel Connectivity change to CONNECTING
2024/12/17 10:04:46 INFO: [core] [Channel #1 SubChannel #2]Subchannel Connectivity change to CONNECTING
2024/12/17 10:04:46 INFO: [core] [Channel #1 SubChannel #2]Subchannel picks a new address "<ip>:443" to connect
2024/12/17 10:04:46 INFO: [pick-first-lb] [pick-first-lb 0x140006ba7e0] Received SubConn state update: 0x140006ba870, {ConnectivityState:CONNECTING ConnectionError:<nil> connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}
2024/12/17 10:05:06 INFO: [transport] Aborting due to connect deadline expiring: context deadline exceeded
2024/12/17 10:05:06 INFO: [core] Creating new client transport to "{Addr: \"<ip>:443\", ServerName: \"fulcio.url.mycompany.com:443\", }": connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
2024/12/17 10:05:06 WARNING: [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "<ip>:443", ServerName: "fulcio.url.mycompany.com:443", }. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
2024/12/17 10:05:06 INFO: [core] [Channel #1 SubChannel #2]Subchannel picks a new address "<ip>:443" to connect
2024/12/17 10:05:06 INFO: [core] Creating new client transport to "{Addr: \"<ip>:443\", ServerName: \"fulcio.url.mycompany.com:443\", }": connection error: desc = "transport: Error while dialing: dial tcp: lookup <proxyURL>: i/o timeout"
2024/12/17 10:05:06 WARNING: [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "<ip>:443", ServerName: "fulcio.url.mycompany.com:443", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup <proxyURL>: i/o timeout"
2024/12/17 10:05:06 INFO: [core] [Channel #1 SubChannel #2]Subchannel picks a new address "<ip>:443" to connect
2024/12/17 10:05:06 INFO: [core] Creating new client transport to "{Addr: \"<ip>:443\", ServerName: \"fulcio.url.mycompany.com:443\", }": connection error: desc = "transport: Error while dialing: dial tcp: lookup <proxyURL>: i/o timeout"
2024/12/17 10:05:06 WARNING: [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "<ip>:443", ServerName: "fulcio.url.mycompany.com:443", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup <proxyURL>: i/o timeout"
2024/12/17 10:05:06 INFO: [core] [Channel #1 SubChannel #2]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
2024/12/17 10:05:06 INFO: [pick-first-lb] [pick-first-lb 0x140006ba7e0] Received SubConn state update: 0x140006ba870, {ConnectivityState:TRANSIENT_FAILURE ConnectionError:connection error: desc = "transport: authentication handshake failed: context deadline exceeded" connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}
2024/12/17 10:05:06 INFO: [core] [Channel #1]Channel Connectivity change to TRANSIENT_FAILURE
INFO    Failed creating signing certificate          
ERROR   failed to create fulcio signer: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded" 
ERROR   failed to load signers: failed to load any signers

~~Also unset the proxies and am now not getting as far as the 0.7.0 binary did with them unset (failing when connecting to the time stamp server)~~

unset HTTPS_PROXY
unset HTTP_PROXY
unset NO_PROXY
go run . run -l debug run $WITNESS_RUN_CLI_OPTS --signer-fulcio-token $IDENTITY_TOKEN -- $WITNESS_RUN_TARGET_COMMANDS 
DEBUG   .witness.yaml does not exist, using command line arguments 
INFO    Failed creating signing certificate          
ERROR   failed to create fulcio signer: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: context deadline exceeded" 
ERROR   failed to load signers: failed to load any signers

Re ran witness 0.6.0 binary to verify token had not expired and it still was working

zprebosnyak-lm · 2024-12-17T19:32:04Z

Ignore the last part about not getting as far with proxies unset I realized I forgot to unset lower case http/s/no _proxy vars. Output with no proxies is actually

go run . run -l debug $WITNESS_RUN_CLI_OPTS --signer-fulcio-token $IDENTITY_TOKEN -- $WITNESS_RUN_TARGET_COMMANDS
DEBUG   .witness.yaml does not exist, using command line arguments 
INFO    Starting prematerial attestors stage...      
INFO    Starting environment attestor...             
INFO    Finished environment attestor... (0.000134583s) 
INFO    Completed prematerial attestors stage...     
INFO    Starting material attestors stage...         
INFO    Starting material attestor...                
INFO    Finished material attestor... (0.059156625s) 
INFO    Completed material attestors stage...        
INFO    Starting execute attestors stage...          
INFO    Starting command-run attestor...             
INFO    Finished command-run attestor... (0.009220458s) 
INFO    Completed execute attestors stage...         
INFO    Starting product attestors stage...          
INFO    Starting product attestor...                 
INFO    Finished product attestor... (0.034915667s)  
INFO    Completed product attestors stage...         
INFO    Starting postproduct attestors stage...      
INFO    Completed postproduct attestors stage...     
ERROR   failed to sign collection: Post "<timeStampServerURL>": dial tcp <ip>:443: i/o timeout 
exit status 1

zprebosnyak-lm added the bug Something isn't working label Dec 16, 2024

jkjell linked a pull request Dec 16, 2024 that will close this issue

bug(fulcio): Fixes Witness #535 in-toto/go-witness#402

Draft

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: NO_PROXY does not seem to be respected #535

[Bug]: NO_PROXY does not seem to be respected #535

zprebosnyak-lm commented Dec 16, 2024

mikhailswift commented Dec 16, 2024

jkjell commented Dec 16, 2024 •

edited

Loading

zprebosnyak-lm commented Dec 16, 2024

ChaosInTheCRD commented Dec 17, 2024

zprebosnyak-lm commented Dec 17, 2024 •

edited

Loading

zprebosnyak-lm commented Dec 17, 2024 •

edited

Loading

[Bug]: NO_PROXY does not seem to be respected #535

[Bug]: NO_PROXY does not seem to be respected #535

Comments

zprebosnyak-lm commented Dec 16, 2024

mikhailswift commented Dec 16, 2024

jkjell commented Dec 16, 2024 • edited Loading

zprebosnyak-lm commented Dec 16, 2024

ChaosInTheCRD commented Dec 17, 2024

zprebosnyak-lm commented Dec 17, 2024 • edited Loading

zprebosnyak-lm commented Dec 17, 2024 • edited Loading

jkjell commented Dec 16, 2024 •

edited

Loading

zprebosnyak-lm commented Dec 17, 2024 •

edited

Loading

zprebosnyak-lm commented Dec 17, 2024 •

edited

Loading