Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] intermittent race conditions in CI on make interchaintest-events #1335

Open
jtieri opened this issue Nov 15, 2023 · 4 comments
Open

[Bug] intermittent race conditions in CI on make interchaintest-events #1335

jtieri opened this issue Nov 15, 2023 · 4 comments
Labels
T: Test TYPE: Test

Comments

@jtieri
Copy link
Member

jtieri commented Nov 15, 2023

Noticed that there is an intermittent race condition detected in CI when running make interchaintest-events. I have not been able to reproduce this issue locally no matter how many runs I have attempted.

Here are some logs:

==================
WARNING: DATA RACE
Write at 0x00c0018c60c0 by goroutine 1027:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:203 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).SetBech32PrefixForAccount()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:89 +0x13d
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SetSDKContext()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/bech32_hack.go:19 +0x2f
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).buildMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:605 +0xbd
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessagesToMempool()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:175 +0x1cf
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages.func2()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:128 +0x144
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:127 +0x46c
  github.com/cosmos/relayer/v2/relayer.CreateClient.func3()
      /home/runner/work/relayer/relayer/relayer/client.go:218 +0xe3
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer.CreateClient()
      /home/runner/work/relayer/relayer/relayer/client.go:215 +0x1370
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients.func4()
      /home/runner/work/relayer/relayer/relayer/client.go:66 +0x190
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x76

Previous read at 0x00c0018c60c0 by goroutine 23:
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).GetBech32AccountAddrPrefix()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:162 +0x7d
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewDefaultSigningOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:94 +0x15c
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfigWithOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:191 +0x[544](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:545)
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfig()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:79 +0x127
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodecConfig()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:98 +0xe7
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodec()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:71 +0x1c4
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.CosmosProviderConfig.NewProvider()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/provider.go:123 +0x31a
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProviderConfig).NewProvider()
      <autogenerated>:1 +0x108
  github.com/cosmos/relayer/v2/cmd.(*ConfigInputWrapper).RuntimeConfig()
      /home/runner/work/relayer/relayer/cmd/config.go:337 +0x305
  github.com/cosmos/relayer/v2/cmd.(*appState).loadConfigFile()
      /home/runner/work/relayer/relayer/cmd/appstate.go:84 +0x391
  github.com/cosmos/relayer/v2/cmd.NewRootCmd.func1()
      /home/runner/work/relayer/relayer/cmd/root.go:77 +0x64
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:954 +0xd25
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x3d1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).RestoreKey()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:106 +0x25
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).configureRelayerKeys()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:431 +0x54f
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).Build()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:263 +0xaa4
  github.com/strangelove-ventures/interchaintest/v8.StartChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/test_setup.go:73 +0x817
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:331 +0x891
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1.2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:257 +0x259
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 1027 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x124
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients()
      /home/runner/work/relayer/relayer/relayer/client.go:63 +0x9d4
  github.com/cosmos/relayer/v2/cmd.createClientsCmd.func1()
      /home/runner/work/relayer/relayer/cmd/tx.go:102 +0x61c
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x2f1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).CreateClients()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:240 +0x2d
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestRelayerSetup.func2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/relayersetup.go:82 +0x20a
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 23 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x82a
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:247 +0x31d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c0018c2988 by goroutine 1027:
  github.com/cosmos/cosmos-sdk/types.(*Config).SetBech32PrefixForAccount()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:89 +0x14c
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SetSDKContext()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/bech32_hack.go:19 +0x2f
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).buildMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:605 +0xbd
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessagesToMempool()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:175 +0x1cf
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages.func2()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:128 +0x144
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:127 +0x46c
  github.com/cosmos/relayer/v2/relayer.CreateClient.func3()
      /home/runner/work/relayer/relayer/relayer/client.go:218 +0xe3
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer.CreateClient()
      /home/runner/work/relayer/relayer/relayer/client.go:215 +0x1370
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients.func4()
      /home/runner/work/relayer/relayer/relayer/client.go:66 +0x190
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x76

Previous read at 0x00c0018c2988 by goroutine 23:
  github.com/cosmos/cosmos-sdk/types.(*Config).GetBech32AccountAddrPrefix()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:162 +0x87
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewDefaultSigningOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:94 +0x15c
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfigWithOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:191 +0x544
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfig()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:79 +0x127
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodecConfig()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:98 +0xe7
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodec()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:71 +0x1c4
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.CosmosProviderConfig.NewProvider()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/provider.go:123 +0x31a
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProviderConfig).NewProvider()
      <autogenerated>:1 +0x108
  github.com/cosmos/relayer/v2/cmd.(*ConfigInputWrapper).RuntimeConfig()
      /home/runner/work/relayer/relayer/cmd/config.go:337 +0x305
  github.com/cosmos/relayer/v2/cmd.(*appState).loadConfigFile()
      /home/runner/work/relayer/relayer/cmd/appstate.go:84 +0x391
  github.com/cosmos/relayer/v2/cmd.NewRootCmd.func1()
      /home/runner/work/relayer/relayer/cmd/root.go:77 +0x64
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:954 +0xd25
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x3d1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).RestoreKey()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:106 +0x25
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).configureRelayerKeys()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:431 +0x54f
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).Build()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/interchain.go:263 +0xaa4
  github.com/strangelove-ventures/interchaintest/v8.StartChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/test_setup.go:73 +0x817
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:331 +0x891
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1.2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:257 +0x259
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 1027 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x124
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients()
      /home/runner/work/relayer/relayer/relayer/client.go:63 +0x9d4
  github.com/cosmos/relayer/v2/cmd.createClientsCmd.func1()
      /home/runner/work/relayer/relayer/cmd/tx.go:102 +0x61c
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x2f1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).CreateClients()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:240 +0x2d
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestRelayerSetup.func2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/relayersetup.go:82 +0x20a
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 23 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x82a
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:247 +0x31d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Write at 0x00c0018c60c0 by goroutine 1028:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:203 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).SetBech32PrefixForAccount()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:89 +0x13d
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SetSDKContext()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/bech32_hack.go:19 +0x2f
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).buildMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:605 +0xbd
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessagesToMempool()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:175 +0x1cf
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages.func2()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:128 +0x144
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProvider).SendMessages()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/tx.go:127 +0x46c
  github.com/cosmos/relayer/v2/relayer.CreateClient.func3()
      /home/runner/work/relayer/relayer/relayer/client.go:218 +0xe3
  github.com/avast/retry-go/v4.Do.func1()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:115 +0x2e
  github.com/avast/retry-go/v4.DoWithData[go.shape.interface {}]()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:179 +0xb0f
  github.com/avast/retry-go/v4.Do()
      /home/runner/go/pkg/mod/github.com/avast/retry-go/[email protected]/retry.go:118 +0x6b
  github.com/cosmos/relayer/v2/relayer.CreateClient()
      /home/runner/work/relayer/relayer/relayer/client.go:215 +0x1370
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients.func5()
      /home/runner/work/relayer/relayer/relayer/client.go:76 +0x190
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x76

Previous read at 0x00c0018c60c0 by goroutine 23:
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.21.3/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/cosmos/cosmos-sdk/types.(*Config).GetBech32AccountAddrPrefix()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/types/config.go:162 +0x7d
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewDefaultSigningOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:94 +0x15c
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfigWithOptions()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:191 +0x544
  github.com/cosmos/cosmos-sdk/x/auth/tx.NewTxConfig()
      /home/runner/go/pkg/mod/github.com/cosmos/[email protected]/x/auth/tx/config.go:79 +0x127
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodecConfig()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:98 +0xe7
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.MakeCodec()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/codec.go:71 +0x1c4
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.CosmosProviderConfig.NewProvider()
      /home/runner/work/relayer/relayer/relayer/chains/cosmos/provider.go:123 +0x31a
  github.com/cosmos/relayer/v2/relayer/chains/cosmos.(*CosmosProviderConfig).NewProvider()
      <autogenerated>:1 +0x108
  github.com/cosmos/relayer/v2/cmd.(*ConfigInputWrapper).RuntimeConfig()
      /home/runner/work/relayer/relayer/cmd/config.go:337 +0x305
  github.com/cosmos/relayer/v2/cmd.(*appState).loadConfigFile()
      /home/runner/work/relayer/relayer/cmd/appstate.go:84 +0x391
  github.com/cosmos/relayer/v2/cmd.(*appState).performConfigLockingOperation()
      /home/runner/work/relayer/relayer/cmd/appstate.go:216 +0x324
  github.com/cosmos/relayer/v2/cmd.pathsNewCmd.func1()
      /home/runner/work/relayer/relayer/cmd/paths.go:246 +0x18d
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x304
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).GeneratePath()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:114 +0x35
  github.com/strangelove-ventures/interchaintest/v8.(*Interchain).Build()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected][592](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:593)933b/interchain.go:281 +0xd84
  github.com/strangelove-ventures/interchaintest/v8.StartChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/test_setup.go:73 +0x817
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestChainPair()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:331 +0x891
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1.2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:257 +0x259
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1[595](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:596) +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 1028 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x124
  github.com/cosmos/relayer/v2/relayer.(*Chain).CreateClients()
      /home/runner/work/relayer/relayer/relayer/client.go:73 +0xcbb
  github.com/cosmos/relayer/v2/cmd.createClientsCmd.func1()
      /home/runner/work/relayer/relayer/cmd/tx.go:102 +0x61c
  github.com/spf13/cobra.(*Command).execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xf2f
  github.com/spf13/cobra.(*Command).ExecuteC()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x604
  github.com/spf13/cobra.(*Command).Execute()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x324
  github.com/spf13/cobra.(*Command).ExecuteContext()
      /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032 +0x31c
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunWithInputC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:83 +0x30e
  github.com/cosmos/relayer/v2/internal/relayertest.(*System).RunC()
      /home/runner/work/relayer/relayer/internal/relayertest/system.go:57 +0x2f1
  github.com/cosmos/relayer/v2/interchaintest.(*Relayer).CreateClients()
      /home/runner/work/relayer/relayer/interchaintest/relayer.go:240 +0x2d
  github.com/strangelove-ventures/interchaintest/v8/conformance.TestRelayerSetup.func2()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/relayersetup.go:82 +0x20a
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44

Goroutine 23 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1[648](https://github.com/cosmos/relayer/actions/runs/6870999378/job/18687001943?pr=1334#step:5:649) +0x82a
  github.com/strangelove-ventures/interchaintest/v8/conformance.Test.func1.1.1()
      /home/runner/go/pkg/mod/github.com/strangelove-ventures/interchaintest/[email protected]/conformance/test.go:247 +0x31d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
==================
    testing.go:1465: race detected during execution of test
@jtieri jtieri added the T: Test TYPE: Test label Nov 15, 2023
@boojamya
Copy link
Contributor

We should look into if this is still needed for our feegrant test:

relayer/relayer/chains/cosmos/bech32_hack.go

Lines 16 to 23 in 7e02760

func (cc *CosmosProvider) SetSDKContext() func() {
sdkConfigMutex.Lock()
sdkConf := sdk.GetConfig()
sdkConf.SetBech32PrefixForAccount(cc.PCfg.AccountPrefix, cc.PCfg.AccountPrefix+"pub")
sdkConf.SetBech32PrefixForValidator(cc.PCfg.AccountPrefix+"valoper", cc.PCfg.AccountPrefix+"valoperpub")
sdkConf.SetBech32PrefixForConsensusNode(cc.PCfg.AccountPrefix+"valcons", cc.PCfg.AccountPrefix+"valconspub")
return sdkConfigMutex.Unlock
}

It seems like this has something to do with it.

@jtieri
Copy link
Member Author

jtieri commented Nov 15, 2023

Good call out, i just dumped the logs and didn't really look into it much since it was end of my day. This method should really not be getting called in many places, it's the source of much pain and suffering. I'm surprised to see that it is used in any of our tests.

Once this is eventually closed out we will be able to kill this once and for all

@akc2267
Copy link
Contributor

akc2267 commented Nov 20, 2023 •
edited
Loading

switch to docker image for e2e tests and use -race during compile in Dockerfile
and then GORACE=halt_on_error=1
at runtime(strangelove-ventures/heighliner#103)

@jtieri
Copy link
Member Author

jtieri commented Sep 12, 2024

Hey @KyleMoser now that this PR in the SDK has been merged, can we remove this code from the test case #1335 (comment)?

It would be nice to fix the race conditions we see from time to time in the e2e test cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
T: Test TYPE: Test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@akc2267 @jtieri @boojamya