Uploading workflows during workflow tutorial has issues when running locally.

[kannon92]

Describe the bug

When following https://github.com/flyteorg/flyte-conference-talks/tree/main/pydata-global-2022 for the local setup, I am getting grpc connection refused when uploading workflows.

I am able to run the example workflow using python but I can't upload packages to my local flyte server.

(flyte) Kevins-MBP:pydata-global-2022 kevinhannon$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v0
 --------------------------------------------------------------------- -------- ------------------------------------------------------- 
| NAME                                                                | STATUS | ADDITIONAL INFO                                       |
 --------------------------------------------------------------------- -------- ------------------------------------------------------- 
| /tmp/register3914406486/00_workflows.example_00_intro.get_data_1.pb | Failed | Error registering file due to rpc error: code =       |
|                                                                     |        | Unavailable desc = connection error: desc =           |
|                                                                     |        | "transport: Error while dialing dial tcp [::1]:30081: |
|                                                                     |        | connect: connection refused"                          |
 --------------------------------------------------------------------- -------- ------------------------------------------------------- 
1 rows
Error: Connection Info: [Endpoint: localhost:30081, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp [::1]:30081: connect: connection refused"
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: localhost:30081, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [::1]:30081: connect: connection refused\"","ts":"2022-12-03T11:02:04-05:00"}

Expected behavior

I expect that my workflows are uploaded and I can view them in the UI.

Additional context to reproduce

uname -a
Darwin Kevins-MBP 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:47:26 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T8101 arm64

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[cosmicBboy]

Hi @kannon92 just a quick sanity check, after running flytectl demo start you should see output like this:

👨‍💻 Flyte is ready! Flyte UI is available at http://localhost:30080/console 🚀 🚀 🎉
Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable
export KUBECONFIG=$KUBECONFIG:/Users/<username>/.kube/config:/Users/<username>/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/Users/<username>/.flyte/config-sandbox.yaml

Did you export the KUBECONFIG and FLYTECTL_CONFIG env vars in your shell?

[kannon92]

I'll rerun it and see. I probably missing that.

[cosmicBboy]

hey @kannon92 just a friendly ping: are you still seeing this issue?

[kannon92]

Thank you for reaching out! I’ll aim to reproduce first thing next week when I’m back from vacation.

[kannon92]

Looking into this now.

I decided to follow https://github.com/flyteorg/flyte-conference-talks/tree/main/pydata-global-2022 directions for the localsetup.

I am getting stuck on starting the sandbox cluster:

(pydata-global-2022) Kevins-MBP:pydata-global-2022 kevinhannon$ flytectl demo start --source .
🧑‍🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
delete existing sandbox cluster [y/n]: y
🐋 Going to use Flyte v1.2.1 release with image cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-bfa1dd4e6057b6fc16272579d61df7b1832b96a7 
🐋 pulling docker image for release cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-bfa1dd4e6057b6fc16272579d61df7b1832b96a7
🧑‍🏭 booting Flyte-sandbox container
Waiting for cluster to come up...

Looking at the logs for docker it seems to not like my environment at all.

The sandbox container has a lot of errors because it can't find the correct information in /etc

I get the following errors:

E0103 19:06:51.555929      63 controller.go:166] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocate IP 10.43.0.1: cannot allocate resources of type serviceipallocations at this time
E0103 19:07:03.314252      63 node.go:152] Failed to retrieve node info: nodes "7b783006f67e" not found
W0103 19:07:03.323181      63 sysinfo.go:203] Nodes topology is not available, providing CPU topology
E0103 19:07:03.324803      63 info.go:114] Failed to get system UUID: open /etc/machine-id: no such file or directory

I am using Docker Desktop on Mac M1. I'm not sure if this was the problem I got a month ago.

One interesting thing to note is that when the sandbox gets into this state, it hangs forever.

[kannon92]

Offline, someone pointed out that I should try flytectl demo start. That got me past the last error.

And then I have to apply export FLYTECTL_CONFIG to the value that is outputted when the sandbox is started. Once I did that I am able to upload workflows.

[cosmicBboy]

great, converting this to a discussion!