ftl dev randomly hangs

[mistermoe]

I'm not sure how to reproduce this one consistently but it happens frequently. project i'm working in has 2 modules

ftlbug2.mov

[wesbillman]

Having trouble reproducing this one. If I add/remove the log files it seems to consistently reload as expected. 🤔

[matt2e]

I think I saw this once or twice but it was when I was neck deep in other changes in this area so I'd blamed my half finished code. I'll have a go at repro-ing as well

[matt2e]

Not sure if this is the full steps or the exact same issue but I did the following:

• ftl dev with the module
• Edit a modules code, save, edit, save, edit, save etc in quick succession, not waiting for build/deploy to finish

What happens:

• I get an error building at some point (may or not be related, and really there wasn't anything bad in what I saved)
• deploys stop finishing. I get "deploying module" and "Building module" messages, but no more "deployed module" logs
• I do still get more of those messages though, unlike how @mistermoe's logs seem to stop

**info:controller
...

error: build and deploy failed for module "echo": failed to build module "echo": exit status 1: failed to build module"echo": exit status 1
debug: Extracting dependencies for module "echo"
info:echo: Building module
...
debug:echo: Compiling
debug:echo: # main
debug:echo: ./main.go:16:26: undefined: echo.Echo7
info:echo: # main
./main.go:16:26: undefined: echo.Echo7

**error: build and deploy failed for module "echo": failed to build module "echo": exit status 1: failed to build module "echo": exit status 1
debug: Extracting dependencies for module "echo"
info:echo: Building module
...
info:echo: Deploying module
...
info:controller0: Stopped echo-0ddc5b5072
...
info:controller0: Deployed echo-0ca0eee264
debug: Extracting dependencies for module "echo"
debug:controller0: Adding 1 replicas
info:echo: Building module
..
info:echo: Deploying module
debug:echo: Uploading 0/1 files
debug:controller0: Created deployment echo-0ca0eee264
debug:controller0: Replace deployment for: echo-0ca0eee264
debug:controller0: Deployment already exists: echo-0ca0eee264
debug:echo: Waiting for deployment echo-0ca0eee264 to become ready
debug: Extracting dependencies for module "echo"
info:echo: Building module
debug:echo: Generating external modules
debug:echo: Extracting schema
debug:echo: Generating main module
debug:echo: Tidying go.mod files
debug:echo: go: found github.com/TBD54566975/ftl/go-runtime/ftl in github.com/TBD54566975/ftl v0.0.0-00010101000000-000000000000
debug:echo: go: found github.com/TBD54566975/ftl/backend/protos/xyz/block/ftl/v1/ftlv1connect in github.com/TBD54566975/ftl v0.0.0-00010101000000-000000000000
debug:echo: go: found github.com/TBD54566975/ftl/common/plugin in github.com/TBD54566975/ftl v0.0.0-00010101000000-000000000000
debug:echo: go: found github.com/TBD54566975/ftl/go-runtime/server in github.com/TBD54566975/ftl v0.0.0-00010101000000-000000000000
debug:echo: Compiling
info:echo: Deploying module
debug:echo: Uploading 0/1 files
debug:controller0: Created deployment echo-0ca0eee264
debug:controller0: Replace deployment for: echo-0ca0eee264
debug:controller0: Deployment already exists: echo-0ca0eee264
debug:echo: Waiting for deployment echo-0ca0eee264 to become ready

[matt2e]

Will look into why this is happening

[alecthomas]

Looks like you're running at debug, you should probably run at trace to get more information.

[matt2e]

Above is a false alarm sorry, looks like it was working as correctly.
When a few changes for a module get queued up, the queue builds the latest it finds on disk, and then subsequent builds find no changes which is why we get the log "Deployment already exists" and no info log saying "deployed module". Subsequent changes do make it up.
So what I repro'd is different than what @mistermoe saw.
Still trying to repro

[alecthomas]

Ah right. That is another bug though - it should exclude repeated builds of the same module. Something like "if the build is more recent than the timestamp of the event, do nothing".

[matt2e]

Wasn't able to repro this issue. Anyone else want to have a try?
Moe's steps leading up to this issue while working on the ledger module:

I created a module rates
ran just dev which led to an infinite build loop
killed ^ and restarted. Things worked fine
added a single verb to rates purely for the sake of calling it from ledger module
Moved back over to implement ftl.Call in ledger module. Finished and hit save
infinite build loop started again. No errors in code
killed and restarted just dev
Made one adjustment to add a log statement
Ran my curl request didn’t see the log I excepted then realized that just dev wasn’t rebuilding

[matt2e]

Other issue found has been moved here: #1210

[wesbillman]

Hey @mistermoe!! I haven't seen this in a while. Are you seeing this with the current versions? I have a feeling it was resolved with some of the other engine stuff, but want to confirm that's true with you :)

[mistermoe]

agreed @wesbillman ! going to close this out and re-open if i run into it agane