Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster API: Improve feedback when things go wrong #497

Open
minrk opened this issue Jun 23, 2021 · 1 comment
Open

Cluster API: Improve feedback when things go wrong #497

minrk opened this issue Jun 23, 2021 · 1 comment

Comments

@minrk
Copy link
Member

minrk commented Jun 23, 2021

The Cluster API traps stdout/stderr of controller and engines because it's meant to be used interactively. But that's where information about why connection failures occurred is going to be, and it's frustrating to need to start a whole new cluster with log_level=logging.DEBUG to get a run with output you can see.

It would be good to be able to get this info after a process died that you didn't expect to. The answer to this is probably to pipe process output to a file and keep track of that file and provide an API to retrieve it.

@minrk
Copy link
Member Author

minrk commented Jul 2, 2021

The SSH launchers are piping to a file and have a get_output() method. This seems to be working well, and we can promote it to the base lanucher and e.g. show output on failure.

Thing I just learned that I thought would be trickier:

f = open("outputfile.txt", "wb")
p = Popen(["command"], stdout=f.fileno())
sys.exit()

works and the subprocess continues to produce output to the file even after the parent exits. That means we can have a very simple implementation of getting output that writes to a file and just read that file when an engine dies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant