Accessing user job log on the working node during execution

[IgorPelevanyuk]

Hello,

I noticed that if I connect to the worknode where DIRACPilot is executing user jobs it is possible to get std.out and std.err but not the log of shell script that was executed.
For example, we set Executable and configure logFile name:
J.setExecutable(script_name, arguments=' '.join(script_arguments), logFile="UserScript.log")

when I connect to the worknode I can access pilot.out, JOBID/std.out .
But I could not find place where is UserScript.log . When the job is finished this file is uploaded with OutputSandbox.

So, the question: is there a place where I can read logs during jobs execution?

[fstagni]

Well it should definitely be in a sub-directory on the WN. How is your application writing it? Does it buffer it before writing it on the output file?

[IgorPelevanyuk]

Exactly, it looks like it buffers it. But I do nothing for that, except adding logFile="UserScript.log" option during setExecutable. Executable is standard bash script.

[FA-VS]

From testing jobs locally, it seems like, by default, the log is only written after the executable ends. Or else it is hidden somewhere else while the executable is still running.

[IgorPelevanyuk]

I can see only one reason for that: to reduce disk IO from several simultaneously running jobs. But it would be useful to know exectly the machanism behind that behavior.