Peek StandardOutput with DIRAC API #7106

FA-VS · 2023-07-12T13:24:03Z

FA-VS
Jul 12, 2023

Hi,
I am testing some processing on the grid with the DIRAC API. However, I get no feedback on the progress of the processing until it either completes or fails. Normally you would be able to "peek" at the standard output from the dashboard, but this does not provide any useful information.
I came across the discussion on the old Google groups ( https://groups.google.com/g/diracgrid-forum/c/UJbG0Ff8xp4 ) which suggested that this was not possible in the standard DIRAC API, but a workaround was found in the ILC DIRAC. This was a few years ago. Did some solution ever get implemented for the standard DIRAC API, or do I have to use "dirac-wms-job-submit" (etc) instead of the API while I am still testing the processing?
Thank you for your help,
Francisco

FA-VS · 2023-07-21T10:07:25Z

FA-VS
Jul 21, 2023
Author

Related to the question above, is it possible to use the dirac command line to run jobs locally? I need to test some processing before sending it to the grid, but would like to see the output as it runs, rather than only after it ends/fails. But in the documentation I can only find how to run jobs locally with the API, which doesn't output as it runs...

0 replies

chaen · 2023-07-24T07:39:50Z

chaen
Jul 24, 2023
Maintainer

Hi,

For the first question, maybe @andresailer or @marianne013 are the best to answer.

Regarding the second one, dirac-production-runjoblocal allows to re-run locally a failed job.
Otherwise, LHCb has a more advanced script that allows to run a job locally prior to submission but it is quite LHCb custom https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/blob/master/src/LHCbDIRAC/ProductionManagementSystem/scripts/dirac_production_request_run_local.py
Not sure if there is anything generic though, but it would surely be nice. The first script in vanilla DIRAC can probably be amended to do what you want, if it does not yet exist

6 replies

FA-VS Jul 24, 2023
Author

Thank you for the replies!

I don't have my own DIRAC client installed, I am using the CVMFS container. I have to look into whether I can modify a copy of that. If the code change is that simple, is there any reason / side effect it's not implemented in base Dirac already?

I apologize for the very basic question, but checking the source code, do I just need to run dirac-production-runjoblocal as follows?
$ dirac-production-runjoblocal -D jobIdOfFailedJob

And then dirac will take care of downloading all required files and running the relevant script based on the information in the original failed job? Will this work even if I am not LHCb? I notice one of the methods specifies it was written specifically for them.

andresailer Jul 28, 2023
Collaborator

If the code change is that simple, is there any reason / side effect it's not implemented in base Dirac already?

This code is not what one wants to have, as it prints everything always. At least one probably would make this log level dependent, and only print everything when the job log level is DEBUG or VERBOSE.

FA-VS Aug 30, 2023
Author

@andresailer : it's taken much longer than expected, but I have managed to get an installation where I can make modifications. I have added those lines into (mypath)/lib/python3.11/site-packages/DIRAC/Workflow/Modules/Script.py , but I still do not see any change in the output as the processing advances when I "peek at standard output". I still see the standard API message; trying a simple echo, I see the API feedback, but not the standard output from the script (in this case, the output from running "echo Testing DIRAC API")

Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1', 'ScriptStep2', 'ScriptStep3']
2023-08-30 12:25:44 UTC dirac-jobexec/Script INFO: Command is: echo Testing Dirac API
2023-08-30 12:25:44 UTC dirac-jobexec/Script INFO: echo Testing Dirac API execution completed with status 0
2023-08-30 12:25:44 UTC dirac-jobexec/Script INFO: Output written to Script1_CodeOutput.log, execution complete.
2023-08-30 12:25:44 UTC dirac-jobexec/Script INFO: echo successful
2023-08-30 12:25:44 UTC dirac-jobexec/Script INFO: ===== Terminating <class 'DIRAC.Workflow.Modules.Script.Script'> =====

Do I have to change anything else before "Peek StandardOutput" on the dashboard shows the standard output from the script itself?

andresailer Aug 30, 2023
Collaborator

@FA-VS This change will only apply for local testing of the jobs!

Dirac().submitJob(j,mode='local')
#                         ^^^^^

It should write out everything to the log file in the working directory.

FA-VS Aug 30, 2023
Author

Ah, I missed that part, thank you. It still does not seem to work, the log file is not created until after the executable is finished.

And just to clarify, there is no similar solution for processes running on the grid? In that case, if I want to see the standard output "live", I need to either run Dirac with the CLI, or modify the executable code so it regularly saves/sends the output to some accessible grid storage location?

fstagni · 2023-07-27T06:41:54Z

fstagni
Jul 27, 2023
Maintainer

I don't have my own DIRAC client installed, I am using the CVMFS container. I have to look into whether I can modify a copy of that. If the code change is that simple, is there any reason / side effect it's not implemented in base Dirac already?

Installing a DIRAC client is rather easy, you can check https://dirac.readthedocs.io/en/latest/UserGuide/GettingStarted/InstallingClient/index.html

Anyway, the CVMFS deployed version would do too.

I apologize for the very basic question, but checking the source code, do I just need to run dirac-production-runjoblocal as follows? $ dirac-production-runjoblocal -D jobIdOfFailedJob

And then dirac will take care of downloading all required files and running the relevant script based on the information in the original failed job?

Yes, exactly.

Will this work even if I am not LHCb? I notice one of the methods specifies it was written specifically for them.

Try first with the vanilla DIRAC one. And the LHCb would work even if you are not LHCb.

1 reply

FA-VS Jul 27, 2023
Author

Thank you very much for the continued help!

Installing a DIRAC client is rather easy, you can check https://dirac.readthedocs.io/en/latest/UserGuide/GettingStarted/InstallingClient/index.html

I have followed the steps there, and tried to install Dirac on my laptop. But I get a number of errors at the configuration level

(base) francisco@francisco-LIFEBOOK-U7413:~/dirac_ui$ dirac-configure
Enter Certificate password: *************
Generating proxy...
No CRL files found for /etc/grid-security/certificates/*.r0. Abort check of CAs
Proxy generated:
subject      : /DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]/CN=4667360497
issuer       : /DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]
identity     : /DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]
timeleft     : 23:59:59
path         : /tmp/x509up_u1000

Choose a DIRAC Setup (press tab for suggestions):
DIRAC-Certification
Choose a configuration server URL (leave blank for default):
dips://nl-dirac-wms.grid.surfsara.nl:9135/Configuration/Server
Configuration is:
  * Setup: DIRAC-Certification
  * Configuration server: dips://nl-dirac-wms.grid.surfsara.nl:9135/Configuration/Server

Are you sure you want to continue? y
Executing: /home/francisco/dirac_ui/diracos/bin/dirac-configure 
Checking DIRAC installation at "/home/francisco/dirac_ui/diracos"
Current hash for bundle CAs in dir /home/francisco/dirac_ui/diracos/etc/grid-security/certificates is ''
Could not select DISET or Tornado client RuntimeError('Option /DIRAC/Setups/DIRAC-Certification/Framework is not defined')
Could not sync dir Cannot get URL for Framework/BundleDelivery in setup DIRAC-Certification: RuntimeError('Option /DIRAC/Setups/DIRAC-Certification/Framework is not defined')
Created vomsdir file /home/francisco/dirac_ui/diracos/etc/grid-security/vomsdir/escape/voms-escape.cloud.cnaf.infn.it.lsc
Created vomses file /home/francisco/dirac_ui/diracos/etc/grid-security/vomses/escape
Created vomsdir file /home/francisco/dirac_ui/diracos/etc/grid-security/vomsdir/km3net.org/voms02.scope.unina.it.lsc
Created vomses file /home/francisco/dirac_ui/diracos/etc/grid-security/vomses/km3net.org
Created vomsdir file /home/francisco/dirac_ui/diracos/etc/grid-security/vomsdir/lofar/voms.grid.sara.nl.lsc
Created vomses file /home/francisco/dirac_ui/diracos/etc/grid-security/vomses/lofar
Created vomsdir file /home/francisco/dirac_ui/diracos/etc/grid-security/vomsdir/lofar/voms.grid.sara.nl.lsc
Created vomses file /home/francisco/dirac_ui/diracos/etc/grid-security/vomses/lofar
Created vomsdir file /home/francisco/dirac_ui/diracos/etc/grid-security/vomsdir/projectmine.com/voms.grid.sara.nl.lsc
Created vomses file /home/francisco/dirac_ui/diracos/etc/grid-security/vomses/projectmine.com
Created vomsdir file /home/francisco/dirac_ui/diracos/etc/grid-security/vomsdir/pvier/voms.grid.sara.nl.lsc
Created vomses file /home/francisco/dirac_ui/diracos/etc/grid-security/vomses/pvier
Created vomsdir file /home/francisco/dirac_ui/diracos/etc/grid-security/vomsdir/projects.nl/voms.grid.sara.nl.lsc
Created vomses file /home/francisco/dirac_ui/diracos/etc/grid-security/vomses/projects.nl
Generating proxy...
Contacting CS...
Can't update from server Error while updating from dips://dirac01.grid.hep.ph.ic.ac.uk:9135/Configuration/Server: Unauthorized query ( 1111 : Unauthorized query)
Reason(s):
	Unauthorized query ( 1111 : Unauthorized query)
Can't contact DIRAC CS: Reason(s):
	Unauthorized query ( 1111 : Unauthorized query) (User possibly not registered with dirac server)

I have tried the same procedure on the UI machine I normally use to connect to the grid, and I get more or less the same output, except a couple of error-sounding lines are not there anymore (the "No CRL files found" line, the "current hash for bundle CAs" line, and everything after "Contacting CS" at the end). In either my laptop or the UI machine, it won't let me submit jobs afterwards. I can start a different thread about this, if it's going too far from the original question?

Try first with the vanilla DIRAC one. And the LHCb would work even if you are not LHCb.

The vanilla DIRAC doesn't work with dirac-production-runjoblocal. It does fetch the input sandbox, but fails to rerun afterwards:

Files retrieved and extracted in /user/pvazquez/gridtest/PythonApiTests/firstjobs/rerunjobs/job140686897temp//InputSandbox140686897
Traceback (most recent call last):
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/site-packages/DIRAC/TransformationSystem/scripts/dirac_production_runjoblocal.py", line 141, in main
    __downloadPilotScripts()
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/site-packages/DIRAC/TransformationSystem/scripts/dirac_production_runjoblocal.py", line 69, in __downloadPilotScripts
    remoteFile = urlopen(
                 ^^^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/bin/dirac-production-runjoblocal", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/Base/Script.py", line 74, in __call__
    return entrypointFunc._func()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/dirac.egi.eu/dirac/v8.0.24/Linux-x86_64/lib/python3.11/site-packages/DIRAC/TransformationSystem/scripts/dirac_production_runjoblocal.py", line 148, in main
    os.rename(_dir + ".dirac.cfg.old", _dir + ".dirac.cfg")
FileNotFoundError: [Errno 2] No such file or directory: '/user/pvazquez/.dirac.cfg.old' -> '/user/pvazquez/.dirac.cfg'

I can't try with the LHCb DIRAC until I figure out what's going wrong with the installation process. In case it's important, I am testing with a basic job:

JDL

Arguments = "hello";
CPUTime = 500;
Executable = "test_script.sh";
InputSandbox = test_script.sh;
JobGroup = km3net.org;
JobName = testjob;
JobType = User;
LogLevel = INFO;
OutputSandbox =·
     {
          std.err,
          std.out,
      };
Priority = 1;
StdError = std.err;
StdOutput = std.out;

test_script.sh

#! /bin/bash

hostname
date
echo $1
 
exit 1

I tried with either exit 0 or exit 1 to force an error, but I get the same output from running dirac-production-runjoblocal in both cases.

fstagni · 2023-07-28T07:48:37Z

fstagni
Jul 28, 2023
Maintainer

Hi,
the real error when installing the client locally is this:

Could not select DISET or Tornado client RuntimeError('Option /DIRAC/Setups/DIRAC-Certification/Framework is not defined')

Which means that the "setup" chosen ("DIRAC-Certification") is not correct. Please refer to the admin on what to choose.

The vanilla DIRAC doesn't work with dirac-production-runjoblocal. It does fetch the input sandbox, but fails to rerun afterwards:

Ah, it's buggy: can you please try this patch? #7138. Thanks!

0 replies

marianne013 · 2023-07-28T09:14:52Z

marianne013
Jul 28, 2023

To elaborate on Federico's answer: When installing a local UI, in the last step it it will pull the configuration from the DIRAC server you are targeting. If you are able to see the configuration (on the webpage) of the server you are targeting, you can check the name of the Setup by navigating to [Toplevel Config] -> DIRAC -> Setups and there you should find the name.

0 replies

FA-VS · 2023-07-28T20:20:24Z

FA-VS
Jul 28, 2023
Author

Thank you both for the help. Specifying the configuration name I find in the Dirac dashboard configuration manager didn't work either. It complains that my DN is not registered. I will check with the admin of the server, and when I get that working I will come back about the original issue.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Peek StandardOutput with DIRAC API #7106

{{title}}

Replies: 6 comments 7 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Peek StandardOutput with DIRAC API #7106

FA-VS Jul 12, 2023

Replies: 6 comments · 7 replies

FA-VS Jul 21, 2023 Author

chaen Jul 24, 2023 Maintainer

FA-VS Jul 24, 2023 Author

andresailer Jul 28, 2023 Collaborator

FA-VS Aug 30, 2023 Author

andresailer Aug 30, 2023 Collaborator

FA-VS Aug 30, 2023 Author

fstagni Jul 27, 2023 Maintainer

FA-VS Jul 27, 2023 Author

fstagni Jul 28, 2023 Maintainer

marianne013 Jul 28, 2023

FA-VS Jul 28, 2023 Author

FA-VS
Jul 12, 2023

Replies: 6 comments 7 replies

FA-VS
Jul 21, 2023
Author

chaen
Jul 24, 2023
Maintainer

FA-VS Jul 24, 2023
Author

andresailer Jul 28, 2023
Collaborator

FA-VS Aug 30, 2023
Author

andresailer Aug 30, 2023
Collaborator

FA-VS Aug 30, 2023
Author

fstagni
Jul 27, 2023
Maintainer

FA-VS Jul 27, 2023
Author

fstagni
Jul 28, 2023
Maintainer

marianne013
Jul 28, 2023

FA-VS
Jul 28, 2023
Author