Here we go again: DIRAC environment leaking into user jobs

[marianne013]

Hi All,
we upgraded our DIRAC server to v7.3.26 last night. It didn't go well:
https://ggus.eu/index.php?mode=ticket_info&ticket_id=158152

When we ran into this problem going from v6 to v7:
https://groups.google.com/g/diracgrid-forum/c/TfiRouxA0XA/m/zTA0Jh1BBAAJ?hl=en-GB
there were a bunch of fixes:
#4565
#4302
Discussion seems to have been lost, though there is a note that we had one:
https://indico.cern.ch/event/854185/note/
(There was probably more, but I cannot find it right now)
We've been running like this for nearly 3 years now.

Unfortunately we only checked the PYTHON/LD_LIBRARY_PATH in testing as this was the issue last time around, so we didn't quite see this coming.
We are now scrambling trying to resolve this, but so far we haven't been able to come up with any solution. We've tried to use python2 pilots, but the issue remains. We've considered downgrading the pilot to v7r2, but at least in first go that didn't work either.
Any ideas ? For us this is an utter disaster.

Daniela

[chaen]

Hi @marianne013

Actually you are right, this is all an unfinished business, maybe closed too soon #4480

And indeed, it is even unclear what the real strategy should be ! And we do cleanup more and more variables (sigh...) DIRACGrid/DIRACOS2#74

So I really do not have a solution now.But isn't the best thing to do what you advice to your users (https://www.gridpp.ac.uk/wiki/Quick_Guide_to_Dirac) ?

Please note that this is the last release that will support python2, please upgrade your scripts!

Also, I just had a very quick look at "snoplus" on github, and it seems their recommended way of doing is through containers. That should be a good enough way to split the environment no ?

[marianne013]

Do you know how hard it is to explain containers to every one, which directories to bind-mount (is that even a word?) etc ? It's all doable, but I would prefer the rather clean setup we had in v7r3 (I meant v7r2) as a starting point. Simon and me had some brief discussion about this, but if we had come up with a master plan you would have heard about it by now.

[sfayer]

Just to suggest a possible solution, I suspect it'll be considered too complicated:

• Right at start of pilot grab ORIG_PATH=$PATH
• After diracos2 install, create scripts dir, symlink bin/dirac-* into scripts dir.
• At start of user payload, set PATH=$ORIG_PATH:<diracos_path>/scripts

(Although actually now I've written it, it's doesn't look quite as bad as I thought it might :-)

That way the user will get the native version of everything, but will still be able to use the DIRAC tools (as they have a wrapper script that points to the diracos python version, so can be run without activating the full environment).

The problem with trying to fix anything at a VO level is that some of our VOs aren't as tightly integrated as some of the bigger ones: There may well be multiple groups with independent stuff, for example I don't think our snoplus users are using containers at all, that's a different sub-group!

This goes a little beyond just python: Things like curl, openssl & perl are also preferentially used from diracos2, so you get a much newer version than you might expect. While I don't have any concrete examples, it's only a matter of time before someone tries to run something like curl and finds it has a slightly different behaviour or command line switches to what they expect (wrong python version is only the visible tip of the iceburg).

Regards,
Simon

[chaen]

I would prefer the rather clean setup we had in v7r3 as a starting point.

I confess being a bit lost in all this, I don't recall of all the changes, but did we REALLY have a clean setup, or did it just look like it, because you had a python2 anyway ?

will still be able to use the DIRAC tools (as they have a wrapper script that points to the diracos python version, so can be run without activating the full environment).

I don't think this is entirely true, as activating the environment also sets and unsets some variables.

Just to make sure we all have the same thing in mind: do we (still) all agree that what we initially want to achieve is that a payload starts with the vanilla environment from the worker node ? And possibly still be able to use dirac commands ? This actually matches perfectly the summary done by Marko

To conclude on this, we had a longer discussion on the last developers meeting, and the majority opinion is that people should not call dirac commands from the payload, but rather execute the payload in a vanilla environment and use the JobWrapper for input output data. Nevertheless dirac offers the possibility to call dirac commands in the payload, and we will not actively disable this option, but if the VO, user, operators decide to follow such an approach, it is their responsibility to ensure the environment separation for calling dirac command and payload execution.

[andresailer]

How do they run the Job? Can we jet the jobDescription.xml (via email, if not public here)?
How did the error message change with the python2 pilots?

[andresailer]

Right, I missed that first if.
Depending on what people call, you also need to store the LD_LIBRARY_PATH or prefix /usr/lib or both, or risk mixing in something from diracos?

[sfayer]

The nice thing about DIRACOS2 is that it doesn't have any LD_LIBRARY_PATH or PYTHONPATH set by default, it's all done by embedding an rpath into the executables: I think the only variable we need to worry about is PATH, which thankfully simplifies things quite a lot.

[andresailer]

Right.
Just give it a go then? https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/Pilots/Pilots3.html#install-non-released-code-from-within-the-pilot

[sfayer]

We tried this out and it did fix the snoplus style use case, but caused new trouble for others (specifically people using DIRAC API in jobs where sites have python3 installed on the WN, it also breaks the diracos version of the gfal tools as they have a slightly dodgy system for finding python).
We've decided to try and stick with the environment how it is for now and instead give our users more documentation on what env they can expect with recipes on how to get other common set-ups. Things will be a little less surprising for everyone once the WLCG upgrades from EL7 to something with only python3 everywhere anyway (EL9?).

Regards,
Simon

[marianne013]

Technically Simon's reply above is the answer (at least for GridPP), but I can only mark this as an answer if I tick Andre's "How do they run the Job? Can we jet the jobDescription.xml (via email, if not public here)?
How did the error message change with the python2 pilots?" comment which is a bit incongruous.

[marianne013]

"... but if the VO, user, operators decide to follow such an approach, it is their responsibility to ensure the environment separation for calling dirac command and payload execution."
Indeed and then Simon and me came up with the "source bashrc wrapper inside command" approach and that fixed the issue in vanilla DIRAC. Maybe I can find teh codez tomorrow, it was a bit of a long day today.
By the looks of it snoplus production uses Ganga, that's a whole other bit of code I won't forgive LHCb in a hurry.

[sfayer]

I've got another suggested fix:
Just before starting any user payloads, we could simply re-order $PATH to put the diracos path(s) at the end. That way the dirac-* tools will still be accessible, but the system versions of python/perl/etc will be favoured in user jobs.