DFC:: list PFNs on SE

[adriansev]

Hi! Is there a way to list the files (LFNs and their PFNs) on a given SE?
It would be really important for recovery and incident response actions.
Thanks a lot!

[chaen]

if you use the DFC, getSEDump is what you want

DIRAC/src/DIRAC/Resources/Catalog/FileCatalogClient.py

Line 660 in 93de7f3

def getSEDump(self, seNames, outputFilename):

You don't get the PFN, given that in principle, the PFN aren't in the catalog. But you can construct them trivially in a second step.

[chaen]

as any other DIRAC script, you need the magic lines first

from DIRAC.Core.Base.Script import parseCommandLine
parseCommandLine()

[adriansev]

actually i have the same error, and it would seem to be a problem with my configuration(???)

(base) ALIBUILD##[Monday 20.02.23 19:10 adrian@sev] ~ $
python
Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:45:29) 
[GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from DIRAC.Core.Base.Script import parseCommandLine
>>> from DIRAC.Resources.Catalog import FileCatalogClient
>>> parseCommandLine()
([], [])
>>> fc = FileCatalogClient.FileCatalogClient()
>>> fc.getSEDump('ROISS-disk','ROISS-disk.csv')
{'OK': False, 'Errno': 0, 'Message': "Cannot get URL for DataManagement/FileCatalog in setup auger: RuntimeError('Option /DIRAC/Setups/auger/DataManagement is not defined')", 'CallStack': ['  File "<stdin>", line 1, in <module>\n', '  File "/home.hdd/adrian/DIRACOS/lib/python3.9/site-packages/DIRAC/Resources/Catalog/FileCatalogClient.py", line 676, in getSEDump\n    dfc = TransferClient(self.serverURL, timeout=3600)\n', '  File "/home.hdd/adrian/DIRACOS/lib/python3.9/site-packages/DIRAC/Core/Tornado/Client/ClientSelector.py", line 81, in ClientSelector\n    rpc = disetClient(*args, **kwargs)\n', '  File "/home.hdd/adrian/DIRACOS/lib/python3.9/site-packages/DIRAC/Core/DISET/private/BaseClient.py", line 95, in __init__\n    result = initFunc()\n', '  File "/home.hdd/adrian/DIRACOS/lib/python3.9/site-packages/DIRAC/Core/DISET/private/BaseClient.py", line 165, in __discoverURL\n    result = self.__findServiceURL()\n', '  File "/home.hdd/adrian/DIRACOS/lib/python3.9/site-packages/DIRAC/Core/DISET/private/BaseClient.py", line 327, in __findServiceURL\n    return S_ERROR(f"Cannot get URL for {self._destinationSrv} in setup {self.setup}: {repr(e)}")\n']}

my dirac.cfg looks like this (and does not have /DIRAC/Setups/auger/DataManagement)

cat dirac.cfg 
DIRAC
{
  VirtualOrganization = auger
  Configuration
  {
    Servers = dips://dirac-conf.in2p3.fr:9135/Configuration/Server
    Servers += dips://ccdirac02.in2p3.fr:9135/Configuration/Server
    Servers += dips://ccdirac04.in2p3.fr:9135/Configuration/Server
    Servers += dips://ccdirac06.in2p3.fr:9135/Configuration/Server
    Servers += dips://ccdirac03.in2p3.fr:9135/Configuration/Server
    Servers += dips://dirac.egi.eu:9135/Configuration/Server
    Servers += dips://dfc-auger.grid.cesnet.cz:9135/Configuration/Server
  }
  Security
  {
    UseServerCertificate = no
  }
  Setup = auger
}
LocalInstallation
{
  VirtualOrganization = auger
  ConfigurationServer = dips://dirac-conf.in2p3.fr:9135/Configuration/Server
  Setup = auger
}

[adriansev]

so, looking for a correct setups value in DIRAC web interface, the dump kind of worked, meaning that the content of csv file is:

cat ROISS-disk.csv 
{"OK": false, "Errno": 0, "Message": "To be implemented on derived class", "CallStack": ["  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/threading.py\", line 937, in _bootstrap\n    self._bootstrap_inner()\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/threading.py\", line 980, in _bootstrap_inner\n    self.run()\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/threading.py\", line 917, in run\n    self._target(*self._args, **self._kwargs)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/concurrent/futures/thread.py\", line 83, in _worker\n    work_item.run()\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/DISET/private/Service.py\", line 407, in _processInThread\n    result = self._processProposal(trid, proposalTuple, handlerObj)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/DISET/private/Service.py\", line 593, in _processProposal\n    result = self._executeAction(trid, proposalTuple, handlerObj)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/DISET/private/Service.py\", line 613, in _executeAction\n    response = handlerObj._rh_executeAction(proposalTuple)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/DISET/RequestHandler.py\", line 134, in _rh_executeAction\n    retVal = self.__doFileTransfer(actionTuple[1])\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/DISET/RequestHandler.py\", line 201, in __doFileTransfer\n    uRetVal = self.transfer_toClient(fileInfo[0], fileInfo[1], fileHelper)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/DataManagementSystem/Service/FileCatalogHandler.py\", line 731, in transfer_toClient\n    res = self.getSEDump(seNames)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/DataManagementSystem/Service/FileCatalogHandler.py\", line 714, in getSEDump\n    return self.fileCatalogDB.getSEDump(seNames)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/DataManagementSystem/DB/FileCatalogDB.py\", line 1188, in getSEDump\n    return self.fileManager.getSEDump(seNames)\n", "  File \"/vo/dirac/versions/v7.3.26-1658502883/Linux-x86_64/lib/python3.9/site-packages/DIRAC/DataManagementSystem/DB/FileCatalogComponents/FileManager/FileManagerBase.py\", line 1350, in getSEDump\n    return S_ERROR(\"To be implemented on derived class\")\n"]}

Any idea on this?
Thanks a lot!!

[chaen]

That must come before anything else, even the other import

from DIRAC.Core.Base.Script import parseCommandLine
parseCommandLine()

As for the To be implemented message, how is your file catalog service configured ? which managers do you use ? maybe @atsareg can answer

[chaen]

I actually just checked and indeed, that method is implemented only for the FileManagerPS manager, which I believe Auger doesn't use.
Maybe a brave soul will implement it for the other manager (not me though :-) )

[chaen]

FYI, I'll very soon upload some dump comparison tools in DIRAC.

[atsareg]

You can easily dump all the LFNs having replicas in a given SE with the dfind command. To see LFNs under a certain PATH you do:

 dfind /auger/user/a/asevcenc "SE=ROISS-disk"

[atsareg]

@chaen By the way, this should also work for LHCb as well, especially now that COMDIRAC is merged into the Core DIRAC

[chaen]

possibly, but not even sure given that the metadata was never something meant to work in the LHCb case.
And also, getSEDump is highly optimized for that, so I see no benefit in using dfind here