Memory request/tag for job through DIRAC API #7202

FA-VS · 2023-09-14T14:26:32Z

FA-VS
Sep 14, 2023

Dear developers,
I am struggling to find the answer to this question in the documentation. If I have a job that requires 16GB memory, what is the proper way to request it with the Python API? From what I could find, I would guess it is something like:

j.setTag( ['16GBMemory'] )

But when I add this, my jobs get stuck waiting forever. I am not sure where the issue could be, but I would like to at least confirm that I am using the right syntax in my script.
Thank you very much for your time!
Francisco

Answered by fstagni

Sep 18, 2023

Am I understanding your reply correctly? The only way requesting memory works is if the administrator of the DIRAC instance has "manually" specified the 16GB tag in the configuration for the site/queue?

Yes, correct.

And if yes, does the memory requirement get transmitted to the site scheduler afterwards?

Unfortunately not. This is a feature that will be implemented only in DiracX, I am afraid.

View full answer

fstagni · 2023-09-15T10:13:26Z

fstagni
Sep 15, 2023
Maintainer

Hello,
tags like the one you set above are basically saying "run this job on a site that is marked with 16GBMemory". I used the the word "marked" because some administrator on the DIRAC instance you are using should have beforehand marked the site with the same tag. So, that would likely not work.

What you can instead do is to ask (in the JDL) a "MaxRAM" parameter (expressed in GB, so MaxRAM=16 for your case). Hope that helps!

0 replies

FA-VS · 2023-09-15T14:34:18Z

FA-VS
Sep 15, 2023
Author

Thank you very much for the help! Just to clarify, there is no way to do this with only the Python API, I have to go through the JDL/CLI? And the tag system is per-site, rather than per-queue?

3 replies

fstagni Sep 15, 2023
Maintainer

I think this would work:

from DIRAC.Interfaces.API.Job import Job
job = Job()
job._addParameter(job.workflow, "MaxRAM", "JDL", 16, "max ram")

If that works I will add it to my comment above and mark that as an answer. Let me know.

FA-VS Sep 18, 2023
Author

@fstagni So far submitting the jobs this way still keeps them hanging "waiting" forever, or at least it's been waiting 60h by now despite asking for short CPU time. Could this be an issue on the request side, on the server side, or somewhere else?

fstagni Sep 18, 2023
Maintainer

Can you post the JDL created for your jobs?

FA-VS · 2023-09-18T10:04:14Z

FA-VS
Sep 18, 2023
Author

Replying to @fstagni 's request for JDLs, this is what got submitted (getting it directly from the dashboard):

[
  Arguments = "jobDescription.xml -o LogLevel=INFO";
  CPUTime = 86399;
  DIRACSetup = "DIRAC-Production";
  Executable = "dirac-jobexec";
  InputSandbox =
  {
    "run-based-data-processing.tar",
    "Run-Snakemake-GridWN.sh",
    "/scratch/fvazquez/data-processing/snakemake-inputs/run_inputs_archives/00000049/KM3NeT_00000049_00007801.run-inputs-archive.v8.0b.tar.gz",
    "SB:ProductionSandboxSE|/SandBox/f/fvazquez.km3net.org_user/519/361/519361a956c8e410b1899bbcb028fd86.tar.bz2"
  };
  JobGroup = "km3net.org";
  JobID = 293124;
  JobName = "Snakemake-00000049-00007801_4cores_c_noshower";
  JobRequirements =
  [
    CPUTime = 86399;
    OwnerDN = "/DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]";
    OwnerGroup = "km3net.org_user";
    Setup = "DIRAC-Production";
    UserPriority = 1;
    VirtualOrganization = "km3net.org";
    Sites = "GRID.NIKHEF.nl";
    JobTypes = "User";
    Tags =
    {
      "4Processors",
      "MultiProcessor",
      "16GB"
    };
  ];
  JobType = "User";
  LogLevel = "INFO";
  MaxNumberOfProcessors = 4;
  MaxRAM = 16;
  NumberOfProcessors = 4;
  OutputSandbox =
  {
    "WN.log",
    "std.out",
    "std.err",
    "snakemake_00000049_00007801_stdouterr.log",
    "snakemake_00000049_00007801_latest.log",
    "Run-Snakemake-GridWN.log"
  };
  Owner = "fvazquez";
  OwnerDN = "/DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]";
  OwnerGroup = "km3net.org_user";
  OwnerName = "fvazquez";
  Priority = 1;
  Site = "GRID.NIKHEF.nl";
  StdError = "std.err";
  StdOutput = "std.out";
  VirtualOrganization = "km3net.org";
  Tags =
  {
    "4Processors",
    "MultiProcessor",
    "16GB"
  };
]

And this is the JDL for a small test job I submitted around the same time:

[
  Arguments = "jobDescription.xml -o LogLevel=INFO";
  CPUTime = 500;
  DIRACSetup = "DIRAC-Production";
  Executable = "dirac-jobexec";
  InputSandbox = "SB:ProductionSandboxSE|/SandBox/f/fvazquez.km3net.org_user/f72/cf0/f72cf0810a58bf1cc7311e3a2f3c5688.tar.bz2";
  JobGroup = "km3net.org";
  JobID = 293126;
  JobName = "testjob";
  JobRequirements =
  [
    CPUTime = 500;
    OwnerDN = "/DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]";
    OwnerGroup = "km3net.org_user";
    Setup = "DIRAC-Production";
    UserPriority = 1;
    VirtualOrganization = "km3net.org";
    Sites = "GRID.NIKHEF.nl";
    JobTypes = "User";
    Tags = "16GB";
  ];
  JobType = "User";
  LogLevel = "INFO";
  MaxRAM = 16;
  OutputSandbox =
  {
    "Script1_CodeOutput.log",
    "Script2_CodeOutput.log",
    "Script3_CodeOutput.log",
    "std.err",
    "std.out"
  };
  Owner = "fvazquez";
  OwnerDN = "/DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]";
  OwnerGroup = "km3net.org_user";
  OwnerName = "fvazquez";
  Priority = 1;
  Site = "GRID.NIKHEF.nl";
  StdError = "std.err";
  StdOutput = "std.out";
  VirtualOrganization = "km3net.org";
  Tags = "16GB";
]

I notice "16GB" gets added as a tag when submitting it through the API the way you proposed. I will try again submitting it through the CLI and only adding the " MaxRAM = 16 " line.

0 replies

fstagni · 2023-09-18T10:34:45Z

fstagni
Sep 18, 2023
Maintainer

I notice "16GB" gets added as a tag when submitting it through the API the way you proposed. I will try again submitting it through the CLI and only adding the " MaxRAM = 16 " line.

So, the result that you are getting should be OK. But, having a second look at the code, I see that there's actually only a simple mechanism implemented, meaning that we are back to the statement of my first reply:

tags like the one you set above are basically saying "run this job on a site that is marked with 16GBMemory". I used the the word "marked" because some administrator on the DIRAC instance you are using should have beforehand marked the site with the same tag. So, that would likely not work.

So, to cut story short, if I was you I would remove the memory requirements/tags and hope for the best!

1 reply

FA-VS Sep 18, 2023
Author

Unfortunately the job does require 14GB, and the default queue is limited at 8GB, so the process crashes eventually if I do not specify anything.

Am I understanding your reply correctly? The only way requesting memory works is if the administrator of the DIRAC instance has "manually" specified the 16GB tag in the configuration for the site/queue?

And if yes, does the memory requirement get transmitted to the site scheduler afterwards? From discussion with administrators of the site, it seems like the queue presented externally can still "hide" different internal queues, some of which don't have the same memory limitations.

fstagni · 2023-09-18T14:28:23Z

fstagni
Sep 18, 2023
Maintainer

Am I understanding your reply correctly? The only way requesting memory works is if the administrator of the DIRAC instance has "manually" specified the 16GB tag in the configuration for the site/queue?

Yes, correct.

And if yes, does the memory requirement get transmitted to the site scheduler afterwards?

Unfortunately not. This is a feature that will be implemented only in DiracX, I am afraid.

0 replies

FA-VS · 2023-09-19T07:53:56Z

FA-VS
Sep 19, 2023
Author

I was going to mark this question as answered, but then I noticed something strange. I have sent two test jobs at different times with the following JDL (as recovered from the DIRAC dashboard):

[
  Arguments = "jobDescription.xml -o LogLevel=INFO";
  CPUTime = 500;
  DIRACSetup = "DIRAC-Production";
  Executable = "dirac-jobexec";
  InputSandbox = "SB:ProductionSandboxSE|/SandBox/f/fvazquez.km3net.org_user/e27/fb8/e27fb8c3f28f7f93933670fd8390d211.tar.bz2";
  JobGroup = "km3net.org";
  JobID = 293550;
  JobName = "testjob";
  JobRequirements =
  [
    CPUTime = 500;
    OwnerDN = "/DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]";
    OwnerGroup = "km3net.org_user";
    Setup = "DIRAC-Production";
    UserPriority = 1;
    VirtualOrganization = "km3net.org";
    Sites = "GRID.NIKHEF.nl";
    JobTypes = "User";
    Tags = "16GB";
  ];
  JobType = "User";
  LogLevel = "INFO";
  MaxRAM = 16;
  OutputSandbox =
  {
    "Script1_CodeOutput.log",
    "Script2_CodeOutput.log",
    "std.err",
    "std.out"
  };
  Owner = "fvazquez";
  OwnerDN = "/DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez [email protected]";
  OwnerGroup = "km3net.org_user";
  OwnerName = "fvazquez";
  Priority = 1;
  Site = "GRID.NIKHEF.nl";
  StdError = "std.err";
  StdOutput = "std.out";
  VirtualOrganization = "km3net.org";
  Tags = "16GB";
]

The only difference was that one included a third test executable ("echo", "hostname", "pwd", that sort of thing). One got scheduled in five minutes, the other (which I submitted first!) is still waiting after almost four days. This is a it confusing, since they both include the MaxRAM and Tags = "16GB" lines, while none of the queues defined or the NIKHEF site in the configuration specify "16GB" as a tag or required tag, so I would have expected neither of the jobs to have found a matching queue. Am I missing anything else?

5 replies

fstagni Sep 20, 2023
Maintainer

Sorry, I am not sure I understand: is there any difference between the JDLs?

FA-VS Sep 20, 2023
Author

@fstagni Not really. Here is the result from doing diff between the two JDLs, as recovered from the DIRAC dashboard:

<   InputSandbox = "SB:ProductionSandboxSE|/SandBox/f/fvazquez.km3net.org_user/f72/cf0/f72cf0810a58bf1cc7311e3a2f3c5688.tar.bz2";
---
>   InputSandbox = "SB:ProductionSandboxSE|/SandBox/f/fvazquez.km3net.org_user/e27/fb8/e27fb8c3f28f7f93933670fd8390d211.tar.bz2";
8c8
<   JobID = 293126;
---
>   JobID = 293550;
29d28
<     "Script3_CodeOutput.log",

So other than the fact that the first job submitted had an additional executable, they were the same. Which makes it surprising that the first one I sent is still waiting, while the second one got scheduled within five minutes.

fstagni Sep 20, 2023
Maintainer

That's weird. Which server version are you running? Can you access to DB content?

FA-VS Sep 20, 2023
Author

I will get in touch with the administrators and see if I can get any more information, but I do not have access directly mysef. I will reply when I have more to share. Thank you for your help!

FA-VS Sep 22, 2023
Author

@fstagni : the version is v7.3.17. I cannot say more about the database itself for now.

IgorPelevanyuk · 2023-09-20T12:19:38Z

IgorPelevanyuk
Sep 20, 2023

This is also interesting question for administrators of DIRAC. What should I add to the configuration of a Site to make this option work? Because right now I create several queues with different tags and it is getting messy.

Another problem appears once we make several tagged queues in addition to untagged queue on the same computing resource.
In that case, if user sends a big bunch of jobs to the resource without specifying tag requirement in the JDL, SiteDirector decides that it can send a bunch of pilots to all available queues: to noTag queue, 4GB tag queue, 8GB, 12GB, 16GB, and so on. That leads to the situation when many pilots are running with exagerated memory requirements and batch system desides that it cannot accept new jobs because memory is already booked for other processes. That leads to idle CPU cores.

2 replies

fstagni Sep 20, 2023
Maintainer

At the moment there's no effective solution to this, as pilots (effectively: DIRAC CEs abstractions) don't have the possibility to administer memory.

IgorPelevanyuk Sep 21, 2023

Ok, thanks! We managed to deal with it somehow. So, lets wait for diracx.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory request/tag for job through DIRAC API #7202

{{title}}

Replies: 7 comments 11 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Memory request/tag for job through DIRAC API #7202

FA-VS Sep 14, 2023

Replies: 7 comments · 11 replies

fstagni Sep 15, 2023 Maintainer

FA-VS Sep 15, 2023 Author

fstagni Sep 15, 2023 Maintainer

FA-VS Sep 18, 2023 Author

fstagni Sep 18, 2023 Maintainer

FA-VS Sep 18, 2023 Author

fstagni Sep 18, 2023 Maintainer

FA-VS Sep 18, 2023 Author

fstagni Sep 18, 2023 Maintainer

FA-VS Sep 19, 2023 Author

fstagni Sep 20, 2023 Maintainer

FA-VS Sep 20, 2023 Author

fstagni Sep 20, 2023 Maintainer

FA-VS Sep 20, 2023 Author

FA-VS Sep 22, 2023 Author

IgorPelevanyuk Sep 20, 2023

fstagni Sep 20, 2023 Maintainer

IgorPelevanyuk Sep 21, 2023

FA-VS
Sep 14, 2023

Replies: 7 comments 11 replies

fstagni
Sep 15, 2023
Maintainer

FA-VS
Sep 15, 2023
Author

fstagni Sep 15, 2023
Maintainer

FA-VS Sep 18, 2023
Author

fstagni Sep 18, 2023
Maintainer

FA-VS
Sep 18, 2023
Author

fstagni
Sep 18, 2023
Maintainer

FA-VS Sep 18, 2023
Author

fstagni
Sep 18, 2023
Maintainer

FA-VS
Sep 19, 2023
Author

fstagni Sep 20, 2023
Maintainer

FA-VS Sep 20, 2023
Author

fstagni Sep 20, 2023
Maintainer

FA-VS Sep 20, 2023
Author

FA-VS Sep 22, 2023
Author

IgorPelevanyuk
Sep 20, 2023

fstagni Sep 20, 2023
Maintainer