Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add PyHPS CLI #3091

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
85d371e
First iteration
germa89 May 13, 2024
1da9ca4
Adding logging
germa89 May 14, 2024
4f32315
fixes to the CLI help
germa89 May 14, 2024
08d2055
Fix codacy issues
germa89 May 14, 2024
08731de
Merge branch 'main' into feat/pyhps-support
germa89 May 14, 2024
d70b090
Apply suggestions from code review
germa89 May 16, 2024
aa8a23c
Apply suggestions from code review
germa89 May 16, 2024
660e717
Adding suggestions
germa89 May 16, 2024
33ac15a
Removing warnings.
germa89 May 17, 2024
4055c60
Adding inputs and outputs (#3112)
germa89 May 22, 2024
f23da00
Adding APDL jobs support (#3111)
germa89 May 22, 2024
0f80a47
Refactoring PyHPS implementation (#3117)
germa89 May 31, 2024
a7adf0b
Merge branch 'main' into feat/pyhps-support
germa89 Jun 5, 2024
dff728d
Apply suggestions from Kathy's code review
germa89 Jun 18, 2024
a9eb0e8
Merge branch 'main' into feat/pyhps-support
germa89 Jun 18, 2024
445949a
Adding changelog entry: 3091.miscellaneous.md
pyansys-ci-bot Jun 18, 2024
4556c49
Adding changelog entry: 3091.miscellaneous.md
pyansys-ci-bot Jun 18, 2024
ac875f1
Renaming argument ``to_json``.
germa89 Jun 18, 2024
1911729
rewriting docstring
germa89 Jun 18, 2024
9bc7612
Merge branch 'main' into feat/pyhps-support
germa89 Jun 24, 2024
0279ffc
Adding changelog entry: 3091.added.md
pyansys-ci-bot Jun 24, 2024
dd0cb4b
Merge branch 'main' into feat/pyhps-support
germa89 Jun 26, 2024
4145688
feat: Detaching logging from main logic (#3205)
germa89 Jun 26, 2024
6807934
fix: codecov suggestions
germa89 Jun 26, 2024
1b53047
feat: renaming PyMAPDLJobSubmissionDefinition class
germa89 Jul 15, 2024
a29589d
docs: improved docstring
germa89 Jul 16, 2024
b9a6c98
feat: renaming to submission.
germa89 Jul 16, 2024
e1b85b8
fix: doc example
germa89 Jul 16, 2024
e0bd844
docs:improve docstring examples
germa89 Jul 16, 2024
6cfa2eb
feat: rename file to match main function 'submit'
germa89 Jul 16, 2024
3b55c89
feat: adding option to pass token to CLI.
germa89 Jul 16, 2024
ef24249
docs: adding API docs
germa89 Jul 17, 2024
82e3209
chore: Merge branch 'main' into feat/pyhps-support
germa89 Jul 17, 2024
7af9298
docs: using other type of reference
germa89 Jul 17, 2024
a78aa4c
feat: adding imports to hpc.__init__
germa89 Jul 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/ansys/mapdl/core/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ def main(ctx):
pass

from ansys.mapdl.core.cli.convert import convert
from ansys.mapdl.core.cli.hpc import submit
from ansys.mapdl.core.cli.list_instances import list_instances
from ansys.mapdl.core.cli.start import start
from ansys.mapdl.core.cli.stop import stop
Expand All @@ -48,6 +49,12 @@ def main(ctx):
main.add_command(stop)
main.add_command(list_instances, name="list")

# HPC commands
# pymapdl hpc submit
# pymapdl hpc list
# pymapdl hpc stop
main.add_command(submit)

def old_pymapdl_convert_script_entry_point():
print(
"""This CLI function has been deprecated. Please use instead:
germa89 marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
271 changes: 271 additions & 0 deletions src/ansys/mapdl/core/cli/hpc.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,271 @@
# Copyright (C) 2024 ANSYS, Inc. and/or its affiliates.
# SPDX-License-Identifier: MIT
#
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

"""Submit PyHPS jobs to a cluster"""

import logging
import os
from typing import Optional, Union

import click

from ansys.mapdl.core.cli import main

logger = logging.getLogger()


@main.command(
short_help="Submit jobs to an HPC cluster using PyHPS package.",
help="""Submit jobs to an HPC cluster using PyHPS package.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.argument("main_file")
@click.option(
"--name",
default=None,
type=str,
help="""Name of the PyHPS project to be created.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--url",
default=None,
type=str,
help="""URL where the HPS cluster is deployed. For example: "https://myserver:3000/hps" """,
)
@click.option(
"--user", default=None, type=str, help="Username to login into the HPC cluster."
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--password",
default=None,
type=str,
help="Password used to login into the HPC cluster.",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--python",
default=None,
type=str,
help="""Set python version to be used to create the virtual environment and
run the python file. By default it uses python3 (default in cluster).""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--output_files",
default=None,
type=str,
help="""Set the output files to be monitored.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--shell_file",
default=None,
type=str,
help="""If desired, you can provide a shell script to execute instead of
the python file. You can call your python file from it if you wish. By default,
it is not used.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--requirements_file",
default=None,
type=str,
help="""If provided, the created virtual environment is installed with the
libraries specified in this file. If not, the activated virtual environment is
cloned through a temporary 'pip list' file. If you are using editable package,
it is recommended you attach your own requirement file using ``pip freeze`` """,
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--extra_files",
default=None,
type=str,
help="""To upload extra files which can be called from your main python file
(or from the shell file).""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--config_file",
default=None,
type=str,
help="""To load job configuration from a file.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--num_cores",
default=None,
type=str,
help="""Set the amount of CPU cores reserved for the job. By default it is 1 CPU.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--memory",
default=None,
type=str,
help="""Set the amount of memory RAM in MB reserved for the job. By default it is 100 MB.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--disk_space",
default=None,
type=str,
help="""Set the amount of hard drive space in MB reserved for the job. By default it is 100 MB.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--exclusive",
default=None,
type=str,
is_flag=False,
flag_value=True,
help="""Set the job to run in a machine exclusively, without sharing it with other jobs. By default it is False""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--max_execution_time",
default=None,
type=str,
germa89 marked this conversation as resolved.
Show resolved Hide resolved
help="""Set the maximum execution time for the job. By default it is zero (unlimited).""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--wait",
default=None,
type=str,
is_flag=False,
flag_value=True,
help="""Set the terminal to wait for job completion before return the control to the user. """,
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--save_config_file",
default=False,
type=bool,
is_flag=False,
flag_value=True,
help="""Writes the configuration to the config file, after successfully
submit the job. It overwrites the configuration file.
The configuration file path is given using ``config_file`` argument.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
@click.option(
"--debug",
default=False,
type=bool,
is_flag=False,
flag_value=True,
help="""Set PyMAPDL to prints the debug logging to the console output.""",
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
def submit(
main_file: str,
name: str = None,
url: str = None,
user: str = None,
password: str = None,
python: float = 3.9,
output_files: Optional[Union[str, list]] = None,
shell_file: str = None,
requirements_file: str = None,
extra_files: Optional[Union[str, list]] = None,
config_file: str = None,
num_cores: int = None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

option to download the result files?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not aware if we can do that from the PyHPS library... i will check it out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, It can be done.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

memory: int = None,
disk_space: int = None,
exclusive: bool = None,
max_execution_time: int = None,
wait: bool = False,
save_config_file: bool = False,
debug: bool = False,
):
"""Example code:
pymapdl submit my_file.sh my_file_01.py my_file_02 --name="my job" --url="https://10.231.106.91:3000/hps" --user=repuser --password=repuser --python=3.9
"""
import json

from ansys.mapdl.core.hpc.pyhps import (
create_pymapdl_pyhps_job,
get_value_from_json_or_default,
wait_for_completion,
)

if debug:
logging.basicConfig(
format="[%(asctime)s | %(levelname)s] %(message)s", level=logging.DEBUG
)

if config_file is None:
config_file = os.path.join(os.getcwd(), "hps_config.json")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@germa89 should the config be in appdirs instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MMhhh... if we use a file in the root directory, we could point to different files.... and hence multiple configurations.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But maybe you are right... maybe it is better to use appdirs.... buttt... it might be a bit hidden for the user...

Copy link
Collaborator Author

@germa89 germa89 May 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And we have many options to be keep in the file:

{
    "url": "https://10.231.106.91:3000/hps",
    "user": "repuser",
    "password": "repuser",
    "python": "3.9",
    "name": "my job",
    "num_cores": 1,
    "memory": 100,
    "disk_space": 100,
    "exclusive": false,
    "max_execution_time": 1000
}

keeping so many important options hidden in a appdir path might bring some complications.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@germa89 use keyring for user/password. I don't think its secure enough to put that in plaintext like this. For everything else I would agree

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree.

logger.debug(f"Using default hps configuration file: {config_file}")
germa89 marked this conversation as resolved.
Show resolved Hide resolved

url = get_value_from_json_or_default(url, config_file, "url", None)
user = get_value_from_json_or_default(user, config_file, "user", None)
password = get_value_from_json_or_default(password, config_file, "password", None)
python = get_value_from_json_or_default(python, config_file, "python", 3)
name = get_value_from_json_or_default(name, config_file, "name", "My PyMAPDL job")

num_cores = get_value_from_json_or_default(num_cores, config_file, "num_cores", 1)
memory = get_value_from_json_or_default(memory, config_file, "memory", 100)
disk_space = get_value_from_json_or_default(
disk_space, config_file, "disk_space", 100
)
exclusive = get_value_from_json_or_default(
exclusive, config_file, "exclusive", False
)
max_execution_time = get_value_from_json_or_default(
max_execution_time, config_file, "max_execution_time", 0
)

proj, _ = create_pymapdl_pyhps_job(
main_file=main_file,
name=name,
url=url,
user=user,
password=password,
python=python,
output_files=output_files,
shell_file=shell_file,
requirements_file=requirements_file,
extra_files=extra_files,
config_file=config_file,
num_cores=num_cores,
memory=memory,
disk_space=disk_space,
exclusive=exclusive,
max_execution_time=max_execution_time,
)

if save_config_file:
germa89 marked this conversation as resolved.
Show resolved Hide resolved
config = {
"url": url,
"user": user,
"password": password,
"python": python,
"name": name,
"num_cores": num_cores,
"memory": memory,
"disk_space": disk_space,
"exclusive": exclusive,
"max_execution_time": max_execution_time,
}

logger.debug(
f"Saving the following configuration to the config file ({config_file}):\n{config}"
germa89 marked this conversation as resolved.
Show resolved Hide resolved
)
with open(config_file, "w") as fid:
json.dump(config, fid)

if wait:
print(f"Waiting for project {name} (id: {proj.id}) to be completed...")
germa89 marked this conversation as resolved.
Show resolved Hide resolved
wait_for_completion(proj, evaluated=True, failed=True)


def list_jobs():
pass


def stop_job():
pass
21 changes: 21 additions & 0 deletions src/ansys/mapdl/core/hpc/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Copyright (C) 2024 ANSYS, Inc. and/or its affiliates.
# SPDX-License-Identifier: MIT
#
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
Loading