hpcugent · itkovian · Sep 9, 2022 · Sep 10, 2022 · Dec 23, 2022 · Jan 3, 2023
diff --git a/.github/workflows/pyslurm.yml b/.github/workflows/pyslurm.yml
@@ -13,9 +13,9 @@ jobs:
       fail-fast: false
     steps:
       - name: Checkout repository code
-        uses: actions/checkout@v2
+        uses: actions/checkout@main
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@main
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install Dependencies

diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,6 @@
 # Ignore Build Directory #
 build/
+site
 
 # Ignore Docs Directory #
 #doc/
@@ -15,6 +16,9 @@ pyslurm/*.pxi~
 pyslurm/*.pxd~
 pyslurm/*.so
 pyslurm/*.c
+pyslurm/**/*.c
+pyslurm/**/*.so
+pyslurm/**/__pycache__
 
 # Ignore vim swap files
 *.swp
@@ -24,6 +28,7 @@ tests/*.pyc
 
 # Ignore pycache (Python 3)
 */__pycache__
+*/**/__pycache__
 
 # Ignore job output files
 *.out

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,100 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## Unreleased on the [23.2.x](https://github.com/PySlurm/pyslurm/tree/23.2.x) branch
+
+- New Classes to interact with Database Associations (WIP)
+    - `pyslurm.db.Association`
+    - `pyslurm.db.Associations`
+- New Classes to interact with Database QoS (WIP)
+    - `pyslurm.db.QualityOfService`
+    - `pyslurm.db.QualitiesOfService`
+- Add `truncate_time` option to `pyslurm.db.JobFilter`, which is the same as -T /
+  --truncate from sacct.
+- Add new Attributes to `pyslurm.db.Jobs` that help gathering statistics for a
+  collection of Jobs more convenient.
+- Fix `allocated_gres` attribute in the `pyslurm.Node` Class returning nothing.
+- Add new `idle_memory` and `allocated_tres` attributes to `pyslurm.Node` class
+- Fix Node State being displayed as `ALLOCATED` when it should actually be
+  `MIXED`.
+
+## [23.2.2](https://github.com/PySlurm/pyslurm/releases/tag/v23.2.2) - 2023-07-18
+
+### Added
+
+- Ability to modify Database Jobs
+- New classes to interact with the Partition API
+    - [pyslurm.Partition][]
+    - [pyslurm.Partitions][]
+- New attributes for a Database Job:
+    - `extra`
+    - `failed_node`
+- Added a new Base Class [MultiClusterMap][pyslurm.xcollections.MultiClusterMap] that some Collections inherit from.
+- Added `to_json` function to all Collections
+
+### Fixed
+
+- Fixes a problem that prevented loading specific Jobs from the Database if
+  the following two conditions were met:
+    - no start/end time was specified
+    - the Job was older than a day
+
+### Changed
+
+- Improved Docs
+- Renamed `JobSearchFilter` to [pyslurm.db.JobFilter][]
+- Renamed `as_dict` function of some classes to `to_dict`
+
+## [23.2.1](https://github.com/PySlurm/pyslurm/releases/tag/v23.2.1) - 2023-05-18
+
+### Added
+
+- Classes to interact with the Job and Submission API
+    - [pyslurm.Job](https://pyslurm.github.io/23.2/reference/job/#pyslurm.Job)
+    - [pyslurm.Jobs](https://pyslurm.github.io/23.2/reference/job/#pyslurm.Jobs)
+    - [pyslurm.JobStep](https://pyslurm.github.io/23.2/reference/jobstep/#pyslurm.JobStep)
+    - [pyslurm.JobSteps](https://pyslurm.github.io/23.2/reference/jobstep/#pyslurm.JobSteps)
+    - [pyslurm.JobSubmitDescription](https://pyslurm.github.io/23.2/reference/jobsubmitdescription/#pyslurm.JobSubmitDescription)
+- Classes to interact with the Database Job API
+    - [pyslurm.db.Job](https://pyslurm.github.io/23.2/reference/db/job/#pyslurm.db.Job)
+    - [pyslurm.db.Jobs](https://pyslurm.github.io/23.2/reference/db/job/#pyslurm.db.Jobs)
+    - [pyslurm.db.JobStep](https://pyslurm.github.io/23.2/reference/db/jobstep/#pyslurm.db.JobStep)
+    - [pyslurm.db.JobFilter](https://pyslurm.github.io/23.2/reference/db/jobsearchfilter/#pyslurm.db.JobFilter)
+- Classes to interact with the Node API
+    - [pyslurm.Node](https://pyslurm.github.io/23.2/reference/node/#pyslurm.Node)
+    - [pyslurm.Nodes](https://pyslurm.github.io/23.2/reference/node/#pyslurm.Nodes)
+- Exceptions added:
+    - [pyslurm.PyslurmError](https://pyslurm.github.io/23.2/reference/exceptions/#pyslurm.PyslurmError)
+    - [pyslurm.RPCError](https://pyslurm.github.io/23.2/reference/exceptions/#pyslurm.RPCError)
+- [Utility Functions](https://pyslurm.github.io/23.2/reference/utilities/#pyslurm.utils)
+
+### Changed
+
+- Completely overhaul the documentation, switch to mkdocs
+- Rework the tests: Split them into unit and integration tests
+
+### Deprecated
+
+- Following classes are superseded by new ones:
+    - [pyslurm.job](https://pyslurm.github.io/23.2/reference/old/job/#pyslurm.job)
+    - [pyslurm.node](https://pyslurm.github.io/23.2/reference/old/node/#pyslurm.node)
+    - [pyslurm.jobstep](https://pyslurm.github.io/23.2/reference/old/jobstep/#pyslurm.jobstep)
+    - [pyslurm.slurmdb_jobs](https://pyslurm.github.io/23.2/reference/old/db/job/#pyslurm.slurmdb_jobs)
+
+## [23.2.0](https://github.com/PySlurm/pyslurm/releases/tag/v23.2.0) - 2023-04-07
+
+### Added
+
+- Support for Slurm 23.02.x ([f506d63](https://github.com/PySlurm/pyslurm/commit/f506d63634a9b20bfe475534589300beff4a8843))
+
+### Removed
+
+- `Elasticsearch` debug flag from `get_debug_flags`
+- `launch_type`, `launch_params` and `slurmctld_plugstack` keys from the
+  `config.get()` output
+- Some constants (mostly `ESLURM_*` constants that do not exist
+  anymore)
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -1,8 +1,8 @@
-include README.rst
+include README.md
 include COPYING.txt
 graft examples
 graft tests
-graft doc
+graft docs
 graft pyslurm/slurm
 graft pyslurm/pydefines
-include pyslurm/alps_cray.h
+recursive-include pyslurm *.pyx *.px[di] *.h
diff --git a/README.md b/README.md
@@ -1,18 +1,23 @@
-# PySlurm
+# <img src="logo.png" alt="PySlurm Logo">
 
-[![PySlurm](https://github.com/PySlurm/pyslurm/actions/workflows/pyslurm.yml/badge.svg?branch=main)](https://github.com/PySlurm/pyslurm/actions/workflows/pyslurm.yml)
+pyslurm is the Python client library for the [Slurm Workload Manager](https://slurm.schedmd.com)
 
-## Overview
-
-PySlurm is the Python client library for the [Slurm](https://slurm.schedmd.com) HPC Scheduler.
-
-## Prerequisites
+## Requirements
 
 * [Slurm](https://slurm.schedmd.com) - Slurm shared library and header files
 * [Python](https://www.python.org) - >= 3.6
 * [Cython](https://cython.org) - >= 0.29.30 but < 3.0
 
-This PySlurm branch is for the Slurm Major-Release 22.05
+This Version is for Slurm 23.02.x
+
+## Versioning
+
+In pyslurm, the versioning scheme follows the official Slurm versioning. The
+first two numbers (`MAJOR.MINOR`) always correspond to Slurms Major-Release,
+for example `23.02`.
+The last number (`MICRO`) is however not tied in any way to Slurms `MICRO`
+version, but is instead PySlurm's internal Patch-Level. For example, any
+pyslurm 23.02.X version should work with any Slurm 23.02.X release.
 
 ## Installation
 
@@ -21,130 +26,33 @@ By default, it is searched inside `/usr/include` for the Header files and in
 For Slurm installations in different locations, you will need to provide
 the corresponding paths to the necessary files.
 
-You can specify these Paths with environment variables, for example:
-
-```shell
-export SLURM_INCLUDE_DIR=/opt/slurm/22.05/include
-export SLURM_LIB_DIR=/opt/slurm/22.05/lib
-```
-
-Then you can proceed to install PySlurm, for example:
+You can specify those with environment variables (recommended), for example:
 
 ```shell
-pip install pyslurm==22.05.0
+export SLURM_INCLUDE_DIR=/opt/slurm/23.02/include
+export SLURM_LIB_DIR=/opt/slurm/23.02/lib
 ```
 
-Or by cloning the repository:
+Then you can proceed to install pyslurm, for example by cloning the Repository:
 
 ```shell
 git clone https://github.com/PySlurm/pyslurm.git && cd pyslurm
-python setup.py install
+scripts/build.sh
 
 # Or simply with pip
 pip install .
 ```
 
 Also see `python setup.py --help`
 
-## Release Versioning
-
-PySlurm's versioning scheme follows the official Slurm versioning. The first
-two numbers (MAJOR.MINOR) always correspond to Slurms Major-Release, for example
-`22.05`. The last number (MICRO) is however not tied in any way to Slurms
-MICRO version. For example, any PySlurm 22.05.X version should work with any
-Slurm 22.05.X release.
-
-## Documentation
-
-The API documentation is hosted at <https://pyslurm.github.io>.
-
-To build the docs locally, use [Sphinx](http://www.sphinx-doc.org) to generate
-the documentation from the reStructuredText based docstrings found in the
-pyslurm module once it is built:
-
-```shell
-cd doc
-make clean
-make html
-```
-
-## Testing
-
-PySlurm requires an installation of Slurm.
-
-### Using a Test Container
-
-To run tests locally without an existing Slurm cluster, `docker` and
-`docker-compose` is required.
-
-Clone the project:
-
-```shell
-git clone https://github.com/PySlurm/pyslurm.git
-cd pyslurm
-```
-
-Start the Slurm container in the background:
-
-```shell
-docker-compose up -d
-```
-
-The cluster takes a few seconds to start all the required Slurm services. Tail
-the logs:
-
-```shell
-docker-compose logs -f
-```
-
-When the cluster is ready, you will see the following log message:
-
-```text
-Cluster is now available
-```
-
-Press CTRL+C to stop tailing the logs. Slurm is now running in a container in
-detached mode. `docker-compose` also bind mounds the git directory inside the
-container at `/pyslurm` so that the container has access to the test cases.
-
-Install test dependencies:
-
-```shell
-pipenv sync --dev
-```
-
-Execute the tests inside the container:
-
-```shell
-pipenv run pytest -sv scripts/run_tests_in_container.py
-```
-
-When testing is complete, stop the running Slurm container:
-
-```shell
-docker-compose down
-```
-
-### Testing on an Existing Slurm Cluster
-
-You may also choose to clone the project and run tests on a node where Slurm is
-already compiled and installed:
-
-```shell
-git clone https://github.com/PySlurm/pyslurm.git
-cd pyslurm
-pip install .
-./scripts/configure.sh
-pipenv sync --dev
-pipenv run pytest -sv
-```
-
 ## Contributors
 
-PySlurm is made by [contributors like
+pyslurm is made by [contributors like
 you](https://github.com/PySlurm/pyslurm/graphs/contributors).
 
-## Help
+## Support
+
+Feel free to ask questions in the [GitHub
+Discussions](https://github.com/orgs/PySlurm/discussions)
 
-Ask questions on the [PySlurm Google
-Group](https://groups.google.com/forum/#!forum/pyslurm)
+Found a bug or you are missing a feature? Feel free to [open an Issue!](https://github.com/PySlurm/pyslurm/issues/new)
diff --git a/UPGRADE_C_API.rst b/UPGRADE_C_API.rst
@@ -51,13 +51,8 @@ Then, simply generate the header definitions like in this example:
     scripts/pyslurm_bindgen.py -D /directoy/with/slurm/headers > pyslurm/slurm/header.pxi
 
 The script outputs everything to `stdout`. Simply redirect the output to the file: :code:`pyslurm/slurm/header.pxi`.
+The headers should now be fully translated.
 
-Now, 99% of the work is done for generating the headers. For the 1% left, you now need to open the generated file, search for the two follwowing statements and comment them out:
-
-- `slurm_addr_t control_addr`
-- `phtread_mutex_t lock`
-
-The compiler will otherwise complain that these are incomplete type definitions.
 
 Compiling, Updating, Testing
 ----------------------------