Feature/antares in memory (#753)

Introduce "study mode". A way to run xpansion and problem generation with a study folder as parameter. In this case Antares will not be run as a stand-alone but as a library by problem generation. - New ADRs - Add a "new" data set for lpnamer E2E tests: SmallTestFiveCandidates, a copy of the example provided by Xpansion. - Refactor test_lpnamerEndToEnd.py : when possible prefer the use multiple @parametrize instead of writing the combinations in a list. - Refactor test_lpnamerEndToEnd.py : copy the test data in a temporary folder and works there. Prevent creating outputs in source code. - Add a test case for Study mode in test_lpnamerEndToEnd.py - Add a new E2E test: short memory - Refactor ProblemGenerationExeOptionsTest to test several combinations of mutually exclusive parameters - In SolverFactory.cpp call Init() on solver objects before returning them. Prevent temporal coupling for Xpress solver where a call to init() was mandatory. - Add Antares as a build dependency Missing pieces (every thing is open to discussion): - [ ] Handle Antares error #888 - [x] Update general data ini - [ ] Revert general data ini #889 - [x] Write ADR - [ ] Update sequence diagram #890 - [ ] Maybe update C4 #890 - [x] Use enum mode in LPFileExtractor #797 - [ ] find/define a way to log before lp dir exist (for antares error for exemple) #888 - [ ] Handle variables.txt directly at Lps level (SignificationMetierDesVariables) - [ ] Factorise variables name and "calculate" time step with week number on xpansion side (A faire dans Antares bien et vite) - [x] Cleanup python to only support memory mode (See ADR, we keep some support for now) - [x] antares named problèmes force mps writing. Peut etre pas besoin de l'option named sachant que les variables sont nommées dans la structure LP - [ ] Mettre à jour les docs: notamment comment build, dépendances, etc #893 - [ ] Investigate to use full run executable with --study option and use fullrun driver #892 - [x] Handle PBGen step with memory - [ ] Add entry in multisolver to rename a range of col/var #753 (comment) #891 - [ ] Run Antares in parallele mode Following other PR: - [ ] Don't overly rename problem in API mode --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: abdoulbari zakir <[email protected]> Co-authored-by: tbittar <[email protected]> Co-authored-by: Thomas Bittar <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AntaresSimulatorTeam · Sep 13, 2024 · 1cf286a · 1cf286a
1 parent 1280d2c
commit 1cf286a
Show file tree

Hide file tree

Showing 53 changed files with 4,718 additions and 4,386 deletions.
diff --git a/.github/workflows/build_centos7.yml b/.github/workflows/build_centos7.yml
@@ -70,6 +70,7 @@ jobs:
     container: 'antaresrte/xpansion-centos7:1.1.0'
     env:
       VCPKG_ROOT: ${{ github.workspace }}/vcpkg
+      ORTOOLS_DIR: $GITHUB_WORKSPACE/or-tools
 
     outputs:
       TGZ_NAME: ${{ steps.export_output.outputs.TGZ_NAME }}
@@ -83,7 +84,11 @@ jobs:
         if: github.event_name == 'release' && github.event.action == 'created'
         id: get_release
         uses: bruceadams/[email protected]
-
+
+      - name: Config OR-Tools URL
+        run: |
+          echo "ORTOOLS_URL=https://github.com/rte-france/or-tools-rte/releases/download/v9.10-rte1.2-test-4/ortools_cxx_centos7_static_sirius.zip" >> $GITHUB_ENV
+
       - uses: actions/checkout@v3 #Keep at 3
         with:
           submodules: true
@@ -102,7 +107,9 @@ jobs:
           antares-version: ${{needs.versions.outputs.antares-version}}
           os: centos7
           os-full-name: CentOS-7.9.2009
-
+          ortools-url: ${{env.ORTOOLS_URL}}
+          ortools-dir: ${{env.ORTOOLS_DIR}}
+
       - name: repo update
         run: |
           yum update
@@ -155,7 +162,7 @@ jobs:
           source /opt/rh/devtoolset-11/enable
           cmake -B _build -S . \
                  -DBUILD_TESTING=ON \
-                 -DCMAKE_PREFIX_PATH=${GITHUB_WORKSPACE}/deps \
+                 -DCMAKE_PREFIX_PATH="${GITHUB_WORKSPACE}/deps;${{env.ORTOOLS_DIR}}/install" \
                  -DCMAKE_BUILD_TYPE=Release \
                  -DCMAKE_INSTALL_PREFIX=_install \
                  -DBUILD_UI=ON \

diff --git a/.github/workflows/build_oracle8.yml b/.github/workflows/build_oracle8.yml
@@ -42,6 +42,9 @@ jobs:
     container: 'oraclelinux:8'
     needs: [ versions ]
 
+    env:
+      ORTOOLS_DIR: $GITHUB_WORKSPACE/or-tools
+
     outputs:
       zip_name: ${{ steps.zip_name.outputs.zip_name }}
       singlefile_name: ${{ steps.zip_name.outputs.singlefile_name }}
@@ -55,6 +58,10 @@ jobs:
           source /opt/rh/gcc-toolset-11/enable
           echo $PATH >> $GITHUB_PATH
 
+      - name: Config OR-Tools URL
+        run: |
+          echo "ORTOOLS_URL=https://github.com/rte-france/or-tools-rte/releases/download/v9.10-rte1.2-test-4/ortools_cxx_oraclelinux-8_static_sirius.zip" >> $GITHUB_ENV
+
       - run: |
           dnf module install -y nodejs:20/common
 
@@ -101,6 +108,8 @@ jobs:
           antares-version: ${{needs.versions.outputs.antares-version}}
           os: Oracle8
           os-full-name: OracleServer-8.10
+          ortools-url: ${{env.ORTOOLS_URL}}
+          ortools-dir: ${{env.ORTOOLS_DIR}}
 
       - name: vcpkg install
         run: |
@@ -128,7 +137,7 @@ jobs:
                  -DBUILD_TESTING=ON \
                  -DCMAKE_BUILD_TYPE=Release \
                  -DCMAKE_INSTALL_PREFIX=_install \
-                 -DCMAKE_PREFIX_PATH=${GITHUB_WORKSPACE}/deps \
+                 -DCMAKE_PREFIX_PATH="${GITHUB_WORKSPACE}/deps;${{env.ORTOOLS_DIR}}/install" \
                  -DBUILD_UI=OFF \
                  -DALLOW_RUN_AS_ROOT=ON \
                  -DVCPKG_TARGET_TRIPLET=x64-linux-release \

diff --git a/.github/workflows/build_ubuntu.yml b/.github/workflows/build_ubuntu.yml
@@ -25,6 +25,7 @@ jobs:
         os: [ ubuntu-20.04 ]
     env:
       VCPKG_BINARY_SOURCES: "clear;files,${{ github.workspace }}/vcpkg_cache,readwrite"
+      ORTOOLS_DIR: $GITHUB_WORKSPACE/or-tools
 
     outputs:
       TGZ_NAME: ${{ steps.export_output.outputs.TGZ_NAME }}
@@ -102,12 +103,18 @@ jobs:
           path: 'antares-version.json'
           key: 'antares_xpansion_version'
 
+      - name: Config OR-Tools URL
+        run: |
+          echo "ORTOOLS_URL=https://github.com/rte-france/or-tools-rte/releases/download/v9.10-rte1.2-test-4/ortools_cxx_ubuntu-20.04_static_sirius.zip" >> $GITHUB_ENV
+
       - name: Download pre-compiled librairies
         uses: ./.github/workflows/download-extract-precompiled-libraries-tgz
         with:
           antares-version: ${{steps.antares-version.outputs.result}}
           os: ${{matrix.os}}
           os-full-name: Ubuntu-20.04
+          ortools-url: ${{env.ORTOOLS_URL}}
+          ortools-dir: ${{env.ORTOOLS_DIR}}
 
       - name: vcpkg install
         run: |
@@ -136,7 +143,7 @@ jobs:
                 -DBUILD_TESTING=ON \
                 -DCMAKE_BUILD_TYPE=Release \
                 -DCMAKE_INSTALL_PREFIX=_install \
-                -DCMAKE_PREFIX_PATH=${{ github.workspace }}/deps \
+                -DCMAKE_PREFIX_PATH="${{ github.workspace }}/deps;${{env.ORTOOLS_DIR}}/install" \
                 -DBUILD_UI=ON  \
                 -DVCPKG_TARGET_TRIPLET=x64-linux-release \
                 -DCMAKE_TOOLCHAIN_FILE=vcpkg/scripts/buildsystems/vcpkg.cmake

diff --git a/.github/workflows/build_windows.yml b/.github/workflows/build_windows.yml
@@ -27,6 +27,7 @@ jobs:
       # Indicates the location of the vcpkg as a Git submodule of the project repository.
       VCPKG_ROOT: ${{ github.workspace }}/vcpkg
       VCPKG_BINARY_SOURCES: "clear;files,${{ github.workspace }}/vcpkg_cache,readwrite"
+      ORTOOLS_DIR: $GITHUB_WORKSPACE/or-tools
 
     outputs:
       zip_name: ${{ steps.zip_name.outputs.zip_name }}
@@ -98,17 +99,25 @@ jobs:
           # Allows to restore a cache when deps have only partially changed (like adding a dependency)
           restore-keys: vcpkg-cache-windows-
 
+      - name: Config OR-Tools URL
+        shell: bash
+        run: |
+          echo "ORTOOLS_URL=https://github.com/rte-france/or-tools-rte/releases/download/v9.10-rte1.2-test-4/ortools_cxx_windows-latest_static.zip" >> $GITHUB_ENV
+
+      - run: echo ${{env.ORTOOLS_URL}}
       - name: Download pre-compiled librairies
         uses: ./.github/workflows/download-extract-precompiled-libraries-zip
         with:
           antares-version: ${{steps.antares-version.outputs.result}}
           os: ${{matrix.os}}
+          ortools-url: ${{env.ORTOOLS_URL}}
+          ortools-dir: ${{env.ORTOOLS_DIR}}
 
       - name: Configure
         shell: bash
         run: >
           cmake -B _build -S .
-          -DCMAKE_PREFIX_PATH="${{github.workspace}}/rte-antares-${{steps.antares-version.outputs.result}}-installer-64bits"
+          -DCMAKE_PREFIX_PATH="${{github.workspace}}/rte-antares-${{steps.antares-version.outputs.result}}-installer-64bits;${{env.ORTOOLS_DIR}}/install;${{github.workspace}}/windows-latest_sirius-solver-install"
           -DBUILD_TESTING=ON
           -DCMAKE_BUILD_TYPE=Release
           -DCMAKE_TOOLCHAIN_FILE="${{env.VCPKG_ROOT}}/scripts/buildsystems/vcpkg.cmake"

diff --git a/.github/workflows/download-extract-precompiled-libraries-tgz/action.yml b/.github/workflows/download-extract-precompiled-libraries-tgz/action.yml
@@ -14,17 +14,60 @@ inputs:
     description: 'build type'
     required: false
     default: 'Release'
-  variant:
-    description: 'extra qualifiers'
-    required: false
-    default: ""
+  ortools-url:
+    description: 'URL for OR-Tools'
+    required: true
+  ortools-dir:
+    description: 'Destination directory for OR-Tools'
+    required: true
 runs:
   using: "composite"
   steps:
-    - id: download-extract
+    - name: download-extract antares
       shell: bash
       run: |
         mkdir -p deps
-        wget https://github.com/AntaresSimulatorTeam/Antares_Simulator/releases/download/v${{inputs.antares-version}}/antares-${{inputs.antares-version}}-${{inputs.os-full-name}}${{inputs.variant}}.tar.gz
-        tar -xvf antares-${{inputs.antares-version}}-${{inputs.os-full-name}}${{inputs.variant}}.tar.gz -C deps --strip-components=1
-        rm -rf antares-${{inputs.antares-version}}-${{inputs.os-full-name}}${{inputs.variant}}.tar.gz
+        wget https://github.com/AntaresSimulatorTeam/Antares_Simulator/releases/download/v${{inputs.antares-version}}/antares-${{inputs.antares-version}}-${{inputs.os-full-name}}.tar.gz
+        tar -xvf antares-${{inputs.antares-version}}-${{inputs.os-full-name}}.tar.gz -C deps --strip-components=1
+        rm -rf antares-${{inputs.antares-version}}-${{inputs.os-full-name}}.tar.gz
+
+    - name: Download & extract OR-Tools
+      shell: bash
+      run: |
+        mkdir -p  ${{inputs.ortools-dir}}
+        cd ${{inputs.ortools-dir}}
+        wget ${{inputs.ortools-url}} -O ortools.zip
+        unzip ortools.zip
+        rm ortools.zip
+
+    #It's simpler to split each case than to try and define a general case and map OS names with Sirius asset names
+    - name: download-extract sirius centos
+      shell: bash
+      if: ${{ inputs.os-full-name == 'Centos-7.9.2009'}}
+      run: |
+        wget https://github.com/rte-france/sirius-solver/releases/download/antares-integration-v1.5/centos-7_sirius-solver.zip
+        unzip centos-7_sirius-solver.zip
+        cp -r centos-7_sirius-solver-install/* deps
+        rm -rf centos-7_sirius-solver.zip
+        rm -rf centos-7_sirius-solver-install
+        
+
+    - name: download-extract sirius ubuntu
+      shell: bash
+      if: ${{ inputs.os-full-name == 'Ubuntu-20.04' }}
+      run: |
+        wget https://github.com/rte-france/sirius-solver/releases/download/antares-integration-v1.5/ubuntu-20.04_sirius-solver.zip
+        unzip ubuntu-20.04_sirius-solver.zip
+        cp -r ubuntu-20.04_sirius-solver-install/* deps
+        rm -rf ubuntu-20.04_sirius-solver.zip
+        rm -rf ubuntu-20.04_sirius-solver-install
+
+    - name: download-extract sirius oracle 8
+      shell: bash
+      if: ${{ inputs.os-full-name == 'OracleServer-8.10'}}
+      run: |
+        wget https://github.com/rte-france/sirius-solver/releases/download/antares-integration-v1.5/oraclelinux-8_sirius-solver.zip
+        unzip oraclelinux-8_sirius-solver.zip
+        cp -r oraclelinux-8_sirius-solver-install/* deps
+        rm -rf oraclelinux-8_sirius-solver.zip
+        rm -rf oraclelinux-8_sirius-solver-install
diff --git a/.github/workflows/download-extract-precompiled-libraries-zip/action.yml b/.github/workflows/download-extract-precompiled-libraries-zip/action.yml
@@ -15,13 +15,35 @@ inputs:
     description: 'extra qualifiers'
     required: false
     default: ""
+  ortools-url:
+    description: 'URL for OR-Tools'
+    required: true
+  ortools-dir:
+    description: 'Destination directory for OR-Tools'
+    required: true
 
 runs:
   using: "composite"
-  steps: 
-  - id: download-extract
-    shell: bash
-    run: |
+  steps:
+    - id: download-extract
+      shell: bash
+      run: |
            wget https://github.com/AntaresSimulatorTeam/Antares_Simulator/releases/download/v${{inputs.antares-version}}/rte-antares-${{inputs.antares-version}}-installer-64bits.zip
            unzip rte-antares-${{inputs.antares-version}}-installer-64bits.zip
            rm -rf rte-antares-${{inputs.antares-version}}-installer-64bits.zip
+
+    - name: Download & extract OR-Tools
+      shell: bash
+      run: |
+        mkdir "${{inputs.ortools-dir}}"
+        cd "${{inputs.ortools-dir}}"
+        wget ${{inputs.ortools-url}} -O ortools.zip
+        unzip ortools.zip
+        rm ortools.zip
+
+    - name: download-extract sirius
+      shell: bash
+      run: |
+        wget https://github.com/rte-france/sirius-solver/releases/download/antares-integration-v1.5/windows-latest_sirius-solver.zip
+        unzip windows-latest_sirius-solver.zip
+        rm -rf windows-latest_sirius-solver.zip
diff --git a/.github/workflows/sonarcloud.yml b/.github/workflows/sonarcloud.yml
@@ -19,6 +19,7 @@ jobs:
         os: [ ubuntu-20.04 ]
     env:
       VCPKG_BINARY_SOURCES: "clear;files,${{ github.workspace }}/vcpkg_cache,readwrite"
+      ORTOOLS_DIR: $GITHUB_WORKSPACE/or-tools
 
     steps:
 
@@ -81,12 +82,19 @@ jobs:
           path: 'antares-version.json'
           key: 'antares_xpansion_version'
 
+      - name: Config OR-Tools URL
+        shell: bash
+        run: |
+          echo "ORTOOLS_URL=https://github.com/rte-france/or-tools-rte/releases/download/v9.10-rte1.2-test-4/ortools_cxx_ubuntu-20.04_static_sirius.zip" >> $GITHUB_ENV
+
       - name: Download pre-compiled librairies
         uses: ./.github/workflows/download-extract-precompiled-libraries-tgz
         with:
           antares-version: ${{steps.antares-version.outputs.result}}
           os: ${{matrix.os}}
           os-full-name: Ubuntu-20.04
+          ortools-url: ${{env.ORTOOLS_URL}}
+          ortools-dir: ${{env.ORTOOLS_DIR}}
 
       - name: Init submodule
         run: |
@@ -118,7 +126,7 @@ jobs:
           -DBUILD_TESTING=ON \
           -DBUILD_antares_solver=OFF \
           -DCMAKE_BUILD_TYPE=Release \
-          -DCMAKE_PREFIX_PATH=${{ github.workspace }}/deps \
+          -DCMAKE_PREFIX_PATH="${{ github.workspace }}/deps;${{env.ORTOOLS_DIR}}/install" \
           -DCMAKE_INSTALL_PREFIX=_install \
           -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
           -DVCPKG_TARGET_TRIPLET=x64-linux-release \