From_step acting as a filter #1981
Conversation
…ers-fix' Docs/jobs key in defining workflows See merge request es/autosubmit!498
|
In GitLab by @dbeltrankyl on Oct 2, 2024, 16:45 added 1 commit
|
|
In GitLab by @dbeltrankyl on Oct 2, 2024, 16:45 Added a unit test since the other related tests are not yet pytest |
|
In GitLab by @dbeltrankyl on Oct 2, 2024, 16:47 added 1 commit
|
|
In GitLab by @dbeltrankyl on Oct 2, 2024, 16:48 requested review from @kinow |
|
In GitLab by @dbeltrankyl on Oct 3, 2024, 11:46 added 1 commit
|
|
In GitLab by @dbeltrankyl on Oct 3, 2024, 11:51 Hello @kinow I've added two additional things:
This should only trigger if the job section depends on itself otherwise it doesn't make sense JOBS:
A:
DEPENDENCIES:
A:
# or A-1:I've tested it with the regression test and found no issues ( commented here https://earth.bsc.es/gitlab/es/autosubmit/-/issues/1430 to implement unit tests that cover all cases ) I'll add this as module autosubmit-dev so it can be tested Related to #1426 @fyi @ainagaya, @lauherrero |
|
In GitLab by @dbeltrankyl on Oct 3, 2024, 12:00 added 1 commit
|
|
In GitLab by @dbeltrankyl on Oct 3, 2024, 12:34 Had an issue on deployment.. Asked Albert, but seems a filesystem issue |
|
In GitLab by @dbeltrankyl on Oct 3, 2024, 14:22 There is no space left due to some modifications taking place. According to Albert, there will be space tomorrow, So tomorrow I'll upload it |
|
In GitLab by @kinow on Oct 4, 2024, 10:31 Will have to review this one next week (trying to finish a review for @franra9, and also working on the deliverable). But I think we might need something in our docs for this? Not sure if we already have an example somewhere (if so, then that won't be needed here I think). Thanks @dbeltrankyl ! |
|
In GitLab by @dbeltrankyl on Oct 4, 2024, 12:12 Tried again after the maintenance, but the same still happens, waiting for Albert input |
|
In GitLab by @dbeltrankyl on Oct 4, 2024, 12:15 It is already documented on https://autosubmit.readthedocs.io/en/master/userguide/defining_workflows/index.html#start-conditions, although the example only shows the most simple way of doing it. Maybe once the workflow is fully done, we can add it as an additional example? Explaining for what the workflow is and well, like a how-to recipe |
|
In GitLab by @kinow on Oct 4, 2024, 12:24
Good idea. Maybe we should create an issue in this milestone so we don't forget too. |
|
In GitLab by @dbeltrankyl on Oct 7, 2024, 08:33 Tried again, They are looking at it, seems that the TMP dir is not mounted for unknown reasons ( and easybuild relies on making the temporal stuff, in the tmp dir ). Albert will warn me when it is solved |
|
In GitLab by @dbeltrankyl on Oct 8, 2024, 11:23 Hello @kinow I tried to deploy the version as Albert told me that tmp issue was ( temporally, they will do a reboot afternoon) solved, and I had an issue with this (non-related to this merge): ImportError: cannot import name 'LiteralString' from 'typing_extensions' (/shared/earth/easybuild/rocky/8.4/x86_64/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages/typing_extensions.py)https://earth.bsc.es/gitlab/es/autosubmit/-/blame/master/autosubmit/monitor/diagram.py#L39 Do you know any alternative? |
|
In GitLab by @kinow on Oct 8, 2024, 13:41 Can't we update I had a look at 4.9.0 in the hub, and it is loading The
|
|
In GitLab by @dbeltrankyl on Oct 8, 2024, 15:47 Hello @kinow , Sorry for the delayed answer Shouldn't it be taking the typing-extensions from the site-packages of autosubmit? When I perform a module load autosubmit/4.1.11.dev0-d6f0d798-foss-2021b-Python-3.9.6, pip is showing. typing==3.7.4.3
typing_extensions==4.9.0Which should be correct but.. autosubmit -v is crashing because it seems to be taking the one in /shared/earth/easybuild/rocky/8.4/x86_64/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages/typing_extensions.py And I can see the egg for the 4.9.0 version inside: /shared/earth/easybuild/rocky/8.4/x86_64/software/autosubmit/4.1.11.dev0-d6f0d798-foss-2021b-Python-3.9.6/lib/python3.9/site-packages/ |
|
In GitLab by @kinow on Oct 9, 2024, 08:05
I thought it should.
The VPN is not working for me right now, but I will try again in a few hours and try the dev module you shared. Thanks Dani! |
|
In GitLab by @kinow on Oct 9, 2024, 08:43 I think the module is not yet available to every user?
Yes, that looks right. Do you know if I need to run some command in order to load this 4.1.11 you created? I tried a couple of |
|
In GitLab by @dbeltrankyl on Oct 9, 2024, 09:20 Hello @kinow , I'm sorry. I tried again yesterday and used EasyBuild's safety check feature, which removes the module if it is not okay. I rebuild it now so that you can use it And if you have eadmin access you can rebuild the module with
The code is the following. module: |
|
In GitLab by @dbeltrankyl on Oct 9, 2024, 09:26 I'm tagging @albertvilabsc and @stamenminkov in case they have any idea. Please look only from this comment and the replies: https://earth.bsc.es/gitlab/es/autosubmit/-/merge_requests/493#note_311051 Strangely, this should have happened before with other Autosubmit modules version, so maybe it is something related to the hub updates? |
|
In GitLab by @stamenminkov on Oct 9, 2024, 10:29 Hi @dbeltrankyl @kinow, I'll check this as soon as I can! Cheers, Stamen |
|
In GitLab by @kinow on Oct 9, 2024, 13:08 It's strange. Looking at the @dbeltrankyl , I had a look at the easybuild.pth file created for easybuild in the AS directory. So these should be loaded, which looks correct. Then the [bdepaula@ bsceshub02 ~]$ cat /shared/earth/easybuild/rocky/8.4/x86_64/software/autosubmit/4.1.11.dev0-d6f0d798-foss-2021b-Python-3.9.6/lib/python3.9/site-packages/setuptools.pth
/shared/earth/easybuild/rocky/8.4/x86_64/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages@stamenminkov , thanks for looking into this issue.
It's a dependency defined in our We use it in some new code, where we simply
Good to know, but I think ideally we should be able to make it so that Autosubmit module uses the dependencies it is declaring in its project configuration :fingers_crossed: without relying on other modules (except Python).
Hmmm, this is a good point! Not sure though, so that will go to @dbeltrankyl :slight_smile: Thanks! |
|
In GitLab by @dbeltrankyl on Oct 9, 2024, 13:31 Thanks to both to looking at this
I tried to use_pip once because I had issues with some libraries, but in the end, it was solved by changing some stuff in our setup.py. I'm afraid I didn't investigate further. To be honest, I inherited this recipe from Domingo/Kim when I arrived at the BSC and have used it forever; if there is any way of doing it with fewer issues, I'm open to changing it. About the I've temporarily commented it so the module "works"; otherwise, the EB shows an "error," and the module won't be available. I did that so Bruno could perform a module load and debug it
In the last module loaded, I've changed that file locally so the dependency is forced, instead of having <=: typing==3.7.4.3
typing_extensions==4.9.0But unfortunately, it was still taking another version. |
|
In GitLab by @kinow on Oct 9, 2024, 13:44 Maybe we could try again now with |
|
In GitLab by @dbeltrankyl on Oct 9, 2024, 13:50 Just tried, now I remember why it is disabled When it is set, it doesn't download any dependency, but maybe it does because I'm missing other stuff in the recipe. Edit: Ah yes, when use_pip is true, I have to add each package manually and I would like to use the setup.py defined ones Edit2: here: |
|
In GitLab by @stamenminkov on Oct 10, 2024, 12:51 Hi @dbeltrankyl @kinow, I'm still digging into this, it is really not clear at all. Also I'm a bit confused by this type of internal installations as most of the times EB goes by either dependencies or extensions. I've done a test installation and now going through the installations logs, although for now not finding anything out of order.
How do you obtain those @dbeltrankyl ? I'm not getting the same, for me it defers directly to
Indeed, but for some reason it's not loading at all the packages from the autosubmit path.
I'm not sure if the order plays anything here, as even if Command examples
Usually
Yep, that's indeed true. In a There's an alternative though, by using a I'm still pending to compare the build logs between using Stamen |
|
In GitLab by @dbeltrankyl on Oct 10, 2024, 13:08 Thanks for looking at it
If you copied the same .eb that I used the last time, I left the
I've rebuilt the package now without the use_pip option, so you should be able to see it with: pip list output |
|
In GitLab by @dbeltrankyl on Oct 10, 2024, 13:09 And the autosubmit -v output takes /shared/earth/easybuild/rocky/8.4/x86_64/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages/typing_extensions.py For some reason |
|
In GitLab by @stamenminkov on Oct 10, 2024, 15:54 Hi @dbeltrankyl,
The tests I did were without the Just checked with the module you've most recently recompiled, and I'm getting the exact same as the other tests ( 😕 This is really confusing. Is it possible that you might have anything in your Thanks! Stamen |
|
In GitLab by @dbeltrankyl on Oct 10, 2024, 16:24 Hello @stamenminkov ,
Ok, sorry for that noise I wasn't getting it with eadmin and checking my files I saw that I had some .local packages which included the typing extensions, after removing it I'm getting the same list This list shows autosubmit installed packages, so the setup.py was triggered. However, it still doesn't explain why it is ignoring these settings: Just to be sure that this change was in place... I've checked the contents of the tar.gz #!/usr/bin/env python3
# Copyright 2014 Climate Forecasting Unit, IC3
# This file is part of Autosubmit.
# Autosubmit is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# Autosubmit is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with Autosubmit. If not, see <http://www.gnu.org/licenses/>.
from os import path
from setuptools import setup
from setuptools import find_packages
here = path.abspath(path.dirname(__file__))
# Get the version number from the relevant file
with open(path.join(here, 'VERSION')) as f:
version = f.read().strip()
install_requires = [
'xlib==0.21',
'setuptools<=68.2.2',
'bscearth.utils<=0.5.2',
'requests<=2.31.0',
'networkx<=2.6.3',
'portalocker<=2.7.0',
'paramiko<=3.4',
'pyparsing==3.1.1',
'matplotlib<=3.8.3',
'packaging<=23.2',
'ruamel.yaml.clib<=0.2.8',
'typing_extensions==4.9.0',
'typing==3.7.4.3',
'psutil<=5.6.1',
'networkx<=2.6.3',
'py3dotplus==1.1.0',
'matplotlib<=3.8.3',
'numpy<2',
'ruamel.yaml==0.17.21',
'rocrate==0.*',
'autosubmitconfigparser==1.0.67',
'configparser',
'setproctitle',
# 'sqlalchemy[mypy]' # TODO: pending Postgres MR
]
pg_require = [
'psycopg2'
]
docs_require = [
'livereload',
'pydata-sphinx-theme==0.15.*',
'sphinx==5.*',
'sphinx-autobuild==2021.3.*',
'sphinx_rtd_theme',
'sphinx-reredirects==0.1.*'
]
tests_require = [
'pytest==8.2.*',
'pytest-cov',
'pytest-mock',
# 'testcontainers' # TODO: pending Postgres MR
]
# You can add more groups, e.g. all_require = tests_require + graph_require, etc...
all_require = tests_require + pg_require
extras_require = {
'postgres': pg_require,
'tests': tests_require,
'docs': docs_require,
'all': all_require
}
setup(
name='autosubmit',
license='GNU GPL v3',
platforms=['GNU/Linux Debian'],
version=version,
description='Autosubmit is a Python-based workflow manager to create, manage and monitor complex tasks involving different substeps, such as scientific computational experiments. These workflows may involve multiple computing systems for their completion, from HPCs to post-processing clusters or workstations. Autosubmit can orchestrate all the tasks integrating the workflow by managing their dependencies, interfacing with all the platforms involved, and handling eventual errors.',
long_description=open('README_PIP.md').read(),
author='Daniel Beltran Mora',
author_email='daniel.beltran@ bsc.es',
url='http://www.bsc.es/projects/earthscience/autosubmit/',
download_url='https://earth.bsc.es/wiki/doku.php?id=tools:autosubmit',
keywords=['climate', 'weather', 'workflow', 'HPC'],
install_requires=install_requires,
extras_require=extras_require,
classifiers=[
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"License :: OSI Approved :: GNU General Public License (GPL)",
"Operating System :: POSIX :: Linux",
],
packages=find_packages(),
include_package_data=True,
package_data={'autosubmit': [
'autosubmit/config/files/autosubmit.conf',
'autosubmit/config/files/expdef.conf',
'autosubmit/database/data/autosubmit.sql',
'README',
'CHANGELOG',
'VERSION',
'LICENSE',
'docs/autosubmit.pdf'
]
},
scripts=['bin/autosubmit']
)And the .eb used has sources = ['%(name)s-%(version)s.tar.gz'] So it should force that version of the typing and ignore the one installed by the system, right? Edit: Also, it doesn't explain why it is not working with .local packages so my confusion level is more or less the same as before |
|
In GitLab by @dbeltrankyl on Oct 10, 2024, 16:27 Btw, the pip list command while using eadmin is much (much) slower than when I prompt it with my user not sure if you're aware or if it is expected |
|
In GitLab by @stamenminkov on Oct 11, 2024, 13:04 Hi @dbeltrankyl @kinow,
I noticed as well 😕 could be the
Indeed, in the build log I can see the I've also tested the build without the following line: which brings back the sanity check for extensions, and then the build fails (for
Yes, it should, but I suppose it's more about how the paths are treated, that additional Anyway, bringing some news here: I've discussed that with the EB developers and they advise to avoid that type of building modules with EB, as it can visibly mess up the environment. That's why I'll try and rewrite that recipe entirely and will turn it into a Will let you know again soon! Thanks, Stamen |
|
In GitLab by @kinow on Oct 11, 2024, 13:10 Thank you Stamen! If we get it working, then maybe we can use it to close this issue/MR https://earth.bsc.es/gitlab/es/autosubmit/-/issues/1394 |
|
In GitLab by @dbeltrankyl on Oct 11, 2024, 13:33 Thanks @stamenminkov ! |
|
In GitLab by @ainagaya on Oct 14, 2024, 11:42 Hi @dbeltrankyl :) In the meantime, I wanted to test this locally in my laptop. This is what I did:
Is this the right way to do it? I think it is my first time doing this and I'm not sure how I can check if I did it right or not (I can also wait but I wanted to do some progress this week in auto-phenomena). Thanks!! |
|
In GitLab by @dbeltrankyl on Oct 14, 2024, 11:54 Hello @ainagaya Usually, I do it with pycharm, and it installs the packages automatically based on the current setup.py in a single venv Without an ide I would do the same as you. Then, if you have never used Autosubmit before on that machine, you have to perform: |
|
In GitLab by @dbeltrankyl on Oct 14, 2024, 11:58 Alternatively, instead of autosubmit configure, you can write this (change $your_user) /etc/autosubmitrc [database]
path = /home/$your_user/autosubmit
filename = autosubmit.db
[local]
path = /home/$your_user/autosubmit
[globallogs]
path = /home/$your_user/autosubmit/logs
[structures]
path = /home/$your_user/autosubmit/metadata/structures
[historicdb]
path = /home/$your_user/autosubmit/metadata/data
[historiclog]
path = /home/$your_user/autosubmit/metadata/logs
[config]
log_recovery_timeout = 120
And |
|
In GitLab by @ainagaya on Oct 14, 2024, 12:59 Thanks Dani!! I think I configured everything fine (I did configure and install). I'm a bit lost with the development (let me know if you prefer me to report somewhere else, but I think that it can be still something to fix in this MR, that's why I'm posting it here). This is my joblist: JOBS:
DOWNLOAD_madrid_traffic:
FILE: templates/download-traffic.sh,templates/downloads/madrid_traffic.yml
PLATFORM: local
FILE_TO_DOWNLOAD: madrid_traffic
RUNNING: chunk
DEPENDENCIES:
DOWNLOAD_madrid_traffic:
CHUNKS_FROM:
"1":
STATUS: "RUNNING"
FROM_STEP: "1"
"2":
STATUS: "RUNNING"
FROM_STEP: "2"In the And when I try to run: (I obviously tried again, also with the create -f, deleted the content of the pkl folder and started again) With a "normal" joblist autosubmit submits the jobs, that's why I think it is something in the joblist I posted above. Let me know if you need some more information, the experiment is on my laptop but I guess that I can do the same steps and run it in the hub if needed (?). Thanks a lot! |
|
In GitLab by @dbeltrankyl on Oct 14, 2024, 13:34 Hello @ainagaya I don't find any of these issues, Do you have sqlite3 installed? I thought it was already deployed on the machine. Could you perform If it says 2.40, you need to update it. A list of pre-requisites can be found in docs https://autosubmit.readthedocs.io/en/master/installation/index.html#
About your jobs.yml, I think you're missing one job. You're setting the download to depend upon himself( each chunk is depending on all the chunks of the section that is why it doesn't work well), which I think is a wrong approach unless I misunderstood something please correct me if not I did an example the other day ( it is in the issue ) JOBS:
DOWNLOAD_madrid_traffic:
FILE: templates/download_traffic.sh,templates/downloads/madrid_traffic.yml
PLATFORM: local
FILE_TO_DOWNLOAD: madrid_traffic
RUNNING: once
DO_WORK:
file: test.sh
platform: local
running: chunk
dependencies:
DOWNLOAD_madrid_traffic:
status: "RUNNING"
CHUNKS_FROM:
"1":
FROM_STEP: "1"
"2":
FROM_STEP: "2"
"3":
FROM_STEP: "3"
"4":
FROM_STEP: "4"
In this example, the download template will run just once, and its task is to download all the necessary stuff. Then another job will depend on a certain download checkpoint to begin to start I can also deploy the module in the climate-dt meanwhile if it is better for you. |
|
In GitLab by @stamenminkov on Oct 16, 2024, 17:03 Hi @dbeltrankyl @kinow, This whole week I'm attending a workshop, so I have limited time, nevertheless I'm working on updating the code! I've however encountered some issues on the hub02, so hoping to get over these soon, I'll let you know when this advances! Thanks, Stamen |
|
In GitLab by @dbeltrankyl on Oct 17, 2024, 10:30 Thanks @stamenminkov ! |
|
In GitLab by @stamenminkov on Oct 18, 2024, 17:10 Hi @dbeltrankyl @kinow , Glad to let you know that this is now sorted! 🎉 🎉 😃 It has taken me some time, as I've had to clean the environments in the hub (some The modulename is the expected one: And here's the new recipe: autosubmit-4.1.11.dev0-d6f0d798-foss-2021b-Python-3.9.6.ebautosubmit-4.1.11.dev0-d6f0d798-foss-2021b-Python-3.9.6.eb Please test this accordingly to see if all the In my case, So, here are a few important points I need to highlight regarding this new config file, especially in case you might require to update it somewhen in the future:
These are the main things for now, if you find anything else, please let me know!! Thanks! Cheers, Stamen |
|
In GitLab by @dbeltrankyl on Oct 21, 2024, 09:59 Hello @stamenminkov! Thanks!
Yep, that is too slow, but it always was the case in all installations in the hub It takes (base) dbeltran@ bsces107894:~$ time autosubmit -v
4.1.10
real 0m1.719s
user 0m1.910s
sys 0m1.033sIn my laptop
Yes, thanks known issue!
autosubmitconfigparser-1.0.67_fix-pathlib-dependence.patch autosubmitconfigparser-1.0.67-foss-2021b-Python-3.9.6.eb Thanks, I'll use variations of this .eb to deploy newer autosubmitconfigparser versions ( it is required by the testing suite )
That seems more convenient, so to deploy official versions in the commit directive I just put a tag, no?
True, I did not notice it. I'll fix it, thanks! The slowness affects all commands; I believe that the difference in the speed is in the process of importing all libraries; after that, the command is more or less as fast. In local: Other than that, the module works ok, thanks! |
|
In GitLab by @dbeltrankyl on Oct 21, 2024, 10:01 Hello @ainagaya In the end, I think you don't use the stuff added in this branch, no? (as_checkpoint) If you tested it, did the developments in this branch work? |
|
In GitLab by @stamenminkov on Oct 21, 2024, 11:02 Hi @dbeltrankyl, That's awesome, glad that it is working properly!!! :raised_hands_tone3: :raised_hands_tone3: :raised_hands_tone3: Regarding the slowness, it makes sense in that case to be caused by the hub itself, hope this to be sorted really really soon with the hubs replication.
Thanks! Please do use that recipe, however take into account that the patch is only for version By the way, the default fetch for EB is first from the source directories on each machine and then via PyPI (for Also, if a new patch needs to be applied in the future, please let me know!
For tags you can directly substitute the Thanks, Stamen |
|
In GitLab by @ainagaya on Oct 21, 2024, 11:13 Hi @dbeltrankyl , we used the |
|
In GitLab by @dbeltrankyl on Oct 21, 2024, 11:16 Thanks @ainagaya The writing part is what this branch is testing, so I'll merge the branch |
|
In GitLab by @dbeltrankyl on Oct 21, 2024, 11:17 Thanks! I'll bookmark that website just in case |
|
In GitLab by @dbeltrankyl on Oct 21, 2024, 11:18 mentioned in commit 56ffbf8 |
In GitLab by @dbeltrankyl on Oct 2, 2024, 16:03
This fixes an issue where the from_step filter acts as a filter even if all the filters are "natural" or not set.
Pending the test ( no time today)#1426 check this issue for info