Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External files management #210

Draft
wants to merge 205 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
205 commits
Select commit Hold shift + click to select a range
0b67809
Divided glidein_startup.sh in modules
terranovafr Aug 9, 2022
7b0f554
Added debug option
terranovafr Aug 9, 2022
4569245
Added manual installation script for testing
terranovafr Aug 9, 2022
b9254e4
Modified menu
terranovafr Aug 9, 2022
39265e5
Solved error printf
terranovafr Aug 9, 2022
31ce38c
Modified manual installation
terranovafr Aug 9, 2022
e7d47d7
Updated functions' comments
terranovafr Aug 10, 2022
bc9a8ee
Fixed some comments
terranovafr Aug 10, 2022
8d55fb0
Marked some useless global variables as local
terranovafr Aug 10, 2022
fdb4a35
Marked more global variables as local
terranovafr Aug 10, 2022
11bc513
Changed debug option
terranovafr Aug 10, 2022
44a88aa
Modified print_header error handling
terranovafr Aug 10, 2022
8216550
Modified print_header error handling
terranovafr Aug 10, 2022
917c2f4
Handled exit status
terranovafr Aug 10, 2022
35b4872
Added first xml change
terranovafr Aug 11, 2022
9f5461c
Added first xml change
terranovafr Aug 11, 2022
fe68b84
Added first xml change
terranovafr Aug 11, 2022
f816366
Added first xml change
terranovafr Aug 11, 2022
1e4af33
Added first xml change
terranovafr Aug 11, 2022
0a2cf42
Added xml changes
terranovafr Aug 11, 2022
566df04
Added xml changes
terranovafr Aug 11, 2022
e669a0f
Added spaces support to create_xml method
terranovafr Aug 11, 2022
5adba35
Substitution of two methods with the call to the create_xml method
terranovafr Aug 11, 2022
5ae02ef
Substitution of two methods with the call to the create_xml method
terranovafr Aug 11, 2022
db70155
Substitution of the methods of utils_http with the call to the create…
terranovafr Aug 11, 2022
21e2aa3
Substitution of the methods of utils_http with the call to the create…
terranovafr Aug 11, 2022
d1b8ead
Substitution of the methods of utils_http with the call to the create…
terranovafr Aug 11, 2022
64f6c5a
Completed substituion of xml methods
terranovafr Aug 11, 2022
880f7dd
Completed substituion of xml methods
terranovafr Aug 11, 2022
49845aa
Methods divided among glidein startup specific methods and general me…
terranovafr Aug 11, 2022
3857ec5
Moved menu of options
terranovafr Aug 11, 2022
4fee000
Moved menu of options
terranovafr Aug 11, 2022
cf63cf9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 11, 2022
5f76a4f
Header and license added to each script file
terranovafr Aug 11, 2022
6c22dad
Merge remote-tracking branch 'origin/master'
terranovafr Aug 11, 2022
1364e8c
Delete manual_installation_wget.sh
terranovafr Aug 11, 2022
40c6e21
Added more controls to the parse_option method and more detailed comm…
terranovafr Aug 12, 2022
891eec6
Merge remote-tracking branch 'origin/master'
terranovafr Aug 12, 2022
e656875
Added more controls to the parse_option method and more detailed comm…
terranovafr Aug 12, 2022
8a4fe6d
Implemented the bats tests of the glidein_cleanup.sh, utils_gs_io.sh,…
terranovafr Aug 16, 2022
85421cf
Updated already existing bats test and implemented utils_gs_signals.s…
terranovafr Aug 16, 2022
e744629
Removed redundant code from some bats tests
terranovafr Aug 16, 2022
0a3b531
Creation of the remaining bats tests and refinement of existing bats …
terranovafr Aug 17, 2022
2122b07
Refinement of existing bats tests
terranovafr Aug 17, 2022
e15bac0
Refinement of existing bats tests
terranovafr Aug 17, 2022
676ce74
Fixed missing end of if statement
terranovafr Aug 17, 2022
a2dd1b1
Changes that fixes some errors
terranovafr Aug 17, 2022
26106d4
Added the removal of the logs directory generated during the test
terranovafr Aug 17, 2022
39118aa
Merge remote-tracking branch 'upstream/master'
terranovafr Aug 18, 2022
4f99912
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 18, 2022
65f885d
Added the tests regarding the write permission
terranovafr Aug 18, 2022
6fe9d34
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 18, 2022
b52480e
Added the tests regarding the write permission
terranovafr Aug 18, 2022
4bce734
Merge remote-tracking branch 'origin/master'
terranovafr Aug 18, 2022
28e9f0a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 18, 2022
a5c4804
Cleaned up code (removed some echos and fixed some local variables)
terranovafr Aug 18, 2022
e2b9911
Merge remote-tracking branch 'origin/master'
terranovafr Aug 18, 2022
63d8eff
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 18, 2022
42ab4ce
Cleaned up code (fixed the scope of some variables)
terranovafr Aug 18, 2022
872ff46
Merge remote-tracking branch 'origin/master'
terranovafr Aug 18, 2022
29c9964
Updated paths of the test files to filenames in /tmp.
terranovafr Aug 22, 2022
2cba72c
Re-committing for testing the timeout feature of the bats tests.
terranovafr Aug 24, 2022
e4e91ba
Applied some of the modifications requested.
terranovafr Aug 24, 2022
4c9d84d
Updated globals/used comments about functions.
terranovafr Aug 24, 2022
74ecac8
Modified some file names and solved some bugs in bats tests.
terranovafr Aug 25, 2022
8e41121
Modified some file names and solved some bugs in bats tests.
terranovafr Aug 25, 2022
f28d339
Added external files management modifications with the creation of su…
terranovafr Aug 31, 2022
b21bae5
Added external files management modifications with the modification o…
terranovafr Sep 2, 2022
559bcb4
Improved descriptor file handling
terranovafr Sep 2, 2022
b22ed5d
Pre-commit modifications
terranovafr Sep 2, 2022
7c573d3
Modified existing bats tests in order to support the modifications
terranovafr Sep 2, 2022
69bb8dc
Corrected create_glidein_config bats test
terranovafr Sep 2, 2022
3b9e395
Added a subset of bats tests regarding the utils_scripts bash script
terranovafr Sep 2, 2022
7483af5
Applied modifications to cWDictFile
terranovafr Sep 2, 2022
9ebbf93
Modified some comments to test the pyunittest
terranovafr Sep 2, 2022
6fd9ad0
Applied modifications to the python scripts handling the xml files co…
terranovafr Sep 2, 2022
608ff29
Applied modifications to the python scripts handling the xml files co…
terranovafr Sep 6, 2022
68d2150
Added compatibility support with older versions, converter to the new…
terranovafr Sep 7, 2022
d3a9f15
Solved some bugs of pyunittest
terranovafr Sep 7, 2022
7df0080
Solved some bugs of pyunittest
terranovafr Sep 7, 2022
3309f57
Solved some bugs that blocked bats tests
terranovafr Sep 7, 2022
ab0bc25
Solved pre-commit reported issues
terranovafr Sep 7, 2022
4fcceee
Solved pre-commit reported issues
terranovafr Sep 8, 2022
cdd3295
Solved pre-commit reported issues
terranovafr Sep 8, 2022
c072989
Solved pyunittest error
terranovafr Sep 8, 2022
712f701
Added some missing modifications
terranovafr Sep 8, 2022
a83fd9f
Added the control to invite to convert if boolean variables are prese…
terranovafr Sep 9, 2022
6d5d91f
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
c634017
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
7a57835
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
90afd58
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
3f28cda
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
1cd915e
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
72d0aaf
Fixed python3 check return value in case of exception and minor pep8 …
mambelli Sep 9, 2022
578d126
Modified HTML website with new modifications
terranovafr Sep 9, 2022
ecc8a9a
Added the two-digits priority handling
terranovafr Sep 12, 2022
58dc40d
Made some changes to the website's HTML section of the external files…
terranovafr Sep 12, 2022
11fc734
Fixed list intersection to work w/ python2 or python3 (no more just p…
mambelli Sep 13, 2022
afdae7a
ready for release v3_9_6_rc4
mambelli Sep 13, 2022
2213ec1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 13, 2022
b9d6d3d
Merge pull request #213 from mambelli/release_v3_9_6_rc4
mambelli Sep 14, 2022
fd71976
Made the modifications requested by the review (glidein_startup and u…
terranovafr Sep 16, 2022
68e4614
Made the modifications requested by the second review (mainly about u…
terranovafr Sep 16, 2022
5c282da
Applied pre-commit
terranovafr Sep 16, 2022
92a263f
Made the modifications requested by the review (mainly glidein_startu…
terranovafr Sep 16, 2022
d97b578
Made the modifications requested by the second review (mainly about u…
terranovafr Sep 16, 2022
f252687
Divided glidein_startup.sh in modules
terranovafr Aug 9, 2022
329c0f4
Added debug option
terranovafr Aug 9, 2022
879ce01
Added manual installation script for testing
terranovafr Aug 9, 2022
39ec146
Modified menu
terranovafr Aug 9, 2022
4bc4472
Solved error printf
terranovafr Aug 9, 2022
72731d7
Modified manual installation
terranovafr Aug 9, 2022
7aab560
Updated functions' comments
terranovafr Aug 10, 2022
30a23db
Fixed some comments
terranovafr Aug 10, 2022
e9e9a0a
Marked some useless global variables as local
terranovafr Aug 10, 2022
fd35145
Marked more global variables as local
terranovafr Aug 10, 2022
5870880
Changed debug option
terranovafr Aug 10, 2022
8790014
Modified print_header error handling
terranovafr Aug 10, 2022
f78013e
Modified print_header error handling
terranovafr Aug 10, 2022
a42d24f
Handled exit status
terranovafr Aug 10, 2022
5175bde
Added first xml change
terranovafr Aug 11, 2022
7a35bb6
Added first xml change
terranovafr Aug 11, 2022
6fb0df2
Added first xml change
terranovafr Aug 11, 2022
e023d0d
Added first xml change
terranovafr Aug 11, 2022
4942be8
Added first xml change
terranovafr Aug 11, 2022
186ec75
Added xml changes
terranovafr Aug 11, 2022
48cfa02
Added xml changes
terranovafr Aug 11, 2022
49cf6cc
Added spaces support to create_xml method
terranovafr Aug 11, 2022
ae23494
Substitution of two methods with the call to the create_xml method
terranovafr Aug 11, 2022
94441ba
Substitution of two methods with the call to the create_xml method
terranovafr Aug 11, 2022
5286261
Substitution of the methods of utils_http with the call to the create…
terranovafr Aug 11, 2022
618f8c5
Substitution of the methods of utils_http with the call to the create…
terranovafr Aug 11, 2022
f36e92c
Substitution of the methods of utils_http with the call to the create…
terranovafr Aug 11, 2022
6ca292a
Completed substituion of xml methods
terranovafr Aug 11, 2022
1f5a834
Completed substituion of xml methods
terranovafr Aug 11, 2022
a0d56bc
Methods divided among glidein startup specific methods and general me…
terranovafr Aug 11, 2022
437f5c9
Moved menu of options
terranovafr Aug 11, 2022
8c55e27
Moved menu of options
terranovafr Aug 11, 2022
71ef813
Header and license added to each script file
terranovafr Aug 11, 2022
785297f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 11, 2022
8440456
Added more controls to the parse_option method and more detailed comm…
terranovafr Aug 12, 2022
e12b460
Delete manual_installation_wget.sh
terranovafr Aug 11, 2022
459f639
Added more controls to the parse_option method and more detailed comm…
terranovafr Aug 12, 2022
1d65e57
Implemented the bats tests of the glidein_cleanup.sh, utils_gs_io.sh,…
terranovafr Aug 16, 2022
0f1bde7
Updated already existing bats test and implemented utils_gs_signals.s…
terranovafr Aug 16, 2022
4cdb090
Removed redundant code from some bats tests
terranovafr Aug 16, 2022
9c7a3a4
Creation of the remaining bats tests and refinement of existing bats …
terranovafr Aug 17, 2022
aa80e20
Refinement of existing bats tests
terranovafr Aug 17, 2022
b87a8f6
Refinement of existing bats tests
terranovafr Aug 17, 2022
8bef45a
Fixed missing end of if statement
terranovafr Aug 17, 2022
daa1c80
Changes that fixes some errors
terranovafr Aug 17, 2022
8a462d5
Added the removal of the logs directory generated during the test
terranovafr Aug 17, 2022
84a57f9
Added the tests regarding the write permission
terranovafr Aug 18, 2022
7349204
Added the tests regarding the write permission
terranovafr Aug 18, 2022
028fd48
Cleaned up code (removed some echos and fixed some local variables)
terranovafr Aug 18, 2022
dbbd7d6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 18, 2022
f0211f7
Cleaned up code (fixed the scope of some variables)
terranovafr Aug 18, 2022
84f9617
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 18, 2022
2480d95
Updated paths of the test files to filenames in /tmp.
terranovafr Aug 22, 2022
3b1a5d7
Re-committing for testing the timeout feature of the bats tests.
terranovafr Aug 24, 2022
3354fa0
Applied some of the modifications requested.
terranovafr Aug 24, 2022
eb788bf
Updated globals/used comments about functions.
terranovafr Aug 24, 2022
8579aea
Modified some file names and solved some bugs in bats tests.
terranovafr Aug 25, 2022
3228c21
Modified some file names and solved some bugs in bats tests.
terranovafr Aug 25, 2022
4bca06b
Corrected create_glidein_config bats test
terranovafr Sep 2, 2022
186849f
Made the modifications requested by the review (mainly glidein_startu…
terranovafr Sep 16, 2022
6ef99b1
Made the modifications requested by the second review (mainly about u…
terranovafr Sep 16, 2022
120d6b3
Applied modifications to the prepare_workdir function considering the…
terranovafr Sep 16, 2022
659d916
Applied modifications considering the original repo's modifications t…
terranovafr Sep 16, 2022
648b48a
Performed last modifications after merging with original repo's behin…
terranovafr Sep 19, 2022
ddeec6a
Merge remote-tracking branch 'origin/master'
terranovafr Sep 19, 2022
df079da
Removed prepare_workdir bats test
terranovafr Sep 19, 2022
1515762
Added external files management modifications with the creation of su…
terranovafr Aug 31, 2022
7e9ef83
Added external files management modifications with the modification o…
terranovafr Sep 2, 2022
dc4acee
Improved descriptor file handling
terranovafr Sep 2, 2022
3338cea
Pre-commit modifications
terranovafr Sep 2, 2022
097d912
Modified existing bats tests in order to support the modifications
terranovafr Sep 2, 2022
00e0cb9
Added a subset of bats tests regarding the utils_scripts bash script
terranovafr Sep 2, 2022
d91a931
Applied modifications to cWDictFile
terranovafr Sep 2, 2022
da02ead
Modified some comments to test the pyunittest
terranovafr Sep 2, 2022
8f18e43
Applied modifications to the python scripts handling the xml files co…
terranovafr Sep 2, 2022
fdf876a
Applied modifications to the python scripts handling the xml files co…
terranovafr Sep 6, 2022
d7ae1b0
Added compatibility support with older versions, converter to the new…
terranovafr Sep 7, 2022
f9686ba
Solved some bugs of pyunittest
terranovafr Sep 7, 2022
6191f9a
Solved some bugs of pyunittest
terranovafr Sep 7, 2022
b0730dd
Solved some bugs that blocked bats tests
terranovafr Sep 7, 2022
be99943
Solved pre-commit reported issues
terranovafr Sep 7, 2022
77a8994
Solved pre-commit reported issues
terranovafr Sep 8, 2022
162f69a
Solved pre-commit reported issues
terranovafr Sep 8, 2022
6650a3f
Solved pyunittest error
terranovafr Sep 8, 2022
f486519
Added some missing modifications
terranovafr Sep 8, 2022
8b84bde
Added the control to invite to convert if boolean variables are prese…
terranovafr Sep 9, 2022
f2676d9
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
00cee4b
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
0b84af7
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
11ec55e
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
38cbe31
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
e50c5b9
Performed some modifications to make the tests work
terranovafr Sep 9, 2022
86c5670
Modified HTML website with new modifications
terranovafr Sep 9, 2022
ef52508
Added the two-digits priority handling
terranovafr Sep 12, 2022
406a20e
Made some changes to the website's HTML section of the external files…
terranovafr Sep 12, 2022
03bf286
Made the modifications requested by the review (glidein_startup and u…
terranovafr Sep 16, 2022
621bb4b
Made the modifications requested by the second review (mainly about u…
terranovafr Sep 16, 2022
3ab922c
Applied pre-commit
terranovafr Sep 16, 2022
a8cc9e5
Merge remote-tracking branch 'origin/external_files_management' into …
terranovafr Sep 19, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,8 @@ bigfiles/*
!bigfiles/push-bigfiles.sh

.idea/*
node_modules/*
output/*
package-lock.json
package.json
test/bats/logs/
8 changes: 5 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,15 @@ Changes since the last release

### Bug Fixes

- Fixed `glidien_config` corrupted by concurrent custom scripts run via HTCSS startd cron (#163)
- Fixed `setup_x509.sh` not to write to stdout when running as periodic script in HTCSS start cron (issues #162, #164 )
- Fixed setup_x509.sh creates proxy file in directory used for tokens (issue 201)
- Fixed glidien_config` corrupted by concurrent custom scripts run via HTCSS startd cron (#163)
- Fixed setup_x509.sh` not to write to stdout when running as periodic script in HTCSS start cron (issues #162, #164 )
- Fixed setup_x509.sh creates proxy file in directory used for tokens (issue #201)
- Fixed GLIDEIN_START_DIR_ORIG and GLIDEIN_WORKSPACE_ORIG values in glidein_config
- Fixed unnecessary proxy/hostcert.pem workaround in frontend config (issue #66)
- Fixed analyze_entries and python3 readiness (issue #194)
- Fixed gwms-renew-proxies service should check if local VOMS cert is expired (issue #21)
- Fixed python3 check return value in case of exception (PR #211)
- Fixed list_get_intersection in singularity_lib.sh that was requiring python2 (PR #212)

### Testing / Development

Expand Down
4 changes: 2 additions & 2 deletions build/packaging/rpm/glideinwms.spec
Original file line number Diff line number Diff line change
Expand Up @@ -1042,10 +1042,10 @@ rm -rf $RPM_BUILD_ROOT
#%config(noreplace) %{_sysconfdir}/condor/scripts/frontend_condortoken

%changelog
* Mon Jul 11 2022 Marco Mambelli <[email protected]> - 3.9.6
* Tue Sep 13 2022 Marco Mambelli <[email protected]> - 3.9.6
- Glideinwms v3.9.6
- Release Notes: http://glideinwms.fnal.gov/doc.v3_9_6/history.html
- Release candidates 3.9.6-01.rc1 to 3.9.6-01.rc2
- Release candidates 3.9.6-01.rc1 to 3.9.6-04.rc4

* Tue May 17 2022 Bruno Coimbra <[email protected]> - 3.9.5
- Glideinwms v3.9.5
Expand Down
5 changes: 5 additions & 0 deletions creation/lib/cWConsts.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,8 @@ def insert_timestr(instr):
BLACKLIST_FILE = "nodes.blacklist"

GRIDMAP_FILE = "grid-mapfile"

# Using two digits for major/minor/patch version
# This is the first version (3.11.0) with this new format
# Change this number only if the DictFile format changes
DICT_FILE_VERSION = "031100"
189 changes: 146 additions & 43 deletions creation/lib/cWDictFile.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
from glideinwms.lib.defaults import BINARY_ENCODING
from glideinwms.lib.util import chmod

from . import cWConsts

########################################
#
# File dictionary classes
Expand Down Expand Up @@ -437,7 +439,7 @@ def file_header(self, want_comments):
str: The file header, a comment containing the file name
"""
if want_comments:
return "# File: %s\n#" % self.fname
return f"# File: {self.fname}\n# Version: {cWConsts.DICT_FILE_VERSION}\n#"
else:
return None

Expand Down Expand Up @@ -992,23 +994,78 @@ class FileDictFile(SimpleFileDictFile):

It is using a dictionary (key, value) from DictFile, serialized to file.
The key is the file ID
The value (line) on file has DATA_LENGTH (7) components: the key and the first DATA_LENGTH-1 attributes below.
The value in memory has DATA_LENGTH components (real_fname,cache/exec,period,prefix,cond_download,config_out, data),
The value (line) on file has DATA_LENGTH (12) components: the key and the first DATA_LENGTH-1 attributes below.
The value in memory has DATA_LENGTH components (real_fname, cache/exec, prefix, time, priority, cond_download, tar_source, config_out, cond_attr, absdir_outattr, data),
the key is used as key for the dictionary and the data (file content) is added reading the file.
Here the attributes stored as tuple in the dictionary value:
1. real_fname, i.e file name
2. cache/exec/... keyword identifying the file type: regular, nocache, exec (:s modifier to run in singularity), untar, wrapper
3. period period in seconds at which an executable is re-invoked (only for periodic executables, 0 otherwise)
4. prefix startd_cron variables prefix (default is GLIDEIN_PS_)
5. cond_download has a special value of TRUE
6. config_out has a special value of FALSE
7. data - String containing the data extracted from the file (real_fname) (not in the serialized dictionary)
2. cache/exec/... keyword identifying the file type:
- regular
- nocache
- executable (:s modifier to run in singularity)
- untar (if a modifier is present, it will be the name of the directory where the tarball will be unpacked, the name of the tarball will be used otherwise)
- wrapper
- source
- library (:shell in case of a shell library)
3. prefix startd_cron variables prefix (default is GLIDEIN_PS_)
4. time - phase of the lifecycle when to execute/source the scripts:
- startup
- pre_job
- after_job
- cleanup
- periodic (:period will specify the period in seconds, 60 by default)
- milestone (:code will specify the milestone code)
- failure (:exit_code will specify the failure code)
Time will be ignored for files when type is different than exec or source.
A combination of time phases can be used.
5. period, 0 if not a periodic script
6. priority - priority in the order of execution inside a time phase [0-99].
Relevant only for executable, source and library files.
Order String code representation Associated integer code (internal representation)
Factory pre_entry e-[g-] 10
Frontend pre_entry pre_group e-g- 20
Frontend pre_entry group e-g= 30
Frontend pre_entry after_group e-g+ 40
entry e=[g-,g+,g=] 50
Frontend after_entry pre_group e+g- 60
Frontend after_entry group e+g= 70
Frontend after_entry after_group e+g+ 80
Factory after_entry e+[g+] 90
A custom value inside the range [0-99] can be specified in order to alter the order of execution.
7. cond_download has a special value of TRUE
8. tar_source - Name of the tarball containing the file, if the file is contained in a tarball, NULL otherwise (default)
9. config_out - has a special value of FALSE
10. cond_attr - Name of a configuration switch. (like "ENABLE_KRB5")
The operation requested on the file will be performed only if that parameter will be set to 1.
If not set (set to NULL), the result will be considered as TRUE (i.e. download)
As result, a file will be downloaded if cond_download == TRUE || $cond_attr == TRUE
11. absdir_outattr - Name of a variable name. (like "KRB5_SUBSYS_DIR")
The variable will be set to the absolute path of the file if the file is not a tarball
or the directory where the tarball was unpacked in case it is a tarball,
this last thing if and only if the unpacking actually happened (else it will not be defined.)
ENTRY_ will be prepended if the <file> directive occurs in an entry.
12. data - String containing the data extracted from the file (real_fname) (not in the serialized dictionary)

For placeholders, the real_name is empty (and the tuple starts w/ an empty string). Placeholders cannot be
serialized (saved into file). Empty strings would cause error when parsed back.
"""

DATA_LENGTH = 7 # Length of value (attributes + data)
PLACEHOLDER_VALUE = ("", "", 0, "", "", "", "") # The tuple should be DATA_LENGTH long and have the correct values
DATA_LENGTH = 12 # Length of value (attributes + data)
OLD_DATA_LENGTH = 7
PLACEHOLDER_VALUE = (
"",
"",
"",
"no_time",
0,
0,
"TRUE",
"NULL",
"FALSE",
"NULL",
"NULL",
"",
) # The tuple should be DATA_LENGTH long and have the correct values

def add_placeholder(self, key, allow_overwrite=True):
# using DictFile, no file content (FileDictFile or SimpleFileDictFile)
Expand All @@ -1018,20 +1075,49 @@ def is_placeholder(self, key):
return self[key][0] == "" # empty real_fname can only be a placeholder

@staticmethod
def make_val_tuple(file_name, file_type, period=0, prefix="GLIDEIN_PS_", cond_download="TRUE", config_out="FALSE"):
def make_val_tuple(
file_name,
file_type,
prefix="GLIDEIN_PS_",
time="no_time",
period=0,
priority=0, # executed first
cond_download="TRUE",
tar_source="NULL",
config_out="FALSE",
cond_attr="NULL",
absdir_outattr="NULL",
):
"""Make a tuple with the DATA_LENGTH-1 attributes in the correct order using the defaults

:param file_name: name of the file (aka real_fname)
:param file_type: type of the file (regular, nocache, exec, untar, wrapper). 'exec allows modifiers like ':s'
:param period: period for periodic executables (ignored otherwise, default: 0)
:param file_type: type of the file (regular, nocache, exec, untar, wrapper, source, library). 'exec allows modifiers like ':s'
:param prefix: prefix for periodic executables (ignored otherwise, default: GLIDEIN_PS_)
:param time: specific phase of the lifecycle where to execute/source the script (startup, pre_job, after_job, cleanup, periodic, milestone, failure)
:param period: period of the periodic execution, 0 if the executable is not periodic
:param priority: priority of execution inside the phase [0-99]
:param cond_download: conditional download (default: 'TRUE')
:param tar_source: name of the tarball containing the file
:param config_out: config out (default: 'FALSE')
:param cond_attr: conditional attribute
:param absdir_outattr: absolute path variable
:return: tuple with the DATA_LENGTH-1 attributes
See class definition for more information about the attributes
"""
# TODO: should it do some value checking? valid constant, int, ...
return file_name, file_type, period, prefix, cond_download, config_out # python constructs the tuple
return (
file_name,
file_type,
prefix,
time,
period,
priority,
cond_download,
tar_source,
config_out,
cond_attr,
absdir_outattr,
) # python constructs the tuple

@staticmethod
def val_to_file_name(val):
Expand Down Expand Up @@ -1066,7 +1152,7 @@ def add_from_str(self, key, val, data, allow_overwrite=False, allow_overwrite_pl

Args:
key (str): file ID
val (tuple): lists of 6 or 7 components (see class definition)
val (tuple): lists of 7 or 8 components (see class definition)
data (str): string w/ data to add
allow_overwrite (bool): if True the existing files can be replaced (default: False)
allow_overwrite_placeholder (bool): if True, placeholder files can be replaced even if allow_overwrite
Expand Down Expand Up @@ -1095,7 +1181,7 @@ def add(

Args:
key (str): file ID
val (tuple): lists of 6 or 7 components (see class definition)
val (tuple): lists of 7 or 8 components (see class definition)
allow_overwrite (bool): if True the existing files can be replaced (default: False)
allow_overwrite_placeholder (bool): if True, placeholder files can be replaced even if allow_overwrite
is False (default: True)
Expand All @@ -1112,13 +1198,6 @@ def add(
# since the other functions from base class know nothing about placeholders, need to force overwrite
allow_overwrite = True

# This will help identify calls not migrated to the new format
# TODO: check parameters!!
try:
int(val[2]) # to check if is integer. Period must be int or convertible to int
except (ValueError, IndexError):
raise DictFileError("Values '%s' not (real_fname,cache/exec,period,prefix,cond_download,config_out)" % val)

if len(val) == self.DATA_LENGTH:
# Alt: return self.add_from_str(key, val[:self.DATA_LENGTH-1], val[self.DATA_LENGTH-1], allow_overwrite)
return DictFile.add(self, key, tuple(val), allow_overwrite)
Expand All @@ -1127,21 +1206,32 @@ def add(
# Maybe check also string length or possible values?
if "\n" in val[-1]:
raise DictFileError(
"Values '%s' not (real_fname,cache/exec,period,prefix,cond_download,config_out)" % val
"Values '%s' not (real_fname,cache/exec,prefix,time,priority,cond_download,tar_source,config_out,cond_attr,absdir_outattr)"
% val
)
return self.add_from_file(key, val, os.path.join(self.dir, self.val_to_file_name(val)), allow_overwrite)
elif len(val) == self.OLD_DATA_LENGTH:
return DictFile.add(self, key, tuple(val), allow_overwrite)
else:
raise DictFileError("Values '%s' not (real_fname,cache/exec,period,prefix,cond_download,config_out)" % val)
raise DictFileError(
"Values '%s' not (real_fname,cache/exec,prefix,time,priority,cond_download,tar_source,config_out,cond_attr,absdir_outattr)"
% val
)

def format_val(self, key, want_comments):
return "{} \t{} \t{} \t{} \t{} \t{} \t{}".format(
return "{} \t{} \t{} \t{} \t{} \t{} \t{} \t{} \t{} \t{} \t{} \t{}".format(
key,
self.vals[key][0],
self.vals[key][1],
self.vals[key][2],
self.vals[key][3],
self.vals[key][4],
self.vals[key][5],
self.vals[key][6],
self.vals[key][7],
self.vals[key][8],
self.vals[key][9],
self.vals[key][10],
)

def file_header(self, want_comments):
Expand All @@ -1150,8 +1240,21 @@ def file_header(self, want_comments):
DictFile.file_header(self, want_comments)
+ "\n"
+ (
"# %s \t%s \t%s \t%s \t%s \t%s \t%s\n"
% ("Outfile", "InFile ", "Cache/exec", "Period", "Prefix", "Condition", "ConfigOut")
"# %s \t%s \t%s \t%s \t%s \t%s \t%s \t%s \t%s \t%s \t%s \t%s \n"
% (
"Outfile",
"InFile ",
"Cache/exec",
"Prefix",
"Time",
"Period",
"Priority",
"Condition",
"TarSource",
"ConfigOut",
"CondAttr",
"AbsdirOutattr",
)
)
+ ("#" * 89)
)
Expand All @@ -1173,22 +1276,22 @@ def parse_val(self, line):
if len(arr[0]) == 0:
return # empty key

if len(arr) != self.DATA_LENGTH:
# compatibility w/ old formats
# 3.2.13 (no prefix): key, fname, type, period, cond_download, config_out
# 3.2.10 (no period, prefix): key, fname, type, cond_download, config_out
# TODO: remove in 3.3 or after a few version (will break upgrade)
if len(arr) == self.DATA_LENGTH - 1:
# For upgrade from 3.2.13 to 3.2.11
return self.add(arr[0], [arr[1], arr[2], arr[3], "GLIDEIN_PS_", arr[4], arr[5]])
elif len(arr) == self.DATA_LENGTH - 2:
# For upgrade from 3.2.10 or earlier
return self.add(arr[0], [arr[1], arr[2], 0, "GLIDEIN_PS_", arr[3], arr[4]])
raise RuntimeError(
"Not a valid file line (expected %i, found %i elements): '%s'" % (self.DATA_LENGTH, len(arr), line)
if len(arr) == self.DATA_LENGTH:
return self.add(arr[0], arr[1:])

# for compatibility with old format
if len(arr) == self.OLD_DATA_LENGTH:
# old format: file, type, period, prefix, condition, configout
return self.add(
arr[0],
[arr[1], arr[2], arr[4], "startup", arr[3], 0, arr[5], "NULL", arr[6], "NULL", "NULL", "NULL"],
)

return self.add(arr[0], arr[1:])
# length error
raise RuntimeError(
"Not a valid file line (expected %i or %i, found %i elements): '%s'"
% (self.DATA_LENGTH, self.OLD_DATA_LENGTH, len(arr), line)
)

def get_immutable_files(self):
mkeys = []
Expand Down
Loading