SAGA Python Internal Developer Notes

These are notes for SAGA-Python developers. If you're writing SAGA-Python adaptors or patching the engine, these could be useful for you!

Notes on using Git to manage branches

How to make bugfixes to devel/master

You can fix small bugs on devel, no problem, and then merge or rebase to the feature branches to benefit from the fix. Larger fixes (i.e. which will likely be worked on for a while and consist of multiple commits) may justify their own temporary branch, to make sure that devel remains in a usable state.

Fixes on master (i.e. on releases) are handled differently: there you always create a hotfix branch, so that you can merge it to both master (for a errata release), and to devel (where you likely need that fix as well). That should be done only for fixes which make a release kind of unusable.

How to merge changes from devel into your branch

So, you should get changes from devel into your feature branch with

git merge devel

or

git rebase devel

whichever you prefer more ('rebase' will delete the commit history of your branch, and squash all commits into one -- it will look like you branched one commit ago, not x commits ago -- 'merge' will keep your history).

Useful Logging

Use ERROR level log messages, raise and log simultaneously

Ole: Looking at existing adaptors, I have noticed that you generally don't write ERROR log messages when you throw an exception. This can be problematic when analyzing logs post-mortem because you simply won't see any errors!

Andre M: I saw your log_error_and_raise, and liked it, but wanted to keep explicit 'raise' calls in the code to make the intent very obvious. So I added a static member to the exception class, and you can now do:

raise saga.BadParameter._log (self._logger, "No no no no NO! No integers, please!!")

see saga.utils.exception.py, where you'll also find some documentation actually). This also keeps the stack trace clean -- which, btw, you can now simply print to the debug log stream with

self._logger.trace ()

Miscellaneous notes / FAQ

Needed files to modify to have your adaptor work

When adding new adaptors, for SAGA-Python to see them you need to make modifications to setup.py, saga/engine/engine.py, and don't forget that you need an empty __init.py__ file in your adaptor directory as well! (AM working on changing this latter requirement, may change) Why do you have to do this? Find out here: http://effbot.org/pyfaq/what-is-init-py-used-for.htm

Checking list of registered adaptors

FWIW, if you want to check the set of adaptors registered by the engine, you can do:

e = saga.engine.engine.Engine ()

e._dump ()

ttype errors

While coding an adaptor if you get an error of type:

Traceback (most recent call last):
  File "examples/jobs/slurm/slurm.py", line 77, in main
    print "Exitcode  :", catjob.exit_code
  File "/home/azebro1/saga-python-env/lib/python2.7/site-packages/saga/attributes.py", line 2236, in __getattr__
    return self._attributes_i_get      (key, flow=self._DOWN)
  File "/home/azebro1/saga-python-env/lib/python2.7/site-packages/saga/attributes.py", line 1017, in _attributes_i_get
    self._attributes_t_call_getter (key)
  File "/home/azebro1/saga-python-env/lib/python2.7/site-packages/saga/attributes.py", line 593, in _attributes_t_call_getter
    if not retries : raise e
TypeError: get_exit_code() got an unexpected keyword argument 'ttype'
Exitcode  :  ***  get_exit_code() got an unexpected keyword argument 'ttype'

then you're probably missing a @SYNC_CALL/@ASYNC_CALL decorator!

    @SYNC_CALL
    def get_exit_code(self) :

Add it in and the error should clear up.

Errors from inadvertently using internal accessor methods externally

You may have something like:

    "job_attributes"   : [saga.job.EXIT_CODE,
                          saga.job.EXECUTION_HOSTS,
                          saga.job.CREATED,
                          saga.job.STARTED,
                          saga.job.FINISHED],

and

    @SYNC_CALL
    def get_created(self) :                                                                     
        return self._created

but the following call from an external script will fail:

print catjob.get_created()

with error message:

Traceback (most recent call last):
  File "examples/jobs/slurmssh.py", line 86, in main
    print catjob.get_created()
  File "/home/azebro1/saga-python-env/lib/python2.7/site-packages/saga/attributes.py", line 2236, in __getattr__
    return self._attributes_i_get      (key, flow=self._DOWN)
  File "/home/azebro1/saga-python-env/lib/python2.7/site-packages/saga/attributes.py", line 1014, in _attributes_i_get
    d = self._attributes_t_init (key)
  File "/home/azebro1/saga-python-env/lib/python2.7/site-packages/saga/attributes.py", line 372, in _attributes_t_init
    raise DoesNotExist ("attribute key is invalid: %s"  %  (key))
DoesNotExist: attribute key is invalid: get_created
 ***  attribute key is invalid: get_created

That's because get_created is an internal accessor method, which is not part of the API. On API level, you should see the attribute job.created, which, if accessed, will call your adaptor method...

So

  print catjob.get_created()

should be

  print catjob.created

and then it will work as intended!