A collection of Python utilities.
Use a virtualenv for development. For example, install kobo and dependencies in editable mode:
virtualenv ~/kobo-dev
. ~/kobo-dev/bin/activate
pip install --editable ~/src/kobo
pip install -rtest-requirements.txt
To run the test suite:
- Run
py.testto run tests against installed versions of python and dependencies, or... - Install and run
toxto run test suite against a matrix of supported python and Django versions (more thorough, slower).
Please submit pull requests against https://github.com/release-engineering/kobo.
- client:
HubProxynow acceptstransport_argsto customize the instantiation of the default transport kobo.pkgset.FileCachenow implements__contains__for improved performance
- hub: introduced
VIEW_RAW_LOG_EXTENSIONSsetting
- hub: get_worker_tasks XMLRPC function now has an option to also return ASSIGNED tasks
- worker: fixed another logging deadlock introduced in kobo 0.36.0
- hub: fixed display of permissions in user details template
- worker: fixed a logging deadlock introduced in kobo 0.36.0
- worker: fixed a locked TaskManager sometimes opening tasks
- worker: on
SIGHUP, will now gracefully await task completion and re-exec
- client: the
--huboption towatch-logcommand is no longer ignored - worker: logging thread is more robust, recovers from many errors
- hub: minor fixes to worker detail UI
- hub: respond with 404 error on missing logs, rather than 500
- other minor fixes
- client: fixed support for proxies (
https_proxy) in XML-RPC client
- Support Django 5.0
- hub: restricted login redirects to same site only
- hub: fixed assignment of tasks to workers with incompatible arch
- hub: fixed display of arches/channels on worker detail page
- hub: avoid SELinux relabelling issue during file upload
- Minor cleanups of obsolete or deprecated code
- hub: the
/character may now be used within usernames
- hub: uploads of the same name but different content are no longer considered duplicates
- hub: added a page for listing failed tasks, alongside the existing Running/Finished task lists.
- hub: introduced
inherit_workerargument tocreate_subtaskAPI, allowing workers to ensure created subtasks are assigned to themselves
- hub: introduced
settings.USERS_ACL_PERMISSIONto adjust access to user list
- hub: minor fixes to page titles
- hub: deleting a worker no longer deletes associated tasks
- hub: decorate Task.save with transaction.atomic
- hub: do not call save twice in Task.save()
- hub: fix Task.cancel_subtasks
- client: allow file upload size to be sent a string
- django: use BigIntegerField for file upload sizes
- hub: return error 404 for non-existent logs
- hub: move filename resolution to _streamed_log_response
- worker: do not repeatedly remove already awaited tasks from subtask list
- django: reimplement django_version_ge in pure Python
- hub: do not use workers without ids in a filter
- hub: replace pipes with shlex
- Make xmlrpc timeout configurable
- django: remove support for Django 1.10 and older
- hub: update
Worker.update_workerdocumentation
- Fixed XML-RPC authentication regression introduced in kobo 0.29.0
- Introduced
create-workercommand - Improved compatibility with Django 4.2
- Improved output from admin commands on XML-RPC errors
- Cleaned up unused code
- Renamed deprecated get_query_set to get_queryset
- Changed repr to properly deserialize tracebacks sent over XML-RPC
- Fixed remaining Django 3 deprecations
- Fixed reporting of failed uploads
- Changed behavior that log decoding failure doesn't raise an error
- Made it possible to interrupt assigned tasks
- Retired python 2 support
- Added Client Credentials Flow OIDC authentication
- Added logging for authN/authZ events
- Added
--huboption to client
- Fixed calculating the number of task's subtasks in new Django versions
- Fixed cli/help_RST compatibility with Python 3
- fixed encoding errors by forcing the usage of UTF-8 in str and bytes functions
- Added an option to make subtask inherit parents priority and make it configurable
- Improved compatibility with Django 4.1
- Improved compatibility with python 3
- Fixed a crash from Django dumpdata command when using Django 2 and later.
- Fixed an encoding error when using a python 2 XML-RPC client against a python 3 kobo hub.
- It is now possible to provide a custom task manager implementation into the kobo worker's main loop. This can be used to alter the worker's task scheduling or execution behavior.
- Introduced
AUTH_METHOD=oidcfor XML-RPC clients. If this authentication method is enabled, the OIDC endpoint introduced by kobo 0.22.0 will be used. HubProxynow respects the system configuration for kerberos realm whenKRB_REALMhas not been set.
- hub: added an endpoint for OIDC login. Note that kobo itself does not implement any OIDC workflow, which may instead be implemented by the hosting web server.
HubProxywill now retry on additional cases of XML-RPC error.
- Fixed a crash bug in
LoggingThreadwhen mixing bytes and text.
- Added methods to
HubProxyfor explicitly logging into hub.
- Many compatibility fixes for modern Python and Django versions.
- Fixed loss of error details when
LoggingThreadfails.
- Improved Django 4.x compatibility
kobo.rpmlib.get_keys_from_headernow correctly handles RPMs using header-only signing. Also fixespkgset.SimpleRpmWrapper.signature.
- kobo workers now maintain a
last_seenfield which may be used to monitor the availability of workers.
- The
resubmit-tasksnow supports adjusting the priority of the resubmitted task. The client and server must both upgrade to kobo 0.19.0 in order to use this feature.
- Fixed backwards-incompatible change in kobo 0.18.0: if a kobo-derived client
provided its own
--profileargument, this could clash with the new argument in kobo.
- The kobo client now supports a
--profileargument to select an alternative configuration file. To enable this functionality, users of kobo should provide adefault_profileandconfiguration_directorywhen creating aCommandOptionParser.
- Fixed a python3 compatibility issue in kobo.http.
- Fixed a crash bug for projects using the default
XMLRPC_TEMPLATEfor the XML-RPC help page with Django > 1.
- Introduced the concept of global variables to
kobo.conf.PyConfigParser. This feature allows a variable to be declared in one file and overridden in a subsequently imported file.
- The
client.get_worker_infoXML-RPC method no longer requires authentication. It was formerly restricted to superusers only.
- The HubProxy auto_logout feature was removed. This feature was unreliable and
could occasionally result in deadlocks, due to the usage of finalizers.
It is recommended to call
logout()from afinallyblock in cases where it's important to ensure a logout.
- Fixed a Django 1.x compatibility issue introduced in kobo 0.15.0
- Improved Django 3.x compatibility
- Introduced
gssapilogin method, for improved Python3 compatibility. This login method requires the hub'skrb5loginendpoint to be enabled. The default login method is unchanged.
- Fixed a Python3 compatibility issue in hub migrations
- Improved Django 1.11 compatibility
- Improved Django 1.11 and 2.x compatibility
- The resubmit-tasks command now accepts a
--nowaitargument
- Fixed usage of
shortcuts.runwith text mode in Python 3 (#133) - Fixed decoding crashes in
shortcuts.runwhen an incomplete multibyte sequence is read (#119) - Fixed rpmlib attempting to decode binary data in RPM headers
- Fixed various Python 3 compatibility issues
- Improved Python 3 compatibility, particularly in xmlrpc and rpmlib modules
- Improved Django 1.11 and 2.x compatibility
- Testing of Python 2.6 has been dropped; some kobo functionality still works with Python 2.6, but may break without warning in later releases
- Support for Django 1.6 has been removed
- Fixed LoggingThread on Python 3 (#66)
- Fixed
kobo.django.xmlrpcmigrations for Django 2.x - Fixed some exceptions discarded without logging (#32)
- Fixed some reliability issues in
kobo.xmlrpc
- Improved Python 3 compatibility
- Improved Django 2.0 compatibility
- Improved tests coverage
- Header produced by kobo.shortcuts.run(show_cmd=True) is now limited to 79 characters length
- Fixed handling of string SERVER_PORT in wsgi requests
- Fixed Worker.timeout_task wrongly setting subtasks to INTERRUPTED (#72)
- Fixed Worker.set_task_weight always crashing (#75)
- Improved Python 3 compatibility
- Improved tests coverage
- kobo worker name no longer needs to match host FQDN
- improved error reporting when loading configuration
- improved error reporting from kobo.shortcuts.run
- reduced memory usage when handling large log files
- models now respect settings.AUTH_USER_MODEL
- fixed crash on xml-rpc client in python <= 2.7.9
- fixed spurious whitespace from kobo.shortcuts.run (#40)
- fixed missing migration for User model
- kobo.shortcuts.run now supports all Popen keyword arguments
- resubmit-tasks has a --force argument to resubmit successful tasks
- new watch-log command for watching a log from CLI
- admin UI now covers user model
- kobo.shortcuts.run now resumes on interrupted system calls
- worker load no longer includes assigned but unstarted tasks
- pkgset.SimpleRpmWrapper has now checksum_type member
- threads.run_in_threads helper function
Django part of kobo was udpated to be compatible with 1.5 release. Lower version are no more supported. As a side-effect, only python 2.6+ is supported by django part.
From same reason there will be no builds of kobo-django package for RHEL 5 and lower as it lacks required python version.
- RemoteUserMiddleware is now used for KrbV authentication
- LimitedRemoreUserMiddleware can be used to authenticate only on entry pages.
- LongnameUser model is used for auth backend. It allows 255 characters long user names (as they come from KrbV)
- Class-based generic views ExtraListView and ExtraDetailView. object_list helper stays for compatibility, but it is deprecated now and will be removed in future.
- State machine implementation - StateEnum, db field, form fields
- Brand new HTML template, media, views, urls and menu
- Menu supports django.root
- Task logs, javascript log watcher, threaded worker stdout logger
- XML-RPC help pages display list of contents
- Kerberos support in CookieTransport
- JSONField to store dicts and lists in database
- Add relative_path() and split_path() functions to shortcuts
file_name attribute renamed to file_path. file_name is now a property which returns actual file name.
Action: Change file_name to file_path in your code.
Username hack is enabled by default now (when kobo.django.auth is used). It changes username to 255 characters and also overrides validation RE.
Action: On postgresql run: ALTER TABLE auth_user ALTER username TYPE VARCHAR(255); Sqlite users have to use db_update-0.2.0-0.3.0 script.
Each worker's name must match it's FQDN now. This prevents cut&paste configuration errors when tasks end in INTERRUPTED state.
Action: Change worker names to FQDN. Change related usernames as well.
Removed 'lower_case' attribute. Plugins are now subclassed when a container is created. Each plugin now contains 'container' attribute.
Action: Remove 'lower_case' attribute from plugins, do whatever is necessary in 'normalize_name() class method instead.
- Improve PluginContainer inheritance. Also add 'container' attribute to each plugin class obtained from a container instance. (Daniel Mach)
- Plugins are now subclassed when a container is created. (Daniel Mach)
Field 'traceback' moved to a file (traceback.log). Field 'result' content dumped to a file (stdout.log), it is supposed to contain actual task result. Field 'args' changed to JSONField and data is directly available without any conversion.
Action: Run db_update-0.2.0-0.3.0 script.
Configuration no longer uses os.environ to get config file path. ClientCommandContainer, HubProxy and TaskManager constructor has a new mandatory 'conf' argument.
Action: config_file = os.environ.get("<PROJECT_NAME>_CONFIG_FILE", "/etc/<project_name>.conf") conf = kobo.conf.PyConfigParser() conf.load_from_file(config_file) ... and pass conf to ClientCommandContainer, HubProxy or TaskManager
TaskBase constructor has now a mandatory argument 'conf', which is automatically set in TaskManager.
Action: Add 'conf' argument to tasks classes with custom constructor.