httpapi connection support (Issue #1086) #1094
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Now download counts are visible via PSF BigQuery.
[cherrypick from master]
These commits were squashed to avoid repo size exploding. [cherrypick from master]
* origin/master: (661 commits) Bump version for release. docs: update Changelog; closes mitogen-hq#481 issue mitogen-hq#481: core: preserve stderr TTY FD if one is present. issue mitogen-hq#481: avoid crash if disconnect occurs during forward_modules() Add a few more important modules to preamble_size.py. .ci: add verbiage for run_batches() too. .ci: add README.md. docs: update thanks docs: lose "approaching stability" language, we're pretty good now docs: fix changelog syntax/order/"20KB" tests: add new compression parameter to mitogen_get_stack results tests: disable affinity_test on Travis :/ issue #508: fix responder stats test due to new smaller parent.py. issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py. tests: fix fallout from 36fb318 issue mitogen-hq#520: add AIX auth failure string to su. tests: move affinity_test to Ansible tests. core: cProfile is not available in 2.4. issue #505: docs: add new detail graph for one scenario. docs: update and re-record profile graphs in docs; closes #505 service: fix PushFileService exception tests: pad out localhost-* service: start pool shutdown on broker shutdown. master: .encode() needed for Py3. ansible: stash PID files in CWD if requested for debugging. issue #508: master: minify_safe_re must be bytes for Py3. bench: tidy up and cpu-pin some more files. tests: add localhost-x100 ansible: double the default pool size. ansible: raise error with correct exception type. issue #508: master: minify all Mitogen/ansible_mitogen sources. parent: PartialZlib docstrings. ansible: hacky parser to alow bools to be specified on command line parent: pre-cache bootstrap if possible. docs: update Changelog. ansible: add mitogen_ssh_compression variable. service: PushFileService never recorded a file as sent. parent: synchronize get_core_source() service: use correct profile aggregation name. SyntaxError. ansible: don't pin controller if <4 cores. tests: make soak testing work reliably on vanilla. docs: changelog tidyups. ansible: document and make affinity stuff portable to non-Linux ansible: fix affinity.py test failure on 2 cores. ansible: preheat PluginLoader caches before fork. tests: make mitogen_shutdown_all be run_once by default. docs: update Changelog. ansible: use Poller for WorkerProcess; closes #491. ansible: new multiplexer/workers configuration docs: update Changelog. docs: update Changelog. ansible: pin connection multiplexer to a single core utils: pad out reset_affinity() and integrate with detach_popen() utils: import reset_affinity() function. master: set Router.profiling if MITOGEN_PROFILING variable present. parent: don't kill children when profiling is active. ansible: hook strategy and worker processes into profiler profiler: import from linear2 branch core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT issue #260: redundant if statement. ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk. ansible/bench: make end= configurable. master: cache sent/forwarded module names Aggregate code coverage data across tox all runs Allow independant control of coverage erase and reporting Fix incorrect attempt to use coverage docs: update Changelog; closes mitogen-hq#527. issue mitogen-hq#527: catch new-style module tracebacks like vanilla. Fix DeprecationWarning in mitogen.utils.run_with_router() Generate coverage report even if some tests fail ci: fix incorrect partition/rpartition from 8a4caea issue #260: hide force-disconnect messages. issue #498: fix shutdown crash issue #260: avoid start_transmit()/on_transmit()/stop_transmit() core: ensure broker profiling output reaches disk master: keep is_stdlib_path() result as negative cache entry ci: Allow DISTROS="debian*32" variable, and KEEP=1 Use develop mode in tox issue mitogen-hq#429: fix sudo regression. misc: rename to scripts. tab completion!! core: Latch._wake improvements issue #498: prevent crash on double 'disconnect' signal. issue #413: don't double-propagate DEL_ROUTE to parent. issue #498: wrap Router dict mutations in a lock issue mitogen-hq#429: enable en_US locale to unbreak debops test. issue mitogen-hq#499: fix another mind-numbingly stupid vanilla inconsistency issue #497: do our best to cope with crap upstream code ssh: fix test to match updated log format. issue mitogen-hq#429: update Changelog. issue mitogen-hq#429: update Changelog. issue mitogen-hq#429: teach sudo about every know i18n password string. issue mitogen-hq#429: install i18n-related bits in test images. ssh: tidy up logs and stream names. tests: ensure file is closed in connection_test. gcloud: small updates tests: give ansible/gcloud/ its own requirements file. issue mitogen-hq#499: another totally moronic implementation difference issue mitogen-hq#499: disable new test on vanilla. docs: update Changelog; closes mitogen-hq#499. ...
* origin/v024: (662 commits) docs: update Changelog release date. Bump version for release. docs: update Changelog; closes mitogen-hq#481 issue mitogen-hq#481: core: preserve stderr TTY FD if one is present. issue mitogen-hq#481: avoid crash if disconnect occurs during forward_modules() Add a few more important modules to preamble_size.py. .ci: add verbiage for run_batches() too. .ci: add README.md. docs: update thanks docs: lose "approaching stability" language, we're pretty good now docs: fix changelog syntax/order/"20KB" tests: add new compression parameter to mitogen_get_stack results tests: disable affinity_test on Travis :/ issue #508: fix responder stats test due to new smaller parent.py. issue #508: tests: skip minify_test Py2.4/2.5 for profiler.py. tests: fix fallout from 36fb318 issue mitogen-hq#520: add AIX auth failure string to su. tests: move affinity_test to Ansible tests. core: cProfile is not available in 2.4. issue #505: docs: add new detail graph for one scenario. docs: update and re-record profile graphs in docs; closes #505 service: fix PushFileService exception tests: pad out localhost-* service: start pool shutdown on broker shutdown. master: .encode() needed for Py3. ansible: stash PID files in CWD if requested for debugging. issue #508: master: minify_safe_re must be bytes for Py3. bench: tidy up and cpu-pin some more files. tests: add localhost-x100 ansible: double the default pool size. ansible: raise error with correct exception type. issue #508: master: minify all Mitogen/ansible_mitogen sources. parent: PartialZlib docstrings. ansible: hacky parser to alow bools to be specified on command line parent: pre-cache bootstrap if possible. docs: update Changelog. ansible: add mitogen_ssh_compression variable. service: PushFileService never recorded a file as sent. parent: synchronize get_core_source() service: use correct profile aggregation name. SyntaxError. ansible: don't pin controller if <4 cores. tests: make soak testing work reliably on vanilla. docs: changelog tidyups. ansible: document and make affinity stuff portable to non-Linux ansible: fix affinity.py test failure on 2 cores. ansible: preheat PluginLoader caches before fork. tests: make mitogen_shutdown_all be run_once by default. docs: update Changelog. ansible: use Poller for WorkerProcess; closes #491. ansible: new multiplexer/workers configuration docs: update Changelog. docs: update Changelog. ansible: pin connection multiplexer to a single core utils: pad out reset_affinity() and integrate with detach_popen() utils: import reset_affinity() function. master: set Router.profiling if MITOGEN_PROFILING variable present. parent: don't kill children when profiling is active. ansible: hook strategy and worker processes into profiler profiler: import from linear2 branch core: tidy up existing profiling code and support MITOGEN_PROFILE_FMT issue #260: redundant if statement. ansible: ensure MuxProcess MITOGEN_PROFILING results reach disk. ansible/bench: make end= configurable. master: cache sent/forwarded module names Aggregate code coverage data across tox all runs Allow independant control of coverage erase and reporting Fix incorrect attempt to use coverage docs: update Changelog; closes mitogen-hq#527. issue mitogen-hq#527: catch new-style module tracebacks like vanilla. Fix DeprecationWarning in mitogen.utils.run_with_router() Generate coverage report even if some tests fail ci: fix incorrect partition/rpartition from 8a4caea issue #260: hide force-disconnect messages. issue #498: fix shutdown crash issue #260: avoid start_transmit()/on_transmit()/stop_transmit() core: ensure broker profiling output reaches disk master: keep is_stdlib_path() result as negative cache entry ci: Allow DISTROS="debian*32" variable, and KEEP=1 Use develop mode in tox issue mitogen-hq#429: fix sudo regression. misc: rename to scripts. tab completion!! core: Latch._wake improvements issue #498: prevent crash on double 'disconnect' signal. issue #413: don't double-propagate DEL_ROUTE to parent. issue #498: wrap Router dict mutations in a lock issue mitogen-hq#429: enable en_US locale to unbreak debops test. issue mitogen-hq#499: fix another mind-numbingly stupid vanilla inconsistency issue #497: do our best to cope with crap upstream code ssh: fix test to match updated log format. issue mitogen-hq#429: update Changelog. issue mitogen-hq#429: update Changelog. issue mitogen-hq#429: teach sudo about every know i18n password string. issue mitogen-hq#429: install i18n-related bits in test images. ssh: tidy up logs and stream names. tests: ensure file is closed in connection_test. gcloud: small updates tests: give ansible/gcloud/ its own requirements file. issue mitogen-hq#499: another totally moronic implementation difference issue mitogen-hq#499: disable new test on vanilla. ...
* origin/docs-fix: stable: fix preamble_size on stable docs.
* origin/dmw: issue #537: disable just the trivial LinuxPolicyTest on Travis. docs: update Changelog; closes #537. ansible: refactor affinity class and add abstract tests. Bump version for release. docs: update Changelog. core: serialize calls to _service_stub_main(). docs: update Changelog; closes #532. issue #532: PushFileService race. docs: more concise Changelog. issue #541: changelog typos. ansible: quiesce boto logger; closes #541. docs: update Changelog. tests/ansible: Spec.port() test & mitogen_via= fix. Update copyright year everywhere. tests/ansible: Spec.become_pass() test. docs: remove top "Table of Contents" link docs: remove a little more top margin wastage tests/ansible: Spec.become_user() test. docs: update Changelog; closes #539. issue #539: disable logger propagation. ansible: capture stderr stream of async tasks. Closes mitogen-hq#540. docs: update Changelog. issue mitogen-hq#536: rework how 2.3-compatible simplejson is served .github: add some more questions to issue template docs: duplicate word docs: update Changelog. tests/ansible: Spec.become_method() test & mitogen_via= fix. setup.py: include LICENSE; closes mitogen-hq#538. tests/ansible: Spec.become() test tests/ansible: Spec.password() test, document interactive pw limitation. tests/ansible: Spec.remote_user() test & mitogen_via= fix. tests/ansible: Spec.remote_addr() test & mitogen_via= fix. tests/ansible: Spec.transport() test. docs: lighter pink docs: add 'Fixes' heading docs: more margin tweaks for changelog docs: tighter <p> margins, even less shouting, red headings docs: tidy up footer and GitHub link docs: enable fixed_sidebar docs: sans-serif fonts, reduce shouty headings issue mitogen-hq#536: add mitogen_via= tests too. ansible: fix a crash on 2.3 when mitogen_via= host is missing. tests: for 2.3 compatibility, disable gcloud.py for now docs: update Changelog; closes mitogen-hq#511, closes mitogen-hq#536. docs: update Changelog release date. issue mitogen-hq#536: disable transport_config tests on vanilla issue mitogen-hq#536: restore correct Python interpreter selection behaviour. issue mitogen-hq#536: connection_delegation/ tests were erroneously broken tests: define MITOGEN_INVENTORY_FILE even if -i unspecified. issue mitogen-hq#536: add tests for each ansible_python_interpreter case. issue mitogen-hq#536: stop defining explicit localhost in inventory. tests: allow running Ansible tests locally without -udmw again. stable: fix preamble_size on stable docs. issue mitogen-hq#481: add test.
* origin/026: docs: update Changelog for release. Bump version for release. issue mitogen-hq#555: ansible: workaround ancient reload(sys) hack. issue mitogen-hq#554: mitogen_action_script fix issue mitogen-hq#554: fix Ansible 2.4 compatibility issue mitogen-hq#554: don't rely on tmp_path autoremoval in test. issue mitogen-hq#554: track and remove multiple make_tmp_path() calls. docs: update Changelog. docs: drastically simplify install/changelog. issue mitogen-hq#552: include process identity in log messages. issue #550: update Changelog. issue #550: parent: add explanatory comment. issue #550: fix up TTY ioctls on WSL 2016 Anniversary Update docs: update Changelog. service: make service list optional. docs: update Changelog; closes mitogen-hq#548. issue mitogen-hq#548: always treat transport=smart as 'ssh' for mitogen_via=. docs: better intro paragraph. .ci: copy private key file to tempdir. os_fork: more doc tweaks os_fork: more doc tweaks os_fork: yet more doc tidyup os_fork: more doc tweaks os_fork: clean up docs .ci: import soak scripts. .ci: allow containers for different jobs to run simultaneously os_fork: python 3 fixes and tests. issue #535: activate Corker on 2.4 in master too. issue #535: update Changelog. issue #535: wire mitogen.os_fork into Broker and Pool. issue #535: parent: add create_socketpair(size=..) parameter. issue #535: introduce mitogen.os_fork module and Corker class. issue #535: docs: update Changelog issue #535: service: support Pool.defer() like Broker.defer() issue #535: core: unicode.encode() may take importer lock on 2.x issue #535: docs: fix up Select doc issue #535: docs: update Changelog. issue #535: core/select: support selecting from Latches. core: increase cookie field lengths to 64-bit; closes mitogen-hq#545. tests: ensure serialization restrictions are in effect tests/bench: set process affinity in throughput.py. docs: update copyright year. docs: update Changelog. core: Make Latch.put(obj=) optional. docs: change 'unreleased' Changelog format and add a hint. docs: update Changelog; closes mitogen-hq#542. issue mitogen-hq#542: return of select poller, new selection logic issue mitogen-hq#542: .ci: move some tests to Azure and enable Mac job. ansible: create stub __init__.py for sdist.
* origin/v027: docs: update Changelog for release. Bump version for release. issue mitogen-hq#587: update Changelog issue mitogen-hq#587: ansible: descriptive version check during startup. ansible: descriptive version check during startup. issue mitogen-hq#581: expose mitogen_mask_remote_name variable. issue #576: fix Kwargs minor version check. issue mitogen-hq#574: fix ISSUE_TEMPLATE link issue mitogen-hq#575: fix exception text rendering docs: remove infringing mark docs: fix config var scope docs: faster stats-preserving redirect docs: update ansible page issue mitogen-hq#570: add firewalld to always-fork list for now. docs: removed excess word docs: fixed message routing example description docs: removed repeated word docs: update Changelog; closes mitogen-hq#557. issue mitogen-hq#557: support correct cpu_set_t size
* origin/v028: (383 commits) Bump version for release. docs: update Changelog for 0.2.8. issue #627: add test and tweak Reaper behaviour. docs: lots more changelog concision docs: changelog concision docs: more changelog tweaks docs: reorder chapters docs: versionless <title> docs: update supported Ansible version, mention unsupported features docs: changelog fixes/tweaks issue #590: update Changelog. issue #621: send ADD_ROUTE earlier and add test for early logging. issue #590: whoops, import missing test modules issue #590: rework ParentEnumerationMethod to recursively handle bad modules issue #627: reduce the default pool size in a child to 2. tests: add a few extra service tests. docs: some more hyperlink joy docs: more hyperlinks docs: add domainrefs plugin to make link aliases everywhere \o/ docs: link IS_DEAD in changelog docs: tweaks to better explain changelog race issue mitogen-hq#533: update routing to account for DEL_ROUTE propagation race tests: use defer_sync() Rather than defer() + ancient sync_with_broker() tests: one case from doas_test was invoking su tests: hide memory-mapped files from lsof output issue #615: remove meaningless test issue #625: ignore SIGINT within MuxProcess issue #625: use exec() instead of subprocess in mitogen_ansible_playbook issue #615: regression test issue #615: update Changelog. issue #615: ensure 4GB max_message_size is configured for task workers. issue #615: update Changelog. issue #615: route a dead message to recipients when no reply is expected issue #615: fetch_file() might be called with AnsibleUnicode. issue #615: redirect 'fetch' action to 'mitogen_fetch'. issue #615: extricate slurp brainwrong from mitogen_fetch issue #615: ansible: import Ansible fetch.py action plug-in issue mitogen-hq#533: include object identity of Stream in repr() docs: lots more changelog issue mitogen-hq#595: add buildah to docs and changelog. docs: a few more internals.rst additions ci: update to Ansible 2.8.3 tests: another random string changed in 2.8.3 tests: fix sudo_flags_failure for Ansible 2.8.3 ci: fix procps command line format warning Whoops, merge together lgtm.yml and .lgtm.yml issue mitogen-hq#440: log Python version during bootstrap. docs: update changelog issue mitogen-hq#558: disable test on OSX to cope with boundless mediocrity issue mitogen-hq#558, mitogen-hq#582: preserve remote tmpdir if caller did not supply one issue #613: must await 'exit' and 'disconnect' in wait=False test Import LGTM config to disable some stuff Fix up another handful of LGTM errors. tests: work around AnsibleModule.run_command() race. docs: mention another __main__ safeguard docs: tweaks formatting error docs: make Sphinx install soft fail on Python 2. issue mitogen-hq#598: allow disabling preempt in terraform issue mitogen-hq#598: update Changelog. issue #605: update Changelog. issue #605: ansible: share a sem_t instead of a pthread_mutex_t issue #613: add tests for all the weird shutdown methods Add mitogen.core.now() and use it everywhere; closes #614. docs: move decorator docs into core.py and use autodecorator preamble_size: make it work on Python 3. docs: upgrade Sphinx to 2.1.2, require Python 3 to build docs. docs: fix Sphinx warnings, add LogHandler, more docstrings docs: tidy up some Changelog text issue #615: fix up FileService tests for new logic issue #615: another Py3x fix. issue #615: Py3x fix. issue #615: update Changelog. issue #615: use FileService for target->controll file transfers issue #482: another Py3 fix ci: try removing exclude: to make Azure jobs work again compat: fix Py2.4 SyntaxError issue #482: remove 'ssh' from checked processes ci: Py3 fix issue #279: add one more test for max_message_size issue #482: ci: add stray process checks to all jobs tests: fix format string error core: MitogenProtocol.is_privileged was not set in children issue #482: tests: fail DockerMixin tests if stray processes exist docs: update Changelog. issue mitogen-hq#586: update Changelog. docs: update Changelog. [security] core: undirectional routing wasn't respected in some cases docs: tidy up Select.all() issue mitogen-hq#612: update Changelog. master: fix TypeError pkgutil: fix Python3 compatibility parent: use protocol for getting remote_id docs: merge signals.rst into internals.rst os_fork: do not attempt to cork the active thread. parent: fix get_log_level() for split out loggers. issue #547: fix service_test failures. issue #547: update Changelog. issue #547: core/service: race/deadlock-free service pool init docs: update Changelog. ...
Release 0.3.0
Prepare 0.3.1
Fix Trove classifier, bump version
Release 0.3.3
Release 0.3.4
(cherry picked from commit 50efa53)
Release v0.3.5
Co-authored-by: Orion Poplawski <[email protected]>
If casting a string fails then raise a TypeError. This is potentially an API breaking change; chosen as the lesser evil vs. allowing silent errors. `cast()` relies on `bytes(obj)` & `str(obj)` returning the respective supertype. That's no longer the case for `AnsibleUnsafeBytes` & `AnsibleUnsafeText`; since fixes/mitigations for CVE-2023-5764. fixes mitogen-hq#1046, refs mitogen-hq#977 See also - GHSA-7j69-qfc3-2fq9 - ansible/ansible#82293
Prep work for ansible_mitogen.utils.unsafe
Follwing fixes in Ansible 7-9 for CVE-2023-5764 cating `AnsibleUnsafeBytes` & `AnsibleUnsafeText` to `bytes()` or `str()` requires special handling. The handling is Ansible specific, so it shouldn't go in the mitogen package but rather the ansible_mitogen package. `ansible_mitogen.utils.unsafe.cast()` is most like `mitogen.utils.cast()`. During development it began as `ansible_mitogen.utils.unsafe.unwrap_var()`, closer to an inverse of `ansible.utils.unsafe_procy.wrap_var()`. Future enhancements may move in this direction. refs mitogen-hq#977, refs mitogen-hq#1046 See also - GHSA-7j69-qfc3-2fq9 - ansible/ansible#82293 - https://github.com/mitogen-hq/mitogen/wiki/AnsibleUnsafe-notes
Release v0.3.7
Release v0.3.8
Release v0.3.9
Release 0.3.10
Release v0.3.11
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, Mitogen supports several connection types like local, ssh, docker and more. However, it lacks support for httpapi.
PR from patch introduced in #1086