Improve logic for restarting kernels with new ports during initial startup

.travis.yml

@@ -17,3 +17,6 @@ after_success:
 matrix:
     allow_failures:
         - python: nightly
+branches:
+  except:
+    - /^auto-backport-of-pr-[0-9]+$/

MANIFEST.in

@@ -5,6 +5,7 @@ include README.md
 # Documentation
 graft docs
 exclude docs/\#*
+exclude docs/_*
 
 # Examples
 graft examples

docs/changelog.rst

@@ -4,6 +4,55 @@
 Changes in Jupyter Client
 =========================
 
+5.2.1
+=====
+
+- Add parenthesis to conditional pytest requirement to work around a bug in the
+  ``wheel`` package, that generate a ``.whl`` which otherwise always depends on
+  ``pytest`` see :ghissue:`324` and :ghpull:`325`
+
+5.2
+===
+
+`5.2 on GitHub <https://github.com/jupyter/jupyter_client/milestones/5.2>`__
+
+- Define Jupyter protocol version 5.3:
+
+  - Kernels can now opt to be interrupted by a message sent on the control channel
+    instead of a system signal. See :ref:`kernelspecs` and :ref:`msging_interrupt`
+    (:ghpull:`294`).
+
+- New ``jupyter kernel`` command to launch an installed kernel by name
+  (:ghpull:`240`).
+- Kernelspecs where the command starts with e.g. ``python3`` or
+  ``python3.6``—matching the version ``jupyter_client`` is running on—are now
+  launched with the same Python executable as the launching process (:ghpull:`306`).
+  This extends the special handling of ``python`` added in 5.0.
+- Command line arguments specified by a kernelspec can now include
+  ``{resource_dir}``, which will be substituted with the kernelspec resource
+  directory path when the kernel is launched (:ghpull:`289`).
+- Kernelspecs now have an optional ``metadata`` field to hold arbitrary metadata
+  about kernels—see :ref:`kernelspecs` (:ghpull:`274`).
+- Make the ``KernelRestarter`` class used by a ``KernelManager`` configurable
+  (:ghpull:`290`).
+- When killing a kernel on Unix, kill its process group (:ghpull:`314`).
+- If a kernel dies soon after starting, reassign random ports before restarting
+  it, in case one of the previously chosen ports has been bound by another
+  process (:ghpull:`279`).
+- Avoid unnecessary filesystem operations when finding a kernelspec with
+  :meth:`.KernelSpecManager.get_kernel_spec` (:ghpull:`311`).
+- :meth:`.KernelSpecManager.get_all_specs` will no longer raise an exception on
+  encountering an invalid ``kernel.json`` file. It will raise a warning and
+  continue (:ghpull:`310`).
+- Check for non-contiguous buffers before trying to send them through ZMQ
+  (:ghpull:`258`).
+- Compatibility with upcoming Tornado version 5.0 (:ghpull:`304`).
+- Simplify setup code by always using setuptools (:ghpull:`284`).
+- Soften warnings when setting the sticky bit on runtime files fails
+  (:ghpull:`286`).
+- Various corrections and improvements to documentation.
+
+
 5.1
 ===
 

docs/conf.py

@@ -34,6 +34,7 @@
     'sphinx.ext.autodoc',
     'sphinx.ext.intersphinx',
     'sphinx.ext.napoleon',
+    'sphinxcontrib_github_alt',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -55,6 +56,8 @@
 copyright = '2015, Jupyter Development Team'
 author = 'Jupyter Development Team'
 
+github_project_url = "https://github.com/jupyter/jupyter_client"
+
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.

docs/environment.yml

@@ -8,3 +8,5 @@ dependencies:
 - jupyter_core
 - sphinx>=1.3.6
 - sphinx_rtd_theme
+- pip:
+  - sphinxcontrib_github_alt

docs/index.rst

@@ -23,6 +23,7 @@ with Jupyter kernels.
 
    kernels
    wrapperkernels
+   kernel_providers
 
 .. toctree::
    :maxdepth: 2

docs/kernel_providers.rst

@@ -0,0 +1,156 @@
+================
+Kernel providers
+================
+
+.. note::
+   This is a new interface under development, and may still change.
+   Not all Jupyter applications use this yet.
+   See :ref:`kernelspecs` for the established way of discovering kernel types.
+
+Creating a kernel provider
+==========================
+
+By writing a kernel provider, you can extend how Jupyter applications discover
+and start kernels. For example, you could find kernels in an environment system
+like conda, or kernels on remote systems which you can access.
+
+To write a kernel provider, subclass
+:class:`jupyter_client.discovery.KernelProviderBase`, giving your provider an ID
+and overriding two methods.
+
+.. class:: MyKernelProvider
+
+   .. attribute:: id
+
+      A short string identifying this provider. Cannot contain forward slash
+      (``/``).
+
+   .. method:: find_kernels()
+
+      Get the available kernel types this provider knows about.
+      Return an iterable of 2-tuples: (name, attributes).
+      *name* is a short string identifying the kernel type.
+      *attributes* is a dictionary with information to allow selecting a kernel.
+
+   .. method:: make_manager(name)
+
+      Prepare and return a :class:`~jupyter_client.KernelManager` instance
+      ready to start a new kernel instance of the type identified by *name*.
+      The input will be one of the names given by :meth:`find_kernels`.
+
+For example, imagine we want to tell Jupyter about kernels for a new language
+called *oblong*::
+
+    # oblong_provider.py
+    from jupyter_client.discovery import KernelProviderBase
+    from jupyter_client import KernelManager
+    from shutil import which
+
+    class OblongKernelProvider(KernelProviderBase):
+        id = 'oblong'
+
+        def find_kernels(self):
+            if not which('oblong-kernel'):
+                return  # Check it's available
+
+            # Two variants - for a real kernel, these could be something like
+            # different conda environments.
+            yield 'standard', {
+                'display_name': 'Oblong (standard)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+            yield 'rounded', {
+                'display_name': 'Oblong (rounded)',
+                'language': {'name': 'oblong'},
+                'argv': ['oblong-kernel'],
+            }
+
+        def make_manager(self, name):
+            if name == 'standard':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '0'})
+            elif name == 'rounded':
+                return KernelManager(kernel_cmd=['oblong-kernel'],
+                                     extra_env={'ROUNDED': '1'})
+            else:
+                raise ValueError("Unknown kernel %s" % name)
+
+You would then register this with an *entry point*. In your ``setup.py``, put
+something like this::
+
+    setup(...
+        entry_points = {
+        'jupyter_client.kernel_providers' : [
+            # The name before the '=' should match the id attribute
+            'oblong = oblong_provider:OblongKernelProvider',
+        ]
+    })
+
+Finding kernel types
+====================
+
+To find and start kernels in client code, use
+:class:`jupyter_client.discovery.KernelFinder`. This uses multiple kernel
+providers to find available kernels. Like a kernel provider, it has methods
+``find_kernels`` and ``make_manager``. The kernel names it works
+with have the provider ID as a prefix, e.g. ``oblong/rounded`` (from the example
+above).
+
+::
+
+    from jupyter_client.discovery import KernelFinder
+    kf = KernelFinder.from_entrypoints()
+
+    ## Find available kernel types
+    for name, attributes in kf.find_kernels():
+        print(name, ':', attributes['display_name'])
+    # oblong/standard : Oblong (standard)
+    # oblong/rounded : Oblong(rounded)
+    # ...
+
+    ## Start a kernel by name
+    manager = kf.make_manager('oblong/standard')
+    manager.start_kernel()
+
+.. module:: jupyter_client.discovery
+
+.. autoclass:: KernelFinder
+
+   .. automethod:: from_entrypoints
+
+   .. automethod:: find_kernels
+
+   .. automethod:: make_manager
+
+Kernel providers included in ``jupyter_client``
+===============================================
+
+``jupyter_client`` includes two kernel providers:
+
+.. autoclass:: KernelSpecProvider
+
+   .. seealso:: :ref:`kernelspecs`
+
+.. autoclass:: IPykernelProvider
+
+Glossary
+========
+
+Kernel instance
+  A running kernel, a process which can accept ZMQ connections from frontends.
+  Its state includes a namespace and an execution counter.
+
+Kernel type
+  The software to run a kernel instance, along with the context in which a
+  kernel starts. One kernel type allows starting multiple, initially similar
+  kernel instances. For instance, one kernel type may be associated with one
+  conda environment containing ``ipykernel``. The same kernel software in
+  another environment would be a different kernel type. Another software package
+  for a kernel, such as ``IRkernel``, would also be a different kernel type.
+
+Kernel provider
+  A Python class to discover kernel types and allow a client to start instances
+  of those kernel types. For instance, one kernel provider might find conda
+  environments containing ``ipykernel`` and allow starting kernel instances in
+  these environments.

docs/kernels.rst

@@ -6,7 +6,7 @@ Making kernels for Jupyter
 
 A 'kernel' is a program that runs and introspects the user's code. IPython
 includes a kernel for Python code, and people have written kernels for
-`several other languages <https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages>`_.
+`several other languages <https://github.com/jupyter/jupyter/wiki/Jupyter-kernels>`_.
 
 When Jupyter starts a kernel, it passes it a connection file. This specifies
 how to set up communications with the frontend.
@@ -132,6 +132,13 @@ JSON serialised dictionary containing the following keys and values:
   is found, a kernel with a matching `language` will be used.
   This allows a notebook written on any Python or Julia kernel to be properly associated
   with the user's Python or Julia kernel, even if they aren't listed under the same name as the author's.
+- **interrupt_mode** (optional): May be either ``signal`` or ``message`` and
+  specifies how a client is supposed to interrupt cell execution on this kernel,
+  either by sending an interrupt ``signal`` via the operating system's
+  signalling facilities (e.g. `SIGINT` on POSIX systems), or by sending an
+  ``interrupt_request`` message on the control channel (see
+  :ref:`msging_interrupt`). If this is not specified
+  the client will default to ``signal`` mode.
 - **env** (optional): A dictionary of environment variables to set for the kernel.
   These will be added to the current environment variables before the kernel is
   started.

docs/messaging.rst

@@ -21,7 +21,7 @@ Versioning
 
 The Jupyter message specification is versioned independently of the packages
 that use it.
-The current version of the specification is 5.2.
+The current version of the specification is 5.3.
 
 .. note::
    *New in* and *Changed in* messages in this document refer to versions of the
@@ -959,6 +959,27 @@ Message type: ``shutdown_reply``::
    socket, they simply send a forceful process termination signal, since a dead
    process is unlikely to respond in any useful way to messages.
 
+.. _msging_interrupt:
+
+Kernel interrupt
+----------------
+
+In case a kernel can not catch operating system interrupt signals (e.g. the used
+runtime handles signals and does not allow a user program to define a callback),
+a kernel can choose to be notified using a message instead. For this to work,
+the kernels kernelspec must set `interrupt_mode` to ``message``. An interruption
+will then result in the following message on the `control` channel:
+
+Message type: ``interrupt_request``::
+
+    content = {}
+
+Message type: ``interrupt_reply``::
+
+    content = {}
+
+.. versionadded:: 5.3
+
 
 Messages on the IOPub (PUB/SUB) channel
 =======================================

jupyter_client/_version.py

@@ -1,5 +1,5 @@
 version_info = (5, 1, 0)
 __version__ = '.'.join(map(str, version_info))
 
-protocol_version_info = (5, 2)
+protocol_version_info = (5, 3)
 protocol_version = "%i.%i" % protocol_version_info

jupyter_client/channels.py

@@ -80,7 +80,10 @@ def __init__(self, context=None, session=None, address=None):
     @staticmethod
     @atexit.register
     def _notice_exit():
-        HBChannel._exiting = True
+        # Class definitions can be torn down during interpreter shutdown.
+        # We only need to set _exiting flag if this hasn't happened.
+        if HBChannel is not None:
+            HBChannel._exiting = True
 
     def _create_socket(self):
         if self.socket is not None: