Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more package index projects #1366

Merged
merged 9 commits into from
Feb 1, 2024
76 changes: 76 additions & 0 deletions source/guides/hosting-your-own-index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,86 @@ directory with autoindex enabled. For an example using the built in Web server
in `Twisted`_, you would simply run ``twistd -n web --path .`` and then
instruct users to add the URL to their installer's configuration.


Existing projects
=================

.. list-table::
:header-rows: 1

* - Project
- Package upload
- PyPI fall-through [2]_
- Additional notes

* - :ref:`devpi`
- ✔
- ✔
- multiple indexes with inheritance, with syncing, replication, fail-over;
mirroring

* - :ref:`simpleindex`
-
- ✔
-

* - :ref:`pypiserver`
- ✔
-
-

* - :ref:`pypiprivate`
-
-
-

* - :ref:`pypicloud`
-
-
- unmaintained; also cached proxying; authentication, authorisation

* - :ref:`pywharf`
-
-
- unmaintained; serve files in GitHub

* - :ref:`pulppython`
- ✔
-
- also mirroring, proxying; plugin for Pulp

* - :ref:`pip2pi`
-
-
- also mirroring; manual synchronisation

* - :ref:`dumb-pypi`
-
-
- not a server, but a static file site generator

* - :ref:`httpserver`
-
-
- standard-library

* - `Apache <https://httpd.apache.org/>`_
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
-
- ✔
- using
`mod_rewrite
<https://httpd.apache.org/docs/current/mod/mod_rewrite.html>`_
and
`mod_cache_disk
<https://httpd.apache.org/docs/current/mod/mod_cache_disk.html>`_,
you can cache requests to package indexes through an Apache server

----

.. [1] For complete documentation of the simple repository protocol, see
:ref:`simple repository API <simple-repository-api>`.

.. [2] Can be configured to fall-back to PyPI (or another package index)
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
if a requested package is missing.

.. _Twisted: https://twistedmatrix.com/
118 changes: 86 additions & 32 deletions source/guides/index-mirrors-and-caches.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,24 @@ Package index mirrors and caches
================================

:Page Status: Incomplete
:Last Reviewed: 2014-12-24
:Last Reviewed: 2023-11-08


Mirroring or caching of PyPI can be used to speed up local package installation,
Mirroring or caching of PyPI (and other
:term:`package indexes <Package Index>`) can be used to speed up local
package installation,
allow offline work, handle corporate firewalls or just plain Internet flakiness.

Three options are available in this area:
There are multiple classes of options in this area:

1. local/hosted caching of package indexes.

2. local/hosted mirroring of a package index. A mirror is a (whole or
partial) copy of a package index, which can be used in place of the
original index.

1. pip provides local caching options,
2. devpi provides higher-level caching option, potentially shared amongst
many users or machines, and
3. bandersnatch provides a local complete mirror of all PyPI :term:`packages
<Distribution Package>`.
3. private package index with fall-through to public package indexes (for
example, to mitigate dependency confusion attacks), also known as a
proxy.


Caching with pip
Expand All @@ -38,26 +43,75 @@ cached copies of :term:`packages <Distribution Package>`:
python3 -m pip install --no-index --find-links=/tmp/wheelhouse SomeProject


Caching with devpi
------------------

devpi is a caching proxy server which you run on your laptop, or some other
machine you know will always be available to you. See the `devpi
documentation for getting started`__.

__ https://devpi.net/docs/devpi/devpi/latest/+d/quickstart-pypimirror.html


Complete mirror with bandersnatch
----------------------------------

bandersnatch will set up a complete local mirror of all PyPI :term:`packages
<Distribution Package>` (externally-hosted packages are not mirrored). See
the `bandersnatch documentation for getting that going`__.

__ https://github.com/pypa/bandersnatch/

A benefit of devpi is that it will create a mirror which includes
:term:`packages <Distribution Package>` that are external to PyPI, unlike
bandersnatch which will only cache :term:`packages <Distribution Package>`
hosted on PyPI.
Existing projects
-----------------

.. list-table::
:header-rows: 1

* - Project
- Cache
- Mirror
- Proxy
- Additional notes

* - :ref:`devpi`
- ✔
- ✔
-
- multiple indexes with inheritance; syncing, replication, fail-over;
package upload

* - :ref:`bandersnatch`
- ✔
- ✔
-
-

* - :ref:`simpleindex`
-
-
- ✔
- custom plugin enables caching; re-routing to other package indexes

* - :ref:`pypicloud`
- ✔
-
- ✔
- unmaintained; authentication, authorisation

* - :ref:`pulppython`
-
- ✔
- ✔
- plugin for Pulp; multiple proxied indexes; package upload

* - :ref:`proxpi`
- ✔
-
- ✔
- multiple proxied indexes

* - :ref:`nginx_pypi_cache`
- ✔
-
- ✔
- multiple proxied indexes

* - :ref:`flaskpypiproxy`
- ✔
-
- ✔
- unmaintained

* - `Apache <https://httpd.apache.org/>`_
- ✔
-
- ✔
- using
`mod_rewrite
<https://httpd.apache.org/docs/current/mod/mod_rewrite.html>`_
and
`mod_cache_disk
<https://httpd.apache.org/docs/current/mod/mod_cache_disk.html>`_,
you can cache requests to package indexes through an Apache server
Loading
Loading