Discussion:
PEP 453 (pip bootstrapping) ready for pronouncement?
(too old to reply)
Nick Coghlan
2013-09-23 11:15:09 UTC
Permalink
With the last round of updates, I believe PEP 453 is ready for
Martin's pronouncement.

HTML: http://www.python.org/dev/peps/pep-0453/
Major diffs: http://hg.python.org/peps/rev/b2993450b32a

Many of the updates are just clarifications in response to questions,
but the actual significant changes are:

- the proposed module name is now "ensurepip" (thanks Antoine for that
suggestion)
- distros are granted considerable latitude to ensure the distro pip
package is used with the system Python installation, so long as they
ensure virtual environments continue to work as expected
- the two open questions regarding uninstallation and tweaking the
directory layout on Windows are decided in favour of the approaches
that were already documented in the PEP (since we hadn't received any
objections to them in earlier discussions)

Regards,
Nick.

=====================
PEP: 453
Title: Explicit bootstrapping of pip in Python installations
Version: $Revision$
Last-Modified: $Date$
Author: Donald Stufft <***@stufft.io>,
Nick Coghlan <***@gmail.com>
BDFL-Delegate: Martin von Löwis
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013, 23-Sep-2013


Abstract
========

This PEP proposes that the `pip`_ package manager be made available by
default when installing CPython and when creating virtual environments
using the standard library's ``venv`` module via the ``pyvenv`` command line
utility).

To clearly demarcate development responsibilities, and to avoid
inadvertently downgrading ``pip`` when updating CPython, the proposed
mechanism to achieve this is to include an explicit `pip`_ bootstrapping
mechanism in the standard library that is invoked automatically by the
CPython installers provided on python.org.

The PEP also strongly recommends that CPython redistributors and other Python
implementations ensure that ``pip`` is available by default, or
at the very least, explicitly document the fact that it is not included.


Proposal
========

This PEP proposes the inclusion of an ``ensurepip`` bootstrapping module in
Python 3.4, as well as in the next maintenance releases of Python 3.3 and
2.7.

This PEP does *not* propose making pip (or any dependencies) directly
available as part of the standard library. Instead, pip will be a
bundled application provided along with CPython for the convenience
of Python users, but subject to its own development life cycle and able
to be upgraded independently of the core interpreter and standard library.


Rationale
=========

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:

* Assume the user already has a suitable cross-platform package manager
installed.
* Duplicate the instructions and tell their users how to install the
package manager.
* Completely forgo the use of dependencies to ease installation concerns
for their users.

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on systems with platform package managers (such as major Linux
distributions). No such assistance is available for Windows and
Mac OS X users. The challenges of dealing with this problem are a
regular feature of feedback the core Python developers receive from
professional educators and others introducing new users to Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.

This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't good (especially on Windows, where downloading and running
the ``get-pip.py`` bootstrap script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of
Python 2 and Python 3), since that makes it harder to create and maintain
good platform specific ``pip`` installers independently of the CPython
installers.

The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By providing a cross-platform package manager by default it will be easier
for users trying to install these third-party packages as well as easier
for the people distributing them as they should now be able to safely assume
that most users will have the appropriate installation tools available.
This is expected to become more important in the future as the Wheel_
package format (deliberately) does not have a built in "installer" in the
form of ``setup.py`` so users wishing to install from a wheel file will want
an installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24 month
feature release cycle, instead of using the general difficulty of installing
third-party packages as justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
these utilities.


Why pip?
--------

``pip`` has been chosen as the preferred default installer, as it
addresses several design and user experience issues with its predecessor
``easy_install`` (these issues can't readily be fixed in ``easy_install``
itself due to backwards compatibility concerns). ``pip`` is also well suited
to working within the bounds of a single Python runtime installation
(including associated virtual environments), which is a desirable feature
for a tool bundled with CPython.

Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their
aims (and hence substantially better than ``pip`` at handling external
binary dependencies), so it makes sense for the Python ecosystem to treat
them more like platform package managers to inter operate with rather than
as the default cross-platform installation tool. This relationship is
similar to that between ``pip`` and platform package management systems
like ``apt`` and ``yum`` (which are also designed to handle arbitrary
binary dependencies).


Explicit bootstrapping mechanism
================================

An additional module called ``ensurepip`` will be added to the standard
library whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a
callable named ``bootstrap()`` as well as offer direct execution via
``python -m ensurepip``.

The bootstrap will *not* contact PyPI, but instead rely on a private copy
of pip stored inside the standard library. Accordingly, only options
related to the installation location will be supported (``--user``,
``--root``, etc).

It is considered desirable that users be strongly encouraged to use the
latest available version of ``pip``, in order to take advantage of the
ongoing efforts to improve the security of the PyPI based ecosystem, as
well as benefiting from the efforts to improve the speed, reliability and
flexibility of that ecosystem.

In order to satisfy this goal of providing the most recent version of
``pip`` by default, the private copy of ``pip`` will be updated in CPython
maintenance releases, which should align well with the 6-month cycle used
for new ``pip`` releases.


Security considerations
-----------------------

The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of the CPython installers for end
users that do not subsequently make use of ``pip``.

The installers will contain all the components of a fully functioning
version of Python, including the ``pip`` installer. The installation
process will *not* require network access, and will *not* rely on
trusting the security of the network connection established between
``pip`` and the Python package index.

Only users that choose to use ``pip`` directly will need to pay
attention to any PyPI related security considerations.


Implementation strategy
-----------------------

To ensure there is no need for network access when installing Python or
creating virtual environments, the ``ensurepip`` module will, as an
implementation detail, include a complete private copy of pip and its
dependencies which will be used to extract pip and install it into the target
environment. It is important to stress that this private copy of pip is
*only* an implementation detail and it should *not* be relied on or
assumed to exist beyond the public capabilities exposed through the
``ensurepip`` module (and indirectly through ``venv``).

There is not yet a reference ``ensurepip`` implementation. The existing
``get-pip.py`` bootstrap script demonstrates an earlier variation of the
general concept, but the standard library version would take advantage of
the improved distribution capabilities offered by the CPython installers
to include private copies of ``pip`` and ``setuptools`` as wheel files
(rather than as embedded base64 encoded data), and would not try to
contact PyPI (instead installing directly from the private wheel files.

Rather than including separate code to handle the bootstrapping, the
``ensurepip`` module will manipulate sys.path appropriately to allow
the wheel files to be used to install themselves, either into the current
Python installation or into a virtual environment (as determined by the
options passed to the bootstrap command).

It is proposed that the implementation be carried out in five separate
steps (all steps after the first are independent of each other and can be
carried out in any order):

* the first step would add the ``ensurepip`` module and the private copies
of the most recently released versions of pip and setuptools, and update
the "Installing Python Modules" documentation. This change
would be applied to Python 2.7, 3.3 and 3.4.
* the Windows installer would be updated to offer the new ``pip``
installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the Mac OS X installer would be updated to offer the new ``pip``
installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the ``venv`` module and ``pyvenv`` command would be updated to make use
of ``ensurepip`` in Python 3.4+
* the PATH handling and ``sysconfig`` directory layout on Windows would be
updated for Python 3.4+


Proposed CLI
------------

The proposed CLI is based on a subset of the existing ``pip install``
options::

Usage:
python -m ensurepip [options]

General Options:
-h, --help Show help.
-v, --verbose Give more output. Option is additive, and
can be used up to 3 times.
-V, --version Show the pip version that would be extracted and exit.
-q, --quiet Give less output.

Installation Options:
-U, --upgrade Upgrade pip and dependencies, even if
already installed
--user Install using the user scheme.
--root <dir> Install everything relative to this
alternate root directory.

In most cases, end users won't need to use this CLI directly, as ``pip``
should have been installed automatically when installing Python or when
creating a virtual environment.

Users that want to retrieve the latest version from PyPI, or otherwise
need more flexibility, should invoke the extracted ``pip`` appropriately.


Proposed module API
-------------------

The proposed ``ensurepip`` module API consists of the following two
functions::

def version():
"""
Returns a string specifying the bundled version of pip.
"""

def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
"""
Bootstrap pip into the current Python installation (or the given root
directory).
"""


Invocation from the CPython installers
--------------------------------------

The CPython Windows and Mac OS X installers will each gain a new option:

* Install pip (the default Python package management utility)?

This option will be checked by default.

If the option is checked, then the installer will invoke the following
command with the just installed Python::

python -m ensurepip --upgrade

This ensures that, by default, installing or updating CPython will ensure
that the installed version of pip is at least as recent as the one included
with that version of CPython. If a newer version of pip has already been
installed then ``python -m ensurepip --upgrade`` will simply return without
doing anything.


Installing from source
----------------------

While the prebuilt binary installers will be updated to run
``python -m ensurepip`` by default, no such change will be made to the
``make install`` and ``make altinstall`` commands of the source
distribution.

``ensurepip`` itself (including the private copy of ``pip`` and its
dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.

Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the ``ensurepip`` invocation avoids any challenges
associated with determining where to install the extracted ``pip``.


Changes to virtual environments
-------------------------------

Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since its release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
*does* include pip installed by default.

To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
suitable replacement.

To handle cases where a user does not wish to have pip bootstrapped into
their virtual environment a ``--without-pip`` option will be
added.

The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
one new parameter: ``with_pip`` (defaulting to ``False``).

The new default for the module API is chosen for backwards compatibility
with the current behaviour (as it is assumed that most invocation of the
``venv`` module happens through third part tools that likely will not
want ``pip`` installed without explicitly requesting it), while the
default for the command line interface is chosen to try to ensure ``pip``
is available in most virtual environments without additional action on the
part of the end user.

This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.


Documentation
-------------

The "Installing Python Modules" section of the standard library
documentation will be updated to recommend the use of the bootstrapped
`pip` installer. It will give a brief description of the most common
commands and options, but delegate to the externally maintained ``pip``
documentation for the full details.

The existing content of the module installation guide will be retained,
but under a new "Invoking distutils directly" subsection.


Bundling CA certificates with CPython
-------------------------------------

The ``ensurepip`` implementation will include the ``pip`` CA bundle along
with the rest of ``pip``. This means CPython effectively includes
a CA bundle that is used solely by ``pip`` after it has been extracted.

This is considered preferable to relying solely on the system
certificate stores, as it ensures that ``pip`` will behave the same
across all supported versions of Python, even those prior to Python 3.4
that cannot access the system certificate store on Windows.


Automatic installation of setuptools
------------------------------------

``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
``ensurepip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
install the private copy in addition to installing ``pip`` itself.

However, this behavior is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.

Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``ensurepip`` in subsequent CPython releases.


Updating the private copy of pip
--------------------------------

In order to keep up with evolutions in packaging as well as providing users
with as recent version a possible the ``ensurepip`` module will be
regularly updated to the latest versions of everything it bootstraps.

After each new ``pip`` release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of this PEP, will be run to ensure the private copies stored in the CPython
source repository have been updated to the latest versions.


Updating the ensurepip module API and CLI
-----------------------------------------

Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI
will be governed by the normal rules for the standard library: no
new features are permitted in maintenance releases.

However, the embedded components may be updated as noted above, so
the extracted ``pip`` may offer additional functionality in maintenance
releases.


Feature addition in maintenance releases
========================================

Adding a new module to the standard library in Python 2.7 and 3.3
maintenance releases breaks the usual policy of "no new features in
maintenance releases".

It is being proposed in this case as the current bootstrapping issues for
the third-party Python package ecosystem greatly affects the experience of
new users, especially on Python 2 where many Python 3 standard library
improvements are available as backports on PyPI, but are not included in
the Python 2 standard library.

By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem,
this change should aid the vast majority of current Python users, rather
than only those with the freedom to adopt Python 3.4 as soon as it is
released.


Uninstallation
==============

No changes are proposed to the uninstallation process by this PEP. The
bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.

While the case can be made for the CPython installers clearing out these
directories automatically, changing that behaviour is considered outside
the scope of this PEP.


Script Execution on Windows
===========================

While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
include the Scripts directory. Independently of this PEP, a proposal has
also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in
order to improve consistency with the typical script installation directory
names on \*nix systems.

Accordingly, in addition to adding the option to extract and install ``pip``
during installation, this PEP proposes that the Windows installer (and
``sysconfig``) in Python 3.4 and later be updated to:

- install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts
- add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the
PATH modification option is enabled during installation

For Python 2.7 and 3.3, it is proposed that the only change be the one
to bootstrap ``pip`` by default.

This means that, for Python 3.3, the most reliable way to invoke pip on
Windows (without tinkering manually with PATH) will actually be
``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both
Python 2 and 3 are installed) rather than simply calling ``pip``.

For Python 2.7 and 3.2, the most reliable mechanism will be to install the
standalone Python launcher for Windows and then use ``py -m pip`` as noted
above.

Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a
non-default version in the parallel installation case (and outside a virtual
environment). This change should also make the ``pyvenv`` command substantially
easier to invoke on Windows, along with all scripts installed by ``pip``,
``easy_install`` and similar tools.

While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
as the Python files in the Scripts directory correctly specify a Python version
in their shebang line or have an adjacent Windows executable (as
``easy_install`` and ``pip`` do).


Recommendations for Downstream Distributors
===========================================

A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
to provide a consistent, user-friendly experience to all users of Python
regardless of how they attained Python this PEP recommends and asks that
downstream distributors:

* Ensure that whenever Python is installed pip is also installed.

* This may take the form of separate packages with dependencies on each
other so that installing the Python package installs the pip package
and installing the pip package installs the Python package.
* Another reasonable way to implement this is to package pip separately but
ensure that there is some sort of global hook that will recommend
installing the separate pip package when a user executes ``pip`` without
it being installed. Systems that choose this option should ensure that
the ``pyvenv`` command still installs pip into the virtual environment
by default, but may modify the ``ensurepip`` module in the system Python
installation to redirect to the platform provided mechanism when
installing ``pip`` globally.

* Do not remove the bundled copy of pip.

* This is required for installation of pip into a virtual environment by the
``venv`` module.
* This is similar to the existing ``virtualenv`` package for which many
downstream distributors have already made exception to the common
"debundling" policy.
* This does mean that if ``pip`` needs to be updated due to a security
issue, so does the private copy in the ``ensurepip`` bootstrap module
* However, altering the private copy of pip to remove the embedded
CA certificate bundle and rely on the system CA bundle instead is a
reasonable change.

* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
using ``setup.py``.

* This will ensure that downstream packages can more easily utilize the
new metadata formats which may not have a ``setup.py``.

* Ensure that all features of this PEP continue to work with any modifications
made to the redistributed version of Python.

* Checking the version of pip that will be bootstrapped using
``python -m ensurepip --version`` or ``ensurepip.version()``.
* Installation of pip into a global or virtual python environment using
``python -m ensurepip`` or ``ensurepip.bootstrap()``.
* ``pip install --upgrade pip`` in a global installation should not affect
any already created virtual environments (but is permitted to affect
future virtual environments, even though it will not do so when using
the upstream version of ``ensurepip``).
* ``pip install --upgrade pip`` in a virtual environment should not affect
the global installation.

In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream ``pip``
based installation instructions into something appropriate for the platform.

Other Python implementations are also encouraged to follow these guidelines
where applicable.


Policies & Governance
=====================

The maintainers of the bootstrapped software and the CPython core team will
work together in order to address the needs of both. The bootstrapped
software will still remain external to CPython and this PEP does not
include CPython subsuming the development responsibilities or design
decisions of the bootstrapped software. This PEP aims to decrease the
burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv``
and other related projects.


Backwards Compatibility
-----------------------

The public API and CLI of the ``ensurepip`` module itself will fall under
the typical backwards compatibility policy of Python for its standard
library. The externally developed software that this PEP bundles does not.

Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.


Security Releases
-----------------

Any security update that affects the ``ensurepip`` module will be shared
prior to release with the Python Security Response Team
(***@python.org). The PSRT will then decide if the reported issue
warrants a security release of CPython with an updated private copy of
``pip``.


Appendix: Rejected Proposals
============================


Automatically contacting PyPI when bootstrapping pip
----------------------------------------------------

Earlier versions of this PEP called the bootstrapping module ``getpip`` and
defaulted to downloading and installing ``pip`` from PyPI, with the private
copy used only as a fallback option or when explicitly requested.

This resulted in several complex edge cases, along with difficulties in
defining a clean API and CLI for the bootstrap module. It also significantly
altered the default trust model for the binary installers published on
python.org, as end users would need to explicitly *opt-out* of trusting
the security of the PyPI ecosystem (rather than opting in to it by
explicitly invoking ``pip`` following installation).

As a result, the PEP was simplified to the current design, where the
bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI
is now always an explicit separate step, with direct access to the full
pip interface.


Implicit bootstrap
------------------

`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at
all. It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through
the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions issues,
if a user inadvertently attempted to bootstrap pip without write access to
the appropriate installation directories.


Including pip directly in the standard library
----------------------------------------------

Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned, through the inclusion and history of
``distutils`` in the standard library, that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a time frame that actually
affects users as any new features will not be available to the general
population for *years*.

Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.

There have also been issues in the past with the "dual maintenance" problem
if a project continues to be maintained externally while *also* having a
fork maintained in the standard library. Since external maintenance of
``pip`` will always be needed to support earlier Python versions, the
proposed bootstrapping mechanism will becoming the explicit responsibility
of the CPython core developers (assisted by the pip developers), while
pip issues reported to the CPython tracker will be migrated to the pip
issue tracker. There will no doubt still be some user confusion over which
tracker to use, but hopefully less than has been seen historically when
including complete public copies of third-party projects in the standard
library.

The approach described in this PEP also avoids some technical issues
related to handling CPython maintenance updates when pip has been
independently updated to a more recent version. The proposed pip-based
bootstrapping mechanism handles that automatically, since pip and the
system installer never get into a fight about who owns the pip
installation (it is always managed through pip, either directly, or
indirectly via the ``ensurepip`` bootstrap module).

Finally, the separate bootstrapping step means it also easy to avoid
installing ``pip`` at all if end users so desire. This is often the case
if integrators are using system packages to handle installation of
components written in multiple languages using a common set of tools.


Defaulting to --user installation
---------------------------------

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behavior would be
surprising (as it differs from the default behavior of pip itself)
and is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).


.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/


References
==========

.. [1] Discussion thread 1 (distutils-sig)
(https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html)

.. [2] Discussion thread 2 (distutils-sig)
(https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html)

.. [3] Discussion thread 3 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128723.html)

.. [4] Discussion thread 4 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128780.html)

.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew <http://brew.sh/>`
.. [#conda] `Conda <http://www.continuum.io/blog/conda>`

Copyright
=========

This document has been placed in the public domain.
--
Nick Coghlan | ***@gmail.com | Brisbane, Australia
Ned Deily
2013-09-23 23:34:54 UTC
Permalink
In general, I think this is a very important usability feature and I
am in favor of the general approach. Good work, all! I do have some
comments, primarily about items that are not currently addressed.
Post by Nick Coghlan
``ensurepip`` itself (including the private copy of ``pip`` and its
dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.
The PEP should address the license implications of this since the PEP will
add the source of pip and setuptools to all new Python releases, both
source and binary installers releases. For Python itself, we go to some
trouble to ensure that all contributions are contributed under a suitable
license. What steps are needed, initially and on-going, to ensure that
these third-party items to be pulled in at each release are compatible
with the license of Python itself? Who will do this: release managers?
Are updates needed in the internal and external documentation for releases
(README, LICENSE)? I took a quick look at the current source releases for
both pip and setuptools. I noted a reference to the LGPL for pip's CA
certs bundle; I didn't see a license file for setuptools.
Post by Nick Coghlan
This PEP proposes that, if pip still requires it as a dependency,
``ensurepip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
install the private copy in addition to installing ``pip`` itself.
However, this behavior is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.
Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``ensurepip`` in subsequent CPython releases.
The PEP does not sufficiently address the issue of the setuptools-provided
easy_install command. I think it is important to do that for several
reasons:

1. The PEP should make clear whether the easy_install scripts are or
are not installed, globally or in a venv, as a side effect of running
ensurepip, at least while pip has the implicit private dependency
on setuptools. By default, the scripts are installed so ensurepip
would need to take action to prevent this if that is not desired.

2. Despite the current strong preference for using pip as a command
line installer, the fact is that much third-party documentation still refers
to and recommends using easy_install for their projects. That is not
going to change overnight. The PEP should identify some steps needed to
facilitate the migration, like perhaps adding something to the "Installing
Python Modules" documentation explicitly deprecating easy_install use
and/or giving pip equivalents for the most common easy_install
scenarios (or linking to other documentation that does so).

3. A specific issue for OS X users is that, as part of OS X, Apple ships
versions of Python 2 and various third-party packages, including
setuptools (older, pre-reunification versions, at that) but not pip.
That means that OS X ships with easy_install and easy_install-2.n
commands in /usr/bin, including 2.7 in recent OS X releases. This has
been an ongoing source of confusion for OS X users who install a newer
version of Python. Depending on the distributor and/or source build
configure options, the new version of Python will be installed to a
different path, like /usr/local/bin. Which instance is used
is generally managed by direct PATH environment manipulations (or by
launcher programs similar to the Windows py launcher). If setuptools
is not installed using the new instance, users invoking the easy_install
command will by default end up with distributions being installed to a
system Python rather than to the new Python. This leads to the
unfortunately common scenario of:

sudo easy_install blah # --> installs to system Python 2.7
python -c 'import blah' # --> fails because using newer Python 2.7

This may very well be the most common current usability problem today
with python.org (and other third-party) Pythons on OS X.

With this PEP, we have an opportunity to fix this problem in one of
two ways. As long as pip requires setuptools and we do not inhibit
the installation of the easy_install scripts is to the same directory
as the python command itself, the new easy_install scripts will
shadow the Apple-supplied ones just as the python commands do. If it
is decided to not install the easy_install scripts now or in the
future (e.g. when pip no longer depends on setuptools),
ensurepip could install dummy easy_install scripts (if there
are none already installed there) that merely emit a message to the user
to encourage migration to pip and suggest, if needed, to use pip to install
setuptools to replace the placeholder easy_install. Either approach
would be a huge usability improvement over the situation today.

(The actual implementation details for either approach are slightly more
complicated for OS X framework builds like the python.org installers
as the actual scripts directory is a ``bin`` directory within the
framework itself with optional symlinks from ``/usr/local/bin`` to
the fw bin directory for specific commands. The installer will now need
to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as
proposed above, for ``easy_install``.)
Post by Nick Coghlan
A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
If you are going to call out Homebrew, you should include the other major
OS X package managers, MacPorts and Fink.
Post by Nick Coghlan
.. [#homebrew] `Homebrew <http://brew.sh/>`
.. [#macports] `MacPorts <http://macports.org>`
.. [#fink] `Fink <http://finkproject.org>`

One final comment is that the PEP does not go into any detail on how it
will be implemented. As it stands, there is a fair amount of one-time
work, including implementing ensurepip, changes to the Windows installer,
changes to the OS X installer, documentation changes, all with testing and
backporting to 2.7.x and 3.3.x. Then there are the on-going process changes
for all future releases, impacting all release team members, which will need
to be documented in PEP 101. Do we have an understanding of who is do the
big pieces and by when? For the record, I am willing to do the extra
one-time and ongoing work for the OS X installers but it would be helpful
to know how we are going to get it all done in time for upcoming releases.
--
Ned Deily,
***@acm.org
Donald Stufft
2013-09-23 23:39:53 UTC
Permalink
Post by Ned Deily
One final comment is that the PEP does not go into any detail on how it
will be implemented. As it stands, there is a fair amount of one-time
work, including implementing ensurepip, changes to the Windows installer,
changes to the OS X installer, documentation changes, all with testing and
backporting to 2.7.x and 3.3.x. Then there are the on-going process changes
for all future releases, impacting all release team members, which will need
to be documented in PEP 101. Do we have an understanding of who is do the
big pieces and by when? For the record, I am willing to do the extra
one-time and ongoing work for the OS X installers but it would be helpful
to know how we are going to get it all done in time for upcoming releases.
I'm in a meeting right now so I only skimmed this email, but I wanted to mention
here that I've been assuming I would be writing the python portions of this. I don't
know how to do the OSX/Window installer pieces and I was hoping that the
relevant people would be at least willing to help me figure it out or would implement
the parts that need to be done inside the installers (mostly running the command).

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft
2013-09-24 00:12:01 UTC
Permalink
Post by Ned Deily
In general, I think this is a very important usability feature and I
am in favor of the general approach. Good work, all! I do have some
comments, primarily about items that are not currently addressed.
Post by Nick Coghlan
``ensurepip`` itself (including the private copy of ``pip`` and its
dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.
The PEP should address the license implications of this since the PEP will
add the source of pip and setuptools to all new Python releases, both
source and binary installers releases. For Python itself, we go to some
trouble to ensure that all contributions are contributed under a suitable
license. What steps are needed, initially and on-going, to ensure that
these third-party items to be pulled in at each release are compatible
with the license of Python itself? Who will do this: release managers?
Are updates needed in the internal and external documentation for releases
(README, LICENSE)? I took a quick look at the current source releases for
both pip and setuptools. I noted a reference to the LGPL for pip's CA
certs bundle; I didn't see a license file for setuptools.
As far as I know the certificate bundle is licensed under either GPL, MPL,
or LGPL. However there is debate about wether a CA bundle *can* be
licensed at all (see https://github.com/pypa/pip/pull/971).

Pip also includes some code taken from other libraries however everything
is licensed as either 1, 2, or 3 clause BSD (besides the aforementioned
certificates). I can't think of us being ok with adding a copyleft license
such as GPL so code we bring in will likely be restricted to things like BSD.
Post by Ned Deily
Post by Nick Coghlan
This PEP proposes that, if pip still requires it as a dependency,
``ensurepip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
install the private copy in addition to installing ``pip`` itself.
However, this behavior is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.
Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``ensurepip`` in subsequent CPython releases.
The PEP does not sufficiently address the issue of the setuptools-provided
easy_install command. I think it is important to do that for several
1. The PEP should make clear whether the easy_install scripts are or
are not installed, globally or in a venv, as a side effect of running
ensurepip, at least while pip has the implicit private dependency
on setuptools. By default, the scripts are installed so ensurepip
would need to take action to prevent this if that is not desired.
2. Despite the current strong preference for using pip as a command
line installer, the fact is that much third-party documentation still refers
to and recommends using easy_install for their projects. That is not
going to change overnight. The PEP should identify some steps needed to
facilitate the migration, like perhaps adding something to the "Installing
Python Modules" documentation explicitly deprecating easy_install use
and/or giving pip equivalents for the most common easy_install
scenarios (or linking to other documentation that does so).
3. A specific issue for OS X users is that, as part of OS X, Apple ships
versions of Python 2 and various third-party packages, including
setuptools (older, pre-reunification versions, at that) but not pip.
That means that OS X ships with easy_install and easy_install-2.n
commands in /usr/bin, including 2.7 in recent OS X releases. This has
been an ongoing source of confusion for OS X users who install a newer
version of Python. Depending on the distributor and/or source build
configure options, the new version of Python will be installed to a
different path, like /usr/local/bin. Which instance is used
is generally managed by direct PATH environment manipulations (or by
launcher programs similar to the Windows py launcher). If setuptools
is not installed using the new instance, users invoking the easy_install
command will by default end up with distributions being installed to a
system Python rather than to the new Python. This leads to the
sudo easy_install blah # --> installs to system Python 2.7
python -c 'import blah' # --> fails because using newer Python 2.7
This may very well be the most common current usability problem today
with python.org (and other third-party) Pythons on OS X.
With this PEP, we have an opportunity to fix this problem in one of
two ways. As long as pip requires setuptools and we do not inhibit
the installation of the easy_install scripts is to the same directory
as the python command itself, the new easy_install scripts will
shadow the Apple-supplied ones just as the python commands do. If it
is decided to not install the easy_install scripts now or in the
future (e.g. when pip no longer depends on setuptools),
ensurepip could install dummy easy_install scripts (if there
are none already installed there) that merely emit a message to the user
to encourage migration to pip and suggest, if needed, to use pip to install
setuptools to replace the placeholder easy_install. Either approach
would be a huge usability improvement over the situation today.
(The actual implementation details for either approach are slightly more
complicated for OS X framework builds like the python.org installers
as the actual scripts directory is a ``bin`` directory within the
framework itself with optional symlinks from ``/usr/local/bin`` to
the fw bin directory for specific commands. The installer will now need
to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as
proposed above, for ``easy_install``.)
So this is an interesting question. On one hand I would say we shouldn't be
installing easy_install as that feels very user facing to me and the fact that
setuptools is being installed is an implementation detail. On the other hand
if we put in stub commands that just simply direct the user to use pip then
people who *want* to use easy_install (perhaps for Egg support) won't be
able too (unless perhaps something is released on PyPI that restores the
easy_install commands?)
Post by Ned Deily
Post by Nick Coghlan
A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
If you are going to call out Homebrew, you should include the other major
OS X package managers, MacPorts and Fink.
Post by Nick Coghlan
.. [#homebrew] `Homebrew <http://brew.sh/>`
.. [#macports] `MacPorts <http://macports.org>`
.. [#fink] `Fink <http://finkproject.org>`
I can add them, I added homebrew because it's what I use :) Didn't really
think to hard about being super inclusive.
Post by Ned Deily
One final comment is that the PEP does not go into any detail on how it
will be implemented. As it stands, there is a fair amount of one-time
work, including implementing ensurepip, changes to the Windows installer,
changes to the OS X installer, documentation changes, all with testing and
backporting to 2.7.x and 3.3.x. Then there are the on-going process changes
for all future releases, impacting all release team members, which will need
to be documented in PEP 101. Do we have an understanding of who is do the
big pieces and by when? For the record, I am willing to do the extra
one-time and ongoing work for the OS X installers but it would be helpful
to know how we are going to get it all done in time for upcoming releases.
To further expand upon my original answer, I'm volunteering to do the initial
work on the ensurepip library, the scripts that will automated the ongoing
maintenance work, and the back porting of both of the above. I can also attempt
to do the OSX Installer and Windows installer but I have zero idea how the
installers actually function so it's probably better for someone else to do that.

Since it sounds like you're willing to do the work for the OSX installer that
saves me from having to flail around trying to figure out how to do that part,
so hopefully MvL or someone can do the same with the Windows installer.

I'm not sure what needs done outside of the up front work, do I just propose
changes to PEP 101? Do I make a whole new PEP? Is there more than
just updating PEP 101?
Post by Ned Deily
--
Ned Deily,
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft
2013-09-24 00:35:11 UTC
Permalink
Post by Donald Stufft
Post by Ned Deily
Post by Nick Coghlan
A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
If you are going to call out Homebrew, you should include the other major
OS X package managers, MacPorts and Fink.
Post by Nick Coghlan
.. [#homebrew] `Homebrew <http://brew.sh/>`
.. [#macports] `MacPorts <http://macports.org>`
.. [#fink] `Fink <http://finkproject.org>`
I can add them, I added homebrew because it's what I use :) Didn't really
think to hard about being super inclusive.
Updated this in PEP453.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Ned Deily
2013-09-24 04:32:37 UTC
Permalink
Post by Donald Stufft
[... license implications ...]
As far as I know the certificate bundle is licensed under either GPL, MPL,
or LGPL. However there is debate about wether a CA bundle *can* be
licensed at all (see https://github.com/pypa/pip/pull/971).
Pip also includes some code taken from other libraries however everything
is licensed as either 1, 2, or 3 clause BSD (besides the aforementioned
certificates). I can't think of us being ok with adding a copyleft license
such as GPL so code we bring in will likely be restricted to things like BSD.
IANAL, but I think it would be good if, at least, the setuptools license were
clearer and the LGPL reference for the cert bundle was changed. It *might* be
a good idea to get an opinion from Van Lindberg. But I'm happy to defer to
Martin's judgement on this.
Post by Donald Stufft
[... easy_install on OS X ...]
So this is an interesting question. On one hand I would say we shouldn't be
installing easy_install as that feels very user facing to me and the fact that
setuptools is being installed is an implementation detail. On the other hand
if we put in stub commands that just simply direct the user to use pip then
people who *want* to use easy_install (perhaps for Egg support) won't be
able too (unless perhaps something is released on PyPI that restores the
easy_install commands?)
I was thinking that, in the latter case, those users who really want to use
easy_install could be told to just use pip to install a PyPI version of
setuptools which would replace the stub commands with the real things -
assuming that works.
Post by Donald Stufft
[... implementation plan ...]
To further expand upon my original answer, I'm volunteering to do the initial
work on the ensurepip library, the scripts that will automated the ongoing
maintenance work, and the back porting of both of the above. I can also attempt
to do the OSX Installer and Windows installer but I have zero idea how the
installers actually function so it's probably better for someone else to do that.
Since it sounds like you're willing to do the work for the OSX installer that
saves me from having to flail around trying to figure out how to do that part,
so hopefully MvL or someone can do the same with the Windows installer.
I'm not sure what needs done outside of the up front work, do I just propose
changes to PEP 101? Do I make a whole new PEP? Is there more than
just updating PEP 101?
I think the thing to do is get review buy-in from the release managers for the
affected active releases (2.7.x = Benjamin, 3.3.x = Georg, 3.4.x = Larry) and
let them worry about updating PEP 101. The key point is that this PEP is
implicitly adding some new responsibilities for the release team; I think they
just need to be explicit.
--
Ned Deily,
***@acm.org
Donald Stufft
2013-09-24 04:35:05 UTC
Permalink
Post by Ned Deily
IANAL, but I think it would be good if, at least, the setuptools license were
clearer and the LGPL reference for the cert bundle was changed. It *might* be
a good idea to get an opinion from Van Lindberg. But I'm happy to defer to
Martin's judgement on this.
After your concern was raised I went ahead and emailed VanL.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Nick Coghlan
2013-09-24 07:52:32 UTC
Permalink
Post by Ned Deily
In general, I think this is a very important usability feature and I
am in favor of the general approach. Good work, all! I do have some
comments, primarily about items that are not currently addressed.
Your reply and Barry's suggest that Betteridge's law [1] applies to
email subject lines, too ;)

As far as easy_install goes, my current plan was actually to tackle
that on the upstream side. If pip still depends on setuptools by the
time of the Python 3.4 release, then it will depend on the *real*
setuptools, easy_install and all. From my perspective, one golden rule
of this integration is that we do *not* mess with the contents of the
wheel files for pip and its dependencies - they're pristine upstream
releases. This is mostly for technical reasons, but it also draws a
sharp line of demarcation for any "aggregation or derivation?"
questions, too.

If pip has been updated by the time of its inclusion to depend on a
cut down setuptools derivative that omits easy_install (or pip has
switched to its own internal replacements instead), so much the
better, but I consider that to be essentially independent of the
CPython bundling situation, since it isn't something we have direct
control over, and I consider the slight downside of potentially
installing easy_install alongside pip to be dwarfed by the benefits of
installing pip.

As far as I am aware, the licensing on setuptools is currently limited
to the "ZPL or PSF" declaration in the distribution metadata.

Cheers,
Nick.

[1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines
--
Nick Coghlan | ***@gmail.com | Brisbane, Australia
Barry Warsaw
2013-09-25 20:50:22 UTC
Permalink
Post by Nick Coghlan
With the last round of updates, I believe PEP 453 is ready for
Martin's pronouncement.
I want to raise an objection to PEP's proposal to add this as a new feature to
Python 2.7 and 3.3. I understand the rationale as stated here:

http://www.python.org/dev/peps/pep-0453/#id38

but I still object. ;)

I think we've learned it's generally pretty risky to add new features in point
releases, and this is a fairly major (and would be the first?) violation of
the principal of Python 2.7's conservative maintenance.

At the very least, I'd like to know why there's no other alternative. For
example, you could provide a PyPI package with this functionality, and let the
binary packagers adopt it into their binary packages, for platforms that care.

So for example, Linux systems which start from the source tarball could opt
out (by doing nothing special) since there are other ways to achieve a similar
benefit. For OS X and Windows users getting binary downloads from
www.python.org, I think it would be acceptable if the .dmg or .msi included
this external package.

Why does it have to be added to the source tree for stable releases?

Cheers,
-Barry
Antoine Pitrou
2013-09-25 21:04:20 UTC
Permalink
On Wed, 25 Sep 2013 16:50:22 -0400
Post by Barry Warsaw
Post by Nick Coghlan
With the last round of updates, I believe PEP 453 is ready for
Martin's pronouncement.
I want to raise an objection to PEP's proposal to add this as a new feature to
http://www.python.org/dev/peps/pep-0453/#id38
but I still object. ;)
I think we've learned it's generally pretty risky to add new features in point
releases, and this is a fairly major (and would be the first?) violation of
the principal of Python 2.7's conservative maintenance.
At the very least, I'd like to know why there's no other alternative. For
example, you could provide a PyPI package with this functionality, and let the
binary packagers adopt it into their binary packages, for platforms that care.
So for example, Linux systems which start from the source tarball could opt
out (by doing nothing special) since there are other ways to achieve a similar
benefit. For OS X and Windows users getting binary downloads from
www.python.org, I think it would be acceptable if the .dmg or .msi included
this external package.
If it's bundled in the official binary installers, it makes little
sense not to add to the source tree, IMHO. First because most people
get one of the installers rather than the source tree (on
http://www.python.org/webstats/, MSI builds win by ten to one). Second
because the few people who get the source tree are experienced users
who will get bitten less by hypothetical quirks or oddities.
Third because otherwise we're probably making things harder for our
beloved packagers ;-)

Regards

Antoine.
Nick Coghlan
2013-09-25 21:14:18 UTC
Permalink
Post by Barry Warsaw
Post by Nick Coghlan
With the last round of updates, I believe PEP 453 is ready for
Martin's pronouncement.
I want to raise an objection to PEP's proposal to add this as a new feature to
http://www.python.org/dev/peps/pep-0453/#id38
but I still object. ;)
I think we've learned it's generally pretty risky to add new features in point
releases, and this is a fairly major (and would be the first?) violation of
the principal of Python 2.7's conservative maintenance.
At the very least, I'd like to know why there's no other alternative. For
example, you could provide a PyPI package with this functionality, and let the
binary packagers adopt it into their binary packages, for platforms that care.
So for example, Linux systems which start from the source tarball could opt
out (by doing nothing special) since there are other ways to achieve a similar
benefit. For OS X and Windows users getting binary downloads from
www.python.org, I think it would be acceptable if the .dmg or .msi included
this external package.
Why does it have to be added to the source tree for stable releases?
If it can get into the installers another way, it doesn't, really. It only
needs to be in the source tree for 3.4+.

Alternatively, we could be explicit that the "don't remove it" guideline
only applies in 3.4+, where it will be needed by pyvenv.

That way, people doing custom builds from source can easily bootstrap pip
and we don't need to come up with an alternate solution for getting them
into the installers.

Cheers,
Nick.
Post by Barry Warsaw
Cheers,
-Barry
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Barry Warsaw
2013-09-25 21:53:48 UTC
Permalink
Post by Nick Coghlan
Post by Barry Warsaw
Why does it have to be added to the source tree for stable releases?
If it can get into the installers another way, it doesn't, really. It only
needs to be in the source tree for 3.4+.
Sure, new features can of course go in the development release. :)

-Barry
Donald Stufft
2013-09-25 21:33:24 UTC
Permalink
Post by Barry Warsaw
Why does it have to be added to the source tree for stable releases?
I think it should be placed in the source tree for the stable releases. The
reasoning is that 2.7 is going to stick around for a long time. Immediately
this won't be ubiquitous but as time goes on you'll be able to be ensured
that a ``python -m ensure pip`` exists so that in situations where you don't
have pip you'll be able to install it.

While not directly relevant to the change I do think this is something users
support. I've received a fair but of feedback as I was writing the original
draft of the PEP and then throughout the process when me and Nick
were working on it. Almost all of it was positive (some of it extremely so)
a fair bit of them pointed out the backport to 2.7 as something they were
*really* wanting.

An early draft of this did not have the backport to 2.7 and when I showed
*that* version around to get feedback people were less enthusiastic
about it and generally viewed it as "nice but worthless to me for N years".

What users want isn't rationale in and of itself but I think it's an important
data point, especially given how long 2.7.LASTEVER is going to be
relevant to end users.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Barry Warsaw
2013-09-25 21:51:29 UTC
Permalink
Post by Donald Stufft
I think it should be placed in the source tree for the stable releases. The
reasoning is that 2.7 is going to stick around for a long time. Immediately
this won't be ubiquitous but as time goes on you'll be able to be ensured
that a ``python -m ensure pip`` exists so that in situations where you don't
have pip you'll be able to install it.
While not directly relevant to the change I do think this is something users
support. I've received a fair but of feedback as I was writing the original
draft of the PEP and then throughout the process when me and Nick
were working on it. Almost all of it was positive (some of it extremely so)
a fair bit of them pointed out the backport to 2.7 as something they were
*really* wanting.
An early draft of this did not have the backport to 2.7 and when I showed
*that* version around to get feedback people were less enthusiastic
about it and generally viewed it as "nice but worthless to me for N years".
Yeah, I get all this, but it's essentially the same reasoning that lead to
Python 2.2.1's addition of True and False. The same flaw occurs though
because you cannot guarantee that invocation of `python -m ensurepip` will
work unless you micro version check (LBYL) or prepare to catch resulting
errors. This essentially means that if you want to write portable Python 2.7
code and scripts, you have to be very cautious, or what works on one box won't
necessarily work on another, even though they both have "Python 2.7".

It also means that anybody who's documenting about this great new feature has
to warn people that, well, maybe it won't work on your machine even though you
have Python 2.7 because you don't have a new enough Python 2.7.

It means that not all Python 2.7's are alike in a rather fundamental and
highly visible way.

I personally think that's a recipe for more problems than it solves, and if I
was the RM for 2.7 I would not allow it. But I'm not.
Post by Donald Stufft
What users want isn't rationale in and of itself but I think it's an important
data point, especially given how long 2.7.LASTEVER is going to be
relevant to end users.
Users want what users want. It's our job to make the best technical decisions
based on all the facts. I understand that Python 2.7 will be around for a
long time, and that it would be very convenient to do this. Why is this not
opening up the door to more new features being added in future Python 2.7
point releases (or any other stable release)?

At least I think the burden should be very high, and the PEP should do a
better job of explaining why *no* other option will accomplish the goal.

Cheers,
-Barry
Donald Stufft
2013-09-25 22:15:18 UTC
Permalink
Post by Barry Warsaw
Post by Donald Stufft
I think it should be placed in the source tree for the stable releases. The
reasoning is that 2.7 is going to stick around for a long time. Immediately
this won't be ubiquitous but as time goes on you'll be able to be ensured
that a ``python -m ensure pip`` exists so that in situations where you don't
have pip you'll be able to install it.
While not directly relevant to the change I do think this is something users
support. I've received a fair but of feedback as I was writing the original
draft of the PEP and then throughout the process when me and Nick
were working on it. Almost all of it was positive (some of it extremely so)
a fair bit of them pointed out the backport to 2.7 as something they were
*really* wanting.
An early draft of this did not have the backport to 2.7 and when I showed
*that* version around to get feedback people were less enthusiastic
about it and generally viewed it as "nice but worthless to me for N years".
Yeah, I get all this, but it's essentially the same reasoning that lead to
Python 2.2.1's addition of True and False. The same flaw occurs though
because you cannot guarantee that invocation of `python -m ensurepip` will
work unless you micro version check (LBYL) or prepare to catch resulting
errors. This essentially means that if you want to write portable Python 2.7
code and scripts, you have to be very cautious, or what works on one box won't
necessarily work on another, even though they both have "Python 2.7".
It also means that anybody who's documenting about this great new feature has
to warn people that, well, maybe it won't work on your machine even though you
have Python 2.7 because you don't have a new enough Python 2.7.
It means that not all Python 2.7's are alike in a rather fundamental and
highly visible way.
I personally think that's a recipe for more problems than it solves, and if I
was the RM for 2.7 I would not allow it. But I'm not.
Well I don't think many scripts will be executing ensurepip, maybe i'm wrong,
but yes it does mean that not all Python 2.7's are alike. Most likely though
at some point 2.7.XXX is going to be near standard as the 2.7 hold outs
stay on it and time progresses. It more or less shifts the "time until projects
can assume people have pip available or easily installable" from "until
3.4+ is ubiquitous and 2.7 is gone (Years?) to "until Python 2.7.6 is ubiquitous"
which will be a much shorter time.
Post by Barry Warsaw
Post by Donald Stufft
What users want isn't rationale in and of itself but I think it's an important
data point, especially given how long 2.7.LASTEVER is going to be
relevant to end users.
Users want what users want. It's our job to make the best technical decisions
based on all the facts. I understand that Python 2.7 will be around for a
long time, and that it would be very convenient to do this. Why is this not
opening up the door to more new features being added in future Python 2.7
point releases (or any other stable release)?
Other than the fact that it in itself is an exception to the rule this actually strengthens
that because it makes it easier for people to install new things into Python 2.7
outside of the release cycle. So things like the new enum module can more easily
be backported and used as an installable than as part of the language. The obvious
exemption to this is language level features, but as this itself isn't a language level
feature i'm not sure how relevant that is.
Post by Barry Warsaw
At least I think the burden should be very high, and the PEP should do a
better job of explaining why *no* other option will accomplish the goal.
This is totally fair and we can expand on it more for sure.
Post by Barry Warsaw
Cheers,
-Barry
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Barry Warsaw
2013-09-25 23:04:10 UTC
Permalink
Post by Donald Stufft
Well I don't think many scripts will be executing ensurepip, maybe i'm wrong,
but yes it does mean that not all Python 2.7's are alike. Most likely though
at some point 2.7.XXX is going to be near standard as the 2.7 hold outs stay
on it and time progresses.
I think it's too optimistic to hope that we'll ever see significant
convergence across the Python 2.7 ecosystem.

For example, I'd be really surprised if you'd see this show up in a stable
release of Debian or Ubuntu (the only ones I can speak somewhat
authoritatively about), even if we wanted to cherrypick updates in newer
Python 2.7 point releases. Linux distros tend to be even more anal about
avoiding new features in stable releases, for the same really good reasons,
IMHO :).

I think for the platforms you're targeting (OS X, Windows), you're going to
see a wide variety for a very long time. It's depressing how infrequently
people upgrade *anything*.

Another reason to oppose this is what I've heard quite often from people
regarding Python 2.7. I've been told that many folks are actually really
happy with using 2.7 precisely because it extremely stable. They don't have
to worry about their stuff breaking or incompatibilities cropping up, because
it *doesn't* change. Python 2.7 is like a long-term maintenance release, with
guaranteed multi-year stability, and I think while that was unintended, it's a
*good* thing. This PEP proposes to break that, and I'm loathe to give up that
good reputation for this particular feature.
Post by Donald Stufft
It more or less shifts the "time until projects can assume people have pip
available or easily installable" from "until 3.4+ is ubiquitous and 2.7 is
gone (Years?) to "until Python 2.7.6 is ubiquitous" which will be a much
shorter time.
Yeah, maybe, but I'm pretty skeptical about the enthusiasm for upgrades. ;)
Post by Donald Stufft
Post by Barry Warsaw
Users want what users want. It's our job to make the best technical
decisions based on all the facts. I understand that Python 2.7 will be
around for a long time, and that it would be very convenient to do this.
Why is this not opening up the door to more new features being added in
future Python 2.7 point releases (or any other stable release)?
Other than the fact that it in itself is an exception to the rule this
actually strengthens that because it makes it easier for people to install
new things into Python 2.7 outside of the release cycle. So things like the
new enum module can more easily be backported and used as an installable than
as part of the language. The obvious exemption to this is language level
features, but as this itself isn't a language level feature i'm not sure how
relevant that is.
So, why shouldn't we add enum to the Python 2.7 stdlib? Or any other new
feature? Just be aware that if this PEP is accepted for Python 2.7, the bar
will be set much lower for other Really Useful Features That Make Users Lives
Better.
Post by Donald Stufft
Post by Barry Warsaw
At least I think the burden should be very high, and the PEP should do a
better job of explaining why *no* other option will accomplish the goal.
This is totally fair and we can expand on it more for sure.
Cool. Maybe in the course of that discussion, a better alternative that
doesn't add a new feature to Python 2.7 will present itself. I really hope
so.

-Barry
Donald Stufft
2013-09-25 23:08:32 UTC
Permalink
Post by Barry Warsaw
So, why shouldn't we add enum to the Python 2.7 stdlib? Or any other new
feature? Just be aware that if this PEP is accepted for Python 2.7, the bar
will be set much lower for other Really Useful Features That Make Users Lives
Better.
Because with PEP453 you can just ``pip install enum34`` it :)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Barry Warsaw
2013-09-25 23:15:05 UTC
Permalink
Post by Donald Stufft
Post by Barry Warsaw
So, why shouldn't we add enum to the Python 2.7 stdlib?
Because with PEP453 you can just ``pip install enum34`` it :)
Of course, pip messing with global Python state on a package managed system
like most Linux distros, is a whole 'nuther happy fun ball of killer
beeswax. :)

mixing-metaphors-ly y'rs,
-Barry
Donald Stufft
2013-09-25 23:18:56 UTC
Permalink
Post by Barry Warsaw
Of course, pip messing with global Python state on a package managed system
like most Linux distros, is a whole 'nuther happy fun ball of killer
beeswax. :)
mixing-metaphors-ly y'rs,
-Barry
<virtualenv invocation not shown>

(For reals a pip and apt-get playing nicely is on my stack of PEPs to do)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Antoine Pitrou
2013-09-26 06:35:36 UTC
Permalink
On Wed, 25 Sep 2013 19:15:05 -0400
Post by Barry Warsaw
Post by Donald Stufft
Post by Barry Warsaw
So, why shouldn't we add enum to the Python 2.7 stdlib?
Because with PEP453 you can just ``pip install enum34`` it :)
Of course, pip messing with global Python state on a package managed system
like most Linux distros, is a whole 'nuther happy fun ball of killer
beeswax. :)
pip install --user enum34 :-)
Stephen J. Turnbull
2013-09-26 01:52:25 UTC
Permalink
Received: from localhost (HELO mail.python.org) (127.0.0.1)
by albatross.python.org with SMTP; 26 Sep 2013 03:52:27 +0200
Received: from mgmt2.sk.tsukuba.ac.jp (unknown [130.158.97.224])
by mail.python.org (Postfix) with ESMTP;
Thu, 26 Sep 2013 03:52:26 +0200 (CEST)
Received: from uwakimon.sk.tsukuba.ac.jp (uwakimon.sk.tsukuba.ac.jp
[130.158.99.156])
by mgmt2.sk.tsukuba.ac.jp (Postfix) with ESMTP id 775019709A0;
Thu, 26 Sep 2013 10:52:25 +0900 (JST)
Received: by uwakimon.sk.tsukuba.ac.jp (Postfix, from userid 1000)
id 2BB081A27EF; Thu, 26 Sep 2013 10:52:25 +0900 (JST)
In-Reply-To: <7FBFFC39-A0AF-47FD-9715-***@stufft.io>
X-Mailer: VM undefined under 21.5 (beta32) "habanero" b0d40183ac79 XEmacs
Lucid (x86_64-unknown-linux)
X-BeenThere: python-***@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Python core developers <python-dev.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-dev>,
<mailto:python-dev-***@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-dev/>
List-Post: <mailto:python-***@python.org>
List-Help: <mailto:python-dev-***@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-dev>,
<mailto:python-dev-***@python.org?subject=subscribe>
Errors-To: python-dev-bounces+python-python-dev=***@python.org
Sender: "Python-Dev"
<python-dev-bounces+python-python-dev=***@python.org>
Archived-At: <http://permalink.gmane.org/gmane.comp.python.devel/142157>
Post by Donald Stufft
Post by Barry Warsaw
So, why shouldn't we add enum to the Python 2.7 stdlib? Or any
other new feature? Just be aware that if this PEP is accepted
for Python 2.7, the bar will be set much lower for other Really
Useful Features That Make Users Lives Better.
Because with PEP453 you can just ``pip install enum34`` it :)
Uh, I've been doing that for years (well, not with ``enum34``, to be
sure). And not just with 2.7, 2.6 and 2.5 too. Who needs PEP 453?
Not me (ironically, I'll probably have a PEP 453 Python within 48
hours of its public release).

OTOH, as Barry points out, the target population doesn't upgrade very
often. What gets them to upgrade (or install in parallel) is the
installation of a package that requires an upgraded Python. But that
typically means a bump from x.y to x.z, and that will never happen to
2.7. So they'll see a failure of some kind anyway if the ensurepip
feature is used! That is going to blowback on distributors, which is
going to make PEP 453 real popular with them. Not.
Nick Coghlan
2013-09-26 04:39:40 UTC
Permalink
Post by Stephen J. Turnbull
Post by Donald Stufft
Post by Barry Warsaw
So, why shouldn't we add enum to the Python 2.7 stdlib? Or any
other new feature? Just be aware that if this PEP is accepted
for Python 2.7, the bar will be set much lower for other Really
Useful Features That Make Users Lives Better.
Because with PEP453 you can just ``pip install enum34`` it :)
Uh, I've been doing that for years (well, not with ``enum34``, to be
sure). And not just with 2.7, 2.6 and 2.5 too. Who needs PEP 453?
Not me (ironically, I'll probably have a PEP 453 Python within 48
hours of its public release).
New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P

Linux and other *nix platforms are affected primarily due to the fact
that we also want pyvenv to start making pip available by default in
3.4+, so we need the PEP to account for making that possible while
still playing nice with the Linux distro vendors.
Post by Stephen J. Turnbull
OTOH, as Barry points out, the target population doesn't upgrade very
often.
If somebody already has Python installed, they're not really part of
the target population, except in the sense that the pip bootstrap
instruction become "upgrade Python to the latest maintenance release"
rather than "got to this website, download and run this bootstrap
script". (Keeping those instructions consistent for 3.3 users as well
would be the advantage of updating 3.3 in addition to 2.7 and 3.4)
Post by Stephen J. Turnbull
What gets them to upgrade (or install in parallel) is the
installation of a package that requires an upgraded Python. But that
typically means a bump from x.y to x.z, and that will never happen to
2.7. So they'll see a failure of some kind anyway if the ensurepip
feature is used! That is going to blowback on distributors, which is
going to make PEP 453 real popular with them. Not.
I'm not sure what usage model you're assuming for _ensurepip, but it
appears to be wrong. End users should be able to just run pip, and
either have it work, or else get a message from the OS vendor telling
them to install the appropriate system package. If it doesn't work,
then the advice will be to upgrade to the latest Python maintenance
release.

Regards,
Nick.
--
Nick Coghlan | ***@gmail.com | Brisbane, Australia
Stephen J. Turnbull
2013-09-27 14:08:33 UTC
Permalink
Post by Nick Coghlan
I'm not sure what usage model you're assuming for _ensurepip, but it
appears to be wrong. End users should be able to just run pip, and
either have it work, or else get a message from the OS vendor telling
them to install the appropriate system package.
I don't understand how you arrange for that message on existing
installs. Wouldn't it be easier to just lobby the distros to make
Python dependent on pip? And speaking of vendors, do you expect Apple
and Microsoft to provide such a message? And such a system package?

If you already are running a Linux distro or MacPorts, you do "apt-get
python-pip" and "port install py-pip" respectively. I bet Cygwin is
Post by Nick Coghlan
New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P
I bet those users are *not* running third-party distros, but rather
are sitting in front of pretty close to plain vanilla factory installs
of the OS, no? And "new users" on Mac OS X already have "old installs"
of Python, no?

That's my model. In that model I don't see backporting PEP 453 to
Python 2.7 as being a sufficiently reliable way to provide a smooth
user experience to justify breaking the "no new features" rule (which
is at the "read my lips" level after the True/False fiasco).

Get a commitment from Apple to put 2.7.6 in their next upgrades for
their OS, and then maybe you'd have enough leverage to tip the
balance. I certainly would concede the point. But without that,
you're telling Mac users "you have to upgrade Python from a 3rd party
site." Is that really the way to make new users participating in a
tutorial session happy? (You tell me, I'm just introspecting here.)

Steve
Paul Moore
2013-09-27 14:26:41 UTC
Permalink
Post by Stephen J. Turnbull
Post by Nick Coghlan
New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P
I bet those users are *not* running third-party distros, but rather
are sitting in front of pretty close to plain vanilla factory installs
of the OS, no? And "new users" on Mac OS X already have "old installs"
of Python, no?
Windows users who don't use a third-party distro like Enthought,
generally download the python.org installer. At the moment, that
doesn't give them a "pip" command. So if they want to install any
third party package, they have to start by installing pip. The
instructions for that are reasonably clear, but non-trivial, largely
because tools like curl are not commonly available on Windows, and by
default running a Python script may not do what you expect.

Rather than try to fix these problems (which are *hard*) the intent is
to have the pip command installed by the python.org installer.

So, for Windows users, installing a package becomes "pip install XXX".

The problem is that this doesn't work for existing releases (2.7 and
3.3). Rather than have a convoluted set of instructions that goes "if
you're not on Python 3.4+, (basically do what we have at the moment)"
the PEP makes it possible to say "If you have a Python older than 3.4,
upgrade to the latest maintenance release of your version of Python,
then use pip". People who have reasons not to use the latest
*maintenance* release are assumed to be special cases who can either
deal with the more complex current process, or can work something out
for themselves. (They may not be, but there's not much we can do in
practice to help them).

Nobody in the basic target group should be running ensurepip (under
any name) manually.

I can't speak for Linux distros or OSX users, but for Windows I do
believe that this is a significant improvement, and worth the (IMO,
negligible) risk involved in adding this to a maintenance release.
There *may* be some mileage in a debate about why we don't just bundle
pip in the installer, rather than having helpers in core Python
itself, but the long & short of it is that ensurepip is needed for
3.4+ to make installing pip in venvs work by default, and why have a
completely different and so less well tested mechanism for older
versions?

Paul.
Mark Lawrence
2013-09-27 14:39:50 UTC
Permalink
Post by Paul Moore
So, for Windows users, installing a package becomes "pip install XXX".
Apologies if this has already been said but it only works if you've
already installed an appropriate compiler. I've lost count of the
number of times I've tried this, got the (rather cyptic) "error: Unable
to find vcvarsall.bat" message and then gone off to find a suitable
binary download.
--
Cheers.

Mark Lawrence
Donald Stufft
2013-09-27 14:42:29 UTC
Permalink
Apologies if this has already been said but it only works if you've already installed an appropriate compiler. I've lost count of the number of times I've tried this, got the (rather cyptic) "error: Unable to find vcvarsall.bat" message and then gone off to find a suitable binary download.
Going forward Wheels are binary packages that pip can install.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Paul Moore
2013-09-27 15:06:33 UTC
Permalink
Post by Paul Moore
So, for Windows users, installing a package becomes "pip install XXX".
Apologies if this has already been said but it only works if you've already
installed an appropriate compiler. I've lost count of the number of times
I've tried this, got the (rather cyptic) "error: Unable to find
vcvarsall.bat" message and then gone off to find a suitable binary download.
I was assuming that you had whatever's necessary to install - a
compiler, any necessary libraries, whatever. Or more plausibly, as
Donald pointed out, a wheel.

But yes, you're right, this doesn't solve the problem of *building*
distributions. It's not intended to (except by enabling communities to
start distributing wheels as a pain-free binary solution). Whether
that's an issue depends on the community (AIUI, it's a major hassle
for scientific users, not so much for web developers, for example)

Paul
R. David Murray
2013-09-27 16:15:31 UTC
Permalink
Post by Paul Moore
Post by Stephen J. Turnbull
Post by Nick Coghlan
New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P
I bet those users are *not* running third-party distros, but rather
are sitting in front of pretty close to plain vanilla factory installs
of the OS, no? And "new users" on Mac OS X already have "old installs"
of Python, no?
Windows users who don't use a third-party distro like Enthought,
generally download the python.org installer. At the moment, that
doesn't give them a "pip" command. So if they want to install any
third party package, they have to start by installing pip. The
instructions for that are reasonably clear, but non-trivial, largely
because tools like curl are not commonly available on Windows, and by
default running a Python script may not do what you expect.
Rather than try to fix these problems (which are *hard*) the intent is
to have the pip command installed by the python.org installer.
[...]
Post by Paul Moore
I can't speak for Linux distros or OSX users, but for Windows I do
believe that this is a significant improvement, and worth the (IMO,
negligible) risk involved in adding this to a maintenance release.
I'm not an OS X user, and probably most people on this list use macports
or something similar, which essentially puts them in the same boat as
the linux users...and there's a section in the PEP about that (that's
where the message about installing pip if you run pip and the distro
didn't include it with python is supposed to come from).

For OS X users *not* using something like macports, I'm pretty sure they
are going to be in a similar boat to the Windows users, with just a touch
of added confusion coming from the fact that an older version of Python
is already installed. But the instructions they will find on the web
for installing package X (once this change hits the field) will be to
install the newest version of 2.7 (or 3) using the python.org installer,
and then they will have the pip command and can go from there.

--David
Daniel Holth
2013-09-27 17:25:40 UTC
Permalink
Post by R. David Murray
Post by Paul Moore
Post by Stephen J. Turnbull
Post by Nick Coghlan
New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P
I bet those users are *not* running third-party distros, but rather
are sitting in front of pretty close to plain vanilla factory installs
of the OS, no? And "new users" on Mac OS X already have "old installs"
of Python, no?
Windows users who don't use a third-party distro like Enthought,
generally download the python.org installer. At the moment, that
doesn't give them a "pip" command. So if they want to install any
third party package, they have to start by installing pip. The
instructions for that are reasonably clear, but non-trivial, largely
because tools like curl are not commonly available on Windows, and by
default running a Python script may not do what you expect.
Rather than try to fix these problems (which are *hard*) the intent is
to have the pip command installed by the python.org installer.
[...]
Post by Paul Moore
I can't speak for Linux distros or OSX users, but for Windows I do
believe that this is a significant improvement, and worth the (IMO,
negligible) risk involved in adding this to a maintenance release.
I'm not an OS X user, and probably most people on this list use macports
or something similar, which essentially puts them in the same boat as
the linux users...and there's a section in the PEP about that (that's
where the message about installing pip if you run pip and the distro
didn't include it with python is supposed to come from).
For OS X users *not* using something like macports, I'm pretty sure they
are going to be in a similar boat to the Windows users, with just a touch
of added confusion coming from the fact that an older version of Python
is already installed. But the instructions they will find on the web
for installing package X (once this change hits the field) will be to
install the newest version of 2.7 (or 3) using the python.org installer,
and then they will have the pip command and can go from there.
--David
OS X and Linux change Python in ways that can be confusing to new and
experienced users, like not installing the stdlib source code by
default, or not installing the profiler, or simply by being out of
date. On these platforms attempting to use the system Python for
development can be a costly mistake instead of a convenience.

Users get a more consistent experience by starting with the installers
from python.org. Hopefully the tutorials will reflect the consistency
added by this PEP.

This PEP only gets pip, an operation which is a recurring
inconvenience for me even though I use Linux. It does not solve any of
the problems you may have after pip has been installed.
Stephen J. Turnbull
2013-09-27 18:14:10 UTC
Permalink
Post by Paul Moore
I can't speak for Linux distros or OSX users, but for Windows I do
believe that this is a significant improvement,
Nobody doubts this.
Post by Paul Moore
and worth the (IMO, negligible) risk involved in adding this to a
maintenance release.
Doesn't that argument apply equally well to any new feature? The
proponents of such changes always characterize them as "worth the
(IMO, negligible) risk involved". I am sure that the proponents of
any such feature are equally sincere about that!

Also, I think that proponents of backporting this PEP are missing
something important. Specifically, why are we encouraging the use of
Python 2.7 for "new users"? Shouldn't we use this as an opportunity
to say, "Move to Python 3.4 and forget the 'agony of installing pip'
issue forever?"
Donald Stufft
2013-09-27 18:22:50 UTC
Permalink
Post by Stephen J. Turnbull
Also, I think that proponents of backporting this PEP are missing
something important. Specifically, why are we encouraging the use of
Python 2.7 for "new users"? Shouldn't we use this as an opportunity
to say, "Move to Python 3.4 and forget the 'agony of installing pip'
issue forever?"
Because reality is that new users are still likely to be using Python 2.7. Python 3
is just now starting to be really usable, however there's a huge corpus of existing
tutorials, course work, books etc for Python 2.7. As Python 3 becomes more usable
that existing corpus of material will be ported over to Python 3 but in the interim
there is still a pretty large hurdle for new users to get over.

That's assuming that they are even able to use Python 3 and whatever they are trying
to do with all of their libraries are ported to Python3. I still think Python 2.7 is a better
target for new users because if you're using Python 3.x theirs a high chance you'll
need to port a library or two still.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Guido van Rossum
2013-09-27 18:31:32 UTC
Permalink
Post by Donald Stufft
Post by Stephen J. Turnbull
Also, I think that proponents of backporting this PEP are missing
something important. Specifically, why are we encouraging the use of
Python 2.7 for "new users"? Shouldn't we use this as an opportunity
to say, "Move to Python 3.4 and forget the 'agony of installing pip'
issue forever?"
Because reality is that new users are still likely to be using Python 2.7. Python 3
is just now starting to be really usable, however there's a huge corpus of existing
tutorials, course work, books etc for Python 2.7. As Python 3 becomes more usable
that existing corpus of material will be ported over to Python 3 but in the interim
there is still a pretty large hurdle for new users to get over.
That's assuming that they are even able to use Python 3 and whatever they are trying
to do with all of their libraries are ported to Python3. I still think
Python 2.7 is a better
target for new users because if you're using Python 3.x theirs a high chance you'll
need to port a library or two still.
Based on my day-to-day experience this is still very true. (And yes, I'm
slowly turning the tide. But it will take a long time and I am committed to
giving users the choice.)
--
--Guido van Rossum (python.org/~guido)
Terry Reedy
2013-09-27 18:50:22 UTC
Permalink
On 9/27/2013 10:26 AM, Paul Moore wrote:
[snip longish post]
I want to summarize the main points of what I believe Paul said and
strongly agree with them.

* For this issue, and especially for backporting to 2.7/3.3, we should
consider Windows, Mac, and *nix distributions as separate cases.

* For Windows users, it would be really good if the PSF installer for
the next releases of all versions at least optionally resulted in the
installation of the most recent pip possible.

I add: just how that is accomplished is not the users concern, except
possibly for using the net to check for a later version of pip. Also, we
can encourage but not command Enthought and ActiveState to do the same
suite in their plus distributions.

* For 2.7/3.3, Windows users have no need to run _ensurepip after
installation.

I add: for 2.7/3.3, there is consequently no need for _ensurepip to be
in /Lib after installation, even if temporarily added*. If it is not
there, there is no change the the stdlib, and hence no violation of the
'no new features' policy. The optional installation of pip is not a
change to Python itself.

* Where the 2.7/3.3 installers merely bundle pip or use the 3.4
procedure with an extra delete step strikes me as an implementation
detail best decided by the installer maker (Martin).
--
Terry Jan Reedy
Donald Stufft
2013-09-27 19:10:43 UTC
Permalink
I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself.
This sounds like a really bad idea to me. You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Terry Reedy
2013-09-27 20:09:01 UTC
Permalink
Received: from localhost (HELO mail.python.org) (127.0.0.1)
by albatross.python.org with SMTP; 27 Sep 2013 22:09:18 +0200
Received: from plane.gmane.org (unknown [80.91.229.3])
(using TLSv1 with cipher AES256-SHA (256/256 bits))
(No client certificate requested)
by mail.python.org (Postfix) with ESMTPS
for <python-***@python.org>; Fri, 27 Sep 2013 22:09:18 +0200 (CEST)
Received: from list by plane.gmane.org with local (Exim 4.69)
(envelope-from <python-python-***@m.gmane.org>) id 1VPeLk-0001BS-9v
for python-***@python.org; Fri, 27 Sep 2013 22:09:16 +0200
Received: from pool-173-75-251-66.phlapa.fios.verizon.net ([173.75.251.66])
by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
id 1AlnuQ-0007hv-00
for <python-***@python.org>; Fri, 27 Sep 2013 22:09:16 +0200
Received: from tjreedy by pool-173-75-251-66.phlapa.fios.verizon.net with
local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00
for <python-***@python.org>; Fri, 27 Sep 2013 22:09:16 +0200
X-Injected-Via-Gmane: http://gmane.org/
Lines: 45
X-Complaints-To: ***@ger.gmane.org
X-Gmane-NNTP-Posting-Host: pool-173-75-251-66.phlapa.fios.verizon.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
rv:24.0) Gecko/20100101 Thunderbird/24.0
In-Reply-To: <2E8C3B8B-808C-40CA-9914-***@stufft.io>
X-BeenThere: python-***@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Python core developers <python-dev.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-dev>,
<mailto:python-dev-***@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-dev/>
List-Post: <mailto:python-***@python.org>
List-Help: <mailto:python-dev-***@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-dev>,
<mailto:python-dev-***@python.org?subject=subscribe>
Errors-To: python-dev-bounces+python-python-dev=***@python.org
Sender: "Python-Dev"
<python-dev-bounces+python-python-dev=***@python.org>
Archived-At: <http://permalink.gmane.org/gmane.comp.python.devel/142209>
Post by Donald Stufft
I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself.
This sounds like a really bad idea to me.
Why would you think that an idea aimed at ending an argument blocking
your proposal is bad. This seems like a really bad response to me.
Post by Donald Stufft
You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not.
The current proposal is to add a new module to the stdlib in a bugfix
release, which looks like a violation of current policy. We agree that
that end result of pip installed would be good. We are arguing over
whether adding '_' to the name makes it not a violation or whether the
good outweighs the bad of a violation. I claim that the arguement is not
necessary and can be ended by not making the addition or by hiding it.

I presume your objection refers to the fact that one can clone the
repository and compile Python on Windows, albeit with some difficulty.
My three responses:

1. I do not consider the the result to be 'installed Python', at least
not as I have used the project file.

2. The ratio of people building Python on Windows to those downloading
and running an installer is so close to 0 that it can be ignored. People
who build Python on Windows are not typical Python beginners.

3. If you do not agree with 1 and 2 and object to _ensurepip being in
/Lib in such limited circumstances, then either put it in /Tools/scripts
or do not use it at all. I already said that the 2.7/3.3 Windows
installer maker (Martin) should decide whether to even use it.

4. The argument for including _ensurepip somewhere in the repository it
that people who *do* build python.exe could then use it to install pip
the first time. /Tools/scripts would be sufficient for this.

5. The result of not having /Lib/_ensurepip in installed Python would,
in any case, be a lessor violation of the policy.
--
Terry Jan Reedy
Donald Stufft
2013-09-27 20:29:34 UTC
Permalink
Post by Donald Stufft
I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself.
This sounds like a really bad idea to me.
Why would you think that an idea aimed at ending an argument blocking your proposal is bad. This seems like a really bad response to me.
Because I think it's a fundamentally bad idea.
Post by Donald Stufft
You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not.
The current proposal is to add a new module to the stdlib in a bugfix release, which looks like a violation of current policy. We agree that that end result of pip installed would be good. We are arguing over whether adding '_' to the name makes it not a violation or whether the good outweighs the bad of a violation. I claim that the arguement is not necessary and can be ended by not making the addition or by hiding it.
1. I do not consider the the result to be 'installed Python', at least not as I have used the project file.
2. The ratio of people building Python on Windows to those downloading and running an installer is so close to 0 that it can be ignored. People who build Python on Windows are not typical Python beginners.
3. If you do not agree with 1 and 2 and object to _ensurepip being in /Lib in such limited circumstances, then either put it in /Tools/scripts or do not use it at all. I already said that the 2.7/3.3 Windows installer maker (Martin) should decide whether to even use it.
4. The argument for including _ensurepip somewhere in the repository it that people who *do* build python.exe could then use it to install pip the first time. /Tools/scripts would be sufficient for this.
5. The result of not having /Lib/_ensurepip in installed Python would, in any case, be a lessor violation of the policy.
If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs? It seems to me your proposal is to add the _ensurepip module… except when they install it via Windows installer.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Zachary Ware
2013-09-27 21:16:50 UTC
Permalink
On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <***@stufft.io> wrote:
<snip>
Post by Donald Stufft
If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs?
If someone is building their own Python from source--regardless of
platform--they're obviously going to have _ensurepip available one way
or another, and such users will need to have it available to match the
capabilities of the installer (or create their own). But if you're
building your own Python, you should already know enough about what's
going on to know when and whether _ensurepip should be used and how.
On the other hand, when you use an installer (be it Windows, OSX, or
otherwise), you really only need _ensurepip one time, ever: at install
time. Even then, you don't actually need to know anything about
_ensurepip at all, just whether to check the box or not. If you
decide you need it later, you can always re-install.
Post by Donald Stufft
It seems to me your [Terry's] proposal is to add the _ensurepip module… except when they install it via Windows installer.
The way I read Terry's proposal, it is to never add the _ensurepip
*module*, but to use (or make available, whichever makes sense in a
given case) the _ensurepip *script* when it is requested at
install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
Mac/, or really anywhere but Lib/ so that builders can find it, but
Python can't. In other words, don't make "import _ensurepip" possible
out of the box, and a lot of the "don't add new features in
maintenance releases" blockade disappears, because you aren't actually
adding any new capability to Python itself or its standard library.

Of course, this proposal only applies to 2.7/3.3. Since _ensurepip
will be used for venv as well as initial install in 3.4, it should be
a full-blown stdlib module in that version, probably even without the
leading underscore.
--
Zach
Brett Cannon
2013-09-28 01:20:10 UTC
Permalink
On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware
Post by Zachary Ware
<snip>
Post by Donald Stufft
If it lives in the source tree how are you going to provent it from
existing when someone installs on Linux? OSX? One of the BSDs?
If someone is building their own Python from source--regardless of
platform--they're obviously going to have _ensurepip available one way
or another, and such users will need to have it available to match the
capabilities of the installer (or create their own). But if you're
building your own Python, you should already know enough about what's
going on to know when and whether _ensurepip should be used and how.
On the other hand, when you use an installer (be it Windows, OSX, or
otherwise), you really only need _ensurepip one time, ever: at install
time. Even then, you don't actually need to know anything about
_ensurepip at all, just whether to check the box or not. If you
decide you need it later, you can always re-install.
Post by Donald Stufft
It seems to me your [Terry's] proposal is to add the _ensurepip module

except when they install it via Windows installer.
The way I read Terry's proposal, it is to never add the _ensurepip
*module*, but to use (or make available, whichever makes sense in a
given case) the _ensurepip *script* when it is requested at
install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
Mac/, or really anywhere but Lib/ so that builders can find it, but
Python can't. In other words, don't make "import _ensurepip" possible
out of the box, and a lot of the "don't add new features in
maintenance releases" blockade disappears, because you aren't actually
adding any new capability to Python itself or its standard library.
Of course, this proposal only applies to 2.7/3.3. Since _ensurepip
will be used for venv as well as initial install in 3.4, it should be
a full-blown stdlib module in that version, probably even without the
leading underscore.
That's how I read the proposal as well. If that works then great, otherwise
naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they
are playing with fire. I think Martin, Ned, etc. would need to weigh in on
whether it's at all an issue having the ensurepip script somewhere that
doesn't get installed but executable from the installer is at all an issue.
For source it can just be in Tools for 2.7/3.3 if that's the route chosen.

Regardless, one of these two options is enough and I suspect we can stop
debating and let Martin make his BDFAP decision.
Donald Stufft
2013-09-28 02:50:19 UTC
Permalink
Post by Zachary Ware
<snip>
Post by Donald Stufft
If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs?
If someone is building their own Python from source--regardless of
platform--they're obviously going to have _ensurepip available one way
or another, and such users will need to have it available to match the
capabilities of the installer (or create their own). But if you're
building your own Python, you should already know enough about what's
going on to know when and whether _ensurepip should be used and how.
On the other hand, when you use an installer (be it Windows, OSX, or
otherwise), you really only need _ensurepip one time, ever: at install
time. Even then, you don't actually need to know anything about
_ensurepip at all, just whether to check the box or not. If you
decide you need it later, you can always re-install.
Post by Donald Stufft
It seems to me your [Terry's] proposal is to add the _ensurepip module… except when they install it via Windows installer.
The way I read Terry's proposal, it is to never add the _ensurepip
*module*, but to use (or make available, whichever makes sense in a
given case) the _ensurepip *script* when it is requested at
install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
Mac/, or really anywhere but Lib/ so that builders can find it, but
Python can't. In other words, don't make "import _ensurepip" possible
out of the box, and a lot of the "don't add new features in
maintenance releases" blockade disappears, because you aren't actually
adding any new capability to Python itself or its standard library.
Of course, this proposal only applies to 2.7/3.3. Since _ensurepip
will be used for venv as well as initial install in 3.4, it should be
a full-blown stdlib module in that version, probably even without the
leading underscore.
That's how I read the proposal as well. If that works then great, otherwise naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they are playing with fire. I think Martin, Ned, etc. would need to weigh in on whether it's at all an issue having the ensurepip script somewhere that doesn't get installed but executable from the installer is at all an issue. For source it can just be in Tools for 2.7/3.3 if that's the route chosen.
Regardless, one of these two options is enough and I suspect we can stop debating and let Martin make his BDFAP decision.
Yes, no matter which way the decision for 2.7 and 3.3 goes the proposal is for "ensurepip" in 3.4+

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Monty Taylor
2013-09-28 02:55:52 UTC
Permalink
Post by Donald Stufft
Post by Brett Cannon
On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware
<snip>
Post by Donald Stufft
If it lives in the source tree how are you going to provent it
from existing when someone installs on Linux? OSX? One of the
BSDs?
If someone is building their own Python from source--regardless
of platform--they're obviously going to have _ensurepip available
one way or another, and such users will need to have it available
to match the capabilities of the installer (or create their own).
But if you're building your own Python, you should already know
enough about what's going on to know when and whether _ensurepip
should be used and how. On the other hand, when you use an
installer (be it Windows, OSX, or otherwise), you really only
need _ensurepip one time, ever: at install time. Even then, you
don't actually need to know anything about _ensurepip at all,
just whether to check the box or not. If you decide you need it
later, you can always re-install.
Post by Donald Stufft
It seems to me your [Terry's] proposal is to add the
_ensurepip
module? except when they install it via Windows installer.
The way I read Terry's proposal, it is to never add the
_ensurepip *module*, but to use (or make available, whichever
makes sense in a given case) the _ensurepip *script* when it is
requested at install-time: specifically, put _ensurepip.py in
Tools/scripts, PC/, Mac/, or really anywhere but Lib/ so that
builders can find it, but Python can't. In other words, don't
make "import _ensurepip" possible out of the box, and a lot of
the "don't add new features in maintenance releases" blockade
disappears, because you aren't actually adding any new capability
to Python itself or its standard library.
Of course, this proposal only applies to 2.7/3.3. Since
_ensurepip will be used for venv as well as initial install in
3.4, it should be a full-blown stdlib module in that version,
probably even without the leading underscore.
That's how I read the proposal as well. If that works then
great, otherwise naming it _ensurepip in 2.7/3.3 should be enough
to warn anyone that they are playing with fire. I think Martin,
Ned, etc. would need to weigh in on whether it's at all an issue
having the ensurepip script somewhere that doesn't get installed
but executable from the installer is at all an issue. For source
it can just be in Tools for 2.7/3.3 if that's the route chosen.
Regardless, one of these two options is enough and I suspect we
can stop debating and let Martin make his BDFAP decision.
Yes, no matter which way the decision for 2.7 and 3.3 goes the
proposal is for "ensurepip" in 3.4+
I'm in favor of whatever causes it to get available in 2.7/3.3 in a
way that's predictable. OpenStack is both very highly invested in pip
installation of software and dependencies and also seemingly
maniacally obsessed with continuing to support the older pythons
people have. (eek, we have to support 2.6 until a new RHEL comes out)

To that end, we wind up bootstrapping into pip via distro packages,
but then it's too old, so then people pip install -U pip - systemwide,
so then pip gets broken on redhat and weird on ubuntu because of fail,
and then they get angry and start yelling and I have to go hide in a hole.

If we could start, in tooling, depending on some predictable manner
other than the distros to bootstrap into a sensible and modern pip, I
believe my life would be so much better that I might grow a herd of
unicorns.

Monty
Ned Deily
2013-09-28 07:20:40 UTC
Permalink
In article
Post by Brett Cannon
On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware
Post by Zachary Ware
The way I read Terry's proposal, it is to never add the _ensurepip
*module*, but to use (or make available, whichever makes sense in a
given case) the _ensurepip *script* when it is requested at
install-time: specifically, put _ensurepip.py in Tools/scripts, PC/,
Mac/, or really anywhere but Lib/ so that builders can find it, but
Python can't. In other words, don't make "import _ensurepip" possible
out of the box, and a lot of the "don't add new features in
maintenance releases" blockade disappears, because you aren't actually
adding any new capability to Python itself or its standard library.
Of course, this proposal only applies to 2.7/3.3. Since _ensurepip
will be used for venv as well as initial install in 3.4, it should be
a full-blown stdlib module in that version, probably even without the
leading underscore.
That's how I read the proposal as well. If that works then great, otherwise
naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they
are playing with fire. I think Martin, Ned, etc. would need to weigh in on
whether it's at all an issue having the ensurepip script somewhere that
doesn't get installed but executable from the installer is at all an issue.
For source it can just be in Tools for 2.7/3.3 if that's the route chosen.
Putting an "ensurepip.py" script (with or without the "_") in Tools/ should be
fine for the needs of 2.7 and/or 3.3 OS X installers. Without venv in the
picture, it doesn't need to do much; the installer build script itself could
handle everything without an ensurepip. There are still other changes in the
installer that would need to be made, particularly if it is necessary to make
the installation of pip (and its dependency, setuptools) optional but all of
those would be isolated to the Mac subdirectory.
--
Ned Deily,
***@acm.org
Nick Coghlan
2013-09-28 00:40:30 UTC
Permalink
Post by Stephen J. Turnbull
Post by Nick Coghlan
I'm not sure what usage model you're assuming for _ensurepip, but it
appears to be wrong. End users should be able to just run pip, and
either have it work, or else get a message from the OS vendor telling
them to install the appropriate system package.
I don't understand how you arrange for that message on existing
installs. Wouldn't it be easier to just lobby the distros to make
Python dependent on pip?
Most distros of interest already emit that message - it's part of a
standard mechanism to check for system packages if a command isn't found
(and I believe even RHEL and derivatives may support this for pip if the
EPEL repo is enabled).
Post by Stephen J. Turnbull
And speaking of vendors, do you expect Apple
and Microsoft to provide such a message? And such a system package?
No, that's why the proposal is to modify the CPython installers for those
platforms.
Post by Stephen J. Turnbull
If you already are running a Linux distro or MacPorts, you do "apt-get
python-pip" and "port install py-pip" respectively. I bet Cygwin is
Post by Nick Coghlan
New users on Windows and Mac OS X. I've heard many more complaints
from folks running tutorials about the pip bootstrapping process than
I ever have from the community at large about the GIL :P
I bet those users are *not* running third-party distros, but rather
are sitting in front of pretty close to plain vanilla factory installs
of the OS, no? And "new users" on Mac OS X already have "old installs"
of Python, no?
No. Instuctors tell users on Mac OS X to install from python.org or use one
of the third party package managers.
Post by Stephen J. Turnbull
That's my model. In that model I don't see backporting PEP 453 to
Python 2.7 as being a sufficiently reliable way to provide a smooth
user experience to justify breaking the "no new features" rule (which
is at the "read my lips" level after the True/False fiasco).
You have confirmed my belief that your model is incorrect. The feedback I
have received is that the majority of beginners are introduced to Python by
downloading the binary installers from python.org for Windows or Mac OS X.
Post by Stephen J. Turnbull
Get a commitment from Apple to put 2.7.6 in their next upgrades for
their OS, and then maybe you'd have enough leverage to tip the
balance. I certainly would concede the point. But without that,
you're telling Mac users "you have to upgrade Python from a 3rd party
site." Is that really the way to make new users participating in a
tutorial session happy? (You tell me, I'm just introspecting here.)
Yes, that's exactly what happens. People don't use the system Python on Mac
OS X the way they do on *nix systems.

Cheers,
Nick.
Post by Stephen J. Turnbull
Steve
Stephen J. Turnbull
2013-09-28 03:48:05 UTC
Permalink
Post by Nick Coghlan
You have confirmed my belief that your model is incorrect.
*shrug* I just think the risks are higher than acknowledged (just
because you have so far failed to imagine a problem doesn't mean it
won't appear), and that the meta effect that "Even Guido admits that
Python 3 isn't ready for prime time" is perverse. We know, even those
who have written blanket statements to that effect in this thread,
that that is false unless conditioned on specific applications.

I understand that the real motivation is that it's churlish to not
relieve the pain of users who have decided for their own good reasons
to use Python 2.7, and perverse to ignore the needs of the teachers
who are going to educate the users about Python 3 at the time they
consider appropriate. But the meta-message *received* by the public
is not going to accurately reflect that motivation, and is not going
to be helpful in encouraging those who already *can* move to Python 3
to do so.

Anyway, clearly this exception is heading for approval, and the PEP
with it. I recommend that the "Feature addition in maintenance
releases" section be amended to read in its entirety:

The additions of the new module to the standard library in the
maintenance releases of 2.7 and 3.3 were granted explicit
exceptions to the rule "no new features in maintenance releases."
These exceptions were explicitly discussed, and approved in
consultation with the affected release managers, separately from
the rest of the PEP. They do not represent a change in policy,
and must not be considered a precedent for other such exceptions.

Just the facts, ma'am. It's a bad idea to include bullshit about the
benefit-cost ratio, because it will be cited in future requests for
similar exceptions. To the extent that people interpret this as a
forecast and support for a long life for Python 2.7, there is
substantial risk of such requests.


Footnotes:
[1] I do it that way myself, always with the most recent Python 3,
but haven't yet gotten to the point of needing "pip" (use cases that
happen to be met by the stdlib).
Donald Stufft
2013-09-28 04:12:50 UTC
Permalink
Post by Stephen J. Turnbull
Post by Nick Coghlan
You have confirmed my belief that your model is incorrect.
*shrug* I just think the risks are higher than acknowledged (just
because you have so far failed to imagine a problem doesn't mean it
won't appear), and that the meta effect that "Even Guido admits that
Python 3 isn't ready for prime time" is perverse. We know, even those
who have written blanket statements to that effect in this thread,
that that is false unless conditioned on specific applications.
I haven't seen anyone say Python 3 isn't ready to be used, just that it
makes more sense for beginners to use 2.7, and I think it does still.
Porting libraries from 2.x to 3.x and translaing the existing corpus of
tutorials, tips, posts, etc isn't a trivial task and one that beginners
are unlikely to grok.
Post by Stephen J. Turnbull
I understand that the real motivation is that it's churlish to not
relieve the pain of users who have decided for their own good reasons
to use Python 2.7, and perverse to ignore the needs of the teachers
who are going to educate the users about Python 3 at the time they
consider appropriate. But the meta-message *received* by the public
is not going to accurately reflect that motivation, and is not going
to be helpful in encouraging those who already *can* move to Python 3
to do so.
Anyway, clearly this exception is heading for approval, and the PEP
with it. I recommend that the "Feature addition in maintenance
The additions of the new module to the standard library in the
maintenance releases of 2.7 and 3.3 were granted explicit
exceptions to the rule "no new features in maintenance releases."
These exceptions were explicitly discussed, and approved in
consultation with the affected release managers, separately from
the rest of the PEP. They do not represent a change in policy,
and must not be considered a precedent for other such exceptions.
Just the facts, ma'am. It's a bad idea to include bullshit about the
benefit-cost ratio, because it will be cited in future requests for
similar exceptions. To the extent that people interpret this as a
forecast and support for a long life for Python 2.7, there is
substantial risk of such requests.
Maybe my understanding of the PEP process is flawed, but isn't
part of the point of it to codify the *reasons* for the decisions that
were made? That's why they include information about dissenting
opinions and such?

I don't think it's dangerous to cite the reasons the decision was came
to. Perhaps it can be toned down but I think it's useful to document
the discussion. I've got a partially done update that tries to capture
the dissenting opinions as well for completeness sake.
Post by Stephen J. Turnbull
[1] I do it that way myself, always with the most recent Python 3,
but haven't yet gotten to the point of needing "pip" (use cases that
happen to be met by the stdlib).
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Nick Coghlan
2013-09-28 06:51:27 UTC
Permalink
Post by Donald Stufft
Post by Stephen J. Turnbull
Post by Nick Coghlan
You have confirmed my belief that your model is incorrect.
*shrug* I just think the risks are higher than acknowledged (just
because you have so far failed to imagine a problem doesn't mean it
won't appear), and that the meta effect that "Even Guido admits that
Python 3 isn't ready for prime time" is perverse. We know, even those
who have written blanket statements to that effect in this thread,
that that is false unless conditioned on specific applications.
I haven't seen anyone say Python 3 isn't ready to be used, just that it
makes more sense for beginners to use 2.7, and I think it does still.
Porting libraries from 2.x to 3.x and translaing the existing corpus of
tutorials, tips, posts, etc isn't a trivial task and one that beginners
are unlikely to grok.
Post by Stephen J. Turnbull
I understand that the real motivation is that it's churlish to not
relieve the pain of users who have decided for their own good reasons
to use Python 2.7, and perverse to ignore the needs of the teachers
who are going to educate the users about Python 3 at the time they
consider appropriate. But the meta-message *received* by the public
is not going to accurately reflect that motivation, and is not going
to be helpful in encouraging those who already *can* move to Python 3
to do so.
Anyway, clearly this exception is heading for approval, and the PEP
with it. I recommend that the "Feature addition in maintenance
The additions of the new module to the standard library in the
maintenance releases of 2.7 and 3.3 were granted explicit
exceptions to the rule "no new features in maintenance releases."
These exceptions were explicitly discussed, and approved in
consultation with the affected release managers, separately from
the rest of the PEP. They do not represent a change in policy,
and must not be considered a precedent for other such exceptions.
Just the facts, ma'am. It's a bad idea to include bullshit about the
benefit-cost ratio, because it will be cited in future requests for
similar exceptions. To the extent that people interpret this as a
forecast and support for a long life for Python 2.7, there is
substantial risk of such requests.
Maybe my understanding of the PEP process is flawed, but isn't
part of the point of it to codify the *reasons* for the decisions that
were made? That's why they include information about dissenting
opinions and such?
I don't think it's dangerous to cite the reasons the decision was came
to. Perhaps it can be toned down but I think it's useful to document
the discussion. I've got a partially done update that tries to capture
the dissenting opinions as well for completeness sake.
We'll put something in pointing out that accepting this change actually
makes it even *harder* to advocate for further feature additions in Python
2.7 maintenance releases as there is *zero* chance of backporting language
changes, and this PEP means library and builtin updates can easily be a pip
install away if someone is willing to create the backport and put it on
PyPI (Brandon Rhodes already created the "backports" namespace package as a
common home for such efforts, or there's the "2" suffix that has been used
in a couple of cases).

Cheers,
Nick.
Post by Donald Stufft
Post by Stephen J. Turnbull
[1] I do it that way myself, always with the most recent Python 3,
but haven't yet gotten to the point of needing "pip" (use cases that
happen to be met by the stdlib).
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
Post by Donald Stufft
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Barry Warsaw
2013-09-28 14:38:48 UTC
Permalink
Post by Stephen J. Turnbull
*shrug* I just think the risks are higher than acknowledged (just
because you have so far failed to imagine a problem doesn't mean it
won't appear), and that the meta effect that "Even Guido admits that
Python 3 isn't ready for prime time" is perverse. We know, even those
who have written blanket statements to that effect in this thread,
that that is false unless conditioned on specific applications.
I understand that the real motivation is that it's churlish to not
relieve the pain of users who have decided for their own good reasons
to use Python 2.7, and perverse to ignore the needs of the teachers
who are going to educate the users about Python 3 at the time they
consider appropriate. But the meta-message *received* by the public
is not going to accurately reflect that motivation, and is not going
to be helpful in encouraging those who already *can* move to Python 3
to do so.
FWIW, +1
Post by Stephen J. Turnbull
Anyway, clearly this exception is heading for approval, and the PEP
with it. I recommend that the "Feature addition in maintenance
The additions of the new module to the standard library in the
maintenance releases of 2.7 and 3.3 were granted explicit
exceptions to the rule "no new features in maintenance releases."
These exceptions were explicitly discussed, and approved in
consultation with the affected release managers, separately from
the rest of the PEP. They do not represent a change in policy,
and must not be considered a precedent for other such exceptions.
Call this the "Bush v. Gore" amendment.

-Barry
Donald Stufft
2013-09-25 23:15:41 UTC
Permalink
Post by Barry Warsaw
Another reason to oppose this is what I've heard quite often from people
regarding Python 2.7. I've been told that many folks are actually really
happy with using 2.7 precisely because it extremely stable. They don't have
to worry about their stuff breaking or incompatibilities cropping up, because
it *doesn't* change. Python 2.7 is like a long-term maintenance release, with
guaranteed multi-year stability, and I think while that was unintended, it's a
*good* thing. This PEP proposes to break that, and I'm loathe to give up that
good reputation for this particular feature.
Maybe I'm just naive but can you expand on how adding a module that nothing
else in the system imports (besides the installers; but you weren't arguing against
adding this to the installers) would break someones use any other module? If
they don't import it (which the vast bulk of people won't directly, nor at all during
the operation of their applications) how does it's existence on the file system
risk a breakage to their system?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Brett Cannon
2013-09-26 14:00:15 UTC
Permalink
Post by Nick Coghlan
Post by Barry Warsaw
Another reason to oppose this is what I've heard quite often from people
regarding Python 2.7. I've been told that many folks are actually really
happy with using 2.7 precisely because it extremely stable. They don't
have
Post by Barry Warsaw
to worry about their stuff breaking or incompatibilities cropping up,
because
Post by Barry Warsaw
it *doesn't* change. Python 2.7 is like a long-term maintenance
release, with
Post by Barry Warsaw
guaranteed multi-year stability, and I think while that was unintended,
it's a
Post by Barry Warsaw
*good* thing. This PEP proposes to break that, and I'm loathe to give
up that
Post by Barry Warsaw
good reputation for this particular feature.
Maybe I'm just naive but can you expand on how adding a module that nothing
else in the system imports (besides the installers; but you weren't arguing against
adding this to the installers) would break someones use any other module? If
they don't import it (which the vast bulk of people won't directly, nor at all during
the operation of their applications) how does it's existence on the file system
risk a breakage to their system?
I think Barry's worry is precisely the fact that users do silly things, so
having a new module suddenly show up in the stdlib can be a problem if
people start using the module. We could conceivably try and not expose the
module on sys.path somehow so that it can't be directly imported by anyone
who doesn't have any business using it unless they jump through some hoops
(e.g. stick it in Tools for Python 2.7 so the installers can get access but
it won't end up in the stdlib install).

Otherwise if Barry is worrying about a command-line tool being installed
for some users of Python 2.7 vs. not then I don't know what the worry is
because that happens on OS upgrades as well, let alone people who just
happen to have done the install themselves vs. not. People are already told
to install pip anyway, so if instructions point to ensurepip on PyPI as the
canonical way to install pip then whether it's installed or not is fine as
it will be mostly a no-op for 2.7.6 users and an actual install for others.
But by including it then it does simplify teaching scenarios and long tail
helps get pip out faster and more easily.
Nick Coghlan
2013-09-25 23:01:12 UTC
Permalink
Post by Barry Warsaw
Post by Donald Stufft
I think it should be placed in the source tree for the stable releases. The
reasoning is that 2.7 is going to stick around for a long time. Immediately
this won't be ubiquitous but as time goes on you'll be able to be ensured
that a ``python -m ensure pip`` exists so that in situations where you don't
have pip you'll be able to install it.
While not directly relevant to the change I do think this is something users
support. I've received a fair but of feedback as I was writing the original
draft of the PEP and then throughout the process when me and Nick
were working on it. Almost all of it was positive (some of it extremely so)
a fair bit of them pointed out the backport to 2.7 as something they were
*really* wanting.
An early draft of this did not have the backport to 2.7 and when I showed
*that* version around to get feedback people were less enthusiastic
about it and generally viewed it as "nice but worthless to me for N years".
Yeah, I get all this, but it's essentially the same reasoning that lead to
Python 2.2.1's addition of True and False. The same flaw occurs though
because you cannot guarantee that invocation of `python -m ensurepip` will
work unless you micro version check (LBYL) or prepare to catch resulting
errors. This essentially means that if you want to write portable Python 2.7
code and scripts, you have to be very cautious, or what works on one box won't
necessarily work on another, even though they both have "Python 2.7".
There's a bit of a difference between adding new builtins and adding a new
utility module, though.

The only regular invocations of ensurepip in 2.7 and 3.3 should be from our
installers and people doing a custom build from source.

Would a leading underscore in the module name make you more comfortable
with the idea? It's really intended mostly as a hidden implementation
detail of the installers and pyvenv anyway, so calling it "_ensurepip"
would help make that explicit while still letting people invoke it directly
if absolutely necessary.

Cheers,
Nick.
Post by Barry Warsaw
It also means that anybody who's documenting about this great new feature has
to warn people that, well, maybe it won't work on your machine even though you
have Python 2.7 because you don't have a new enough Python 2.7.
It means that not all Python 2.7's are alike in a rather fundamental and
highly visible way.
I personally think that's a recipe for more problems than it solves, and if I
was the RM for 2.7 I would not allow it. But I'm not.
Post by Donald Stufft
What users want isn't rationale in and of itself but I think it's an important
data point, especially given how long 2.7.LASTEVER is going to be
relevant to end users.
Users want what users want. It's our job to make the best technical decisions
based on all the facts. I understand that Python 2.7 will be around for a
long time, and that it would be very convenient to do this. Why is this not
opening up the door to more new features being added in future Python 2.7
point releases (or any other stable release)?
At least I think the burden should be very high, and the PEP should do a
better job of explaining why *no* other option will accomplish the goal.
Cheers,
-Barry
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Barry Warsaw
2013-09-25 23:08:54 UTC
Permalink
Post by Nick Coghlan
Would a leading underscore in the module name make you more comfortable
with the idea? It's really intended mostly as a hidden implementation
detail of the installers and pyvenv anyway, so calling it "_ensurepip"
would help make that explicit while still letting people invoke it directly
if absolutely necessary.
If it was also undocumented, and we can guarantee that it will have no impact
outside of the module itself, then maybe. I'm not sure it would be enough for
me to accept it if I were the RM, but I'm not (thankfully. :).

I still think it's optimistic to pin any hopes on its widespread availability
in future 2.7 point releases on people's actual systems.

-Barry
Nick Coghlan
2013-09-26 04:30:18 UTC
Permalink
Post by Barry Warsaw
Post by Nick Coghlan
Would a leading underscore in the module name make you more comfortable
with the idea? It's really intended mostly as a hidden implementation
detail of the installers and pyvenv anyway, so calling it "_ensurepip"
would help make that explicit while still letting people invoke it directly
if absolutely necessary.
If it was also undocumented, and we can guarantee that it will have no impact
outside of the module itself, then maybe. I'm not sure it would be enough for
me to accept it if I were the RM, but I'm not (thankfully. :).
I still think it's optimistic to pin any hopes on its widespread availability
in future 2.7 point releases on people's actual systems.
The recommended user experience for 2.7 users on Windows and Mac OS X
will be "Install the latest Python 2.7 from python.org". This feature
is *not* aimed at existing users *at all*. For Python 2.7, the
benefits are aimed almost entirely at beginners installing Python for
the first time, regardless of platform. There's a *side effect* for
existing users, where for Windows and Mac OS X users, the pip
bootstrapping instructions become "install the latest maintenance
release" rather than downloading and running the bootstrap script from
pip-installer.org.

The presence or absence of _ensurepip should be completely irrelevant
to end users, as it will be there in 2.7 solely to make the new
feature of the installer work.

Your alternative proposal is to instead come up with a *new* mechanism
to support the feature, specifically for 2.7. That's increasing the
chance of breakage, not reducing it, *because the _ensurepip tests
would no longer be executed by the Python 2.7 buildbots*.

That said, there are changes that I think are definitely worth making
due to the concerns you raise:

- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)

The benefits are most obvious in the case of 2.7, so both Donald and I
are also fine with skipping making any changes to Python 3.3. The kind
of environment that could make it difficult to upgrade from Python
3.3. to 3.4 is unlikely to condone the installation and use of pip
anyway.

There is one other change which is potentially reasonable, and that's
changing the default state of the "Install pip (the default Python
package management utility)?" checkbox. I believe it should be checked
by default in all versions, but if Benjamin decides that it should be
unchecked by default in 2.7, I'd be willing to go along with that.

Regards,
Nick.
--
Nick Coghlan | ***@gmail.com | Brisbane, Australia
Nick Coghlan
2013-09-26 04:54:49 UTC
Permalink
Post by Nick Coghlan
That said, there are changes that I think are definitely worth making
- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)
Donald pointed out it makes more sense to continue with the idea of a
properly documented public "ensurepip" module in 3.4+, and have the
"_ensurepip" version as an implementation detail of the 2.7 and 3.3
installers that is included in the stdlib primarily so it can be
covered by the existing buildbot fleet.

Redistributors will be free to remove "_ensurepip" from 2.7 and 3.3,
but the PEP will continue to advise strongly against removing
"ensurepip" from 3.4+ (since it will be a documented feature and a
dependency of pyvenv)
Post by Nick Coghlan
The benefits are most obvious in the case of 2.7, so both Donald and I
are also fine with skipping making any changes to Python 3.3. The kind
of environment that could make it difficult to upgrade from Python
3.3. to 3.4 is unlikely to condone the installation and use of pip
anyway.
In replying to Stephen I realised the benefit of making a similar
change in the 3.3 installer is that it means the preferred pip
bootstrapping instructions for both 2.7 and 3.3 can be to update to
the latest maintenance release of CPython, rather than needing to tell
3.3 users to consider upgrading to 3.4 when that release contains
potentially breaking changes to the way file descriptors are handled.

Cheers,
Nick.
--
Nick Coghlan | ***@gmail.com | Brisbane, Australia
Antoine Pitrou
2013-09-26 14:10:46 UTC
Permalink
Le Thu, 26 Sep 2013 14:54:49 +1000,
Post by Nick Coghlan
Post by Nick Coghlan
That said, there are changes that I think are definitely worth
- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)
Donald pointed out it makes more sense to continue with the idea of a
properly documented public "ensurepip" module in 3.4+, and have the
"_ensurepip" version as an implementation detail of the 2.7 and 3.3
installers that is included in the stdlib primarily so it can be
covered by the existing buildbot fleet.
Hmm, but what is the point of "_ensurepip" exactly? Are people supposed
to type "python -m _ensurepip"?

With all due respect, Barry's argument looks rather paranoid to me.
I would suggest a clear choice:
- either having "ensurepip" in 2.7 is useful and we endorse it as a
public module (not something hidden somewhere) - which I personally
think is reasonable
- or it's not useful and we don't introduce it at all

A middleground doesn't make sense here, except in a broken "design by
committee" sense.

Regards

Antoine.
Donald Stufft
2013-09-26 14:22:55 UTC
Permalink
Ideally people won't be typing either of them because it'll be installed automatically. They might in some cases (accidentally uninstalled pip?)

I agree that it seems there is paranoia going on here and that the risk is low and making it just be a special cased new feature is ok. However the point of the underscore prefix on 2.7 and 3.3 is to more effectively communicate that on these pythons you shouldn't rely on that module existing. I think that's worse situation then just making it ensurepip everywhere but better than 2.7 not getting it at all or Barry's suggestion.
Post by Antoine Pitrou
Le Thu, 26 Sep 2013 14:54:49 +1000,
Post by Nick Coghlan
Post by Nick Coghlan
That said, there are changes that I think are definitely worth
- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)
Donald pointed out it makes more sense to continue with the idea of a
properly documented public "ensurepip" module in 3.4+, and have the
"_ensurepip" version as an implementation detail of the 2.7 and 3.3
installers that is included in the stdlib primarily so it can be
covered by the existing buildbot fleet.
Hmm, but what is the point of "_ensurepip" exactly? Are people supposed
to type "python -m _ensurepip"?
With all due respect, Barry's argument looks rather paranoid to me.
- either having "ensurepip" in 2.7 is useful and we endorse it as a
public module (not something hidden somewhere) - which I personally
think is reasonable
- or it's not useful and we don't introduce it at all
A middleground doesn't make sense here, except in a broken "design by
committee" sense.
Regards
Antoine.
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
Antoine Pitrou
2013-09-26 14:28:42 UTC
Permalink
Le Thu, 26 Sep 2013 10:22:55 -0400,
Post by Donald Stufft
Ideally people won't be typing either of them because it'll be
installed automatically. They might in some cases (accidentally
uninstalled pip?)
Installing from source perhaps.
Post by Donald Stufft
I agree that it seems there is paranoia going on here and that the
risk is low and making it just be a special cased new feature is ok.
However the point of the underscore prefix on 2.7 and 3.3 is to more
effectively communicate that on these pythons you shouldn't rely on
that module existing.
That's a fair argument, although it also makes the thing a bit ugly.

Regards

Antoine.
Donald Stufft
2013-09-26 16:40:39 UTC
Permalink
Post by Antoine Pitrou
Le Thu, 26 Sep 2013 10:22:55 -0400,
Post by Donald Stufft
Ideally people won't be typing either of them because it'll be
installed automatically. They might in some cases (accidentally
uninstalled pip?)
Installing from source perhaps.
Ah yea, installing from source is the time when they'd execute it,
that slipped my mind.
Post by Antoine Pitrou
Post by Donald Stufft
I agree that it seems there is paranoia going on here and that the
risk is low and making it just be a special cased new feature is ok.
However the point of the underscore prefix on 2.7 and 3.3 is to more
effectively communicate that on these pythons you shouldn't rely on
that module existing.
That's a fair argument, although it also makes the thing a bit ugly.
Yea It's not my preferred thing to do and I think that calling it just ensurepip
is fine, however I think if folks are worried about it, it could offer a compromise.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Nick Coghlan
2013-09-26 21:20:49 UTC
Permalink
Post by Antoine Pitrou
Le Thu, 26 Sep 2013 14:54:49 +1000,
Post by Nick Coghlan
Post by Nick Coghlan
That said, there are changes that I think are definitely worth
- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)
Donald pointed out it makes more sense to continue with the idea of a
properly documented public "ensurepip" module in 3.4+, and have the
"_ensurepip" version as an implementation detail of the 2.7 and 3.3
installers that is included in the stdlib primarily so it can be
covered by the existing buildbot fleet.
Hmm, but what is the point of "_ensurepip" exactly? Are people supposed
to type "python -m _ensurepip"?
With all due respect, Barry's argument looks rather paranoid to me.
- either having "ensurepip" in 2.7 is useful and we endorse it as a
public module (not something hidden somewhere) - which I personally
think is reasonable
- or it's not useful and we don't introduce it at all
A middleground doesn't make sense here, except in a broken "design by
committee" sense.
The *only* reason to have _ensurepip in the older versions is to make the
"install pip" option in the installers work. By contrast, in 3.4+ it will
be integrated into pyvenv and a fully supported API.

The two levels make sense to me, especially since pyvenv in 3.3 *won't* be
updated to invoke it automatically. 2.7 and 3.3 users will still need to
install and use virtualenv to get pip automatically installed in virtual
environments.

Cheers,
Nick.
Post by Antoine Pitrou
Regards
Antoine.
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Barry Warsaw
2013-09-28 00:44:17 UTC
Permalink
Post by Nick Coghlan
- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)
I'm sorry that I probably won't be able to engage more on this thread, but I
think my points should be fairly well understood by the PEP czar and RM.
Should the PEP be accepted, I think both of these are good changes.

-Barry
Benjamin Peterson
2013-09-28 01:19:57 UTC
Permalink
Post by Barry Warsaw
Post by Nick Coghlan
- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)
I'm sorry that I probably won't be able to engage more on this thread, but I
think my points should be fairly well understood by the PEP czar and RM.
Should the PEP be accepted, I think both of these are good changes.
And FWIW, I generally agree with Barry about adding things to 2.7.
--
Regards,
Benjamin
Eli Bendersky
2013-09-28 12:56:46 UTC
Permalink
Post by Benjamin Peterson
Post by Barry Warsaw
Post by Nick Coghlan
- the module name should be "_ensurepip" in all versions
- the PEP should explicitly state that the "don't remove _ensurepip
and it's wheel files" caveat for redistributors applies only in 3.4+
(where removing it will break pyvenv)
I'm sorry that I probably won't be able to engage more on this thread,
but I
Post by Barry Warsaw
think my points should be fairly well understood by the PEP czar and RM.
Should the PEP be accepted, I think both of these are good changes.
And FWIW, I generally agree with Barry about adding things to 2.7.
+1. Not to mention the general problem of adding new features in
maintenance releases (what? it doesn't work for you? Check it's 2.7.6 and
not 2.7.5!) , at this point in the trajectory of Python 3.x and 2.7 it
sends a very wrong message.

Eli
Martin v. Löwis
2013-09-30 09:01:45 UTC
Permalink
Post by Donald Stufft
An early draft of this did not have the backport to 2.7 and when I
showed *that* version around to get feedback people were less
enthusiastic about it and generally viewed it as "nice but
worthless to me for N years".
I'm leaning towards the people that oppose addition of this feature to
older Python releases. In particular, the objection that the new PEP is
worthless for a long time to come, is in itself not really relevant:

It is always the case that features proposed by a PEP reach users only
years after they have been implemented.
Post by Donald Stufft
What users want isn't rationale in and of itself but I think it's
an important data point, especially given how long 2.7.LASTEVER is
going to be relevant to end users.
Well, I really really don't like this idea. 2.7 should not get new
features; users who want new features need to switch to 3.x.

Regards,
Martin
Donald Stufft
2013-09-30 11:18:29 UTC
Permalink
Signed PGP part
Post by Donald Stufft
An early draft of this did not have the backport to 2.7 and when I
showed *that* version around to get feedback people were less
enthusiastic about it and generally viewed it as "nice but
worthless to me for N years".
I'm leaning towards the people that oppose addition of this feature to
older Python releases. In particular, the objection that the new PEP is
It is always the case that features proposed by a PEP reach users only
years after they have been implemented.
Well the point we tried to get across in the PEP is that a normal feature
you can typically just install a backport from PyPI to gain it early. This
isin't so much driven by "well it'd be nice for the stdlib to have X", but
"well this is a real and valid pain point that causes pain for a lot of users".
The 2.7 backport was driven by just how painful this particular pain point
can be. I've personally had feedback that in tutorials at like PyCon or
meet ups that easily 1/3 of the time can be spent in getting users setup
with Python, setuptools, and pip.

Obviously you're the delegate for this PEP and it's your final decision and
either way If the PEP is accepted I'll do the implementation for 3.4.
Post by Donald Stufft
What users want isn't rationale in and of itself but I think it's
an important data point, especially given how long 2.7.LASTEVER is
going to be relevant to end users.
Well, I really really don't like this idea. 2.7 should not get new
features; users who want new features need to switch to 3.x.
Regards,
Martin
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Martin v. Löwis
2013-09-30 11:50:51 UTC
Permalink
Post by Donald Stufft
Well the point we tried to get across in the PEP is that a normal
feature you can typically just install a backport from PyPI to gain
it early. This isin't so much driven by "well it'd be nice for the
stdlib to have X", but "well this is a real and valid pain point
that causes pain for a lot of users".
I see it exactly vice versa. A normal new language feature, you *cannot*
get early from PyPI, e.g. if it's a syntax extension (e.g. the with
statement). OTOH, *this* particular feature you can easily get from
PyPI - just download and run the PIP installer.
Post by Donald Stufft
The 2.7 backport was driven by just how painful this particular
pain point can be. I've personally had feedback that in tutorials
at like PyCon or meet ups that easily 1/3 of the time can be spent
in getting users setup with Python, setuptools, and pip.
I'm sure there is something that can be done about it. What operating
system are people using who have difficulties to set this all up?

I see that pip doesn't have a Windows installer on PyPI. That looks
like the real culprit to me. I'd be willing to work with you to
provide one (that ideally bundles all dependencies).
Post by Donald Stufft
Obviously you're the delegate for this PEP and it's your final
decision and either way If the PEP is accepted I'll do the
implementation for 3.4.
The backporting is the main stumbling block. Otherwise, I'm in favor
of the PEP.

Regards,
Martin
Nick Coghlan
2013-09-30 13:07:36 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Post by Donald Stufft
Well the point we tried to get across in the PEP is that a normal
feature you can typically just install a backport from PyPI to gain
it early. This isin't so much driven by "well it'd be nice for the
stdlib to have X", but "well this is a real and valid pain point
that causes pain for a lot of users".
I see it exactly vice versa. A normal new language feature, you *cannot*
get early from PyPI, e.g. if it's a syntax extension (e.g. the with
statement).
It's standard library improvements (like SSL hostname matching) that are
relevant to this comparison, rather than the kinds of changes that were
barred by the language moratorium.
OTOH, *this* particular feature you can easily get from
PyPI - just download and run the PIP installer.
Post by Donald Stufft
The 2.7 backport was driven by just how painful this particular
pain point can be. I've personally had feedback that in tutorials
at like PyCon or meet ups that easily 1/3 of the time can be spent
in getting users setup with Python, setuptools, and pip.
I'm sure there is something that can be done about it. What operating
system are people using who have difficulties to set this all up?
I see that pip doesn't have a Windows installer on PyPI. That looks
like the real culprit to me. I'd be willing to work with you to
provide one (that ideally bundles all dependencies).
We'll take you up on that offer. Windows is the main problem, since other
platforms offer wget/curl out of the box so bootstrapping is easy by
comparison.
Post by Donald Stufft
Obviously you're the delegate for this PEP and it's your final
decision and either way If the PEP is accepted I'll do the
implementation for 3.4.
The backporting is the main stumbling block. Otherwise, I'm in favor
of the PEP.
My plan now is to split the PEP in two, so the 3.4 changes can be accepted
as non-controversial, including the offer of core dev assistance in
creating and maintaining a Windows installer for pip to better support
earlier versions. The backporting PEP will be deferred, for reconsideration
some time in the future after the initial PEP has been implemented.

Regards,
Nick.
Regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlJJZZsACgkQavBT8H2dyNIAWQCeNi3OcIXaNGmFuDfPcTEfnBjV
vTMAnAv7diLGOsJ2oNUqE7BHG1wIwMCP
=r+wP
-----END PGP SIGNATURE-----
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Barry Warsaw
2013-09-30 14:02:56 UTC
Permalink
Post by Nick Coghlan
My plan now is to split the PEP in two, so the 3.4 changes can be accepted
as non-controversial, including the offer of core dev assistance in
creating and maintaining a Windows installer for pip to better support
earlier versions. The backporting PEP will be deferred, for reconsideration
some time in the future after the initial PEP has been implemented.
+1, since I think there's little disagreement about adding this to 3.4.

-Barry
Skip Montanaro
2013-09-30 14:35:24 UTC
Permalink
Splitting into two pieces also means you can implement it for 3.4
first and identify possible problems caused by preexisting pip
installs before deciding whether to add it to 2.7 and 3.3.

Skip
Nick Coghlan
2013-09-30 14:47:11 UTC
Permalink
Post by Skip Montanaro
Splitting into two pieces also means you can implement it for 3.4
first and identify possible problems caused by preexisting pip
installs before deciding whether to add it to 2.7 and 3.3.
One of the key reasons for using the bootstrap mechanism rather than
just bundling pip is so that the installers for maintenance releases
won't have any more trouble with pre-existing installs of pip than pip
itself does :)

But yes, there are several advantages to splitting the PEP:

- the Python 3.4 changes are non-controversial, so it makes sense to
get them officially accepted
- the Python 3.4 beta deadline is closer than the deadlines for the
3.3.3 and 2.7.6 maintenance releases
- it allows the full extent of the changes proposed for backporting to
be clear prior to the reconsideration of the separated PEP
- a Windows installer for pip could be made available well in advance
of the 3.3.3 and 2.7.6 maintenance releases
- it allows a chance to see if a separate Windows installer for pip
and documenting the bootstrapping instructions in the "Installing
Python Modules" guide for older releases is sufficient to get new
users over the "getting started with PyPI" barrier

Cheers,
Nick.
--
Nick Coghlan | ***@gmail.com | Brisbane, Australia
Nick Coghlan
2013-09-30 11:32:13 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Post by Donald Stufft
An early draft of this did not have the backport to 2.7 and when I
showed *that* version around to get feedback people were less
enthusiastic about it and generally viewed it as "nice but
worthless to me for N years".
I'm leaning towards the people that oppose addition of this feature to
older Python releases. In particular, the objection that the new PEP is
It is always the case that features proposed by a PEP reach users only
years after they have been implemented.
Post by Donald Stufft
What users want isn't rationale in and of itself but I think it's
an important data point, especially given how long 2.7.LASTEVER is
going to be relevant to end users.
Well, I really really don't like this idea. 2.7 should not get new
features; users who want new features need to switch to 3.x.
Before you make your final decision on this front, I'd like to record in
the PEP my fallback plan for if the backporting portion of the PEP is
rejected while the rest is accepted. After all, you should know what it is
you're actually choosing between.

I just need to run the details of the fallback proposal past Donald before
publishing it under both our names.

Regards,
Nick.
Regards,
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlJJPfkACgkQavBT8H2dyNL9SQCfceoTqIeCiwq0GMdaWeUSn/nV
6E4AmwRfmjFcSXjdmGLQbewVqxxYwRit
=tLof
-----END PGP SIGNATURE-----
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Raymond Hettinger
2013-09-27 18:15:10 UTC
Permalink
Post by Nick Coghlan
With the last round of updates, I believe PEP 453 is ready for
Martin's pronouncement.
Personally, I'm very excited and happy that this or something like it
is coming close to fruition.

My experiences in userland suggest that this will be a big step
forward for Python usability.


Raymond
Loading...