Python 2.7 patch levels turning two digit

[M.-A. Lemburg]

With PEP 466 and the constant flow of OpenSSL security fixes
which are currently being handled via Python patch level releases,
we will soon reach 2.7.10 and quickly go beyond that (also see
http://bugs.python.org/issue21308).

This opens up a potential backwards incompatibility with existing
tools that assume the Python release version number to use the
"x.y.z" single digit approach, e.g. code that uses sys.version[:5]
for the Python version or relies on the lexicographic ordering
of the version string (sys.version > '2.7.2').

Some questions we should probably ask ourselves (I've added my
thoughts inline):

* Is it a good strategy to ship to Python releases for every
single OpenSSL security release or is there a better way to
handle these 3rd party issues ?

I think we should link to the OpenSSL libs dynamically rather
than statically in Python 2.7 for Windows so that it's possible
to provide drop-in updates for such issues.

* Should we try to avoid two digit patch level release numbers
by using some other mechanism such as e.g. a release date
after 2.7.9 ?

Grepping through our code, this will introduce some breakage,
but not much. Most older code branches on minor versions,
not patch levels. More recent code uses sys.python_info so
is not affected.

* Should we make use of the potential breakage with 2.7.10
to introduce a new Windows compiler version for Python 2.7 ?

I think this would be a good chance to update the compiler
to whatever we use for Python 3 at the time.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jun 21 2014)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 11 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Nick Coghlan]

On 21 June 2014 20:27, M.-A. Lemburg <***@egenix.com> wrote:
> With PEP 466 and the constant flow of OpenSSL security fixes
> which are currently being handled via Python patch level releases,
> we will soon reach 2.7.10 and quickly go beyond that (also see
> http://bugs.python.org/issue21308).
>
> This opens up a potential backwards incompatibility with existing
> tools that assume the Python release version number to use the
> "x.y.z" single digit approach, e.g. code that uses sys.version[:5]
> for the Python version or relies on the lexicographic ordering
> of the version string (sys.version > '2.7.2').

Such code has an easy fix available, though, as sys.version_info has
existed since 2.0, and handles two digit micro releases just fine. The
docs for sys.version also have this explicit disclaimer: "Do not
extract version information out of it, rather, use version_info and
the functions provided by the platform module."

Making it harder to tell whether or not someone's Python installation
is affected by an OpenSSL CVE is also an undesirable outcome. On a
Linux distro, folks will check the distro package database directly
for the OpenSSL version, but on Windows, no such centralised audit
mechanism is available by default. With OpenSSL statically linked,
Python versions can just be mapped to OpenSSL versions (so, for
example, 2.7.7 has 1.0.1g)

Cheers,
Nick.

--
Nick Coghlan | ***@gmail.com | Brisbane, Australia

[M.-A. Lemburg]

On 21.06.2014 12:51, Nick Coghlan wrote:
> On 21 June 2014 20:27, M.-A. Lemburg <***@egenix.com> wrote:
>> With PEP 466 and the constant flow of OpenSSL security fixes
>> which are currently being handled via Python patch level releases,
>> we will soon reach 2.7.10 and quickly go beyond that (also see
>> http://bugs.python.org/issue21308).
>>
>> This opens up a potential backwards incompatibility with existing
>> tools that assume the Python release version number to use the
>> "x.y.z" single digit approach, e.g. code that uses sys.version[:5]
>> for the Python version or relies on the lexicographic ordering
>> of the version string (sys.version > '2.7.2').
>
> Such code has an easy fix available, though, as sys.version_info has
> existed since 2.0, and handles two digit micro releases just fine. The
> docs for sys.version also have this explicit disclaimer: "Do not
> extract version information out of it, rather, use version_info and
> the functions provided by the platform module."

I don't think that's a good argument. Of course, there are
better ways to figure out the version number, but fact is,
existing code, even in the stdlib, does use and parse
the sys.version string version.

During Python's lifetime, we've always avoided two digit
version numbers, so people have been relying on this, even
if it was never (AFAIK) documented anywhere.

> Making it harder to tell whether or not someone's Python installation
> is affected by an OpenSSL CVE is also an undesirable outcome. On a
> Linux distro, folks will check the distro package database directly
> for the OpenSSL version, but on Windows, no such centralised audit
> mechanism is available by default. With OpenSSL statically linked,
> Python versions can just be mapped to OpenSSL versions (so, for
> example, 2.7.7 has 1.0.1g)

I have to disagree here as well :-)

If people cannot upgrade to a higher patch level for whatever
reason (say a patch level release introduced some other bugs),
but still need to upgrade to the current OpenSSL version, they'd
be stuck if we continue to bind the Python version number to
some OpenSSL release version.

We should definitely make it possible to address OpenSSL
bugs without having to upgrade Python and it's not hard to
do: just replace the static binding with dynamic binding
and include the two OpenSSL DLLs with the Windows installer.

People can then drop in new versions of those DLLs
as needed, without having the core devs do a complete
new release every time someone finds a new problem those
libs. Security libs simply have a much higher release
rate (if they are well maintained) than most other
software.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jun 21 2014)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 11 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Ned Deily]

In article <***@egenix.com>,
"M.-A. Lemburg" <***@egenix.com> wrote:
> > Making it harder to tell whether or not someone's Python installation
> > is affected by an OpenSSL CVE is also an undesirable outcome. On a
> > Linux distro, folks will check the distro package database directly
> > for the OpenSSL version, but on Windows, no such centralised audit
> > mechanism is available by default. With OpenSSL statically linked,
> > Python versions can just be mapped to OpenSSL versions (so, for
> > example, 2.7.7 has 1.0.1g)
>
> I have to disagree here as well :-)
>
> If people cannot upgrade to a higher patch level for whatever
> reason (say a patch level release introduced some other bugs),
> but still need to upgrade to the current OpenSSL version, they'd
> be stuck if we continue to bind the Python version number to
> some OpenSSL release version.
>
> We should definitely make it possible to address OpenSSL
> bugs without having to upgrade Python and it's not hard to
> do: just replace the static binding with dynamic binding
> and include the two OpenSSL DLLs with the Windows installer.
>
> People can then drop in new versions of those DLLs
> as needed, without having the core devs do a complete
> new release every time someone finds a new problem those
> libs. Security libs simply have a much higher release
> rate (if they are well maintained) than most other
> software.

I agree that with Nick and Barry that, due to the extended support
period for 2.7, we have no choice but to bite the bullet and deal with
micro levels exceeding 9. On the other hand, it would also be good to
be better able to deal with third-party library revisions that only
affect the Windows or OS X binary installers we supply. I don't know
that we've ever had a process/policy in place to do that, certainly not
recently. Even for statically linked libraries, we presumably could
supply replacement re-linked files or even carefully repacked installers
with the updated files. This might be something to discuss and add to
PEP 101 or a new PEP.

Up to now, this hasn't been a major concern since there have usually
been other reasons to do full releases as well, e.g. source regressions.
Given the still relatively high churn rate for changes going into 2.7
and the growing distance between the 2.7 and 3.x code bases (among other
things, leading to more frequent inadvertent backporting errors), we'll
probably need to keep making relatively frequent 2.7 releases unless we
can further slow down the 2.7 change rate.

--
Ned Deily,
***@acm.org

[Chris Angelico]

On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg <***@egenix.com> wrote:
> On 21.06.2014 12:51, Nick Coghlan wrote:
>> Such code has an easy fix available, though, as sys.version_info has
>> existed since 2.0, and handles two digit micro releases just fine. The
>> docs for sys.version also have this explicit disclaimer: "Do not
>> extract version information out of it, rather, use version_info and
>> the functions provided by the platform module."
>
> I don't think that's a good argument. Of course, there are
> better ways to figure out the version number, but fact is,
> existing code, even in the stdlib, does use and parse
> the sys.version string version.
>
> During Python's lifetime, we've always avoided two digit
> version numbers, so people have been relying on this, even
> if it was never (AFAIK) documented anywhere.

It's going to be a broken-code-breaking change that's introduced in a
point release, but since PEP 404 implicitly says that there won't be a
2.10.0, there's no way around that. Although actually, a glance at the
stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than
2.7.10 would break - there are places where sys.version[:3] is used
(or equivalents like "... %.3s ..." % sys.version), or a whole-string
comparison is done against a two-part version string (eg: sys.version
>= "2.6"), and at least one place that checks sys.version[0] for the
major version number, but I didn't find any that look at
sys.version[:5] or equivalent. Everything that cares about the
three-part version number seems to either look at
sys.version.split()[0] or sys.version_info. Do you know where this
problematic code is?

I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy
system, for what it's worth.

ChrisA

[Oleg Broytman]

On Sun, Jun 22, 2014 at 06:34:23AM +1000, Chris Angelico <***@gmail.com> wrote:
> Do you know where this problematic code is?

In many places:

https://encrypted.google.com/search?q=%22sys.version[%3A3]%22

https://encrypted.google.com/search?q=%22sys.version[%3A5]%22

Oleg.
--
Oleg Broytman http://phdru.name/ ***@phdru.name
Programmers don't die, they just GOSUB without RETURN.

[M.-A. Lemburg]

On 21.06.2014 22:34, Chris Angelico wrote:
> On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg <***@egenix.com> wrote:
>> On 21.06.2014 12:51, Nick Coghlan wrote:
>>> Such code has an easy fix available, though, as sys.version_info has
>>> existed since 2.0, and handles two digit micro releases just fine. The
>>> docs for sys.version also have this explicit disclaimer: "Do not
>>> extract version information out of it, rather, use version_info and
>>> the functions provided by the platform module."
>>
>> I don't think that's a good argument. Of course, there are
>> better ways to figure out the version number, but fact is,
>> existing code, even in the stdlib, does use and parse
>> the sys.version string version.
>>
>> During Python's lifetime, we've always avoided two digit
>> version numbers, so people have been relying on this, even
>> if it was never (AFAIK) documented anywhere.
>
> It's going to be a broken-code-breaking change that's introduced in a
> point release, but since PEP 404 implicitly says that there won't be a
> 2.10.0, there's no way around that. Although actually, a glance at the
> stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than
> 2.7.10 would break - there are places where sys.version[:3] is used
> (or equivalents like "... %.3s ..." % sys.version), or a whole-string
> comparison is done against a two-part version string (eg: sys.version
>> = "2.6"), and at least one place that checks sys.version[0] for the
> major version number, but I didn't find any that look at
> sys.version[:5] or equivalent. Everything that cares about the
> three-part version number seems to either look at
> sys.version.split()[0] or sys.version_info. Do you know where this
> problematic code is?
>
> I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy
> system, for what it's worth.

There are no places in the stdlib that parse sys.version in a
way that would break wtih 2.7.10, AFAIK. I was just referring
to the statement that Nick quoted. sys.version *is* used for
parsing the Python version or using parts of it to build
e.g. filenames and that's really no surprise.

That said, and I also included this in my answers to the questions
that Nick removed in his reply, I don't think that a lot of
code would be affected by this. I do believe that we can use
this potential breakage as a chance for improvement. See the last
question (listed here again)...

1. Is it a good strategy to ship to Python releases for every
single OpenSSL security release or is there a better way to
handle these 3rd party issues ?

2. Should we try to avoid two digit patch level release numbers
by using some other mechanism such as e.g. a release date
after 2.7.9 ?

3. Should we make use of the potential breakage with 2.7.10
to introduce a new Windows compiler version for Python 2.7 ?

My answers to these are: 1. We should use dynamic linking
instead and not let OpenSSL bugs trigger Python releases; 2.
It's not a big problem; 3. Yes, please, since it is difficult
for people to develop and debug their extensions with a
2008 compiler, when the rest of the world has long moved on.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jun 21 2014)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 11 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Phil Thompson]

On 21/06/2014 10:37 pm, M.-A. Lemburg wrote:
> That said, and I also included this in my answers to the questions
> that Nick removed in his reply, I don't think that a lot of
> code would be affected by this. I do believe that we can use
> this potential breakage as a chance for improvement. See the last
> question (listed here again)...
>
> 1. Is it a good strategy to ship to Python releases for every
> single OpenSSL security release or is there a better way to
> handle these 3rd party issues ?

Isn't this only a packaging issue? There is no change to the Python API
or implementation, so there is no need to change the version number. So
just make new Windows packages.

The precedent is to add a dash and a package number. I can't remember
what version this was applied to before - but I got a +1 from Guido for
suggesting it :)

Phil

[Ethan Furman]

On 06/21/2014 02:37 PM, M.-A. Lemburg wrote:
>
> My answers to these are: 1. We should use dynamic linking
> instead and not let OpenSSL bugs trigger Python releases; 2.
> It's not a big problem; 3. Yes, please, since it is difficult
> for people to develop and debug their extensions with a
> 2008 compiler, when the rest of the world has long moved on.

+1 (assuming not incredibly difficult and those that can are willing ;)

--
~Ethan~

[Ethan Furman]

On 06/21/2014 02:48 PM, Ethan Furman wrote:
> On 06/21/2014 02:37 PM, M.-A. Lemburg wrote:
>>
>> My answers to these are: 1. We should use dynamic linking
>> instead and not let OpenSSL bugs trigger Python releases; 2.
>> It's not a big problem; 3. Yes, please, since it is difficult
>> for people to develop and debug their extensions with a
>> 2008 compiler, when the rest of the world has long moved on.
>
> +1 (assuming not incredibly difficult and those that can are willing ;)

Revising this to:

+1, -0, -1

It seems to me the intention of supporting 2.7 for so long was not to give ourselves additional nightmares, but to
provide a basic level of support for those who are needing longer time before migrating. One of the reasons to migrate
is to avoid future pain (pain is an excellent motivator -- it's why we don't go to the doctor when we're healthy, right?
;) If getting new or updated modules becomes more painful then that's motivation to upgrade -- not motivation for us to
make both our lives (with the extra work) and everyone's else lives (why isn't this module working? oh, wrong compiler)
more difficult.

--
~Ethan~

[Steve Dower]

We can always lie about the version in sys.version. Existing code is unaffected and new code will have to use version_info (Windows developers will know that Windows pulls tricks like this every other version... doesn't make it a great idea, but it works).

Changing compiler without changing at least the install directory and preventing in place upgrades is a really bad idea, and with those mitigations is only pretty bad. I'm torn here, because I know the current situation hurts, but it'd probably only move to VC10 which will hurt just as much in a few years... there are better tooling solutions (yes, I'm working on some behind the scenes).

A separate distro of _ssl and _hashlib wouldn't be too hard and has the same effect as a dynamically linked OpenSSL. Maybe we can make these PyPI updateable?

Top-posted from my Windows Phone
________________________________
From: M.-A. Lemburg<mailto:***@egenix.com>
Sent: ý6/ý21/ý2014 14:38
To: Chris Angelico<mailto:***@gmail.com>
Cc: Python-Dev<mailto:python-***@python.org>
Subject: Re: [Python-Dev] Python 2.7 patch levels turning two digit

On 21.06.2014 22:34, Chris Angelico wrote:
> On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg <***@egenix.com> wrote:
>> On 21.06.2014 12:51, Nick Coghlan wrote:
>>> Such code has an easy fix available, though, as sys.version_info has
>>> existed since 2.0, and handles two digit micro releases just fine. The
>>> docs for sys.version also have this explicit disclaimer: "Do not
>>> extract version information out of it, rather, use version_info and
>>> the functions provided by the platform module."
>>
>> I don't think that's a good argument. Of course, there are
>> better ways to figure out the version number, but fact is,
>> existing code, even in the stdlib, does use and parse
>> the sys.version string version.
>>
>> During Python's lifetime, we've always avoided two digit
>> version numbers, so people have been relying on this, even
>> if it was never (AFAIK) documented anywhere.
>
> It's going to be a broken-code-breaking change that's introduced in a
> point release, but since PEP 404 implicitly says that there won't be a
> 2.10.0, there's no way around that. Although actually, a glance at the
> stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than
> 2.7.10 would break - there are places where sys.version[:3] is used
> (or equivalents like "... %.3s ..." % sys.version), or a whole-string
> comparison is done against a two-part version string (eg: sys.version
>> = "2.6"), and at least one place that checks sys.version[0] for the
> major version number, but I didn't find any that look at
> sys.version[:5] or equivalent. Everything that cares about the
> three-part version number seems to either look at
> sys.version.split()[0] or sys.version_info. Do you know where this
> problematic code is?
>
> I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy
> system, for what it's worth.

There are no places in the stdlib that parse sys.version in a
way that would break wtih 2.7.10, AFAIK. I was just referring
to the statement that Nick quoted. sys.version *is* used for
parsing the Python version or using parts of it to build
e.g. filenames and that's really no surprise.

That said, and I also included this in my answers to the questions
that Nick removed in his reply, I don't think that a lot of
code would be affected by this. I do believe that we can use
this potential breakage as a chance for improvement. See the last
question (listed here again)...

1. Is it a good strategy to ship to Python releases for every
single OpenSSL security release or is there a better way to
handle these 3rd party issues ?

2. Should we try to avoid two digit patch level release numbers
by using some other mechanism such as e.g. a release date
after 2.7.9 ?

3. Should we make use of the potential breakage with 2.7.10
to introduce a new Windows compiler version for Python 2.7 ?

My answers to these are: 1. We should use dynamic linking
instead and not let OpenSSL bugs trigger Python releases; 2.
It's not a big problem; 3. Yes, please, since it is difficult
for people to develop and debug their extensions with a
2008 compiler, when the rest of the world has long moved on.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jun 21 2014)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 11 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Donald Stufft]

On Jun 21, 2014, at 6:00 PM, Steve Dower <***@microsoft.com> wrote:

> We can always lie about the version in sys.version. Existing code is unaffected and new code will have to use version_info (Windows developers will know that Windows pulls tricks like this every other version... doesn't make it a great idea, but it works).
>
> Changing compiler without changing at least the install directory and preventing in place upgrades is a really bad idea, and with those mitigations is only pretty bad. I'm torn here, because I know the current situation hurts, but it'd probably only move to VC10 which will hurt just as much in a few years... there are better tooling solutions (yes, I'm working on some behind the scenes).
>
> A separate distro of _ssl and _hashlib wouldn't be too hard and has the same effect as a dynamically linked OpenSSL. Maybe we can make these PyPI updateable?

Stuff from PyPI installs later on in the sys.path than the stdlib. I wish it were different but it means without sys.path shenanigans you can’t replace the stdlib with something from PyPI.

>
> Top-posted from my Windows Phone
> From: M.-A. Lemburg
> Sent: ý6/ý21/ý2014 14:38
> To: Chris Angelico
> Cc: Python-Dev
> Subject: Re: [Python-Dev] Python 2.7 patch levels turning two digit
>
> On 21.06.2014 22:34, Chris Angelico wrote:
> > On Sun, Jun 22, 2014 at 2:57 AM, M.-A. Lemburg <***@egenix.com> wrote:
> >> On 21.06.2014 12:51, Nick Coghlan wrote:
> >>> Such code has an easy fix available, though, as sys.version_info has
> >>> existed since 2.0, and handles two digit micro releases just fine. The
> >>> docs for sys.version also have this explicit disclaimer: "Do not
> >>> extract version information out of it, rather, use version_info and
> >>> the functions provided by the platform module."
> >>
> >> I don't think that's a good argument. Of course, there are
> >> better ways to figure out the version number, but fact is,
> >> existing code, even in the stdlib, does use and parse
> >> the sys.version string version.
> >>
> >> During Python's lifetime, we've always avoided two digit
> >> version numbers, so people have been relying on this, even
> >> if it was never (AFAIK) documented anywhere.
> >
> > It's going to be a broken-code-breaking change that's introduced in a
> > point release, but since PEP 404 implicitly says that there won't be a
> > 2.10.0, there's no way around that. Although actually, a glance at the
> > stdlib suggests that 2.10.0 (or 3.10.0) would break a lot more than
> > 2.7.10 would break - there are places where sys.version[:3] is used
> > (or equivalents like "... %.3s ..." % sys.version), or a whole-string
> > comparison is done against a two-part version string (eg: sys.version
> >> = "2.6"), and at least one place that checks sys.version[0] for the
> > major version number, but I didn't find any that look at
> > sys.version[:5] or equivalent. Everything that cares about the
> > three-part version number seems to either look at
> > sys.version.split()[0] or sys.version_info. Do you know where this
> > problematic code is?
> >
> > I checked this in the 2.7.3 stdlib as packaged on my Debian Wheezy
> > system, for what it's worth.
>
> There are no places in the stdlib that parse sys.version in a
> way that would break wtih 2.7.10, AFAIK. I was just referring
> to the statement that Nick quoted. sys.version *is* used for
> parsing the Python version or using parts of it to build
> e.g. filenames and that's really no surprise.
>
> That said, and I also included this in my answers to the questions
> that Nick removed in his reply, I don't think that a lot of
> code would be affected by this. I do believe that we can use
> this potential breakage as a chance for improvement. See the last
> question (listed here again)...
>
> 1. Is it a good strategy to ship to Python releases for every
> single OpenSSL security release or is there a better way to
> handle these 3rd party issues ?
>
> 2. Should we try to avoid two digit patch level release numbers
> by using some other mechanism such as e.g. a release date
> after 2.7.9 ?
>
> 3. Should we make use of the potential breakage with 2.7.10
> to introduce a new Windows compiler version for Python 2.7 ?
>
> My answers to these are: 1. We should use dynamic linking
> instead and not let OpenSSL bugs trigger Python releases; 2.
> It's not a big problem; 3. Yes, please, since it is difficult
> for people to develop and debug their extensions with a
> 2008 compiler, when the rest of the world has long moved on.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source (#1, Jun 21 2014)
> >>> Python Projects, Consulting and Support ... http://www.egenix.com/
> >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
> ________________________________________________________________________
> 2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
> 2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57
> 2014-07-02: Python Meeting Duesseldorf ... 11 days to go
>
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
> http://www.egenix.com/company/contact/
> _______________________________________________
> Python-Dev mailing list
> Python-***@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
> _______________________________________________
> Python-Dev mailing list
> Python-***@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Chris Angelico]

On Sun, Jun 22, 2014 at 8:00 AM, Steve Dower <***@microsoft.com> wrote:
> We can always lie about the version in sys.version. Existing code is
> unaffected and new code will have to use version_info (Windows developers
> will know that Windows pulls tricks like this every other version... doesn't
> make it a great idea, but it works).

I'd prefer a change of format to an outright lie. Something like
"2.7._10" which will sort after "2.7.9". But ideally, nothing at all -
just go smoothly to "2.7.10" and let broken code be broken. It'll
think it's running on 2.7.1, and if anything needs to distinguish
between 2.7.1 and 2.7.x, hopefully it's using version_info.

ChrisA

[Chris Angelico]

On Sun, Jun 22, 2014 at 7:37 AM, M.-A. Lemburg <***@egenix.com> wrote:
> There are no places in the stdlib that parse sys.version in a
> way that would break wtih 2.7.10, AFAIK. I was just referring
> to the statement that Nick quoted. sys.version *is* used for
> parsing the Python version or using parts of it to build
> e.g. filenames and that's really no surprise.

Right, good to know. I thought you were implying that stuff would
break. Yes, stuff definitely does parse out the version number from
sys.version, lots of that happens.

ChrisA

[Barry Warsaw]

On Jun 21, 2014, at 12:27 PM, M.-A. Lemburg wrote:

>This opens up a potential backwards incompatibility with existing
>tools that assume the Python release version number to use the
>"x.y.z" single digit approach, e.g. code that uses sys.version[:5]
>for the Python version or relies on the lexicographic ordering
>of the version string (sys.version > '2.7.2').

Patient: Doctor, it hurts when I do this.
Doctor: Don't do that!

> * Should we try to avoid two digit patch level release numbers
> by using some other mechanism such as e.g. a release date
> after 2.7.9 ?
>
> Grepping through our code, this will introduce some breakage,
> but not much. Most older code branches on minor versions,
> not patch levels. More recent code uses sys.python_info so
> is not affected.

s/sys.python_info/sys.version_info/ and yes the latter has been preferred for
a long time now.

Given that 2.7 is a long term support release, it's inevitable that we'll
break the 2-digit micro release number barrier. So be it. A 2.7.10 isn't the
end of the world.

-Barry

[Martin v. Löwis]

> * Is it a good strategy to ship to Python releases for every
> single OpenSSL security release or is there a better way to
> handle these 3rd party issues ?

At least for Windows, a new release certainly needs to be made.
It could be possible to produce MSI patch files, but this would
still be a new release.

> I think we should link to the OpenSSL libs dynamically rather
> than statically in Python 2.7 for Windows so that it's possible
> to provide drop-in updates for such issues.

It is possible to provide drop-in updates regardless of whether the
OpenSSL libs are dynamically linked, as the _ssl module itself is a
dynamic lib.

> * Should we try to avoid two digit patch level release numbers
> by using some other mechanism such as e.g. a release date
> after 2.7.9 ?

If it was for me, then yes, certainly: the development of 2.7 should
just stop :-)

> * Should we make use of the potential breakage with 2.7.10
> to introduce a new Windows compiler version for Python 2.7 ?

Assuming it is a good idea to continue producing Windows binaries
for 2.7, I think it would be a bad idea to switch compilers. It will
cause severe breakage of 2.7 installations, much more problematic
than switching to two-digit version numbers.

Regards,
Martin

[Donald Stufft]

On Jun 23, 2014, at 2:09 AM, Martin v. Löwis <***@v.loewis.de> wrote:

>>
>> * Should we make use of the potential breakage with 2.7.10
>> to introduce a new Windows compiler version for Python 2.7 ?
>
> Assuming it is a good idea to continue producing Windows binaries
> for 2.7, I think it would be a bad idea to switch compilers. It will
> cause severe breakage of 2.7 installations, much more problematic
> than switching to two-digit version numbers.

I agree with this, we’ve just finally started getting things to the point where
it makes a lot of sense for binary distributions for Windows. Breaking all
of them on 2.7 would be very bad.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[M.-A. Lemburg]

On 23.06.2014 18:09, Donald Stufft wrote:
>
> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis <***@v.loewis.de> wrote:
>
>>>
>>> * Should we make use of the potential breakage with 2.7.10
>>> to introduce a new Windows compiler version for Python 2.7 ?
>>
>> Assuming it is a good idea to continue producing Windows binaries
>> for 2.7, I think it would be a bad idea to switch compilers. It will
>> cause severe breakage of 2.7 installations, much more problematic
>> than switching to two-digit version numbers.
>
> I agree with this, we’ve just finally started getting things to the point where
> it makes a lot of sense for binary distributions for Windows. Breaking all
> of them on 2.7 would be very bad.

Not sure what you mean. We've had binary wininst distributions
for Windows for more than a decade, and egg and msi distributions
for 8 years :-)

But without access to the VS 2008 compiler that is needed to
compile those extensions, it will become increasingly difficult
for package authors to provide such binary packages, so we have to
ask ourselves:

What's worse: breaking old Windows binaries for Python 2.7
or not having updated and new Windows binaries for Python 2.7
at all in a few years ?

Switching to a newer compiler will make things easier for everyone
and we'd see more binary packages for Windows again.

Given that Python 2.7 support was extended for another 5 years at the
recent Python Language Summit to 2020, we have to face this
breakage sooner or later anyway. Extended support for VS 2008
will end in 2018 (but then: Python developers usually don't have
extended support contracts with MS). Service pack support has already
ended in 2009.

Depending on how you see it, using such an old compiler also
poses security risks. The last security update for VS 2008 dates
back to 2011 (http://support.microsoft.com/kb/2538243).

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jun 23 2014)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
2014-07-02: Python Meeting Duesseldorf ... 9 days to go
2014-07-21: EuroPython 2014, Berlin, Germany ... 28 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Ned Deily]

In article <***@egenix.com>,
"M.-A. Lemburg" <***@egenix.com> wrote:
[...]
> But without access to the VS 2008 compiler that is needed to
> compile those extensions, it will become increasingly difficult
> for package authors to provide such binary packages, so we have to
> ask ourselves:
>
> What's worse: breaking old Windows binaries for Python 2.7
> or not having updated and new Windows binaries for Python 2.7
> at all in a few years ?
>
> Switching to a newer compiler will make things easier for everyone
> and we'd see more binary packages for Windows again.

It does seem like a conundrum. As I have no deep Windows experience to
be able to have an appreciation of all of the technical issues involved,
I ask out of ignorance: would it be possible and desirable to provide a
transition period of n 2.7.x maintenance releases (where n is between 1
and, say, 3) where we would provide 2 sets of Windows installers, one
set (32- and 64-bit) with the older compiler and CRT and one with the
newer, and campaign to get users and packagers who provide binary
extensions to migrate? Would that mitigate the pain, assuming that
Steve (or someone else) would be willing to build the additional
installers for the transition period? I've done something similar on a
smaller scale with the OS X 32-bit installer for 2.7.x but that impact
is much less as the audience for that installer is much smaller.

--
Ned Deily,
***@acm.org

[Martin v. Löwis]

Am 23.06.14 21:53, schrieb Ned Deily:
> It does seem like a conundrum. As I have no deep Windows experience to
> be able to have an appreciation of all of the technical issues involved,
> I ask out of ignorance: would it be possible and desirable to provide a
> transition period of n 2.7.x maintenance releases (where n is between 1
> and, say, 3) where we would provide 2 sets of Windows installers, one
> set (32- and 64-bit) with the older compiler and CRT and one with the
> newer, and campaign to get users and packagers who provide binary
> extensions to migrate?

The question is how exactly you implement the transition. I see two
alternatives:
1. "Hijack" the 2.7 name space, in particular the name "python27.dll",
along with registry keys, the .pyd extension, etc.
Doing so would permit users to mix binaries from different compilers,
and doing so would lead to crashes. Users would have to be careful
to either install packages built for the old compiler or packages
for the new compiler, and never mix.
2. "Sandbox" the 2.7 name space; come up with new names for everything.
E.g. "python27vs13.dll", ".pydvs13" (or "_vs13.pyd"),
"C:\Python27vs13", along with the corresponding changes to PyPI,
pip, buildout, etc. which would need to learn to look for the right
variant of a Python 2.7 package.
This should work, but might take several years to implement: you
need to find all the places in existing code that refer to the
"old" names.
If you do it right, you are done about the time when VS 13 becomes
unavailable, so you'd then do another such transition to VS 2015,
which promises forward-binary compatibility to future releases of
Visual Studio.

> Would that mitigate the pain, assuming that
> Steve (or someone else) would be willing to build the additional
> installers for the transition period? I've done something similar on a
> smaller scale with the OS X 32-bit installer for 2.7.x but that impact
> is much less as the audience for that installer is much smaller.

Well, the question really is whether precompiled extension modules
available from PyPI would work on both compilers. I understand that
for OSX, you typically don't have precompiled binaries for the extension
modules, so installation compiles the modules from scratch. This is
easier, as it can use the ABI of the Python which will be installed
to.

If you go the "parallel ABIs" route, extension authors have to provide
two parallel sets of packages as well. Given 32-bit and 64-bit packages,
this will make actually two additional packages - just as if they had
to support another Python version.

Regards,
Martin

[Donald Stufft]

On Jun 23, 2014, at 4:31 PM, Martin v. Löwis <***@v.loewis.de> wrote:

>>
>> Would that mitigate the pain, assuming that
>> Steve (or someone else) would be willing to build the additional
>> installers for the transition period? I've done something similar on a
>> smaller scale with the OS X 32-bit installer for 2.7.x but that impact
>> is much less as the audience for that installer is much smaller.
>
> Well, the question really is whether precompiled extension modules
> available from PyPI would work on both compilers. I understand that
> for OSX, you typically don't have precompiled binaries for the extension
> modules, so installation compiles the modules from scratch. This is
> easier, as it can use the ABI of the Python which will be installed
> to.
>
> If you go the "parallel ABIs" route, extension authors have to provide
> two parallel sets of packages as well. Given 32-bit and 64-bit packages,
> this will make actually two additional packages - just as if they had
> to support another Python version.

As far as I know, stuff on OSX is generally built for “X compiler or later”
so binary compatibility is kept as long as you’re using an “or later” but
I could be wrong about that. Using binary packages on OSX is a much
less frequent thing I think though since getting a working compiler toolchain
is easier there.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Ned Deily]

In article <14DE41E2-5314-4E49-BE93-***@stufft.io>,
Donald Stufft <***@stufft.io> wrote:

> On Jun 23, 2014, at 4:31 PM, Martin v. Lowis <***@v.loewis.de> wrote:
>
> >>
> >> Would that mitigate the pain, assuming that
> >> Steve (or someone else) would be willing to build the additional
> >> installers for the transition period? I've done something similar on a
> >> smaller scale with the OS X 32-bit installer for 2.7.x but that impact
> >> is much less as the audience for that installer is much smaller.
> >
> > Well, the question really is whether precompiled extension modules
> > available from PyPI would work on both compilers. I understand that
> > for OSX, you typically don't have precompiled binaries for the extension
> > modules, so installation compiles the modules from scratch. This is
> > easier, as it can use the ABI of the Python which will be installed
> > to.
> >
> > If you go the "parallel ABIs" route, extension authors have to provide
> > two parallel sets of packages as well. Given 32-bit and 64-bit packages,
> > this will make actually two additional packages - just as if they had
> > to support another Python version.
>
> As far as I know, stuff on OSX is generally built for "X compiler or later"
> so binary compatibility is kept as long as you're using an "or later" but
> I could be wrong about that. Using binary packages on OSX is a much
> less frequent thing I think though since getting a working compiler toolchain
> is easier there.

Both points are generally true on OS X so, yes, binary extensions are
much less of an issue there. Where binary distributions on OS X are
most used, I think, is when there are dependencies on third-party
non-Python libraries that are not shipped by Apple with OS X.

But, yes, if we were to go down the route of two sets of installers,
that could mean two sets of third-party packages. I suppose there could
potentially be some pip / wheel / possibly Distutils help by
conditioning the platform name or other component used to generate the
egg / wheel / and/or bdist file names on the CRT version (or compiler
version), much as what we do today with OS X deployment target. Again,
I'm speculating in ignorance here. If that were feasible, things built
with the old toolchain could have unchanged names. And, clearly, we
would want to keep that "n" number of releases with two sets of
installers to be as small as possible, like 1. While there would be a
certain amount of unavoidable disruption for some folks, others *might*
welcome the opportunity to no longer have to keep around old versions of
the tool chain, particularly if they now could use the same tool chain
to produce binaries for both Py2 and Py3.

--
Ned Deily,
***@acm.org

[Antoine Pitrou]

Le 23/06/2014 15:27, M.-A. Lemburg a écrit :
>
> Not sure what you mean. We've had binary wininst distributions
> for Windows for more than a decade, and egg and msi distributions
> for 8 years :-)
>
> But without access to the VS 2008 compiler that is needed to
> compile those extensions,

It does seem to be available:
http://www.microsoft.com/en-us/download/details.aspx?id=13276

What am I missing?

Regards

Antoine.

[Ethan Furman]

On 06/23/2014 01:04 PM, Antoine Pitrou wrote:
> Le 23/06/2014 15:27, M.-A. Lemburg a écrit :
>>
>> Not sure what you mean. We've had binary wininst distributions
>> for Windows for more than a decade, and egg and msi distributions
>> for 8 years :-)
>>
>> But without access to the VS 2008 compiler that is needed to
>> compile those extensions,
>
> It does seem to be available:
> http://www.microsoft.com/en-us/download/details.aspx?id=13276
>
> What am I missing?

Is that VS 2008 /with/ the SP, or just the SP?

--
~Ethan~

[Martin v. Löwis]

Am 23.06.14 22:04, schrieb Antoine Pitrou:
> Le 23/06/2014 15:27, M.-A. Lemburg a écrit :
>>
>> Not sure what you mean. We've had binary wininst distributions
>> for Windows for more than a decade, and egg and msi distributions
>> for 8 years :-)
>>
>> But without access to the VS 2008 compiler that is needed to
>> compile those extensions,
>
> It does seem to be available:
> http://www.microsoft.com/en-us/download/details.aspx?id=13276
>
> What am I missing?

I believe (without testing) that this is just the service pack.
Installing it requires a pre-existing installation of Visual
Studio 2008, or else the installer will refuse to do anything.
Note that it also won't install on top of Visual Studio Express:
you need a licensed copy of Visual Studio to install the service
pack.

Visual Studio 2008 still *is* available to MSDN users. It's
just not available through regular sales channels anymore.

Regards,
Martin

[Steve Dower]

> Antoine Pitrou wrote:
> Le 23/06/2014 15:27, M.-A. Lemburg a écrit :
>>
>> Not sure what you mean. We've had binary wininst distributions for
>> Windows for more than a decade, and egg and msi distributions for 8
>> years :-)
>>
>> But without access to the VS 2008 compiler that is needed to compile
>> those extensions,
>
> It does seem to be available:
> http://www.microsoft.com/en-us/download/details.aspx?id=13276
>
> What am I missing?

That's the service pack, which will only install if you already have VS 2008 installed.

The only official source for VS 2008 these days is through an MSDN subscription, though there's a link floating around that will get to an ISO of VC 2008 Express (but it could disappear or move at any time, which would break the link for good).

It's also possible to get VC9 standalone through the Windows SDK for Windows 7 and .NET 3.5, but this installer has bugs, and distutils does not detect VC installs properly (it only detects Visual Studio and then assumes VC). This is fixable with a few extra files and registry keys, but not simple.

The best answer here is making VC9 available in a long-term, unsupported manner (support is the main MSFT concern - simply throwing products out there and forgetting about them is very counter-cultural). I'm working on getting people to recognize the importance of keeping the old compilers available, but it's an uphill battle. Obviously I'll post here as soon as I have something I can officially share. :)

Cheers,
Steve

> Regards
>
> Antoine.

[Donald Stufft]

On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg <***@egenix.com> wrote:

> On 23.06.2014 18:09, Donald Stufft wrote:
>>
>> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis <***@v.loewis.de> wrote:
>>
>>>>
>>>> * Should we make use of the potential breakage with 2.7.10
>>>> to introduce a new Windows compiler version for Python 2.7 ?
>>>
>>> Assuming it is a good idea to continue producing Windows binaries
>>> for 2.7, I think it would be a bad idea to switch compilers. It will
>>> cause severe breakage of 2.7 installations, much more problematic
>>> than switching to two-digit version numbers.
>>
>> I agree with this, we’ve just finally started getting things to the point where
>> it makes a lot of sense for binary distributions for Windows. Breaking all
>> of them on 2.7 would be very bad.

Err, sorry that “We” was with my pip hat on.

>
> Not sure what you mean. We've had binary wininst distributions
> for Windows for more than a decade, and egg and msi distributions
> for 8 years :-)

Nonetheless, changing the compiler will not only break pip, but every
automated installer tool (easy_install, buildout) that i’m aware of. The
blow back for binary installation is going to be huge I think.

>
> But without access to the VS 2008 compiler that is needed to
> compile those extensions, it will become increasingly difficult
> for package authors to provide such binary packages, so we have to
> ask ourselves:
>
> What's worse: breaking old Windows binaries for Python 2.7
> or not having updated and new Windows binaries for Python 2.7
> at all in a few years ?

At the risk of getting Guido to post his slide again, I still think the
solution to the old compiler is to just roll a 2.8 with minimal changes.
It could even be a good place to move to the ssl backport changes
too since they were the riskier set of changes in PEP466.

But either way, if a compiler does change in a 2.7 release we’ll need
to update a lot of tooling to cope with that, so any plan to do that should
include that and a timeline for adoption of that.

>
> Switching to a newer compiler will make things easier for everyone
> and we'd see more binary packages for Windows again.
>
> Given that Python 2.7 support was extended for another 5 years at the
> recent Python Language Summit to 2020, we have to face this
> breakage sooner or later anyway. Extended support for VS 2008
> will end in 2018 (but then: Python developers usually don't have
> extended support contracts with MS). Service pack support has already
> ended in 2009.
>
> Depending on how you see it, using such an old compiler also
> poses security risks. The last security update for VS 2008 dates
> back to 2011 (http://support.microsoft.com/kb/2538243).
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source (#1, Jun 23 2014)
>>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
> ________________________________________________________________________
> 2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
> 2014-07-02: Python Meeting Duesseldorf ... 9 days to go
> 2014-07-21: EuroPython 2014, Berlin, Germany ... 28 days to go
>
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
> http://www.egenix.com/company/contact/


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Barry Warsaw]

On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote:

>At the risk of getting Guido to post his slide again, I still think the
>solution to the old compiler is to just roll a 2.8 with minimal changes.

No. It's not going to happen, for all the reasons discussed previously.
Python 2.8 is not a solution to anything.

If a new, incompatible compiler suite is required, why can't there just be
multiple Windows downloads on https://www.python.org/download/releases/2.7.7/
? Well, on reason is that you'd have to convince MvL or someone else to take
over the work that would require, but that's gotta be *much* lighter weight
than releasing a Python 2.8.

-Barry

[Donald Stufft]

On Jun 23, 2014, at 4:31 PM, Barry Warsaw <***@python.org> wrote:

> On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote:
>
>> At the risk of getting Guido to post his slide again, I still think the
>> solution to the old compiler is to just roll a 2.8 with minimal changes.
>
> No. It's not going to happen, for all the reasons discussed previously.
> Python 2.8 is not a solution to anything.
>
> If a new, incompatible compiler suite is required, why can't there just be
> multiple Windows downloads on https://www.python.org/download/releases/2.7.7/
> ? Well, on reason is that you'd have to convince MvL or someone else to take
> over the work that would require, but that's gotta be *much* lighter weight
> than releasing a Python 2.8.

As far as I am aware, a 2.7 with a different compiler, even if it’s just an option
is an attractive nuisance. None of the tooling right now differentiates between
binary compatibility by anything other than “CPython 2.7”.

The end result of having a 2.7 which is built with the old compiler, and a 2.7 built
with the new compiler is that you’ll end up with binary distributions which work
sometimes if you’re lucky and the creator of the binary distribution and you
happened to pick the same “variant” of 2.7. Most likely result is all the binary
distributions will *mostly* still depend on using the old compiler because of the
corpus of existing binary packages that depend on that. Which means that the
2.7 with new compiler will exist entirely to act as a footgun to anyone who picks
it and also wants to use binary packages.

>
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-***@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Martin v. Löwis]

Am 23.06.14 22:31, schrieb Barry Warsaw:
> On Jun 23, 2014, at 04:20 PM, Donald Stufft wrote:
>
>> At the risk of getting Guido to post his slide again, I still think the
>> solution to the old compiler is to just roll a 2.8 with minimal changes.
>
> No. It's not going to happen, for all the reasons discussed previously.
> Python 2.8 is not a solution to anything.
>
> If a new, incompatible compiler suite is required, why can't there just be
> multiple Windows downloads on https://www.python.org/download/releases/2.7.7/
> ? Well, on reason is that you'd have to convince MvL or someone else to take
> over the work that would require, but that's gotta be *much* lighter weight
> than releasing a Python 2.8.

See my other message. It's actually heavier, since it requires changes
to distutils, PyPI, pip, buildout etc., all which know how to deal with
Python minor version numbers, but are unaware of the notion of competing
ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit).

Regards,
Martin

[Chris Angelico]

On Tue, Jun 24, 2014 at 6:42 AM, "Martin v. Löwis" <***@v.loewis.de> wrote:
> See my other message. It's actually heavier, since it requires changes
> to distutils, PyPI, pip, buildout etc., all which know how to deal with
> Python minor version numbers, but are unaware of the notion of competing
> ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit).

Is it possible to hijack the "deal with 32-bit vs 64-bit"ness of
things to handle the different compilers? So, for instance, there
might be a "32-bit-NewCompiler" and a "64-bit-NewCompiler", two new
architectures, just as if someone came out with a 128-bit Windows and
built Python 2.7 for it. Would packaging be able to handle that more
easily than a compiler change within the same architecture?

ChrisA

[Donald Stufft]

On Jun 23, 2014, at 5:48 PM, Chris Angelico <***@gmail.com> wrote:

> On Tue, Jun 24, 2014 at 6:42 AM, "Martin v. Löwis" <***@v.loewis.de> wrote:
>> See my other message. It's actually heavier, since it requires changes
>> to distutils, PyPI, pip, buildout etc., all which know how to deal with
>> Python minor version numbers, but are unaware of the notion of competing
>> ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit).
>
> Is it possible to hijack the "deal with 32-bit vs 64-bit"ness of
> things to handle the different compilers? So, for instance, there
> might be a "32-bit-NewCompiler" and a "64-bit-NewCompiler", two new
> architectures, just as if someone came out with a 128-bit Windows and
> built Python 2.7 for it. Would packaging be able to handle that more
> easily than a compiler change within the same architecture?
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-***@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I’m not sure about this FWIW. I’d have to look at the implementations of
stuff to see how they’d cope with a new thing like that.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Martin v. Löwis]

Am 23.06.14 22:31, schrieb Barry Warsaw:
> Well, on reason is that you'd have to convince MvL or someone else to take
> over the work that would require, but that's gotta be *much* lighter weight
> than releasing a Python 2.8.

Just to point this out in a separate message: it will have to be
somebody else. I stepped down as the Windows release maintainer for 2.7
when I learned about the extended life of 2.7, much because I feared
that exactly the thing would happen that we see happen now - and I
didn't want to be the one who would have to deal with it. It is a mess,
and it will get bigger the more time passes.

Playing-the-role-of-Cassandra-ly y'rs,
Martin

[M.-A. Lemburg]

On 23.06.2014 22:20, Donald Stufft wrote:
>
> On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg <***@egenix.com> wrote:
>
>> On 23.06.2014 18:09, Donald Stufft wrote:
>>>
>>> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis <***@v.loewis.de> wrote:
>>>
>>>>>
>>>>> * Should we make use of the potential breakage with 2.7.10
>>>>> to introduce a new Windows compiler version for Python 2.7 ?
>>>>
>>>> Assuming it is a good idea to continue producing Windows binaries
>>>> for 2.7, I think it would be a bad idea to switch compilers. It will
>>>> cause severe breakage of 2.7 installations, much more problematic
>>>> than switching to two-digit version numbers.
>>>
>>> I agree with this, we’ve just finally started getting things to the point where
>>> it makes a lot of sense for binary distributions for Windows. Breaking all
>>> of them on 2.7 would be very bad.
>
> Err, sorry that “We” was with my pip hat on.
>
>>
>> Not sure what you mean. We've had binary wininst distributions
>> for Windows for more than a decade, and egg and msi distributions
>> for 8 years :-)
>
> Nonetheless, changing the compiler will not only break pip, but every
> automated installer tool (easy_install, buildout) that i’m aware of. The
> blow back for binary installation is going to be huge I think.
>
>> But without access to the VS 2008 compiler that is needed to
>> compile those extensions, it will become increasingly difficult
>> for package authors to provide such binary packages, so we have to
>> ask ourselves:
>>
>> What's worse: breaking old Windows binaries for Python 2.7
>> or not having updated and new Windows binaries for Python 2.7
>> at all in a few years ?
>
> At the risk of getting Guido to post his slide again, I still think the
> solution to the old compiler is to just roll a 2.8 with minimal changes.
> It could even be a good place to move to the ssl backport changes
> too since they were the riskier set of changes in PEP466.
>
> But either way, if a compiler does change in a 2.7 release we’ll need
> to update a lot of tooling to cope with that, so any plan to do that should
> include that and a timeline for adoption of that.

Sure, and we'd need to hash out possible solutions to minimize
breakage, but first we'll have to see whether we want to consider
this step or not.


BTW: It's strange that I'm arguing for breaking things. I'm usually
on the other side of such arguments :-)

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Donald Stufft]

On Jun 23, 2014, at 5:07 PM, M.-A. Lemburg <***@egenix.com> wrote:

> On 23.06.2014 22:20, Donald Stufft wrote:
>>
>> On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg <***@egenix.com> wrote:
>>
>>> On 23.06.2014 18:09, Donald Stufft wrote:
>>>>
>>>> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis <***@v.loewis.de> wrote:
>>>>
>>>>>>
>>>>>> * Should we make use of the potential breakage with 2.7.10
>>>>>> to introduce a new Windows compiler version for Python 2.7 ?
>>>>>
>>>>> Assuming it is a good idea to continue producing Windows binaries
>>>>> for 2.7, I think it would be a bad idea to switch compilers. It will
>>>>> cause severe breakage of 2.7 installations, much more problematic
>>>>> than switching to two-digit version numbers.
>>>>
>>>> I agree with this, we’ve just finally started getting things to the point where
>>>> it makes a lot of sense for binary distributions for Windows. Breaking all
>>>> of them on 2.7 would be very bad.
>>
>> Err, sorry that “We” was with my pip hat on.
>>
>>>
>>> Not sure what you mean. We've had binary wininst distributions
>>> for Windows for more than a decade, and egg and msi distributions
>>> for 8 years :-)
>>
>> Nonetheless, changing the compiler will not only break pip, but every
>> automated installer tool (easy_install, buildout) that i’m aware of. The
>> blow back for binary installation is going to be huge I think.
>>
>>> But without access to the VS 2008 compiler that is needed to
>>> compile those extensions, it will become increasingly difficult
>>> for package authors to provide such binary packages, so we have to
>>> ask ourselves:
>>>
>>> What's worse: breaking old Windows binaries for Python 2.7
>>> or not having updated and new Windows binaries for Python 2.7
>>> at all in a few years ?
>>
>> At the risk of getting Guido to post his slide again, I still think the
>> solution to the old compiler is to just roll a 2.8 with minimal changes.
>> It could even be a good place to move to the ssl backport changes
>> too since they were the riskier set of changes in PEP466.
>>
>> But either way, if a compiler does change in a 2.7 release we’ll need
>> to update a lot of tooling to cope with that, so any plan to do that should
>> include that and a timeline for adoption of that.
>
> Sure, and we'd need to hash out possible solutions to minimize
> breakage, but first we'll have to see whether we want to consider
> this step or not.
>
>
> BTW: It's strange that I'm arguing for breaking things. I'm usually
> on the other side of such arguments :-)

Ok. I’m just making sure that any proposal to do this includes how
it plans to work around/minimize that. I agree with Martin (I think)
that trying to fix the entire ecosystem to cope with that change is
going to be far more work than folks realize and that it needs to
be an explicit part of the discussion when deciding how to solve
the problem.

Normally when I see someone suggest that switching compilers
in 2.7.x is likely to be less work than releasing a 2.8 It normally
appears to me they haven’t looked at the impact on the packaging
tooling.

>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source
>>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
> ________________________________________________________________________
>
> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>
>
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
> http://www.egenix.com/company/contact/


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Barry Warsaw]

On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote:

>Normally when I see someone suggest that switching compilers
>in 2.7.x is likely to be less work than releasing a 2.8 It normally
>appears to me they haven’t looked at the impact on the packaging
>tooling.

Just to be clear, releasing a Python 2.8 has enormous impact outside of just
the amount of work to do it. It's an exceedingly bad idea.

-Barry

[Donald Stufft]

On Jun 23, 2014, at 5:22 PM, Barry Warsaw <***@python.org> wrote:

> On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote:
>
>> Normally when I see someone suggest that switching compilers
>> in 2.7.x is likely to be less work than releasing a 2.8 It normally
>> appears to me they haven’t looked at the impact on the packaging
>> tooling.
>
> Just to be clear, releasing a Python 2.8 has enormous impact outside of just
> the amount of work to do it. It's an exceedingly bad idea.

Can you clarify?

Also FWIW I’m not really married to the 2.8 thing, it’s mostly that, on Windows, the X.Y release
prior to the ABI thing in 3.x _was_ the ABI so all the tooling builds on that. So you need to
either

1) Stick with the old Compiler
2) Release 2.8
3) Do all the work to fix all the tooling to cope with the fact that X.Y isn’t the ABI on 2.x anymore

I don’t think a reasonable option is:

4) Just switch compilers and leave it on someone else’s doorsteps to fix the entire packaging
tool chain to cope.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Barry Warsaw]

On Jun 23, 2014, at 05:28 PM, Donald Stufft wrote:

>Can you clarify?

What support guarantees will we make about Python 2.8? Will it be supported
as long as Python 2.7? Longer? Will we now have two long-term support
versions or change *years* of expectations that users should transition off of
Python 2.7 onto Python 2.8? Will all the LTS policies for 2.7 (e.g. PEP 466)
be retired for 2.7 and/or adopted completely into 2.8?

What should Linux distros do? Should they support both 2.7 and 2.8 or begin
the long and potentially arduous process of certifying and transitioning to
2.8? What about other operating systems and package managers, including
commercial redistributors?

Who is going to do the work to make sure patch are forward ported from 2.7 to
2.8? Who is going to be the 2.8 release manager? Will they be strong enough
to reject any and all new features that wouldn't have already made it into 2.7
(due to the already approved, narrow exemptions)? Or will we open the flood
gates to Just One More Little New Feature To Make It Easier To Port to Python
3?

How will we manage the PR surrounding our backtracking on Python 2.8? How
will we manage expectations that it's only released to support a new Windows
compiler? Should non-Windows users just ignore it (much like the Python 1.6
releases were mostly ignored)?

How do you know which tools, workflows, and processes that will break with a
Python 2.8 release? What assumptions about 2.7 being EOL for Python 2 are
baked into the ecosystems outside of core Python?

I could probably go on, but I'm exhausted.

-Barry

[Nick Coghlan]

On 24 Jun 2014 07:29, "Donald Stufft" <***@stufft.io> wrote:
>
>
> On Jun 23, 2014, at 5:22 PM, Barry Warsaw <***@python.org> wrote:
>
> > On Jun 23, 2014, at 05:15 PM, Donald Stufft wrote:
> >
> >> Normally when I see someone suggest that switching compilers
> >> in 2.7.x is likely to be less work than releasing a 2.8 It normally
> >> appears to me they haven’t looked at the impact on the packaging
> >> tooling.
> >
> > Just to be clear, releasing a Python 2.8 has enormous impact outside of
just
> > the amount of work to do it. It's an exceedingly bad idea.
>
> Can you clarify?
>
> Also FWIW I’m not really married to the 2.8 thing, it’s mostly that, on
Windows, the X.Y release
> prior to the ABI thing in 3.x _was_ the ABI so all the tooling builds on
that. So you need to
> either
>
> 1) Stick with the old Compiler

This is what we're going with. Steve is working on making that more
manageable from the Visual Studio side, and there are some folks in the
numeric/scientific community looking at improving the usability of the
MinGW toolchain for the purpose of building Python 2.7 C extensions.

> 2) Release 2.8

Impractical for the various reasons Barry listed.

> 3) Do all the work to fix all the tooling to cope with the fact that X.Y
isn’t the ABI on 2.x anymore

Impractical for the various reasons you listed.

> I don’t think a reasonable option is:
>
> 4) Just switch compilers and leave it on someone else’s doorsteps to fix
the entire packaging
> tool chain to cope.

Agreed. We discussed this option in detail when the Stackless folks asked
about it a while ago, and the conclusion was that the risk of obscure
breakage was just too high.

Cheers,
Nick.

>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-***@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>

[Chris Kaynor]

Not being a Python developer, I normally just lurk on Py-Dev, but I figured
I'd throw this out there for this thread:

Recent version of Maya embed Python 2.x, and the newer version of Maya (I
believe 2012 was the first version) embeds a Python 2.7 compiled with VS
2010. From my experience, most C extensions work across compiler versions,
however when they don't, it's generally a fairly difficult to debug issue -
at least unless you know what to look for in the call stacks, and have
access to the symbol files.



Chris


On Mon, Jun 23, 2014 at 2:07 PM, M.-A. Lemburg <***@egenix.com> wrote:

> On 23.06.2014 22:20, Donald Stufft wrote:
> >
> > On Jun 23, 2014, at 3:27 PM, M.-A. Lemburg <***@egenix.com> wrote:
> >
> >> On 23.06.2014 18:09, Donald Stufft wrote:
> >>>
> >>> On Jun 23, 2014, at 2:09 AM, Martin v. Löwis <***@v.loewis.de>
> wrote:
> >>>
> >>>>>
> >>>>> * Should we make use of the potential breakage with 2.7.10
> >>>>> to introduce a new Windows compiler version for Python 2.7 ?
> >>>>
> >>>> Assuming it is a good idea to continue producing Windows binaries
> >>>> for 2.7, I think it would be a bad idea to switch compilers. It will
> >>>> cause severe breakage of 2.7 installations, much more problematic
> >>>> than switching to two-digit version numbers.
> >>>
> >>> I agree with this, we’ve just finally started getting things to the
> point where
> >>> it makes a lot of sense for binary distributions for Windows. Breaking
> all
> >>> of them on 2.7 would be very bad.
> >
> > Err, sorry that “We” was with my pip hat on.
> >
> >>
> >> Not sure what you mean. We've had binary wininst distributions
> >> for Windows for more than a decade, and egg and msi distributions
> >> for 8 years :-)
> >
> > Nonetheless, changing the compiler will not only break pip, but every
> > automated installer tool (easy_install, buildout) that i’m aware of. The
> > blow back for binary installation is going to be huge I think.
> >
> >> But without access to the VS 2008 compiler that is needed to
> >> compile those extensions, it will become increasingly difficult
> >> for package authors to provide such binary packages, so we have to
> >> ask ourselves:
> >>
> >> What's worse: breaking old Windows binaries for Python 2.7
> >> or not having updated and new Windows binaries for Python 2.7
> >> at all in a few years ?
> >
> > At the risk of getting Guido to post his slide again, I still think the
> > solution to the old compiler is to just roll a 2.8 with minimal changes.
> > It could even be a good place to move to the ssl backport changes
> > too since they were the riskier set of changes in PEP466.
> >
> > But either way, if a compiler does change in a 2.7 release we’ll need
> > to update a lot of tooling to cope with that, so any plan to do that
> should
> > include that and a timeline for adoption of that.
>
> Sure, and we'd need to hash out possible solutions to minimize
> breakage, but first we'll have to see whether we want to consider
> this step or not.
>
>
> BTW: It's strange that I'm arguing for breaking things. I'm usually
> on the other side of such arguments :-)
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source
> >>> Python/Zope Consulting and Support ... http://www.egenix.com/
> >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
> ________________________________________________________________________
>
> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>
>
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> Registered at Amtsgericht Duesseldorf: HRB 46611
> http://www.egenix.com/company/contact/
> _______________________________________________
> Python-Dev mailing list
> Python-***@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ckaynor%40zindagigames.com
>