PEP 3121, 384 Refactoring Issues

[Mark Lawrence]

I'm just curious as to why there are 54 open issues after both of these
PEPs have been accepted and 384 is listed as finished. Did we hit some
unforeseen technical problem which stalled development?

For these and any other open issues if you need some Windows testing
doing please feel free to put me on the nosy list and ask for a test run.

[Brett Cannon]

[for those that don't know, 3121 is extension module inti/finalization and
384 is the stable ABI]

Post by Mark Lawrence
I'm just curious as to why there are 54 open issues after both of these
PEPs have been accepted and 384 is listed as finished. Did we hit some
unforeseen technical problem which stalled development?

No, the PEPs were fine and were accepted properly. A huge portion of the
open issues are from Robin Schreiber who as part of GSoC 2012 --
https://www.google-melange.com/gsoc/project/details/google/gsoc2012/robin_hood/5668600916475904
-- went through and updated the stdlib to follow the new practices
introduced in the two PEPs. Not sure if there was some policy decision made
that updating the code wasn't worth it or people simply didn't get around
to applying the patches.

-Brett

Post by Mark Lawrence
For these and any other open issues if you need some Windows testing
doing please feel free to put me on the nosy list and ask for a test run.
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
brett%40python.org

[Guido van Rossum]

I don't know the details, but I suspect that was the result of my general
guideline "don't start projects cleaning up lots of stdlib code just to
satisfy some new style rule or just to use a new API" -- which came from
hard-won experience where such a cleanup project introduced some new bugs
that weren't found by review nor by tests. Though that was admittedly a
long time. Still, such a project can really sap reviewer resources for
relatively little benefit.

Post by Brett Cannon
[for those that don't know, 3121 is extension module inti/finalization and
384 is the stable ABI]

Post by Mark Lawrence
I'm just curious as to why there are 54 open issues after both of these
PEPs have been accepted and 384 is listed as finished. Did we hit some
unforeseen technical problem which stalled development?

No, the PEPs were fine and were accepted properly. A huge portion of the
open issues are from Robin Schreiber who as part of GSoC 2012 --
https://www.google-melange.com/gsoc/project/details/google/gsoc2012/robin_hood/5668600916475904
-- went through and updated the stdlib to follow the new practices
introduced in the two PEPs. Not sure if there was some policy decision made
that updating the code wasn't worth it or people simply didn't get around
to applying the patches.
-Brett

Post by Mark Lawrence
For these and any other open issues if you need some Windows testing
doing please feel free to put me on the nosy list and ask for a test run.
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
---
This email is free from viruses and malware because avast! Antivirus
protection is active.
http://www.avast.com
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
brett%40python.org

_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/options/python-dev/guido%40python.org

[Stefan Krah]

No, the PEPs were fine and were accepted properly. A huge portion of the open
issues are from Robin Schreiber who as part of GSoC 2012 -- https://
www.google-melange.com/gsoc/project/details/google/gsoc2012/robin_hood/
5668600916475904 -- went through and updated the stdlib to follow the new
practices introduced in the two PEPs. Not sure if there was some policy
decision made that updating the code wasn't worth it or people simply didn't
get around to applying the patches.

Due to the frequent state lookups there is a performance problem though,
which is quite significant for _decimal. Otherwise I think I would have
implemented the changes already.

http://bugs.python.org/issue15722


I think for speed sensitive applications it may be an idea to create
a new C function (METH_STATE flag) which gets the state passed in by
ceval.

Other than that, looking up the state inside the module but cache it (like
it's done for the _decimal context) also has reasonable performance.



Also I hit the same issues that Eli mentioned here a while ago:

https://mail.python.org/pipermail/python-dev/2013-August/127862.html



Stefan Krah

[Alexander Belopolsky]

Post by Mark Lawrence
I'm just curious as to why there are 54 open issues after both of these
PEPs have been accepted and 384 is listed as finished. Did we hit some
unforeseen technical problem which stalled development?

I tried to bring some sanity to that effort by opening a "meta issue":

http://bugs.python.org/issue15787

My enthusiasm, however, vanished after I reviewed the refactoring for the
datetime module:

http://bugs.python.org/issue15390

My main objections are to following PEP 384
<http://www.python.org/dev/peps/pep-0384/> (Stable ABI) within stdlib
modules. I see little benefit for the stdlib (which is shipped fresh with
every new version of Python) from following those guidelines.

[Ethan Furman]

Post by Alexander Belopolsky

Post by Mark Lawrence
I'm just curious as to why there are 54 open issues after both of
these PEPs have been accepted and 384 is listed as finished. Did
we hit some unforeseen technical problem which stalled development?

http://bugs.python.org/issue15787
http://bugs.python.org/issue15390
My main objections are to following PEP 384 <http://www.python.org/dev/peps/pep-0384/> (Stable ABI) within stdlib
modules. I see little benefit for the stdlib (which is shipped fresh with every new version of Python) from following
those guidelines.

If we aren't going to implement the changes (and I agree there's little value for the stdlib to do so), let's mark the
issues as "won't fix" and close them.

And thanks, Mark, for bringing it up.

--
~Ethan~

[Nick Coghlan]

Post by Alexander Belopolsky

Post by Mark Lawrence
I'm just curious as to why there are 54 open issues after both of these

PEPs have been accepted and 384 is listed as finished. Did we hit some
unforeseen technical problem which stalled development?

Post by Alexander Belopolsky
http://bugs.python.org/issue15787
My enthusiasm, however, vanished after I reviewed the refactoring for the
http://bugs.python.org/issue15390
My main objections are to following PEP 384 (Stable ABI) within stdlib

modules. I see little benefit for the stdlib (which is shipped fresh with
every new version of Python) from following those guidelines.

The main downside of "do as we say, not as we do" in this case is that we
miss out on the feedback loop of what the stable ABI is like to *use*. For
example, the docs problem, where it's hard to tell whether an API is part
of the stable ABI or not, or the performance problem Stefan mentions.

Using the stable ABI for standard library extensions also serves to
decouple them further from the internal details of the CPython runtime,
making it more likely they will be able to run correctly on alternative
interpreters (since emulating or otherwise supporting the limited API is
easier than supporting the whole thing).

Cheers,
Nick.

Post by Alexander Belopolsky
_______________________________________________
Python-Dev mailing list
https://mail.python.org/mailman/listinfo/python-dev

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

[Alexander Belopolsky]

Post by Nick Coghlan
The main downside of "do as we say, not as we do" in this case is that we
miss out on the feedback loop of what the stable ABI is like to *use*.

I good start for improving the situation would be to convert the extension
module templates that we ship with the Python source:

http://bugs.python.org/issue15848 (xxsubtype module)
http://bugs.python.org/issue15849 (xx module)

[Martin v. Löwis]

Post by Nick Coghlan
Using the stable ABI for standard library extensions also serves to
decouple them further from the internal details of the CPython runtime,
making it more likely they will be able to run correctly on alternative
interpreters (since emulating or otherwise supporting the limited API is
easier than supporting the whole thing).

There are two features to be gained for the standard library from that

A. with proper module shutdown support, it will be possible to release
objects that are currently held in C global/static variables, as the
C global variables will go away. This, in turn, is a step forward in
the desire to allow for proper leak-free interpreter shutdown, and
in the desire to base interpreter shutdown on GC.

B. with proper use of heap types (instead of the static type objects),
support for the multiple-interpreter feature will be improved, since
type objects will be per-interpreter, instead of being global. This,
in turn, is desirable since otherwise state changes can leak from
one interpreter to the other.

So I still maintain that the change is desirable even for the standard
library.

Regards,
Martin

[Brett Cannon]

Post by Martin v. Löwis

Post by Nick Coghlan
Using the stable ABI for standard library extensions also serves to
decouple them further from the internal details of the CPython runtime,
making it more likely they will be able to run correctly on alternative
interpreters (since emulating or otherwise supporting the limited API is
easier than supporting the whole thing).

There are two features to be gained for the standard library from that
A. with proper module shutdown support, it will be possible to release
objects that are currently held in C global/static variables, as the
C global variables will go away. This, in turn, is a step forward in
the desire to allow for proper leak-free interpreter shutdown, and
in the desire to base interpreter shutdown on GC.
B. with proper use of heap types (instead of the static type objects),
support for the multiple-interpreter feature will be improved, since
type objects will be per-interpreter, instead of being global. This,
in turn, is desirable since otherwise state changes can leak from
one interpreter to the other.
So I still maintain that the change is desirable even for the standard
library.

I agree for PEP 3121 which is the initialization/finalization work. The
stable ABi is not necessary. So maybe we should re-examine the patches and
accept the bits that clean up init/finalization and leave out any
ABi-related changes.

[Alexander Belopolsky]

So maybe we should re-examine the patches and accept the bits that clean
up init/finalization and leave out any ABI-related changes.

This is precisely what I suggested two years ago.

http://bugs.python.org/issue15390#msg170249

I am not against ABI-related changes in principle, but I think these
changes should be carefully considered on a case by case basis and not
applied wholesale.

[Nick Coghlan]

Post by Brett Cannon
I agree for PEP 3121 which is the initialization/finalization work. The

stable ABi is not necessary. So maybe we should re-examine the patches and
accept the bits that clean up init/finalization and leave out any
ABi-related changes.

Martin's right about improving the subinterpreter support - every type
declaration we move from a static struct to the dynamic type creation API
is one that isn't shared between subinterpreters any more.

That argument is potentially valid even for *builtin* modules and types,
not just those in extension modules.

Cheers,
Nick.