Criticism of execfile() removal in Python3

[Paul Sokolovsky]

Hello,

I was pleasantly surprised with the response to recent post about
MicroPython implementation details
(https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I
hope that discussion means that posts about alternative implementations
are not unwelcome here, so I would like to bring up another (of many)
issues we faced while implementing MicroPython.

execfile() builtin function was removed in 3.0. This brings few
problems:

1. It hampers interactive mode - instead of short and easy to type
execfile("file.py") one needs to use exec(open("file.py").read()). I'm
sure that's not going to bother a lot of people - after all, the
easiest way to execute a Python file is to drop back to shell and
restart python with file name, using all wonders of tab completion. But
now imagine that Python interpreter runs on bare hardware, and its REPL
is the only shell. That's exactly what we have with MicroPython's
Cortex-M port. But it's not really MicroPython-specific, there's
CPython port to baremetal either - http://www.pycorn.org/ .

2. Ok, assuming that exec(open().read()) idiom is still a way to go,
there's a problem - it requires to load entire file to memory. But
there can be not enough memory. Consider 1Mb file with 900Kb comments
(autogenerated, for example). execfile() could easily parse it, using
small buffer. But exec() requires to slurp entire file into memory, and
1Mb is much more than heap sizes that we target.


Comments, suggestions? Just to set a productive direction, please
kindly don't consider the problems above as MicroPython's. I very much
liked how last discussion went: I was pointed that
https://docs.python.org/3/reference/index.html is not really a CPython
reference, it's a *Python* reference, and there were even motion to
clarify in it some points which came out from MicroPython discussion.
So, what about https://docs.python.org/3/library/index.html - is it
CPython, or Python standard library specification? Assuming the latter,
what we have is that, by removal of previously available feature,
*Python* became less friendly for interactive usage and less scalable.


Thanks,
Paul mailto:***@gmail.com

[Steven D'Aprano]

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:

> execfile() builtin function was removed in 3.0. This brings few
> problems:
>
> 1. It hampers interactive mode - instead of short and easy to type
> execfile("file.py") one needs to use exec(open("file.py").read()).

If the amount of typing is the problem, that's easy to solve:

# do this once
def execfile(name):
exec(open("file.py").read())

Another possibility is:

os.system("python file.py")


> 2. Ok, assuming that exec(open().read()) idiom is still a way to go,
> there's a problem - it requires to load entire file to memory. But
> there can be not enough memory. Consider 1Mb file with 900Kb comments
> (autogenerated, for example). execfile() could easily parse it, using
> small buffer. But exec() requires to slurp entire file into memory, and
> 1Mb is much more than heap sizes that we target.

There's nothing stopping alternative implementations having their own
implementation-specific standard library modules.

***@orac:/home/s$ jython
Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19)
[OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27
Type "help", "copyright", "credits" or "license" for more information.
>>> import java
>>>


So you could do this:

from upy import execfile
execfile("file.py")

So long as you make it clear that this is a platform specific module,
and don't advertise it as a language feature, I see no reason why you
cannot do that.



--
Steven

[Terry Reedy]

On 6/9/2014 11:03 PM, Steven D'Aprano wrote:
> On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:
>
>> execfile() builtin function was removed in 3.0.

Because it was hardly ever used. For short bits of code, it is usually
inferior to exec with a string in the file. For substantial bits of
code, it is generally inferior to 'from file import *' and does not have
the option of other forms of import. For startup code that you want
every session, it is inferior to PYTHONSTARTUP or custom site module.

>> This brings few problems:
>> 1. It hampers interactive mode - instead of short and easy to type
>> execfile("file.py") one needs to use exec(open("file.py").read())

> If the amount of typing is the problem, that's easy to solve:
>
> # do this once
> def execfile(name):
> exec(open("file.py").read())
>
> Another possibility is:
>
> os.system("python file.py")
>
>
>> 2. Ok, assuming that exec(open().read()) idiom is still a way to go,
>> there's a problem - it requires to load entire file to memory. But
>> there can be not enough memory. Consider 1Mb file with 900Kb comments
>> (autogenerated, for example). execfile() could easily parse it, using
>> small buffer. But exec() requires to slurp entire file into memory, and
>> 1Mb is much more than heap sizes that we target.

Execfile could slurp the whole file into memory too. Next parse the
entire file. Then execute the entire bytecode. Finally toss the bytecode
so that the file has to be reparsed next time it is used.

> There's nothing stopping alternative implementations having their own
> implementation-specific standard library modules.
...
> So you could do this:
>
> from upy import execfile
> execfile("file.py")
>
> So long as you make it clear that this is a platform specific module,
> and don't advertise it as a language feature, I see no reason why you
> cannot do that.

If you want execfile as a substitute for 'python -i file' on the
unavailable command console, you should have the option to restore
globals to initial condition. Something like (untested)

# startup entries in globals in CPython 3.4.1
startnames={'__spec__', '__name__', '__builtins__',
'__doc__', '__loader__', '__package__'}

def execfile(file, encoding='utf-8', restart=<True/False>):
glodict = globals()
code = open(file, 'r', encoding=encoding)
# don't restart if the file does not open
if restart:
for name in list(glodict):
if name not in startnames:
del glodict(name)
for statement in statements(code): # statements is statement iterator
exec(statement,...globals=glodict, locals=glodict)

--
Terry Jan Reedy

[Jim Baker]

On Mon, Jun 9, 2014 at 9:03 PM, Steven D'Aprano <***@pearwood.info> wrote:

> ...
> There's nothing stopping alternative implementations having their own
> implementation-specific standard library modules.
>
> ***@orac:/home/s$ jython
> Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19)
> [OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import java
> >>>
>
>
Small nit: Jython does implement a number of implementation-specific
modules in its version of the standard library; jarray comes to mind, which
is mostly but not completely superseded by the standard array module.
However, the java package namespace is not part of the standard library,
it's part of the standard Java ecosystem and it's due to a builtin import
hook:

Jython 2.7b3+ (default:6cee6fef06f0, Jun 9 2014, 22:29:14)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_60
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/jbaker/jythondev/jython27/dist/Lib', '__classpath__',
'__pyclasspath__/', '/home/jbaker/.local/lib/jython2.7/site-packages',
'/home/jbaker/jythondev/jython27/dist/Lib/site-packages']

The entry __classpath__ means search CLASSPATH for Java packages; this
includes the Java runtime, rt.jar, from which you get package namespaces as
java.*, javax.*, sun.*, etc.

Another behavior that you get for free in Jython is being able to also
import the org.python.* namespace, which is Jython's own runtime. Some of
the implementations of standard library modules, such as threading, take
advantage of this support.

- Jim

[Paul Sokolovsky]

Hello,

On Tue, 10 Jun 2014 13:03:03 +1000
Steven D'Aprano <***@pearwood.info> wrote:

> On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:
>
> > execfile() builtin function was removed in 3.0. This brings few
> > problems:
> >
> > 1. It hampers interactive mode - instead of short and easy to type
> > execfile("file.py") one needs to use exec(open("file.py").read()).
>
> If the amount of typing is the problem, that's easy to solve:
>
> # do this once
> def execfile(name):
> exec(open("file.py").read())

So, you here propose to workaround removal of core language feature
either a) on end user side, or b) on "system integrator" side. But such
solution is based on big number of assumptions, like: user wants to
workaround that at all (hint: they don't, they just want to use it);
you say "do this once", but actually it's "do it in each interactive
session again and again", and user may not have knowledge to "do it
once" instead; that if system integrator does that, the the function is
called "execfile": if system integrator didn't have enough Python
experience, and read only Python3 spec, they might call it something
else, and yet users with a bit of Python experience will expect it be
called exactly "execfile" and not anything else.

>
> Another possibility is:
>
> os.system("python file.py")
>
>
> > 2. Ok, assuming that exec(open().read()) idiom is still a way to go,
> > there's a problem - it requires to load entire file to memory. But
> > there can be not enough memory. Consider 1Mb file with 900Kb
> > comments (autogenerated, for example). execfile() could easily
> > parse it, using small buffer. But exec() requires to slurp entire
> > file into memory, and 1Mb is much more than heap sizes that we
> > target.
>
> There's nothing stopping alternative implementations having their own
> implementation-specific standard library modules.

And here you propose to workaround it on particular implementation's
level. But in my original mail, in excerpt that you removed, I kindly
asked to skip obvious suggestions (like that particular implementation
can do anything it wants).

I don't see how working around the issue on user, particular
distribution, or particular implementation level help *Python* language
in general, and *Python community* in general. So, any bright ideas how
to workaround the issue of execfile() removal on *language level*?

[]
> So you could do this:
>
> from upy import execfile
> execfile("file.py")
>
> So long as you make it clear that this is a platform specific module,
> and don't advertise it as a language feature, I see no reason why you
> cannot do that.

The case we discuss is clearly different. It's not about "platform
specific module", it's about functionality which was in Python all the
time, and was suddenly removed in Python3, for not fully clear, or
alternatively, not severe enough, reasons. If some implementation is to
re-add it, the description like above seems the most truthful way to
represent that function.


--
Best regards,
Paul mailto:***@gmail.com

[Skip Montanaro]

> you say "do this once", but actually it's "do it in each interactive
> session again and again", ...

That's what your Python startup file is for. I have been running with
several tweaked builtin functions for years. Never have to consciously load
them. If I wanted execfile badly enough, I'd define it there.

I don't think I've used execfile more than a handful of times in the 20-odd
years I've been using Python. Perhaps our personal approaches to executing
code at the interpreter prompt are radically different, but I think if the
lack of execfile is such a big deal for you, you might want to check around
to see how other people use interactive mode.

Skip

[Chris Barker]

On Sat, Jun 14, 2014 at 1:11 PM, Paul Sokolovsky <***@gmail.com> wrote:


> > > 1. It hampers interactive mode - instead of short and easy to type
> > > execfile("file.py") one needs to use exec(open("file.py").read()).

>
> > If the amount of typing is the problem, that's easy to solve:
> >
> > # do this once
> > def execfile(name):
> > exec(open("file.py").read())
>

FWIW, when I started using python (15?) years ago -- the first thing I
looked for was a way to "just run a file", at the interactive prompt, like
I had in MATLAB. I found and used execfile().

However, it wasn't long before I discovered that excecfile() was really
kind of a pain, you've got namespaces, and all sorts of stuff that made it
often not work like I wanted, and was a pain to type. I stopped using it
all together

More recently, I discovered iPython and its "run" function -- very nice, it
does the obvious stuff for you the way you'd expect.

My conclusions:

1) runfile() is not really very usefull, it's fine to hve removed it.

2) the built-in interactive python interpreter is really pretty lame. If
you want a good interactive experience, you need something more anyway
(iPython, for instance) -- putting execfile() back is only one tiny
improvement that's not worth it.


So if this is about micropython -- I think it would serve the project very
well to have a micropython-specific interactive mode. iPython is fabulous,
but though I imagine too heavy weight. But perhaps you could borrow some
things from it -- like "run" , for example.

-Chris


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Ethan Furman]

On 06/16/2014 10:40 AM, Chris Barker wrote:
>
> My conclusions:
>
> 1) runfile() is not really very usefull, it's fine to hve removed it.

s/runfile/execfile

--
~Ethan~

[Nick Coghlan]

On 17 Jun 2014 03:42, "Chris Barker" <***@noaa.gov> wrote:
>
> On Sat, Jun 14, 2014 at 1:11 PM, Paul Sokolovsky <***@gmail.com>
wrote:
>
>>
>> > > 1. It hampers interactive mode - instead of short and easy to type
>> > > execfile("file.py") one needs to use exec(open("file.py").read()).
>>
>> >
>> > If the amount of typing is the problem, that's easy to solve:
>> >
>> > # do this once
>> > def execfile(name):
>> > exec(open("file.py").read())
>
>
> FWIW, when I started using python (15?) years ago -- the first thing I
looked for was a way to "just run a file", at the interactive prompt, like
I had in MATLAB. I found and used execfile().

Yes, if people are looking for a MATLAB replacement, they want IPython
rather than the default REPL.

The default one is deliberately minimal, IPython is designed to be a
comprehensive numeric and scientific workspace.

Cheers,
Nick.

[Chris Barker]

On Mon, Jun 16, 2014 at 3:39 PM, Nick Coghlan <***@gmail.com> wrote:

> > FWIW, when I started using python (15?) years ago -- the first thing I
> looked for was a way to "just run a file", at the interactive prompt, like
> I had in MATLAB. I found and used execfile().
>
> Yes, if people are looking for a MATLAB replacement, they want IPython
> rather than the default REPL.
>
I didn't meant o distract the conversation here -- what I meant was that
even before iPython existed, I still dropped using execfile("") it was
hardly ever the right thing.

And for the micropython example, I'm proposing that a micropython
interactive environment would be a really nice thing to build -- and worth
doing, even if execfile() was still there.

By the way: iPython, while coming from, and heavily used by, the
scientific/numeric computing community, is a great tool for all sorts of
other python development as well. But probably too heavyweight for
micropython.

-CHB




--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Nick Coghlan]

On 18 Jun 2014 01:59, "Chris Barker" <***@noaa.gov> wrote:
>
> By the way: iPython, while coming from, and heavily used by, the
scientific/numeric computing community, is a great tool for all sorts of
other python development as well. But probably too heavyweight for
micropython.

(we're drifting off topic, so this will be my last addition to this
subthread)

Yes, as great as IPython is, when it's considered out of scope for the
standard installers, it's unlikely to be a good fit for a version of Python
aimed at running *on* a microcontroller. Running on a Raspberry Pi or
remote PC and *talking* to an associated microcontroller is a different
story, though.

Cheers,
Nick.

>
> -CHB
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R [phone] voice
> 7600 Sand Point Way NE [phone] fax
> Seattle, WA 98115 [phone] main reception
>
> ***@noaa.gov

[Nick Coghlan]

On 10 June 2014 12:23, Paul Sokolovsky <***@gmail.com> wrote:
> 1. It hampers interactive mode - instead of short and easy to type
> execfile("file.py") one needs to use exec(open("file.py").read()). I'm
> sure that's not going to bother a lot of people - after all, the
> easiest way to execute a Python file is to drop back to shell and
> restart python with file name, using all wonders of tab completion. But
> now imagine that Python interpreter runs on bare hardware, and its REPL
> is the only shell. That's exactly what we have with MicroPython's
> Cortex-M port. But it's not really MicroPython-specific, there's
> CPython port to baremetal either - http://www.pycorn.org/ .

https://docs.python.org/3/library/runpy.html#runpy.run_path

import runpy
file_globals = runpy.run_path("file.py")

The standard implementation of run_path reads the whole file into
memory, but MicroPython would be free to optimise that and do
statement by statement execution instead (while that will pose some
challenges in terms of handling encoding cookies, future imports, etc
correctly, it's certainly feasible).

Cheers,
Nick.

--
Nick Coghlan | ***@gmail.com | Brisbane, Australia

[Paul Moore]

On 10 June 2014 08:36, Nick Coghlan <***@gmail.com> wrote:
> The standard implementation of run_path reads the whole file into
> memory, but MicroPython would be free to optimise that and do
> statement by statement execution instead (while that will pose some
> challenges in terms of handling encoding cookies, future imports, etc
> correctly, it's certainly feasible).

... and if they did optimise that way, I would imagine that the patch
would be a useful contribution back to the core Python stdlib, rather
than remaining a MicroPython-specific optimisation.

Paul

[Nick Coghlan]

On 10 Jun 2014 18:41, "Paul Moore" <***@gmail.com> wrote:
>
> On 10 June 2014 08:36, Nick Coghlan <***@gmail.com> wrote:
> > The standard implementation of run_path reads the whole file into
> > memory, but MicroPython would be free to optimise that and do
> > statement by statement execution instead (while that will pose some
> > challenges in terms of handling encoding cookies, future imports, etc
> > correctly, it's certainly feasible).
>
> ... and if they did optimise that way, I would imagine that the patch
> would be a useful contribution back to the core Python stdlib, rather
> than remaining a MicroPython-specific optimisation.

I believe it's a space/speed trade-off, so I'd be surprised if it made
sense for CPython in general. There are also some behavioural differences
when it comes to handling syntax errors.

Now that I think about the idea a bit more, if the MicroPython folks can
get a low memory usage incremental file execution model working, the
semantic differences mean it would likely make the most sense as a separate
API in runpy, rather than as an implicit change to run_path.

Cheers,
Nick.

>
> Paul

[R. David Murray]

On Tue, 10 Jun 2014 19:07:40 +1000, Nick Coghlan <***@gmail.com> wrote:
> On 10 Jun 2014 18:41, "Paul Moore" <***@gmail.com> wrote:
> >
> > On 10 June 2014 08:36, Nick Coghlan <***@gmail.com> wrote:
> > > The standard implementation of run_path reads the whole file into
> > > memory, but MicroPython would be free to optimise that and do
> > > statement by statement execution instead (while that will pose some
> > > challenges in terms of handling encoding cookies, future imports, etc
> > > correctly, it's certainly feasible).
> >
> > ... and if they did optimise that way, I would imagine that the patch
> > would be a useful contribution back to the core Python stdlib, rather
> > than remaining a MicroPython-specific optimisation.
>
> I believe it's a space/speed trade-off, so I'd be surprised if it made
> sense for CPython in general. There are also some behavioural differences
> when it comes to handling syntax errors.
>
> Now that I think about the idea a bit more, if the MicroPython folks can
> get a low memory usage incremental file execution model working, the
> semantic differences mean it would likely make the most sense as a separate
> API in runpy, rather than as an implicit change to run_path.

If it is a separate API, it seems like there's no reason it couldn't be
contributed back to CPython. There might be other contexts in which
low memory would be the right tradeoff. Although, if key bits end
up working at the C level, "contributing back" might require writing
separate C for CPython, so that might not happen.

--David

[Nick Coghlan]

On 10 June 2014 23:05, R. David Murray <***@bitdance.com> wrote:
> On Tue, 10 Jun 2014 19:07:40 +1000, Nick Coghlan <***@gmail.com> wrote:
>> I believe it's a space/speed trade-off, so I'd be surprised if it made
>> sense for CPython in general. There are also some behavioural differences
>> when it comes to handling syntax errors.
>>
>> Now that I think about the idea a bit more, if the MicroPython folks can
>> get a low memory usage incremental file execution model working, the
>> semantic differences mean it would likely make the most sense as a separate
>> API in runpy, rather than as an implicit change to run_path.
>
> If it is a separate API, it seems like there's no reason it couldn't be
> contributed back to CPython. There might be other contexts in which
> low memory would be the right tradeoff. Although, if key bits end
> up working at the C level, "contributing back" might require writing
> separate C for CPython, so that might not happen.

Yeah, as a separate API it could make sense in CPython - I just didn't
go back and revise the first paragraph after writing the second one :)

Cheers,
Nick.

--
Nick Coghlan | ***@gmail.com | Brisbane, Australia

[Paul Sokolovsky]

Hello,

On Tue, 10 Jun 2014 17:36:02 +1000
Nick Coghlan <***@gmail.com> wrote:

> On 10 June 2014 12:23, Paul Sokolovsky <***@gmail.com> wrote:
> > 1. It hampers interactive mode - instead of short and easy to type
> > execfile("file.py") one needs to use exec(open("file.py").read()).
> > I'm sure that's not going to bother a lot of people - after all, the
> > easiest way to execute a Python file is to drop back to shell and
> > restart python with file name, using all wonders of tab completion.
> > But now imagine that Python interpreter runs on bare hardware, and
> > its REPL is the only shell. That's exactly what we have with
> > MicroPython's Cortex-M port. But it's not really
> > MicroPython-specific, there's CPython port to baremetal either -
> > http://www.pycorn.org/ .
>
> https://docs.python.org/3/library/runpy.html#runpy.run_path
>
> import runpy
> file_globals = runpy.run_path("file.py")

Thanks, it's the most productive response surely. So, at least there's
alternative to removed execfile(). Unfortunately, I don't think it's
good alternative to execfile() in all respects. It clearly provides API
for that functionality, but is that solution of least surprise and is
it actually known by users at all (to be useful for them)?

Googling for "execfile python 3", top 3 hits I see are stackoverflow
questions, *none* of which mentions runpy. So, people either don't
consider it viable alternative to execfile, or don't know about it at
all (my guess it's the latter).


Like with previous discussion, its meaning goes beyond just Python
realm - there's competition all around. And internets bring funny
examples, like for example http://www.red-lang.org/p/contributions.html
(scroll down to diagram, or here's direct link:
http://3.bp.blogspot.com/-xhOP35Dm99w/UuXFKgY2dlI/AAAAAAAAAGA/YQu98_pPDjw/s1600/reichart-abstraction-diagram.png)
So, didn't you know that Ruby can be used for OS-level development, and
Python can't? Or that JavaScript DSL capabilities are better than
Python's (that's taking into account that JavaScript DSL capabilities
are represented by JSON, whose creators were so arrogant as to disallow
even usage of comments in it).

So, now suppose there's a discussion of how good different languages are
for interactive usage (out of the box apparently). It would be a little
hard to defend claim that Python is *excellent* interactive language,
if its latest series got -1 on that scale, by removing feature which
may be indispensable at times. Knowing that, one subconsciously may
start to wonder if Ruby or JavaScript are doing it (in wide sense)
better than Python.


--
Best regards,
Paul mailto:***@gmail.com

[Nick Coghlan]

On 15 Jun 2014 06:52, "Paul Sokolovsky" <***@gmail.com> wrote:
>
> Hello,
>
> On Tue, 10 Jun 2014 17:36:02 +1000
> Nick Coghlan <***@gmail.com> wrote:
>
> > On 10 June 2014 12:23, Paul Sokolovsky <***@gmail.com> wrote:
> > > 1. It hampers interactive mode - instead of short and easy to type
> > > execfile("file.py") one needs to use exec(open("file.py").read()).
> > > I'm sure that's not going to bother a lot of people - after all, the
> > > easiest way to execute a Python file is to drop back to shell and
> > > restart python with file name, using all wonders of tab completion.
> > > But now imagine that Python interpreter runs on bare hardware, and
> > > its REPL is the only shell. That's exactly what we have with
> > > MicroPython's Cortex-M port. But it's not really
> > > MicroPython-specific, there's CPython port to baremetal either -
> > > http://www.pycorn.org/ .
> >
> > https://docs.python.org/3/library/runpy.html#runpy.run_path
> >
> > import runpy
> > file_globals = runpy.run_path("file.py")
>
> Thanks, it's the most productive response surely. So, at least there's
> alternative to removed execfile(). Unfortunately, I don't think it's
> good alternative to execfile() in all respects. It clearly provides API
> for that functionality, but is that solution of least surprise and is
> it actually known by users at all (to be useful for them)?

We don't want people instinctively reaching for execfile (or run_path for
that matter). It's almost always the wrong answer to a problem (because it
runs code in a weird, ill-defined environment and has undefined behaviour
when used inside a function), meeting the definition of "attractive
nuisance".

We moved reload() to imp.reload() and reduce() to functools.reduce() for
similar reasons - they're too rarely the right answer to justify having
them globally available by default.

> Googling for "execfile python 3", top 3 hits I see are stackoverflow
> questions, *none* of which mentions runpy. So, people either don't
> consider it viable alternative to execfile, or don't know about it at
> all (my guess it's the latter).

Given the relative age of the two APIs, that seems likely. Adding answers
pointing users to the runpy APIs could be useful.

> Like with previous discussion, its meaning goes beyond just Python
> realm - there's competition all around. And internets bring funny
> examples, like for example http://www.red-lang.org/p/contributions.html
> (scroll down to diagram, or here's direct link:
>
http://3.bp.blogspot.com/-xhOP35Dm99w/UuXFKgY2dlI/AAAAAAAAAGA/YQu98_pPDjw/s1600/reichart-abstraction-diagram.png
)
> So, didn't you know that Ruby can be used for OS-level development, and
> Python can't? Or that JavaScript DSL capabilities are better than
> Python's (that's taking into account that JavaScript DSL capabilities
> are represented by JSON, whose creators were so arrogant as to disallow
> even usage of comments in it).

There's a lot of misinformation on the internet. While there is certainly
room for the PSF to do more in terms of effectively communicating Python's
ubiquity and strengths (and we're working on that), "people with no clue
post stuff on the internet" doesn't make a compelling *technical* argument
(which is what is needed to get new builtins added).

> So, now suppose there's a discussion of how good different languages are
> for interactive usage (out of the box apparently). It would be a little
> hard to defend claim that Python is *excellent* interactive language,
> if its latest series got -1 on that scale, by removing feature which
> may be indispensable at times. Knowing that, one subconsciously may
> start to wonder if Ruby or JavaScript are doing it (in wide sense)
> better than Python.

Yes, people get upset when we tell them we consider some aspects of their
software designs to be ill-advised. Running other code in the *current*
namespace is such a thing - it is typically preferable to run it in a
*different* namespace and then access the results, rather than implicitly
overwriting the contents of the current namespace.

That said, a question still worth asking is whether there is scope for
additional runpy APIs that are designed to more easily implement Python 2
and IPython style modes of operation where independent units of code
manipulate a shared namespace? That's actually a possibility, but any such
proposals need to be presented on python-ideas in terms of the *use case*
to be addressed, rather than the fact that execfile() happened to be the
preferred solution in Python 2.

Regards,
Nick.

>
>
> --
> Best regards,
> Paul mailto:***@gmail.com

[Markus Unterwaditzer]

On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:
> Hello,
>
> I was pleasantly surprised with the response to recent post about
> MicroPython implementation details
> (https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I
> hope that discussion means that posts about alternative implementations
> are not unwelcome here, so I would like to bring up another (of many)
> issues we faced while implementing MicroPython.
>
> execfile() builtin function was removed in 3.0. This brings few
> problems:
>
> 1. It hampers interactive mode - instead of short and easy to type
> execfile("file.py") one needs to use exec(open("file.py").read()). I'm
> sure that's not going to bother a lot of people - after all, the
> easiest way to execute a Python file is to drop back to shell and
> restart python with file name, using all wonders of tab completion. But
> now imagine that Python interpreter runs on bare hardware, and its REPL
> is the only shell. That's exactly what we have with MicroPython's
> Cortex-M port. But it's not really MicroPython-specific, there's
> CPython port to baremetal either - http://www.pycorn.org/ .

As far as i can see, minimizing the amount of characters to type was never a
design goal of the Python language. And because that goal never mattered as
much for the designers as it seems to do for you, the reason for it to get
removed -- reducing the amount of builtins without reducing functionality --
was the only one left.

> 2. Ok, assuming that exec(open().read()) idiom is still a way to go,
> there's a problem - it requires to load entire file to memory. But
> there can be not enough memory. Consider 1Mb file with 900Kb comments
> (autogenerated, for example). execfile() could easily parse it, using
> small buffer. But exec() requires to slurp entire file into memory, and
> 1Mb is much more than heap sizes that we target.

That is a valid concern, but i believe violating the language specification and
adding your own execfile implementation (either as a builtin or in a new stdlib
module) here is justified, even if it means you will have to modify your
existing Python 3 code to use it -- i don't think the majority of software
written in Python will be able to run under such memory constraints without
major modifications anyway.

> Comments, suggestions? Just to set a productive direction, please
> kindly don't consider the problems above as MicroPython's.

A new (not MicroPython-specific) stdlib module containing functions such as
execfile could be considered. Not really for Python-2-compatibility, but for
performance-critical situations.

I am not sure if this is a good solution. Not at all. Even though it's
separated from the builtins, i think it would still sacrifice the purity of the
the language (by which i mean having a minimal composable API), because people
are going to use it anyway. It reminds me of the situation in Python 2 where
developers are trying to use cStringIO with a fallback to StringIO as a matter
of principle, not because they actually need that kind of performance.

Another, IMO better idea which shifts the problem to the MicroPython devs is to
"just" detect code using

exec(open(...).read())

and transparently rewrite it to something more memory-efficient. This is the
idea i actually think is a good one.


> I very much liked how last discussion went: I was pointed that
> https://docs.python.org/3/reference/index.html is not really a CPython
> reference, it's a *Python* reference, and there were even motion to
> clarify in it some points which came out from MicroPython discussion.
> So, what about https://docs.python.org/3/library/index.html - is it
> CPython, or Python standard library specification? Assuming the latter,
> what we have is that, by removal of previously available feature,
> *Python* became less friendly for interactive usage and less scalable.

"Less friendly for interactive usage" is a strong and vague statement. If
you're going after the amount of characters required to type, yes, absolutely,
but by that terms one could declare Bash and Perl to be superior languages.
Look at it from a different perspective: There are fewer builtins to remember.

>
>
> Thanks,
> Paul mailto:***@gmail.com
> _______________________________________________
> Python-Dev mailing list
> Python-***@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net

[Fabio Zadrozny]

On Sat, Jun 14, 2014 at 6:00 PM, Markus Unterwaditzer <
***@unterwaditzer.net> wrote:

> On Tue, Jun 10, 2014 at 05:23:12AM +0300, Paul Sokolovsky wrote:
> > Hello,
> >
> > I was pleasantly surprised with the response to recent post about
> > MicroPython implementation details
> > (https://mail.python.org/pipermail/python-dev/2014-June/134718.html). I
> > hope that discussion means that posts about alternative implementations
> > are not unwelcome here, so I would like to bring up another (of many)
> > issues we faced while implementing MicroPython.
> >
> > execfile() builtin function was removed in 3.0. This brings few
> > problems:
> >
> > 1. It hampers interactive mode - instead of short and easy to type
> > execfile("file.py") one needs to use exec(open("file.py").read()). I'm
> > sure that's not going to bother a lot of people - after all, the
> > easiest way to execute a Python file is to drop back to shell and
> > restart python with file name, using all wonders of tab completion. But
> > now imagine that Python interpreter runs on bare hardware, and its REPL
> > is the only shell. That's exactly what we have with MicroPython's
> > Cortex-M port. But it's not really MicroPython-specific, there's
> > CPython port to baremetal either - http://www.pycorn.org/ .
>
> As far as i can see, minimizing the amount of characters to type was never
> a
> design goal of the Python language. And because that goal never mattered as
> much for the designers as it seems to do for you, the reason for it to get
> removed -- reducing the amount of builtins without reducing functionality
> --
> was the only one left.
>
> > 2. Ok, assuming that exec(open().read()) idiom is still a way to go,
> > there's a problem - it requires to load entire file to memory. But
> > there can be not enough memory. Consider 1Mb file with 900Kb comments
> > (autogenerated, for example). execfile() could easily parse it, using
> > small buffer. But exec() requires to slurp entire file into memory, and
> > 1Mb is much more than heap sizes that we target.
>
> That is a valid concern, but i believe violating the language
> specification and
> adding your own execfile implementation (either as a builtin or in a new
> stdlib
> module) here is justified, even if it means you will have to modify your
> existing Python 3 code to use it -- i don't think the majority of software
> written in Python will be able to run under such memory constraints without
> major modifications anyway.
>
> > Comments, suggestions? Just to set a productive direction, please
> > kindly don't consider the problems above as MicroPython's.
>
> A new (not MicroPython-specific) stdlib module containing functions such as
> execfile could be considered. Not really for Python-2-compatibility, but
> for
> performance-critical situations.
>
> I am not sure if this is a good solution. Not at all. Even though it's
> separated from the builtins, i think it would still sacrifice the purity
> of the
> the language (by which i mean having a minimal composable API), because
> people
> are going to use it anyway. It reminds me of the situation in Python 2
> where
> developers are trying to use cStringIO with a fallback to StringIO as a
> matter
> of principle, not because they actually need that kind of performance.
>
> Another, IMO better idea which shifts the problem to the MicroPython devs
> is to
> "just" detect code using
>
> exec(open(...).read())
>
> and transparently rewrite it to something more memory-efficient. This is
> the
> idea i actually think is a good one.
>
>
> > I very much liked how last discussion went: I was pointed that
> > https://docs.python.org/3/reference/index.html is not really a CPython
> > reference, it's a *Python* reference, and there were even motion to
> > clarify in it some points which came out from MicroPython discussion.
> > So, what about https://docs.python.org/3/library/index.html - is it
> > CPython, or Python standard library specification? Assuming the latter,
> > what we have is that, by removal of previously available feature,
> > *Python* became less friendly for interactive usage and less scalable.
>
> "Less friendly for interactive usage" is a strong and vague statement. If
> you're going after the amount of characters required to type, yes,
> absolutely,
> but by that terms one could declare Bash and Perl to be superior languages.
> Look at it from a different perspective: There are fewer builtins to
> remember.
>
> >
>

Well, I must say that the exec(open().read()) is not really a proper
execfile implementation because it may fail because of encoding issues...
(i.e.: one has to check the file encoding to do the open with the proper
encoding, otherwise it's possible to end up with gibberish).

The PyDev debugger has an implementation (see:
https://github.com/fabioz/Pydev/blob/development/plugins/org.python.pydev/pysrc/_pydev_execfile.py)
which considers the encoding so that the result is ok (but it still has a
bug related to utf-8 with bom:
https://sw-brainwy.rhcloud.com/tracker/PyDev/346 which I plan to fix
soon...)

Personally, it's one thing that I think should be restored as the proper
implementation is actually quite tricky and the default recommended
solution does not work properly on some situations (and if micropython can
provide an optimized implementation which'd conform to Python, that'd be
one more point to add it back)...

Best Regards,

Fabio

[Greg Ewing]

Fabio Zadrozny wrote:
> Well, I must say that the exec(open().read()) is not really a proper
> execfile implementation because it may fail because of encoding
> issues...

It's not far off, though -- all it needs is an optional
encoding parameter.

--
Greg

[Steve Dower]

I think the point is that the encoding may be embedded in the file as a coding comment and there's no obvious way to deal with that.

Top-posted from my Windows Phone
________________________________
From: Greg Ewing<mailto:***@canterbury.ac.nz>
Sent: ý6/ý14/ý2014 16:19
To: python-***@python.org<mailto:python-***@python.org>
Subject: Re: [Python-Dev] Criticism of execfile() removal in Python3

Fabio Zadrozny wrote:
> Well, I must say that the exec(open().read()) is not really a proper
> execfile implementation because it may fail because of encoding
> issues...

It's not far off, though -- all it needs is an optional
encoding parameter.

--
Greg

[Nick Coghlan]

On 15 Jun 2014 09:37, "Steve Dower" <***@microsoft.com> wrote:
>
> I think the point is that the encoding may be embedded in the file as a
coding comment and there's no obvious way to deal with that.

Opening source files correctly is the intended use case for tokenize.open().

Cheers,
Nick.

[Steve Dower]

So is exec(tokenize.open(file).read()) the actual replacement for execfile()? Not too bad, but still not obvious (or widely promoted - I'd never heard of it).

Top-posted from my Windows Phone
________________________________
From: Nick Coghlan<mailto:***@gmail.com>
Sent: ý6/ý14/ý2014 18:31
To: Steve Dower<mailto:***@microsoft.com>
Cc: Greg Ewing<mailto:***@canterbury.ac.nz>; python-***@python.org<mailto:python-***@python.org>
Subject: Re: [Python-Dev] Criticism of execfile() removal in Python3


On 15 Jun 2014 09:37, "Steve Dower" <***@microsoft.com<mailto:***@microsoft.com>> wrote:
>
> I think the point is that the encoding may be embedded in the file as a coding comment and there's no obvious way to deal with that.

Opening source files correctly is the intended use case for tokenize.open().

Cheers,
Nick.

[Nick Coghlan]

On 15 June 2014 13:15, Steve Dower <***@microsoft.com> wrote:
> So is exec(tokenize.open(file).read()) the actual replacement for
> execfile()? Not too bad, but still not obvious (or widely promoted - I'd
> never heard of it).

Yes, that's pretty close. It's still a dubious idea due to the
implicit modification of the local namespace (and the resulting
differences in behaviour at function level due to the fact that
writing to locals() doesn't actually update the local namespace).

That said, the "implicit changes to the local namespace are a bad
idea" concern applies to exec() in general, so it was the "it's just a
shorthand for a particular use of exec" aspect that tipped in the
balance in the demise of execfile (this is also implied by the
phrasing of the relevant bullet point in PEP 3100).

Cheers,
Nick.

--
Nick Coghlan | ***@gmail.com | Brisbane, Australia

[Joseph Martinot-Lagarde]

Le 15/06/2014 05:15, Steve Dower a écrit :
> So is exec(tokenize.open(file).read()) the actual replacement for
> execfile()? Not too bad, but still not obvious (or widely promoted - I'd
> never heard of it).
>
Another way is to open the file in binary, then exec() checks itself if
an encoding is defined in the file. This is what is used in spyder:

exec(open(file, 'rb').read())

Here is the discussion for reference:
https://bitbucket.org/spyder-ide/spyderlib/pull-request/3/execution-on-current-spyder-interpreter/diff

This behavior is not indicated in the documentation but is somehow
confirmed on stackoverflow:
http://stackoverflow.com/questions/6357361/alternative-to-execfile-in-python-3-2/6357418?noredirect=1#comment30467918_6357418

---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com

[Paul Moore]

On 19 June 2014 20:39, Joseph Martinot-Lagarde
<joseph.martinot-***@m4x.org> wrote:
> Another way is to open the file in binary, then exec() checks itself if an
> encoding is defined in the file. This is what is used in spyder:
>
> exec(open(file, 'rb').read())
>
> Here is the discussion for reference:
> https://bitbucket.org/spyder-ide/spyderlib/pull-request/3/execution-on-current-spyder-interpreter/diff

It would be good to document this. Could you open a docs bug to get this added?

Paul