Can't have unbuffered text I/O in Python 3.0?

[Fabio Zadrozny]

Hi,

I'm currently having problems to get the output of Python 3.0 into the
Eclipse console (integrating it into Pydev).

The problem appears to be that stdout and stderr are not running
unbuffered (even passing -u or trying to set PYTHONUNBUFFERED), and
the content only appears to me when a flush() is done or when the
process finishes.

So, in the search of a solution, I found a suggestion from
http://stackoverflow.com/questions/107705/python-output-buffering

to use the following construct:

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

But that gives the error below in Python 3.0:

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
File "D:\bin\Python30\lib\os.py", line 659, in fdopen
return io.open(fd, *args, **kwargs)
File "D:\bin\Python30\lib\io.py", line 243, in open
raise ValueError("can't have unbuffered text I/O")
ValueError: can't have unbuffered text I/O

So, I'd like to know if there's some way I can make it run unbuffered
(to get the output contents without having to flush() after each
write).

Thanks,

Fabio

[Brett Cannon]

Post by Fabio Zadrozny
Hi,
I'm currently having problems to get the output of Python 3.0 into the
Eclipse console (integrating it into Pydev).
The problem appears to be that stdout and stderr are not running
unbuffered (even passing -u or trying to set PYTHONUNBUFFERED), and
the content only appears to me when a flush() is done or when the
process finishes.
So, in the search of a solution, I found a suggestion from
http://stackoverflow.com/questions/107705/python-output-buffering
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
File "D:\bin\Python30\lib\os.py", line 659, in fdopen
return io.open(fd, *args, **kwargs)
File "D:\bin\Python30\lib\io.py", line 243, in open
raise ValueError("can't have unbuffered text I/O")
ValueError: can't have unbuffered text I/O
So, I'd like to know if there's some way I can make it run unbuffered
(to get the output contents without having to flush() after each
write).

Notice how the exception specifies test I/O cannot be unbuffered. This
restriction does not apply to bytes I/O. Simply open it as 'wb'
instead of 'w' and it works.

-Brett

[Nick Coghlan]

Post by Brett Cannon
Notice how the exception specifies test I/O cannot be unbuffered. This
restriction does not apply to bytes I/O. Simply open it as 'wb'
instead of 'w' and it works.

s/test/text/ :)

(For anyone else that is like me and skipped over the exception detail
on first reading, thus becoming a little confused...)

Cheers,
Nick.

[Fabio Zadrozny]

You're right, thanks (guess I'll use that option then).

Now, is it a bug that Python 3.0 doesn't run unbuffered when
specifying -u or PYTHONUNBUFFERED, or was this support dropped?

Thanks,

Fabio

Post by Brett Cannon

Post by Fabio Zadrozny
Hi,
I'm currently having problems to get the output of Python 3.0 into the
Eclipse console (integrating it into Pydev).
The problem appears to be that stdout and stderr are not running
unbuffered (even passing -u or trying to set PYTHONUNBUFFERED), and
the content only appears to me when a flush() is done or when the
process finishes.
So, in the search of a solution, I found a suggestion from
http://stackoverflow.com/questions/107705/python-output-buffering
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
File "D:\bin\Python30\lib\os.py", line 659, in fdopen
return io.open(fd, *args, **kwargs)
File "D:\bin\Python30\lib\io.py", line 243, in open
raise ValueError("can't have unbuffered text I/O")
ValueError: can't have unbuffered text I/O
So, I'd like to know if there's some way I can make it run unbuffered
(to get the output contents without having to flush() after each
write).

Notice how the exception specifies test I/O cannot be unbuffered. This
restriction does not apply to bytes I/O. Simply open it as 'wb'
instead of 'w' and it works.
-Brett

[b***@gmail.com]

Post by Fabio Zadrozny
You're right, thanks (guess I'll use that option then).
Now, is it a bug that Python 3.0 doesn't run unbuffered when
specifying -u or PYTHONUNBUFFERED, or was this support dropped?

Well, ``python -h`` still lists it. That means either the output for -h
needs to be fixed or the feature needs to be supported.

-Brett

[Antoine Pitrou]

Post by b***@gmail.com
Well, ``python -h`` still lists it.

Precisely, it says:

-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'

Note the "binary". And indeed:

./python -u
Python 3.1a0 (py3k:67839M, Dec 18 2008, 17:56:54)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by b***@gmail.com

import sys
sys.stdout.buffer.write(b"y")

y1
I don't know what it would take to enable unbuffered text IO while keeping the
current TextIOWrapper implementation...

Regards

Antoine.

[Guido van Rossum]

Fror truly unbuffered text output you'd have to make changes to the
io.TextIOWrapper class to flush after each write() call. That's an API
change -- the constructor currently has a line_buffering option but no
option for completely unbuffered mode. It would also require some
changes to io.open() which currently rejects buffering=0 in text mode.
All that suggests that it should wait until 3.1.

However it might make sense to at least turn on line buffering when -u
or PYTHONUNBUFFERED is given; that doesn't require API changes and so
can be considered a bug fix.

--Guido van Rossum (home page: http://www.python.org/~guido/)

Post by Antoine Pitrou

Post by b***@gmail.com
Well, ``python -h`` still lists it.

-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
./python -u
Python 3.1a0 (py3k:67839M, Dec 18 2008, 17:56:54)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by b***@gmail.com

import sys
sys.stdout.buffer.write(b"y")

y1
I don't know what it would take to enable unbuffered text IO while keeping the
current TextIOWrapper implementation...
Regards
Antoine.
_______________________________________________
Python-Dev mailing list
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

[Fabio Zadrozny]

It appears that this bug was already reported: http://bugs.python.org/issue4705

Any chance that it gets in the next 3.0.x bugfix release?

Just as a note, if I do: sys.stdout._line_buffering = True, it also
works, but doesn't seem right as it's accessing an internal attribute.

Note 2: the solution that said to pass 'wb' does not work, because I
need the output as text and not binary or text becomes garbled when
it's not ascii.

Thanks,

Fabio

Post by Guido van Rossum
Fror truly unbuffered text output you'd have to make changes to the
io.TextIOWrapper class to flush after each write() call. That's an API
change -- the constructor currently has a line_buffering option but no
option for completely unbuffered mode. It would also require some
changes to io.open() which currently rejects buffering=0 in text mode.
All that suggests that it should wait until 3.1.
However it might make sense to at least turn on line buffering when -u
or PYTHONUNBUFFERED is given; that doesn't require API changes and so
can be considered a bug fix.
--Guido van Rossum (home page: http://www.python.org/~guido/)

Post by Antoine Pitrou

Post by b***@gmail.com
Well, ``python -h`` still lists it.

-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
./python -u
Python 3.1a0 (py3k:67839M, Dec 18 2008, 17:56:54)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by b***@gmail.com

import sys
sys.stdout.buffer.write(b"y")

y1
I don't know what it would take to enable unbuffered text IO while keeping the
current TextIOWrapper implementation...
Regards
Antoine.
_______________________________________________
Python-Dev mailing list
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

_______________________________________________
Python-Dev mailing list
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/fabiofz%40gmail.com

[Brett Cannon]

Post by Fabio Zadrozny
It appears that this bug was already reported: http://bugs.python.org/issue4705
Any chance that it gets in the next 3.0.x bugfix release?
Just as a note, if I do: sys.stdout._line_buffering = True, it also
works, but doesn't seem right as it's accessing an internal attribute.
Note 2: the solution that said to pass 'wb' does not work, because I
need the output as text and not binary or text becomes garbled when
it's not ascii.

Can't you decode the bytes after you receive them?

-Brett

Post by Fabio Zadrozny
Thanks,
Fabio

Post by Guido van Rossum
Fror truly unbuffered text output you'd have to make changes to the
io.TextIOWrapper class to flush after each write() call. That's an API
change -- the constructor currently has a line_buffering option but no
option for completely unbuffered mode. It would also require some
changes to io.open() which currently rejects buffering=0 in text mode.
All that suggests that it should wait until 3.1.
However it might make sense to at least turn on line buffering when -u
or PYTHONUNBUFFERED is given; that doesn't require API changes and so
can be considered a bug fix.
--Guido van Rossum (home page: http://www.python.org/~guido/)

Post by Antoine Pitrou

Post by b***@gmail.com
Well, ``python -h`` still lists it.

-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
see man page for details on internal buffering relating to '-u'
./python -u
Python 3.1a0 (py3k:67839M, Dec 18 2008, 17:56:54)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by b***@gmail.com

import sys
sys.stdout.buffer.write(b"y")

y1
I don't know what it would take to enable unbuffered text IO while keeping the
current TextIOWrapper implementation...
Regards
Antoine.
_______________________________________________
Python-Dev mailing list
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

_______________________________________________
Python-Dev mailing list
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/fabiofz%40gmail.com

_______________________________________________
Python-Dev mailing list
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

[Fabio Zadrozny]

Post by Brett Cannon

Post by Fabio Zadrozny
It appears that this bug was already reported: http://bugs.python.org/issue4705
Any chance that it gets in the next 3.0.x bugfix release?
Just as a note, if I do: sys.stdout._line_buffering = True, it also
works, but doesn't seem right as it's accessing an internal attribute.
Note 2: the solution that said to pass 'wb' does not work, because I
need the output as text and not binary or text becomes garbled when
it's not ascii.

Can't you decode the bytes after you receive them?

Well, in short, no (long answer is that I probably could if I spent a
long time doing my own console instead of relying on what's already
done and working in Eclipse for all the current available languages it
supports, but that just doesn't seem right).

Also, it's seems easily solvable (enabling line buffering for the
python streams when -u is passed) in the Python side... My current
workaround is doing that on a custom site-initialization when a Python
3 interpreter is found, but I find that this is not the right way for
doing it, and it really feels like a Python bug.

-- Fabio

[Antoine Pitrou]

[...]

And I realize I should have thought a bit before giving that "proof".
Sorry!