Returning Windows file attribute information via os.stat()

[Ben Hoyt]

Received: from localhost (HELO mail.python.org) (127.0.0.1)
by albatross.python.org with SMTP; 10 Jun 2014 06:02:21 +0200
Received: from mail-qc0-x231.google.com (unknown
[IPv6:2607:f8b0:400d:c01::231])
(using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits))
(No client certificate requested)
by mail.python.org (Postfix) with ESMTPS
for <python-***@python.org>; Tue, 10 Jun 2014 06:02:20 +0200 (CEST)
Received: by mail-qc0-f177.google.com with SMTP id i17so1624297qcy.22
for <python-***@python.org>; Mon, 09 Jun 2014 21:02:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
h=mime-version:date:message-id:subject:from:to:content-type;
bh=+AsoZKcj985DIwzl0nEJcqp5ACkhZFgu7zr6djRUaY0=;
b=hNPPtb7HGMVLJsjdFj6OQBVtYIEKTI0tUDTm3Lrhrm0IjywNP9p78A6n3fqehEZH9D
iKPUNZvWsyBtzfEqia8rTOvBW8NNQKM56IMKMznz38Jcw/tn6PBTnBLSb/2Yybbxw2qO
DNvlcysHw1yKI8kbuNOXF6vwK1YZ430ZZsm3zibM3kKaMZxOefLxHjmFPFWGcgtwo/Wa
eu1oEJIYQhK9kqLd9jFiFEe/kmVYiWE8kckrl50aYNJccbmGSnCh8AQX+OEYTsnbSWJh
+lJZTz/V0xxcwjoiW7ypwV3BQpUjhuQhdKlqqe09aiE8n4S8jzJEMGI3jKXbD6mN9fER
M8mw==
X-Received: by 10.140.29.34 with SMTP id a31mr36278811qga.95.1402372934509;
Mon, 09 Jun 2014 21:02:14 -0700 (PDT)
Received: by 10.229.174.6 with HTTP; Mon, 9 Jun 2014 21:02:14 -0700 (PDT)
X-BeenThere: python-***@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Python core developers <python-dev.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-dev>,
<mailto:python-dev-***@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-dev/>
List-Post: <mailto:python-***@python.org>
List-Help: <mailto:python-dev-***@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-dev>,
<mailto:python-dev-***@python.org?subject=subscribe>
Errors-To: python-dev-bounces+python-python-dev=***@python.org
Sender: "Python-Dev"
<python-dev-bounces+python-python-dev=***@python.org>
Archived-At: <http://permalink.gmane.org/gmane.comp.python.devel/148165>

Hi folks,

As pointed out to me recently in an issue report [1] on my scandir
module, Python's os.stat() simply discards most of the file attribute
information fetched via the Win32 system calls. On Windows, os.stat()
calls CreateFile to open the file and get the dwFileAttributes value,
but it throws it all away except the FILE_ATTRIBUTE_DIRECTORY and
FILE_ATTRIBUTE_READONLY bits. See CPython source at [2].

Given that os.stat() returns extended, platform-specific file
attributes on Linux and OS X platforms (see [3] -- for example,
st_blocks, st_rsize, etc), it seems that Windows is something of a
second-class citizen here.

There are several questions on StackOverflow about how to get this
information on Windows, and one has to resort to ctypes. For example,
[4].

To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?

Then, similarly to existing code like hasattr(st, 'st_blocks') on
Linux, you could write a cross-platform function to determine if a
file was hidden, something like so:

FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h

def is_hidden(path):
if startswith(os.path.basename(path), '.'):
return True
st = os.stat(path)
if hasattr(st, 'st_winattrs') and st.st_winattrs & FILE_ATTRIBUTE_HIDDEN:
return True
return False

I'd be interested to hear people's thoughts on this.

Thanks,
Ben.

[1]: https://github.com/benhoyt/scandir/issues/22
[2]: https://github.com/python/cpython/blob/master/Modules/posixmodule.c#L1462
[3]: https://docs.python.org/3.4/library/os.html#os.stat
[4]: http://stackoverflow.com/a/6365265

[Paul Moore]

Post by Ben Hoyt
To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?

+1. Given the precedent of Linux- and OS X-specific attributes, this
seems like a no-brainer to me.

Paul

[Victor Stinner]

Post by Ben Hoyt
To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?
(...)
FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h

I don't like such API, it requires to import constants, use masks, etc.

I would prefer something like:

if st.win_hidden: ...

Or maybe:

if st.winattrs.hidden: ...

Victor

[Ben Hoyt]

Post by Victor Stinner

Post by Ben Hoyt
FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h

I don't like such API, it requires to import constants, use masks, etc.
if st.win_hidden: ...
if st.winattrs.hidden: ...

Yes, fair call. However, it looks like the precent for the attributes
in os.stat()'s return value has long since been set -- this is
OS-specific stuff. For example, what's in "st_flags"? It's not
documented, but comes straight from the OS. Same with st_rdev,
st_type, etc -- the documentation doesn't define them, and it looks
like they're OS-specific values.

I don't think the st.win_hidden approach gains us much, because the
next person is going to ask for the FILE_ATTRIBUTE_ENCRYPTED or
FILE_ATTRIBUTE_COMPRESSED flag. So we really need all the bits or
nothing. I don't mind the st.st_winattrs.hidden approach, except that
we'd need 17 sub-attributes, and they'd all have to be documented. And
if Windows added another attribute, Python wouldn't have it, etc. So I
think the OS-defined constant is the way to go.

Because these are fixed-forever constants, I suspect in library code
and the like people would just KISS and use an integer literal and a
comment, avoiding the import/constant thing:

if getattr(st, 'st_winattrs', 0) & 2: # FILE_ATTRIBUTE_HIDDEN
...

-Ben

[Paul Moore]

Post by Ben Hoyt
Because these are fixed-forever constants, I suspect in library code
and the like people would just KISS and use an integer literal and a

The stat module exposes a load of constants - why not add the
(currently known) ones there? Finding the values of Windows constants
if you don't have access to the C headers can be a pain, so having
them defined *somewhere* as named values is useful.

Paul

[Ben Hoyt]

Post by Paul Moore
The stat module exposes a load of constants - why not add the
(currently known) ones there? Finding the values of Windows constants
if you don't have access to the C headers can be a pain, so having
them defined *somewhere* as named values is useful.

So stat.FILE_ATTRIBUTES_HIDDEN and the like? Alternatively they could
go in ctypes.wintypes, but I think stat makes more sense in this case.

-Ben

[Paul Moore]

Post by Ben Hoyt
So stat.FILE_ATTRIBUTES_HIDDEN and the like?

Yep. (Maybe WIN_FILE_ATTRIBUTES_HIDDEN, but the Unix ones don't have
an OA name prefix, so I'd go with your original).

Paul

[MRAB]

On 2014-06-10 05:02, Ben Hoyt wrote:
[snip]

Post by Ben Hoyt
FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h
return True
st = os.stat(path)
return True
return False

[Ben Hoyt]

Post by Ben Hoyt
if hasattr(st, 'st_winattrs') and st.st_winattrs &
return True
return False

Yes, good call. Or one further:

return getattr(st, 'st_winattrs', 0) & FILE_ATTRIBUTE_HIDDEN != 0

-Ben

[Ethan Furman]

Post by Ben Hoyt
To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?

+1 to the idea, whatever the exact implementation.

--
~Ethan~

[Zachary Ware]

Post by Ethan Furman

Post by Ben Hoyt
To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?

+1 to the idea, whatever the exact implementation.

Agreed.

[Ben Hoyt]

Post by Ethan Furman

Post by Ben Hoyt
To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?

+1 to the idea, whatever the exact implementation.

Cool.

I think we should add a st_winattrs integer attribute (on Windows) and
then also add the FILE_ATTRIBUTES_* constants to stat.py per Paul
Moore.

What would be the next steps to get this to happen? Open an issue on
bugs.python.org and submit a patch with tests?

-Ben

[Zachary Ware]

Post by Ben Hoyt

Post by Ethan Furman

Post by Ben Hoyt
To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?

+1 to the idea, whatever the exact implementation.

Cool.
I think we should add a st_winattrs integer attribute (on Windows) and
then also add the FILE_ATTRIBUTES_* constants to stat.py per Paul
Moore.

Add to _stat.c rather than stat.py.

Post by Ben Hoyt
What would be the next steps to get this to happen? Open an issue on
bugs.python.org and submit a patch with tests?

Yep!

[Ben Hoyt]

Post by Ben Hoyt
What would be the next steps to get this to happen? Open an issue on
bugs.python.org and submit a patch with tests?

Yep!

Okay, I've done step one (opened an issue on bugs.python.org), and
hope to provide a patch in the next few weeks if no-one else does
(I've never compiled CPython on Windows before):

http://bugs.python.org/issue21719

-Ben

[Terry Reedy]

Post by Ben Hoyt

Post by Ben Hoyt
What would be the next steps to get this to happen? Open an issue on
bugs.python.org and submit a patch with tests?

Yep!

Okay, I've done step one (opened an issue on bugs.python.org), and
hope to provide a patch in the next few weeks if no-one else does
http://bugs.python.org/issue21719

If you have problems compiling, the core-mentorship list is one place to
ask. For 3.4+, I believe the devguide instructions are correct. If not,
say something.