Issue 21671: CVE-2014-0224 OpenSSL upgrade to 1.0.1h on Windows required

[Yates, Andy (CS Houston, TX)]

Python Dev,
Andy here. I have a Windows product based on Python and I'm getting hammered to release a version that includes the fix in OpenSSL 1.0.1h. My product is built on a Windows system using Python installed from the standard Python installer at Python.org. I would be grateful if I could get some advice on my options. Will Python.org be releasing a Windows installer with the fix any time soon or will it be at the next scheduled release in November? If it is November, there's no way I can wait that long. Now what? Would it be best to build my own Python? Is it possible to drop in new OpenSSL versions on Windows without rebuilding Python? Looking for some guidance on how to handle these OpenSSL issues on Windows.

Thanks!
Andy Yates

[Steve Dower]

Yates, Andy (CS Houston, TX) wrote:
> Python Dev,
> Andy here. I have a Windows product based on Python and I'm getting hammered to
> release a version that includes the fix in OpenSSL 1.0.1h. My product is built
> on a Windows system using Python installed from the standard Python installer at
> Python.org. I would be grateful if I could get some advice on my options. Will
> Python.org be releasing a Windows installer with the fix any time soon or will
> it be at the next scheduled release in November? If it is November, there's no
> way I can wait that long. Now what? Would it be best to build my own Python? Is
> it possible to drop in new OpenSSL versions on Windows without rebuilding
> Python? Looking for some guidance on how to handle these OpenSSL issues on
> Windows.

You'll only need to rebuild the _ssl and _hashlib extension modules with the new OpenSSL version. The easiest way to do this is to build from source (which has already been updated for 1.0.1h if you use the externals scripts in Tools\buildbot), and you should just be able to drop _ssl.pyd and _hashlib.pyd on top of a normal install.

Aside: I wonder if it's worth changing to dynamically linking to OpenSSL? It would make this kind of in-place upgrade easier when people need to do it. Any thoughts? (Does OpenSSL even support it?)

Cheers,
Steve

> Thanks!
> Andy Yates

[M.-A. Lemburg]

On 17.06.2014 20:27, Steve Dower wrote:
> Yates, Andy (CS Houston, TX) wrote:
>> Python Dev,
>> Andy here. I have a Windows product based on Python and I'm getting hammered to
>> release a version that includes the fix in OpenSSL 1.0.1h. My product is built
>> on a Windows system using Python installed from the standard Python installer at
>> Python.org. I would be grateful if I could get some advice on my options. Will
>> Python.org be releasing a Windows installer with the fix any time soon or will
>> it be at the next scheduled release in November? If it is November, there's no
>> way I can wait that long. Now what? Would it be best to build my own Python? Is
>> it possible to drop in new OpenSSL versions on Windows without rebuilding
>> Python? Looking for some guidance on how to handle these OpenSSL issues on
>> Windows.
>
> You'll only need to rebuild the _ssl and _hashlib extension modules with the new OpenSSL version. The easiest way to do this is to build from source (which has already been updated for 1.0.1h if you use the externals scripts in Tools\buildbot), and you should just be able to drop _ssl.pyd and _hashlib.pyd on top of a normal install.
>
> Aside: I wonder if it's worth changing to dynamically linking to OpenSSL? It would make this kind of in-place upgrade easier when people need to do it. Any thoughts? (Does OpenSSL even support it?)

Yes, no problem at all, but you'd still have to either do a new release
every time a new OpenSSL problem is found (don't think that's an
option for Python) or provide new compiled versions
compatible with the Python modules needing the OpenSSL libs or
instructions on how to build these.

Note that the hash routines are rarely affected by these OpenSSL
bugs. They usually only affect the SSL/TLS protocol parts.

Alternatively, you could make use of our pyOpenSSL distribution,
which includes pyOpenSSL and the OpenSSL libs (also for Windows):

http://www.egenix.com/products/python/pyOpenSSL/

We created this to address the problem of having to update
OpenSSL rather often. It doesn't support Python 3 yet, but
on the plus side, you do get OpenSSL libs which are compiled
with the same compiler versions used for the Python.org
installers.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Antoine Pitrou]

Le 17/06/2014 14:55, M.-A. Lemburg a écrit :
>
> Alternatively, you could make use of our pyOpenSSL distribution,
> which includes pyOpenSSL and the OpenSSL libs (also for Windows):
>
> http://www.egenix.com/products/python/pyOpenSSL/
>
> We created this to address the problem of having to update
> OpenSSL rather often.

This is very nice, but does it also upgrade the OpenSSL version used by
the _ssl and _hashlib modules?

Regards

Antoine.

[M.-A. Lemburg]

On 17.06.2014 22:36, Antoine Pitrou wrote:
> Le 17/06/2014 14:55, M.-A. Lemburg a écrit :
>>
>> Alternatively, you could make use of our pyOpenSSL distribution,
>> which includes pyOpenSSL and the OpenSSL libs (also for Windows):
>>
>> http://www.egenix.com/products/python/pyOpenSSL/
>>
>> We created this to address the problem of having to update
>> OpenSSL rather often.
>
> This is very nice, but does it also upgrade the OpenSSL version used by the _ssl and _hashlib modules?

On Unix, tt will if you load pyOpenSSL before importing _ssl or
_hashlib (and those modules are built as shared libs).

Alternatively, you can set LD_LIBRARY_PATH to lib/python2.7/OpenSSL
to have the system linker use the embedded libs before starting
Python. Then it will always use the up-to-date libs.

On Windows, this won't work, because _ssl and _hashlib are
statically linked against the OpenSSL libs.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jun 17 2014)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2014-06-17: Released eGenix PyRun 2.0.0 ... http://egenix.com/go58
2014-06-09: Released eGenix pyOpenSSL 0.13.3 ... http://egenix.com/go57
2014-07-02: Python Meeting Duesseldorf ... 15 days to go

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[Ned Deily]

In article
<***@BLUPR03MB389.namprd03.prod.outlook.com
>,
Steve Dower <***@microsoft.com> wrote:
> You'll only need to rebuild the _ssl and _hashlib extension modules with the
> new OpenSSL version. The easiest way to do this is to build from source
> (which has already been updated for 1.0.1h if you use the externals scripts
> in Tools\buildbot), and you should just be able to drop _ssl.pyd and
> _hashlib.pyd on top of a normal install.

Should we consider doing a re-spin of the Windows installers for 2.7.7
with 1.0.1h? Or consider doing a 2.7.8 in the near future to address
this and various 2.7.7 regressions that have been identified so far
(Issues 21652 and 21672)?

> Aside: I wonder if it's worth changing to dynamically linking to OpenSSL? It
> would make this kind of in-place upgrade easier when people need to do it.
> Any thoughts? (Does OpenSSL even support it?)

OpenSSL is often dynamically linked in Python builds on various other
platforms, for example, on Linux or OS X.

--
Ned Deily,
***@acm.org

[Benjamin Peterson]

On Tue, Jun 17, 2014, at 12:03, Ned Deily wrote:
> In article
> <***@BLUPR03MB389.namprd03.prod.outlook.com
> >,
> Steve Dower <***@microsoft.com> wrote:
> > You'll only need to rebuild the _ssl and _hashlib extension modules with the
> > new OpenSSL version. The easiest way to do this is to build from source
> > (which has already been updated for 1.0.1h if you use the externals scripts
> > in Tools\buildbot), and you should just be able to drop _ssl.pyd and
> > _hashlib.pyd on top of a normal install.
>
> Should we consider doing a re-spin of the Windows installers for 2.7.7
> with 1.0.1h? Or consider doing a 2.7.8 in the near future to address
> this and various 2.7.7 regressions that have been identified so far
> (Issues 21652 and 21672)?

I think we should do a 2.7.8 soon to pick up the openssl upgrade and
recent CGI security fix. I would like to see those two regressions fixed
first, though.

[Yates, Andy (CS Houston, TX)]

Thanks for all the good information. We ended up building _ssl and _hashlib and dropping those into the existing Python on our build server. That seems to be working fine.

>From my perspective ssl libraries are a special case. I think I could handle any other included library having a flaw for weeks or months, but my management and customers are sensitive to releasing software with known ssl vulnerabilities. For Windows Python it looks like the only option for updating OpenSSL is to build from source. For us that turned out to be no big deal. However, it may be beyond the reach of some, either technically or due to the lack of access to Dev Studio. There's also some concern that a custom build of Python may not have some secret sauce or complier switch that could cause unexpected behavior.

That said, I'd like to see Python spin within a short period of time after a recognized OpenSSL vulnerability is fixed if is statically linked. This would limit exposure to the unsuspecting user who downloads Windows Python from Python.org. The next best thing would be to dynamically link to Windows OpenSSL DLLs allowing users to drop in which ever version they like.

Thanks again!!

Andy


-----Original Message-----
From: Python-Dev [mailto:python-dev-bounces+ayates=***@python.org] On Behalf Of Benjamin Peterson
Sent: Tuesday, June 17, 2014 2:07 PM
To: Ned Deily; python-***@python.org
Subject: Re: [Python-Dev] Issue 21671: CVE-2014-0224 OpenSSL upgrade to 1.0.1h on Windows required

On Tue, Jun 17, 2014, at 12:03, Ned Deily wrote:
> In article
> <***@BLUPR03MB389.namprd03.prod.outlook.c
> om
> >,
> Steve Dower <***@microsoft.com> wrote:
> > You'll only need to rebuild the _ssl and _hashlib extension modules
> > with the new OpenSSL version. The easiest way to do this is to build
> > from source (which has already been updated for 1.0.1h if you use
> > the externals scripts in Tools\buildbot), and you should just be
> > able to drop _ssl.pyd and _hashlib.pyd on top of a normal install.
>
> Should we consider doing a re-spin of the Windows installers for 2.7.7
> with 1.0.1h? Or consider doing a 2.7.8 in the near future to address
> this and various 2.7.7 regressions that have been identified so far
> (Issues 21652 and 21672)?

I think we should do a 2.7.8 soon to pick up the openssl upgrade and recent CGI security fix. I would like to see those two regressions fixed first, though.

[Martin v. Löwis]

Am 17.06.14 20:27, schrieb Steve Dower:
> You'll only need to rebuild the _ssl and _hashlib extension modules
> with the new OpenSSL version. The easiest way to do this is to build
> from source (which has already been updated for 1.0.1h if you use the
> externals scripts in Tools\buildbot), and you should just be able to
> drop _ssl.pyd and _hashlib.pyd on top of a normal install.
>
> Aside: I wonder if it's worth changing to dynamically linking to
> OpenSSL? It would make this kind of in-place upgrade easier when
> people need to do it. Any thoughts? (Does OpenSSL even support it?)

We originally considered using prebuilt binaries, such as

http://slproweb.com/products/Win32OpenSSL.html

This is tricky because of CRT issues: they will likely bind to a
different version of the CRT, and
a) it is unclear whether this would reliably work, and
b) requires the Python installer to include a different version of
the CRT, which we would not have a license to include (as the
CRT redistribution license only applies to the version of the CRT
that Python was built with)

There was also the desire to use the same compiler for all code
distributed, to use the same optimizations on all of it. In addition,
for OpenSSL, there is compile time configuration wrt. to the algorithms
built into the binaries where Python's build deviates from the default.

Having a separate project to build a DLL within pcbuild.sln was never
implemented. Doing so possibly increases the risk of DLL hell, if Python
picks up the wrong version of OpenSSL (e.g. if Python gets embedded
into some other application).

Regards,
Martin

[Steve Dower]

Yeah, the fact that it really has to be our own copy of the DLL negates the advantage. If someone can rebuild that, they could rebuild the modules that statically link it.

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Martin v. Löwis<mailto:***@v.loewis.de>
Sent: ‎6/‎18/‎2014 2:46
To: Steve Dower<mailto:***@microsoft.com>; Yates, Andy (CS Houston, TX)<mailto:***@hp.com>; Python-***@python.org<mailto:Python-***@python.org>
Subject: Re: [Python-Dev] Issue 21671: CVE-2014-0224 OpenSSL upgrade to 1.0.1h on Windows required

Am 17.06.14 20:27, schrieb Steve Dower:
> You'll only need to rebuild the _ssl and _hashlib extension modules
> with the new OpenSSL version. The easiest way to do this is to build
> from source (which has already been updated for 1.0.1h if you use the
> externals scripts in Tools\buildbot), and you should just be able to
> drop _ssl.pyd and _hashlib.pyd on top of a normal install.
>
> Aside: I wonder if it's worth changing to dynamically linking to
> OpenSSL? It would make this kind of in-place upgrade easier when
> people need to do it. Any thoughts? (Does OpenSSL even support it?)

We originally considered using prebuilt binaries, such as

http://slproweb.com/products/Win32OpenSSL.html

This is tricky because of CRT issues: they will likely bind to a
different version of the CRT, and
a) it is unclear whether this would reliably work, and
b) requires the Python installer to include a different version of
the CRT, which we would not have a license to include (as the
CRT redistribution license only applies to the version of the CRT
that Python was built with)

There was also the desire to use the same compiler for all code
distributed, to use the same optimizations on all of it. In addition,
for OpenSSL, there is compile time configuration wrt. to the algorithms
built into the binaries where Python's build deviates from the default.

Having a separate project to build a DLL within pcbuild.sln was never
implemented. Doing so possibly increases the risk of DLL hell, if Python
picks up the wrong version of OpenSSL (e.g. if Python gets embedded
into some other application).

Regards,
Martin

[Cory Benfield]

On 17 June 2014 17:41, Yates, Andy (CS Houston, TX) <***@hp.com> wrote:
> Is it possible to drop in new OpenSSL versions
> on Windows without rebuilding Python?

If you think this is a problem you're going to have more than once,
you'll want to look hard at whether it's worth using pyOpenSSL
(either the egenix version or the PyCA one[1]) instead, and delivering
binary releases with a bundled copy of OpenSSL. PyOpenSSL from PyCA is
actually considering bundling OpenSSL on Windows anyway[2], so you
might find this problem goes away.

[1] https://github.com/pyca/pyopenssl
[2] https://github.com/pyca/cryptography/issues/1121

[Martin v. Löwis]

Am 17.06.14 18:41, schrieb Yates, Andy (CS Houston, TX):
> Python Dev,
>
> Andy here. I have a Windows product based on Python and I’m getting
> hammered to release a version that includes the fix in OpenSSL 1.0.1h.
> My product is built on a Windows system using Python installed from the
> standard Python installer at Python.org. I would be grateful if I could
> get some advice on my options.

Can you please report
- what version of Python you are distributing?
- why it absolutely has to be 1.0.1h that is included?

According to the CVE, 0.9.8za and 1.0.0m would work as well (and in our
case, would be preferred for older versions of Python).

Regards,
Martin