ArchiveOrangemail archive

List which receives bug reports on Python


python-bugs-list.python.org
(List home) (Recent threads) (100 other Python lists)

Subscription Options

  • RSS or Atom: Read-only subscription using a browser or aggregator. This is the recommended way if you don't need to send messages to the list. You can learn more about feed syndication and clients here.
  • Conventional: All messages are delivered to your mail address, and you can reply. To subscribe, send an email to the list's subscribe address with "subscribe" in the subject line, or visit the list's homepage here.
  • Very high traffic list: 100+ messages per day
  • This list contains about 238,971 messages, beginning May 1999
  • 82 messages added yesterday
Report the Spam
This button sends a spam report to the moderator. Please use it sparingly. For other removal requests, read this.
Are you sure? yes no

Fatal LookupError: unknown encoding: cp0 on Windows embedded startup.

Ad
Graham Dumpleton 1247826092Fri, 17 Jul 2009 10:21:32 +0000 (UTC)
New submission from Graham Dumpleton :

When using Python 3.1 for Apache/mod_wsgi (3.0c4) on Windows, Apache will 
crash on startup because Python is forcing the process to exit with:

Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: cp0

I first mentioned this on issue6393, but have now created it as a separate 
issue as appears to be distinct from the issue on MacOS X, athough possibly 
related.

In the Windows case there is actually an encoding, that of 'cp0' where as on 
MacOS X, the encoding name was empty.

The same mod_wsgi code works fine under Python 3.1 on MacOS X.

----------
components: Interpreter Core, Windows
messages: 90616
nosy: grahamd
severity: normal
status: open
title: Fatal LookupError: unknown encoding: cp0 on Windows embedded startup.
type: crash
versions: Python 3.1
Amaury Forgeot d'Arc 1247829597Fri, 17 Jul 2009 11:19:57 +0000 (UTC)
Amaury Forgeot d'Arc  added the comment:

- Apache is not a Console application, so the Windows GetConsoleCP()
function returns zero (and os.device_encoding(1) returns 'cp0').
- pythonw.exe has no console either; but in pythonrun.c, the test
(fileno(stdin) < 0) is true, and the standard streams are all set to None.
- It is probable that Apache has redefined stdin & co, so the previous
test does not work there.

As a workaround, I suggest to set the environment variable
PYTHONIOENCODING before starting Apache, or before the call to
Py_Initialize.

----------
nosy: +amaury.forgeotdarc
Graham Dumpleton 1247830334Fri, 17 Jul 2009 11:32:14 +0000 (UTC)
Graham Dumpleton  added the comment:

Yes, Apache remaps stdout and stderr to the Apache error log to still 
capture anything that errant modules don't log via the Apache error log 
functions. In mod_wsgi it replaces sys.stdout and sys.stderr with special 
file like objects that redirect via Apache error logging functions. This 
though obviously happens after Python first initialises sys.stdout and 
sys.stderr.

What would be an appropriate value to set PYTHONIOENCODING to on Windows 
as a workaround?

----------
Amaury Forgeot d'Arc 1247863892Fri, 17 Jul 2009 20:51:32 +0000 (UTC)
Amaury Forgeot d'Arc  added the comment:

On a Western Windows, I suggest
    PYTHONIOENCODING=cp1252:backslashreplace

But 
    PYTHONIOENCODING=mbcs
is also OK, except that characters outside the Windows code page will be 
replaced with '?'

----------
Graham Dumpleton 1248264915Wed, 22 Jul 2009 12:15:15 +0000 (UTC)
Graham Dumpleton  added the comment:

The workaround of using:

#if defined(WIN32) && PY_MAJOR_VERSION >= 3
        _wputenv(L"PYTHONIOENCODING=cp1252:backslashreplace");
#endif

        Py_Initialize();

gets around the crash on startup.

I haven't done sufficient testing to know if this may introduce any other 
problems given that one is overriding default I/O encoding for whole 
process.

----------
Antoine Pitrou 1249470068Wed, 05 Aug 2009 11:01:08 +0000 (UTC)
Antoine Pitrou  added the comment:

Graham, is the workaround ok for you or do you think this is something
Python itself should handle?

----------
nosy: +pitrou
Graham Dumpleton 1249474177Wed, 05 Aug 2009 12:09:37 +0000 (UTC)
Graham Dumpleton  added the comment:

Python should be as robust as possible and thus should be fixed, but I am 
happy with using the workaround so if this is more a question of what 
priority to mark this, I wouldn't see it as being urgent.

----------
Florent Xicluna 1261319726Sun, 20 Dec 2009 14:35:26 +0000 (UTC)
Florent Xicluna  added the comment:

Patch to prevent crash when PYTHONIOENCODING is invalid:

~ $ PYTHONIOENCODING= ./python 
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: 
Abandon

----------
keywords: +patch
nosy: +flox
versions: +Python 3.2
Added file: http://bugs.python.org/file15623/issue6501_PY...
Antoine Pitrou 1261331101Sun, 20 Dec 2009 17:45:01 +0000 (UTC)
Antoine Pitrou  added the comment:

Well, this might prevent the crash but how does the system behave
afterwards? Do the standard streams use utf-8 by default?
At the minimum, we should still output a warning on stderr.

----------
Alexander Belopolsky 1296682361Wed, 02 Feb 2011 21:32:41 +0000 (UTC)
Alexander Belopolsky  added the comment:

The issue is not Windows specific, so I am changing the title to reflect that.   On OSX, for example, I get

$ PYTHONIOENCODING=xyz ./python.exe
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: xyz
Abort trap

I agree that abort() is too drastic for a typo in the environment variable setting, but ignoring it silently is not a good option either.  Someone setting PYTHONIOENCODING most likely does it for a reason and giving him or her some sort of default behavior for mistyped encoding is not helpful.  (Note that this is how many C libraries treat TZ environment variable setting and this is often very frustrating.)

I think errors in environment variables that can be detected on startup should be treated the same way as the command line typos: a descriptive message on C stderr and exit(1).

Currently different environment variables are treated differently.  For example, mistakes in PYTHONHOME and PYTHONIOENCODING cause fatal error while an error in PYTHONSTARTUP is reported but does not terminate python:

$ PYTHONSTARTUP=xyz.py ./python.exe
Python 3.2rc2+ (py3k:88320, Feb  2 2011, 14:07:18) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Could not open PYTHONSTARTUP
IOError: [Errno 2] No such file or directory: 'xyz.py'
>>>

----------
nosy: +belopolsky
title: Fatal LookupError: unknown encoding: cp0 on Windows embedded startup. -> Fatal error on startup with invalid PYTHONIOENCODING_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Éric Araujo 1300673276Mon, 21 Mar 2011 02:07:56 +0000 (UTC)
Changes by Éric Araujo :


----------
nosy: +haypo_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Daniel Goertzen 1301670281Fri, 01 Apr 2011 15:04:41 +0000 (UTC)
Daniel Goertzen  added the comment:

I run into this problem when I start a Python app as a subprocess from Erlang (open_port() function).  The PYTHONIOENCODING fix works when I launch my py app via pythonw.exe, but it does *not* work when I use the cx-freeze version of the app.

I am using the Win32GUI base for cx-freeze which appears to be a thin WinMain() wrapper around Py_Initialize().  I am going to continue investigating the cx-freeze related problems.

I am using Python 3.2 under Windows.  The failure is basically silent under Windows (generic MSVC runtime error), so I wasted a lot time figuring out what the problem actually was.  Python 2 doesn't seem to have this problem.

----------
nosy: +Daniel.Goertzen_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Anthony Tuininga 1301681886Fri, 01 Apr 2011 18:18:06 +0000 (UTC)
Changes by Anthony Tuininga :


----------
nosy: +atuining_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Daniel Goertzen 1301802026Sun, 03 Apr 2011 03:40:26 +0000 (UTC)
Daniel Goertzen  added the comment:

It turns out that cx-freeze deliberately sets Py_IgnoreEnvironmentFlag to ensure that the final executable is really an isolated, standalone executable (ie, it can't be subverted by setting PYTHONPATH.)  Therefore the PYTHONIOENCODING work-around does not work in this situation.

I am currently using a cx-freeze work-around from the author to enable the PYTHONIOENCODING work-around.  Altogether not that pleasant.

Could Python 3 could just default to some reasonable encoding and keep on chugging?

----------_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
STINNER Victor 1304293585Sun, 01 May 2011 23:46:25 +0000 (UTC)
STINNER Victor  added the comment:

> Fatal Python error: Py_Initialize: can't initialize sys standard streams
> LookupError: unknown encoding: cp0That's a bug in os.device_encoding(): os.device_encoding(sys.stdout.fileno()) should return None if the application has no console (if sys.stdout is not a Windows console stream).

Attached device_encoding.patch should fix this issue. (I didn't test the patch yet.)

----------
Added file: http://bugs.python.org/file21849/device_encod..._______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
STINNER Victor 1304293948Sun, 01 May 2011 23:52:28 +0000 (UTC)
STINNER Victor  added the comment:

> On a Western Windows, I suggest
>    PYTHONIOENCODING=cp1252:backslashreplaceWhy using this very small charset whereas a web server can use UTF-8?

I don't think that using backslashreplace on stdout is a good idea.> But 
>    PYTHONIOENCODING=mbcs
> is also OK, except that characters outside the Windows code
> page will be replaced with '?'Starting at Python 3.2, you should use mbcs:replace to replace unencodable characters by '?'. The strict error handler is now strict: it raises a UnicodeEncodeError if a character is not encodable to mbcs.

Note: mbcs is the ANSI code page.

--

Using device_encoding.patch, I suppose that sys.std* streams will use the ANSI code page (mbcs, which is the code page 1252 on a Western Windows setup) in grahamd's usecase (Python program running in Apache).

----------_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Roundup Robot 1306167219Mon, 23 May 2011 16:13:39 +0000 (UTC)
Roundup Robot <devnull@devnull> added the comment:

New changeset 5783a55a2418 by Victor Stinner in branch 'default':
Issue #6501: os.device_encoding() returns None on Windows if the application
http://hg.python.org/cpython/rev/5783a55a2418

----------
nosy: +python-dev_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
STINNER Victor 1306167448Mon, 23 May 2011 16:17:28 +0000 (UTC)
STINNER Victor  added the comment:

@grahamd: Can you try the development version of Python 3.3, or try to patch your version using device_encoding.patch? You will not get cp0 encoding anymore.

If the patch fixes your issue, I will backport it. I don't see anything interesting to do for this issue.

----------_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Éric Araujo 1321362412Tue, 15 Nov 2011 13:06:52 +0000 (UTC)
Éric Araujo  added the comment:

> Currently different environment variables are treated differently.  For example,
> mistakes in PYTHONHOME and PYTHONIOENCODING cause fatal error while an error in
> PYTHONSTARTUP is reported but does not terminate python:If PYTHONSTARTUP is the only envvar with non-fatal errors, I think it’s okay.  PYTHONHOME contains vital information, PYTHONIOENCODING is set by the programmer/admin and their code probably depends on it, but PYTHONSTARTUP is just niceties for the interactive interpreter, so non-vital IMO.

----------
nosy: +eric.araujo
versions: +Python 2.7, Python 3.3 -Python 3.1_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Mark Lawrence 1386354460Fri, 06 Dec 2013 18:27:40 +0000 (UTC)
Mark Lawrence added the comment:

Is there anything to backport as referred to in msg136670 or can this be closed?

----------
nosy: +BreamoreBoy_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Mark Lawrence 1391437777Mon, 03 Feb 2014 14:29:37 +0000 (UTC)
Changes by Mark Lawrence :


----------
nosy:  -BreamoreBoy_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
STINNER Victor 1392283144Thu, 13 Feb 2014 09:19:04 +0000 (UTC)
STINNER Victor added the comment:

> Is there anything to backport as referred to in msg136670 or can this be closed?

The fix is present in Python 3.3 since Python 3.3.0 according to the changelog. Python 3.2 doesn't accept bugfixes anymore.

Python 2.7 doesn't have the function os.device_encoding(), I don't think that it is affected.

There is nothing more to do, I'm closing the bug. Thanks for the report.

----------
resolution:  -> fixed
status: open -> closed
versions:  -Python 2.7, Python 3.2_______________________________________
Python tracker 
<http://bugs.python.org/issue6501>
Ad
Home | About | Privacy