ArchiveOrangemail archive

bazaar discussion


bazaar.lists.canonical.com
(List home) (Recent threads) (4 other Canonical lists)

Subscription Options

  • RSS or Atom: Read-only subscription using a browser or aggregator. This is the recommended way if you don't need to send messages to the list. You can learn more about feed syndication and clients here.
  • Conventional: All messages are delivered to your mail address, and you can reply. To subscribe, send an email to the list's subscribe address with "subscribe" in the subject line, or visit the list's homepage here.
  • Low traffic list: less than 3 messages per day
  • This list contains about 75,577 messages, beginning Mar 2005
  • 1 messages added yesterday
Report the Spam
This button sends a spam report to the moderator. Please use it sparingly. For other removal requests, read this.
Are you sure? yes no

Python 3

Ad
Robert Collins 1277243285Tue, 22 Jun 2010 21:48:05 +0000 (UTC)
I've done a couple of trivial patches moving in the direction of being
able to play with Python 3 and bzr.

I'd like to document some changes to our coding style, to make
eventual Python 3 migration easier - but they aren't all no-brainers
so I thought I'd raise stuff here and we can discuss it.

The no brainers (simple, not terribly ugly, not used a lot in our code):
octal 0666 ->0o666
print foo, bar -> print("%s %s" % (foo, bar))
exec foo, locals() -> exec(foo, locals())

Somewhat ugly - we're going to see a lot of these:
except Foo, e:
->
except Foo:
    e = sys.exc_info()[1]

bytestrings:
This has the potential to slow load times slightly: in 3 to get a
bytestring one says b'foo', but you can't do that in 2, so for
single-source compatibility it gets written as:
 _b('foo')

which on 2 is a no-op, and on 3 reencodes using latin-1 (so everything
works). Or we could split out separate files to import on 2 and 3, but
I think the extra seek would make it a wash perf wise.

really ugly, but very few occurences:
    raise type, val, tb
->
    bzrlib.util.py._builtin._reraise(type, val, tb)

Now, there are a lot of other things that we will have to solve and
talk about, this is really top level mechanical stuff and should not
be taken as the whole list or a magic bullet.

However, I think changing our coding style this and not much more will
be enough to let interested people slowly push forward and get things
like:
 - the test suite
 - C / pyrex modules
and so forth working on 3.

I think that ideally, in a year or so we'd be in a position to make a
concerted push to make 3 a first class citizen (because 3 is getting
considerable upstream and in-distribution attention).

If the consensus from this thread is that this is ok, I'll update
HACKING docs appropriately.

-Rob
Barry Warsaw 1277245268Tue, 22 Jun 2010 22:21:08 +0000 (UTC)
On Jun 23, 2010, at 09:47 AM, Robert Collins wrote:

>I've done a couple of trivial patches moving in the direction of being
>able to play with Python 3 and bzr.\o/(quoting out of order...)

>If the consensus from this thread is that this is ok, I'll update
>HACKING docs appropriately.HACKING.txt doesn't describe the minimum Python version for which
compatibility must be maintained.  Perhaps it should?  Maybe it's described in
a different document?  It does *imply* in a few places that Python 2.4 is the
minimum version.  Understandable of course, but darn.>which on 2 is a no-op, and on 3 reencodes using latin-1 (so everything
>works). Or we could split out separate files to import on 2 and 3, but
>I think the extra seek would make it a wash perf wise.Or split it out for Python 2.4/2.6+ compatibility?  At least you can write
idioms for the latter that 2to3 can more easily transform.  E.g...>The no brainers (simple, not terribly ugly, not used a lot in our code):
>octal 0666 ->0o666
>print foo, bar -> print("%s %s" % (foo, bar))from __future__ import print_function
print(foo, bar)>exec foo, locals() -> exec(foo, locals())
>
>Somewhat ugly - we're going to see a lot of these:
>except Foo, e:except Foo as e:>->
>except Foo:
>    e = sys.exc_info()[1]
>
>bytestrings:
>This has the potential to slow load times slightly: in 3 to get a
>bytestring one says b'foo', but you can't do that in 2, so for
>single-source compatibility it gets written as:
> _b('foo')b'foo' FTW. :)>
>really ugly, but very few occurences:
>    raise type, val, tb
>->
>    bzrlib.util.py._builtin._reraise(type, val, tb)raise

-Barry
Robert Collins 1277246460Tue, 22 Jun 2010 22:41:00 +0000 (UTC)
Thanks for the feedback; some data inline below.On Wed, Jun 23, 2010 at 10:20 AM, Barry Warsaw  wrote:

>>If the consensus from this thread is that this is ok, I'll update
>>HACKING docs appropriately.
>
> HACKING.txt doesn't describe the minimum Python version for which
> compatibility must be maintained.  Perhaps it should?  Maybe it's described in
> a different document?  It does *imply* in a few places that Python 2.4 is the
> minimum version.  Understandable of course, but darn.perhaps it should, certainly we should have it written down somewhere.
overview and plugin-api reference 2.4 but there is no statement - we
should also advise our users more clearly.
https://bugs.edge.launchpad.net/bzr/+bug/5974...>>which on 2 is a no-op, and on 3 reencodes using latin-1 (so everything
>>works). Or we could split out separate files to import on 2 and 3, but
>>I think the extra seek would make it a wash perf wise.
>
> Or split it out for Python 2.4/2.6+ compatibility?  At least you can write
> idioms for the latter that 2to3 can more easily transform.  E.g...I think if we're going for separate files, we'd want a real perf win,
because adding (say) 10% more files to bzrlib would suck, and user
strings (vs protocol definitions) are fairly mutable. That is, we
tweak a lot.>>The no brainers (simple, not terribly ugly, not used a lot in our code):
>>octal 0666 ->0o666
>>print foo, bar -> print("%s %s" % (foo, bar))
>
> from __future__ import print_function
> print(foo, bar)Its a little nicer, but for print statements this implies two copies
of the literal foo to update - see under maintenance burden.>>exec foo, locals() -> exec(foo, locals())
>>
>>Somewhat ugly - we're going to see a lot of these:
>>except Foo, e:
>
> except Foo as e:that is nicer, but two copies of the same code isn't. If we could hook
into the python compiler and just monkey patch it on 2.4->2.5, that
would be nice.>>->
>>except Foo:
>>    e = sys.exc_info()[1]
>>
>>bytestrings:
>>This has the potential to slow load times slightly: in 3 to get a
>>bytestring one says b'foo', but you can't do that in 2, so for
>>single-source compatibility it gets written as:
>> _b('foo')
>
> b'foo' FTW. :)Yes. For protocol stuff where this is relevant, we could split the
source - as above though, thats a pretty significant burden to wear.>>
>>really ugly, but very few occurences:
>>    raise type, val, tb
>>->
>>    bzrlib.util.py._builtin._reraise(type, val, tb)
>
> raiseNo, doesn't do what we need in any python version :)

We use 'raise', where we need to, but we also do reraise in a couple
of places, and switching it to raise there wouldn't work. Or we'd use
'raise' already :).

-Rob
Barry Warsaw 1277254060Wed, 23 Jun 2010 00:47:40 +0000 (UTC)
On Jun 23, 2010, at 10:40 AM, Robert Collins wrote:

>that is nicer, but two copies of the same code isn't. If we could hook
>into the python compiler and just monkey patch it on 2.4->2.5, that
>would be nice.Indeed, that would be!
-Barry
John Arbash Meinel 1277247381Tue, 22 Jun 2010 22:56:21 +0000 (UTC)
Barry Warsaw wrote:
> On Jun 23, 2010, at 09:47 AM, Robert Collins wrote:
> 
>> I've done a couple of trivial patches moving in the direction of being
>> able to play with Python 3 and bzr.
> 
> \o/
> 
> (quoting out of order...)
> 
>> If the consensus from this thread is that this is ok, I'll update
>> HACKING docs appropriately.
> 
> HACKING.txt doesn't describe the minimum Python version for which
> compatibility must be maintained.  Perhaps it should?  Maybe it's described in
> a different document?  It does *imply* in a few places that Python 2.4 is the
> minimum version.  Understandable of course, but darn.
> 
>> which on 2 is a no-op, and on 3 reencodes using latin-1 (so everything
>> works). Or we could split out separate files to import on 2 and 3, but
>> I think the extra seek would make it a wash perf wise.
> 
> Or split it out for Python 2.4/2.6+ compatibility?  At least you can write
> idioms for the latter that 2to3 can more easily transform.  E.g...So we discussed moving to even 2.5 a while back, and basically Redhat's
latest Enterprise versus (RHEL5) is supporting python2.4 until 2014 or so...

http://www.redhat.com/docs/manuals/enterprise...
and
http://www.redhat.com/security/updates/errata...

And RHEL6 isn't even out yet (and will move to python2.6, IIRC).

Down at the bottom it shows that RHEL5 is in primary support until March
2011, and secondary/tertiary support until 2014. And if you track it
through, RHEL5 only has 2.4 available.

Being tied to 2.4 for at least 1 more year, if not 4 more years, is
really going to hurt switching to python3. As having 2.4+3 compatibility
is just hard.

John
=:->
Barry Warsaw 1277253786Wed, 23 Jun 2010 00:43:06 +0000 (UTC)
On Jun 22, 2010, at 05:55 PM, John Arbash Meinel wrote:

>Being tied to 2.4 for at least 1 more year, if not 4 more years, is
>really going to hurt switching to python3. As having 2.4+3 compatibility
>is just hard.Definitely.  It doesn't just affect Bazaar though, it hurts every Python
application that wants to run on supported versions of RHEL5.

The only remotely possible solution would be to cut off Python < 2.6 support
in Bazaar at some point but ensure that newer versions are still wire
compatible with older versions for as long as you need.  Then, RHEL5 users
could still run the old clients and servers while folks that talk to them with
newer versions of Bazaar would still be able to do so.

Not fun though.
- -Barry
Robert Collins 1277254088Wed, 23 Jun 2010 00:48:08 +0000 (UTC)
On Wed, Jun 23, 2010 at 12:42 PM, Barry Warsaw  wrote:
>>Being tied to 2.4 for at least 1 more year, if not 4 more years, is
>>really going to hurt switching to python3. As having 2.4+3 compatibility
>>is just hard.
>
> Definitely.  It doesn't just affect Bazaar though, it hurts every Python
> application that wants to run on supported versions of RHEL5.Yup. And see recent threads on Python-dev too. Even now, years later,
the complete break in Python 3 is still a source of angst.> The only remotely possible solution would be to cut off Python < 2.6 support
> in Bazaar at some point but ensure that newer versions are still wire
> compatible with older versions for as long as you need.  Then, RHEL5 users
> could still run the old clients and servers while folks that talk to them with
> newer versions of Bazaar would still be able to do so.
>
> Not fun though.Very not fun - it transfers a burden (work with older python) to a
burden (under no circumstances stop supporting XYZ network call); it
also adds a burden to plugin developers - 'if you want your plugin to
be usable with a contributor on RHEL, write two versions, because you
have to use two bzrib versions'.

It would be nice to avoid that. Perhaps 2.8 or 3.3 could be more compatible?

-Rob
Barry Warsaw 1277255380Wed, 23 Jun 2010 01:09:40 +0000 (UTC)
On Jun 23, 2010, at 12:47 PM, Robert Collins wrote:

>It would be nice to avoid that. Perhaps 2.8 or 3.3 could be more compatible?If only.  But there almost definitely won't be a Python 2.8.

-Barry
Stephen J. Turnbull 1277260819Wed, 23 Jun 2010 02:40:19 +0000 (UTC)
Robert Collins writes:

> Yup. And see recent threads on Python-dev too. Even now, years later,
> the complete break in Python 3 is still a source of angst.True, but the "even now" is inappropriate.  This angst was predictable
(though nobody said "we expect much weeping and gnashing of teeth" out
loud that I recall), and *is expected to continue* for at least another
3-5 years as various libraries transition to Python 3 (thus enabling
new generations of app developers to make their own transitions and
start gnashing their teeth as they do).

I commend you personally for your forward-looking spirit.  But Bazaar
as a project shouldn't hurry at this point.> It would be nice to avoid that. Perhaps 2.8 or 3.3 could be more
> compatible?As Barry says, 2.8 is clearly a dead letter within python-dev, and as
yet I've seen no calls (not even trolling) for a separate project to
do a 2.8.  Of course, 2.7 is a pretty nice language, most libraries
will maintain compatible versions at least on PyPI, so pressure for a
2.8 (from any source) is at least a year away, I think.  It's not
clear what the theme of 3.3 is going to be, but with the ongoing
language freeze, I suspect compatibility efforts would not fare well.

The most common library strategy seems to be supporting two library
versions, with multiple versions of some files and a gradual
refactoring concentrating wrappers for the incompatible interfaces in
a small number of modules.  As Robert proposed in his initial post.
So that's good.

But 2to3 gets better all the time, and I think 3to2 is expected for
3.2?  So it's not at all a panacea, but instead of targeting "one code
base for Python 2 and Python 3", maybe the thing to do is to target
"one code base for Python 2 and 2to3", and try to autogenerate as much
of the Python 3 code base as possible.  Presumably as 3to2 improves,
you should be able to transition that to target "one code base for
3to2 and Python 3".

But what worries me is the Python 2 str vs. Python 3 str set of
issues, because they're hard to flag, let alone automatically
transition.  Do you have a sense for how pervasive that is in the
Bazaar codebase?  Is Unicode usage mostly encapsulated in a few
functions, primarily used in the UI?  Or is it more widespread?  How
about functionality like diff?  Is that affected?

(Again, this isn't something you need to deal with immediately, a
Python 3 transition will take years, but I think it's worth thinking a
little about.)
Martin Pool 1277254713Wed, 23 Jun 2010 00:58:33 +0000 (UTC)
On 23 June 2010 08:20, Barry Warsaw  wrote:
>>If the consensus from this thread is that this is ok, I'll update
>>HACKING docs appropriately.
>
> HACKING.txt doesn't describe the minimum Python version for which
> compatibility must be maintained.  Perhaps it should?  Maybe it's described in
> a different document?  It does *imply* in a few places that Python 2.4 is the
> minimum version.  Understandable of course, but darn.It is described in
<http://doc.bazaar.canonical.com/bzr.dev/devel...>.
 I added that a while ago in response to some merge proposals that
were not 2.4 compatible because people didn't realize we needed it.

We had a discussion a while ago that settled that we wanted to keep
2.4 compatibility at least for a while.

Based on what I read at the time it seemed to me the next step was to
get things clean with -3, then perhaps test it remains so in Babune,
then try getting the tests to run under 2to3 in at least pure python
mode.  Many of these cleanups will improve our code even under 2.x.Robert said
>  - 2to3 means an indeterministic program is actually runIn what sense is it indeterminisitic?
Martin Pool 1277254934Wed, 23 Jun 2010 01:02:14 +0000 (UTC)
On 23 June 2010 10:58, Martin Pool  wrote:
> Many of these cleanups will improve our code even under 2.x.On the other hand, Robert's patch has things like

-except locale.Error, e:
+except locale.Error:
+    e = sys.exc_info()[1]

which is clearly worse, and clearly also the kind of thing that can be
easily mechanically translated by 2to3.  I wouldn't like to see that
kind of thing merged until we've decided we really want a single
codebase that works on both with no translation.
John Arbash Meinel 1277254999Wed, 23 Jun 2010 01:03:19 +0000 (UTC)
Martin Pool wrote:
> On 23 June 2010 10:58, Martin Pool  wrote:
>>  Many of these cleanups will improve our code even under 2.x.
> 
> On the other hand, Robert's patch has things like
> 
> -except locale.Error, e:
> +except locale.Error:
> +    e = sys.exc_info()[1]
> 
> which is clearly worse, and clearly also the kind of thing that can be
> easily mechanically translated by 2to3.  I wouldn't like to see that
> kind of thing merged until we've decided we really want a single
> codebase that works on both with no translation.
>I thought 2to3 doesn't actually handle that. As it expects you've
written python2.6 compatible code which uses:

 except locale.Error as e:

Which we *can't* do for 2.4 or 2.5

John
=:->
Barry Warsaw 1277255277Wed, 23 Jun 2010 01:07:57 +0000 (UTC)
On Jun 22, 2010, at 08:02 PM, John Arbash Meinel wrote:

>I thought 2to3 doesn't actually handle that. As it expects you've
>written python2.6 compatible code which uses:
>
> except locale.Error as e:
>
>Which we *can't* do for 2.4 or 2.5Note that 2to3 is extensible, and it's not too hard to write custom fixers.
Not sure that will help here, but just FYI.

- -Barry
Martin Pool 1277255639Wed, 23 Jun 2010 01:13:59 +0000 (UTC)
On 23 June 2010 11:07, Barry Warsaw  wrote:>
> On Jun 22, 2010, at 08:02 PM, John Arbash Meinel wrote:
>
>>I thought 2to3 doesn't actually handle that. As it expects you've
>>written python2.6 compatible code which uses:
>>
>> except locale.Error as e:
>>
>>Which we *can't* do for 2.4 or 2.5It does actually handle this already:

--- /tmp/t.py (original)
+++ /tmp/t.py (refactored)
@@ -1,4 +1,4 @@
 try:
     pass
-except Exception, e:
-    print e
+except Exception as e:
+    print(e)
RefactoringTool: Files that need to be modified:
RefactoringTool: /tmp/t.py

If we need to make changes that either don't work under 2.[456] or
that make the code worse, and that are simple to automate, we would be
better off putting that into 2to3 than doing it once-off through a
search and replace.
Robert Collins 1277257348Wed, 23 Jun 2010 01:42:28 +0000 (UTC)
2to3 is indeterministic in the sense that users will have different
versions of it, with different capabilities.

I think 2to3 is a fine tool for small projects with no serious quirks,
but we aren't one of those. I enumerated all the issues I see with
2to3 earlier in the thread, so I won't repeat that now; but I think
that they need to be addressed before we can consider using 2to3 for
anything other than mechanical translation. Fixing things that are
outside of what 2to3 can do is likely a good idea anyway.

For the bytes stuff I've mentioned - see
http://bugs.python.org/issue5425 - 2to3 converts str to 'string' not
'bytes', so we have to mark up everything, and only Python 2.6
supports b'foo'. If we change the importer to go to bytes for 'str',
then we end up with all our literals that are meant for humans as
bytes, which is also wrong. And while we support < 2.6, we can't
annotate strings properly for bytes recognition without something like
a domain specific language - e.g. a _b() function.

We're a library, with very different concerns about when we stop
supporting a version of Python to a simple end user application that
doesn't care about embedding or reuse - and typically ships an entire
Python stack anyway. AFAICT 2to3 is *entirely* driven by 'hey, you can
*switch* over easily', not by ongoing maintenance concerns; the 'add
2to3 as a compile step in your application' is deeply unsatisfying

http://www.slideshare.net/regebro/python-3-co...
and http://wiki.python.org/moin/PortingDjangoTo3k are interesting
reads and had some hand in shaping my opinion on this. Note that the
latter is a patch - you can get an impression of how much stuff 2to3
*didn't* do there.

-Rob
Martin Pool 1277259152Wed, 23 Jun 2010 02:12:32 +0000 (UTC)
On 23 June 2010 11:42, Robert Collins  wrote:
> 2to3 is indeterministic in the sense that users will have different
> versions of it, with different capabilities.That's a funny definition of "indeterministic" but anyhow, it's true
they may vary in the same kind of way that people will have different
pythons, kernels, pyrexes, libcs, etc.  What the magnitude of those
differences may be, I don't know.  It first came in to python2.6
afaict so we only have that and then later updates in 3.0 and 3.1.

I suppose like for Pyrex we would have the choice of either
distributing the converted code or getting people to run it.  If we
distribute the converted code then we could fix it to some particular
version.  If we expect the user to run it then perhaps we could say
"bzr is supported on python3.0 using python3.0-2to3".>
> I think 2to3 is a fine tool for small projects with no serious quirks,
> but we aren't one of those. I enumerated all the issues I see with
> 2to3 earlier in the thread, so I won't repeat that now; but I think
> that they need to be addressed before we can consider using 2to3 for
> anything other than mechanical translation. Fixing things that are
> outside of what 2to3 can do is likely a good idea anyway.You mention that

> - folk will want 3 before there are no platforms that have python2.x and not python3.x support.

True but irrelevant

> - 2to3 means an indeterministic program is actually run

It's an algorithm, it's not indeterministic.  There may be different
versions with different bugs but I doubt it will be a larger problem
than we've had with other dependencies, and probably smaller.

> - I like to be able to properly support things

Me too, but this is also irrelevant.  We can say we support particular
2to3 versions in the same way we support particular interpreter
versions.  But more to the point we can get things passing on 3 before
we commit to supporting anything.

> - distros would prefer to ship a single 'bzr' package rather than two separate ones - and because they run with precompiled files not writable by the user...

This is going to be complicated for distros, but it's a somewhat
similar problem to needing binary extensions built for different
interpreters, which they can do.>
> For the bytes stuff I've mentioned - see
> http://bugs.python.org/issue5425 - 2to3 converts str to 'string' not
> 'bytes', so we have to mark up everything, and only Python 2.6
> supports b'foo'. If we change the importer to go to bytes for 'str',
> then we end up with all our literals that are meant for humans as
> bytes, which is also wrong. And while we support < 2.6, we can't
> annotate strings properly for bytes recognition without something like
> a domain specific language - e.g. a _b() function.This seems like a problem with ever wanting a single codebase to run
across 2.4..3.x, whether you support 3.x directly or by 2to3, but
probably soluble along the lines discussed in the Django page: add a
new clearer marker for "really bytes".>
> We're a library, with very different concerns about when we stop
> supporting a version of Python to a simple end user application that
> doesn't care about embedding or reuse - and typically ships an entire
> Python stack anyway. AFAICT 2to3 is *entirely* driven by 'hey, you can
> *switch* over easily', not by ongoing maintenance concerns; the 'add
> 2to3 as a compile step in your application' is deeply unsatisfyingThe document Barry quoted earlier is not suggesting that you switch,
but rather you run it as a compile step.  Additional compile steps are
not great but not awful, and better than having bad code that you have
to actually edit.> http://www.slideshare.net/regebro/python-3-co...
> and http://wiki.python.org/moin/PortingDjangoTo3k are interesting
> reads and had some hand in shaping my opinion on this. Note that the
> latter is a patch - you can get an impression of how much stuff 2to3
> *didn't* do there.Slide 11 and slide 50 suggest libraries develop on 2 and used 2to3 to
support 3.

So the course I would like to follow is the one suggested by python.org:

 * get things clean with -3
 * get the test suite running under 2to3
 * assess at this point whether we want to support actual use under 3
and/or have a separate branch and/or cut over to requiring 2.6 or 3
Stephen J. Turnbull 1277269159Wed, 23 Jun 2010 04:59:19 +0000 (UTC)
Martin Pool writes:

> That's a funny definition of "indeterministic" but anyhow, it's true
> they may vary in the same kind of way that people will have different
> pythons, kernels, pyrexes, libcs, etc.  What the magnitude of those
> differences may be, I don't know.  It first came in to python2.6
> afaict so we only have that and then later updates in 3.0 and 3.1.The differences across versions were pretty large in my experience
(all pre-3.1, though; maybe it's more stable now).  You might want to
distribute a particular version with Bazaar to get a reasonably
reliable output.  I'm not sure how much the 2to3 application itself
changes, but the ruleset gets better.

You'd also probably really want to distribute specific customized
rules to handle bzr-ish idioms, anyway.> I suppose like for Pyrex we would have the choice of either
> distributing the converted code or getting people to run it.I don't think having people run 2to3 themselves is a reasonable idea.
It is not intended to be a Python2 to Python3 compiler the way Pyrex
is a Python to C compiler, at least not yet.  At least at first (ie
during the alpha period when you're still learning about bzr idioms
that 2to3 doesn't handle OOTB and developing bzr-oriented custom
rules), there will need to be some post 2to3 hand-tweaking.  Cf.
Martin van Löwis's Django port and patch (cited by Robert).> > - distros would prefer to ship a single 'bzr' package rather than
> >   two separate ones - and because they run with precompiled files
> >   not writable by the user...
> 
> This is going to be complicated for distros, but it's a somewhat
> similar problem to needing binary extensions built for different
> interpreters, which they can do.They're also going to need to do this for other libraries, such as
numpy, and probably a number of applications.  The distros can handle
it.
Robert Collins 1277272467Wed, 23 Jun 2010 05:54:27 +0000 (UTC)
On Wed, Jun 23, 2010 at 2:11 PM, Martin Pool  wrote:

Meta: I'm not pushing for 3, I did a quick test to see how hard it was
likely to be; we can stop this conversation and just wait and see, at
any point.

I'm fine with the steps you're proposing we do now; waiting and seeing
is a fine strategy, we certainly won't be significantly better or
worse off if we wait (though we may end up with more time-pressure if
3.0 support were to become an important issue in the future for some
reason).

However, I'm going to tug on the threads a little more, because I
think folk are misassessing some of the risks I see.> On 23 June 2010 11:42, Robert Collins  wrote:
>> 2to3 is indeterministic in the sense that users will have different
>> versions of it, with different capabilities.
>
> That's a funny definition of "indeterministic" but anyhow, it's true
> they may vary in the same kind of way that people will have different
> pythons, kernels, pyrexes, libcs, etc.  What the magnitude of those
> differences may be, I don't know.  It first came in to python2.6
> afaict so we only have that and then later updates in 3.0 and 3.1.
>
> I suppose like for Pyrex we would have the choice of either
> distributing the converted code or getting people to run it.  If we
> distribute the converted code then we could fix it to some particular
> version.  If we expect the user to run it then perhaps we could say
> "bzr is supported on python3.0 using python3.0-2to3".From the perspective of bug reports, I think it fits the dictionary
reasonably well; for the case where other people run 2to3 - the output
is not dependent purely on the input we supply (the bzr code) but
anyhow; the point was made and supported by Stephen - at least
historically, 2to3 has been a very variable thing.>> - folk will want 3 before there are no platforms that have python2.x and not python3.x support.
>
> True but irrelevantI dispute the irrelevancy; I wasn't clear enough, but AIUI 2to3 only
helps with 2.6->3.x, so if we still support 2.4, we will have
significant trouble supporting 3 from a single source tree using 2to3;
other options like a ported-trunk may work better at that point (and
the port could be to 2.6 w/2to3 at that stage)>> - 2to3 means an indeterministic program is actually run
>
> It's an algorithm, it's not indeterministic.  There may be different
> versions with different bugs but I doubt it will be a larger problem
> than we've had with other dependencies, and probably smaller.See above, 2to3 is a moving target, rather like rpython (what it can
handle is rpython, what it can't isn't).>> - I like to be able to properly support things
>
> Me too, but this is also irrelevant.  We can say we support particular
> 2to3 versions in the same way we support particular interpreter
> versions.  But more to the point we can get things passing on 3 before
> we commit to supporting anything.We've had bugs with pyrex output already, and they took more time to
analyse that would have been ideal - because the users pyrex version
output different, unexpected, things. I see this as similar - we have
an open task for someone to start versioning the pyrex files we put in
our tarballs, and that issue is part of it. So again, I don't see this
as irrelevant. It would be irrelevant if the migration and [python
version] support strategy we use had no impact on our ability to
diagnose and solve bugs.>> - distros would prefer to ship a single 'bzr' package rather than two separate ones - and because they run with precompiled files not writable by the user...
>
> This is going to be complicated for distros, but it's a somewhat
> similar problem to needing binary extensions built for different
> interpreters, which they can do.Stephen says numpy will force such support, but I fear that it will be
spotty, hard on users, and not very nice. It is a similar problem, but
in a different domain (package dependencies) and I can pretty much
guarantee that at least the Debian answer will be 'not pretty'. At
best. Of course this isn't a dominating factor for us, but its worth
thinking about a little.>> For the bytes stuff I've mentioned - see
>> http://bugs.python.org/issue5425 - 2to3 converts str to 'string' not
>> 'bytes', so we have to mark up everything, and only Python 2.6
>> supports b'foo'. If we change the importer to go to bytes for 'str',
>> then we end up with all our literals that are meant for humans as
>> bytes, which is also wrong. And while we support < 2.6, we can't
>> annotate strings properly for bytes recognition without something like
>> a domain specific language - e.g. a _b() function.
>
> This seems like a problem with ever wanting a single codebase to run
> across 2.4..3.x, whether you support 3.x directly or by 2to3, but
> probably soluble along the lines discussed in the Django page: add a
> new clearer marker for "really bytes".Its quite solvable yes, but using explicit markers - such as the _b() function.>> We're a library, with very different concerns about when we stop
>> supporting a version of Python to a simple end user application that
>> doesn't care about embedding or reuse - and typically ships an entire
>> Python stack anyway. AFAICT 2to3 is *entirely* driven by 'hey, you can
>> *switch* over easily', not by ongoing maintenance concerns; the 'add
>> 2to3 as a compile step in your application' is deeply unsatisfying
>
> The document Barry quoted earlier is not suggesting that you switch,
> but rather you run it as a compile step.  Additional compile steps are
> not great but not awful, and better than having bad code that you have
> to actually edit.I think this depends on the degree to which you trust your compile
steps :). I don't have a huge degree of trust in 2to3, based on the
anecdotal evidence of lurking in #python-dev and on the main python
developer list.> So the course I would like to follow is the one suggested by python.org:
>
>  * get things clean with -3I think that this is good to do regardless.

>  * get the test suite running under 2to3

No problem with doing that either :- but I do think that close care
should be paid to the output of 2to3>  * assess at this point whether we want to support actual use under 3
> and/or have a separate branch and/or cut over to requiring 2.6 or 3I'm fine with this too.

I really do believe that the dragons under the hood are all going to
be bytes vs string type related, and that we're going to find out the
hard way - e.g. by our test suite being internally consistent, but
externally faulty, due to the test suite getting cast up to string
too.

-Rob
Martin Pool 1277274675Wed, 23 Jun 2010 06:31:15 +0000 (UTC)
On 23 June 2010 15:54, Robert Collins  wrote:
> On Wed, Jun 23, 2010 at 2:11 PM, Martin Pool  wrote:
>
> Meta: I'm not pushing for 3, I did a quick test to see how hard it was
> likely to be; we can stop this conversation and just wait and see, at
> any point.
>
> I'm fine with the steps you're proposing we do now; waiting and seeing
> is a fine strategy, we certainly won't be significantly better or
> worse off if we wait (though we may end up with more time-pressure if
> 3.0 support were to become an important issue in the future for some
> reason).I think it's good to start getting some back-burner experience with
python3.  If nothing else it may improve the code in the same way gz's
non-cpython patches have.

It seems we have to choose between
 0 - use 2to3 as a compile step, and update the code so that it works
on 2.[456] and 2to3

You may well be right that 2to3 doesn't work well, or doesn't work
consistently across all installations, or leaves enough hard problems
unsolved that it is infeasible to have a single codebase across
2.3..3.1.  In that case we would have to choose between

 1- having one branch written in the common subset of 2.3..3.1
 2- dropping support for 2.4 (or also 2.5) in some bzr series
 3- having a separate 3.x branch

#2 and #3 are clearly possible but a bit unattractive, especially #2.
#0 and #1 may be either impossible or hard.

I'm reluctant to merge ugly or intrusive changes to trunk until we
know that we do want to do #1.  I think we can get closer before
deciding.>>> - folk will want 3 before there are no platforms that have python2.x and not python3.x support.
>>
>> True but irrelevant
>
> I dispute the irrelevancy; I wasn't clear enough, but AIUI 2to3 only
> helps with 2.6->3.x, so if we still support 2.4, we will have
> significant trouble supporting 3 from a single source tree using 2to3;
> other options like a ported-trunk may work better at that point (and
> the port could be to 2.6 w/2to3 at that stage)I guess we'll have to see whether there are any constructs where the
only way to spell it in 2.4 can't be translated by 2to3.>> So the course I would like to follow is the one suggested by python.org:
>>
>>  * get things clean with -3
>
> I think that this is good to do regardless.
>
>>  * get the test suite running under 2to3
>
> No problem with doing that either :- but I do think that close care
> should be paid to the output of 2to3
>
>>  * assess at this point whether we want to support actual use under 3
>> and/or have a separate branch and/or cut over to requiring 2.6 or 3
>
> I'm fine with this too.Great, let's do this then.> I really do believe that the dragons under the hood are all going to
> be bytes vs string type related, and that we're going to find out the
> hard way - e.g. by our test suite being internally consistent, but
> externally faulty, due to the test suite getting cast up to string
> too.Probably.  On the bright side it could make things cleaner to have a
clearer distinction.
Andrew Bennetts 1277276086Wed, 23 Jun 2010 06:54:46 +0000 (UTC)
Martin Pool wrote:
[...]
> You may well be right that 2to3 doesn't work well, or doesn't work
> consistently across all installations, or leaves enough hard problems
> unsolved that it is infeasible to have a single codebase across
> 2.3..3.1.  In that case we would have to choose between
> 
>  1- having one branch written in the common subset of 2.3..3.1
>  2- dropping support for 2.4 (or also 2.5) in some bzr series
>  3- having a separate 3.x branchOr, if the problem is just “not consistent across all installations” (as
opposed to 2to3 not working well for us at all):

 4- run 2to3 ourselves (using whichever version and extensions we
    consider best), and distribute the output.

IIRC, we already distribute tarballs with .c files that we have
generated from our .pyx source.  This doesn't strike me as much
different.

-Andrew.
Gordon Tyler 1277302297Wed, 23 Jun 2010 14:11:37 +0000 (UTC)
On 23/06/2010 2:54 AM, Andrew Bennetts wrote:
> Or, if the problem is just “not consistent across all installations” (as
> opposed to 2to3 not working well for us at all):
> 
>  4- run 2to3 ourselves (using whichever version and extensions we
>     consider best), and distribute the output.
> 
> IIRC, we already distribute tarballs with .c files that we have
> generated from our .pyx source.  This doesn't strike me as much
> different.This sounds like the best option to me.

Ciao,
Gordon
Stephen J. Turnbull 1277282784Wed, 23 Jun 2010 08:46:24 +0000 (UTC)
Robert Collins writes:

> >> - distros would prefer to ship a single 'bzr' package rather
> >>   than two separate ones - and because they run with precompiled
> >>   files not writable by the user...
> >
> > This is going to be complicated for distros, but it's a somewhat
> > similar problem to needing binary extensions built for different
> > interpreters, which they can do.
> 
> Stephen says numpy will force such support, but I fear that it will be
> spotty, hard on users, and not very nice.But note I also think the time frame for Bazaar to really get going on
Python 3 support is 2-3 years.  Yes, Debian & Ubuntu will take a long
time (~5 years?) to shake down on this, but numpy is not the only such
package.  They *have* to do something.  (More precisely, their Python
cabals need to do something, or the distro as a whole will move on.)
Gordon Tyler 1277301925Wed, 23 Jun 2010 14:05:25 +0000 (UTC)
On 22/06/2010 9:42 PM, Robert Collins wrote:
> For the bytes stuff I've mentioned - see
> http://bugs.python.org/issue5425 - 2to3 converts str to 'string' not
> 'bytes', so we have to mark up everything, and only Python 2.6
> supports b'foo'. If we change the importer to go to bytes for 'str',
> then we end up with all our literals that are meant for humans as
> bytes, which is also wrong. And while we support < 2.6, we can't
> annotate strings properly for bytes recognition without something like
> a domain specific language - e.g. a _b() function.Why aren't all string literals meant for users marked as unicode in the
current codebase?

Ciao,
Gordon
John Arbash Meinel 1277307987Wed, 23 Jun 2010 15:46:27 +0000 (UTC)
Gordon Tyler wrote:
> On 22/06/2010 9:42 PM, Robert Collins wrote:
>> For the bytes stuff I've mentioned - see
>> http://bugs.python.org/issue5425 - 2to3 converts str to 'string' not
>> 'bytes', so we have to mark up everything, and only Python 2.6
>> supports b'foo'. If we change the importer to go to bytes for 'str',
>> then we end up with all our literals that are meant for humans as
>> bytes, which is also wrong. And while we support < 2.6, we can't
>> annotate strings properly for bytes recognition without something like
>> a domain specific language - e.g. a _b() function.
> 
> Why aren't all string literals meant for users marked as unicode in the
> current codebase?
> 
> Ciao,
> Gordon
> 
>I would say the #1 reason (by a large margin) is that they "didn't need
to be". As such, someone needs to put in the time to evaluate whether it
is worthwhile.

There are also things like "unicode + str == unicode" so if you have
your hard-coded strings as plain 'str', then if you ever concatenate you
don't accidentally auto-cast.

There are also things like diff headers, where it is a bit more unclear
whether it is valid to have them as Unicode. (even though users see them)

I also don't know how long __unicode__ has existed, but certainly in the
*bzrlib* codebase we haven't done unicode(exception) we've done
str(exception).

This also gets a bit messy when dealing with OS errors. Where it is very
common to have the OS return a localized error message in some 8-bit
string, which we then want to combine with say a Unicode path.

John
=:->
Gordon Tyler 1277314133Wed, 23 Jun 2010 17:28:53 +0000 (UTC)
On 23/06/2010 11:45 AM, John Arbash Meinel wrote:
> There are also things like diff headers, where it is a bit more unclear
> whether it is valid to have them as Unicode. (even though users see them)This is perhaps due to my being a Java programmer for 10 years but it
seems quite logical to have the internal representation of a string in
Unicode and encode that to the required encoding when it "exits" the
application, i.e. written to stdout/file/socket/etc.

Of course, it may be tricky getting an external input into Unicode if
you're not certain what the encoding is.> I also don't know how long __unicode__ has existed, but certainly in the
> *bzrlib* codebase we haven't done unicode(exception) we've done
> str(exception).Apparently unicode(exception) won't do what you want it to do:
http://groups.google.com/group/comp.lang.pyth...> This also gets a bit messy when dealing with OS errors. Where it is very
> common to have the OS return a localized error message in some 8-bit
> string, which we then want to combine with say a Unicode path.The boilerplate to convert to unicode could be a somewhat tiresome, I
suppose. Although, would it not be possible to monkeypatch Exception to
add a __unicode__ method that does the right thing?

Ciao,
Gordon
Martin Pool 1277334000Wed, 23 Jun 2010 23:00:00 +0000 (UTC)
On 24 June 2010 03:28, Gordon Tyler  wrote:
> On 23/06/2010 11:45 AM, John Arbash Meinel wrote:
>> There are also things like diff headers, where it is a bit more unclear
>> whether it is valid to have them as Unicode. (even though users see them)(I think the illuminating example there is to consider a UTF-16
encoded diff - istm the headers should be UTF-16, but whether patch
can actually read such a thing is a different question.)> This is perhaps due to my being a Java programmer for 10 years but it
> seems quite logical to have the internal representation of a string in
> Unicode and encode that to the required encoding when it "exits" the
> application, i.e. written to stdout/file/socket/etc.
>
> Of course, it may be tricky getting an external input into Unicode if
> you're not certain what the encoding is.Java was designed from the start with a clear separation between
strings which are Unicode and byte arrays.  Python supported both but
in a kind of fuzzy way, with the default for literals being a byte
string in 2.0.

http://blog.labix.org/2009/07/02/screwing-up-...

Our general approach is also to normally convert to/from byte encoding
on the boundary but the 2.x environment means that either sometimes we
couldn't follow that approach consistently, or there may be latent
inadvertent cases where we don't follow it.  This is one reason why
getting the tests to pass under 2to3, even if we don't want to
officially support that, may find some interesting bugs.
John Arbash Meinel 1277337132Wed, 23 Jun 2010 23:52:12 +0000 (UTC)
...> http://blog.labix.org/2009/07/02/screwing-up-...
> 
> Our general approach is also to normally convert to/from byte encoding
> on the boundary but the 2.x environment means that either sometimes we
> couldn't follow that approach consistently, or there may be latent
> inadvertent cases where we don't follow it.  This is one reason why
> getting the tests to pass under 2to3, even if we don't want to
> officially support that, may find some interesting bugs.
>I'll also note that we used to have file-ids and revision-ids as Unicode
strings in memory, but intentionally switched back to 8-bit strings.
Mostly for performance/memory consumption. (Default Unicode on most
Unixes is UCS-4, which means we bloated every revision-id and file-id by
approximately 4x. And when you have 100,000+ of them, that gets pretty big)


John
=:->
Benjamin Peterson 1277258963Wed, 23 Jun 2010 02:09:23 +0000 (UTC)
John Arbash Meinel <john <at> arbash-meinel.com> writes:
> I thought 2to3 doesn't actually handle that. As it expects you've
> written python2.6 compatible code which uses:This is incorrect. 2to3 does translate it.
Russel Winder 1277272865Wed, 23 Jun 2010 06:01:05 +0000 (UTC)
Martin,On Wed, 2010-06-23 at 11:01 +1000, Martin Pool wrote:

> On the other hand, Robert's patch has things like
> 
> -except locale.Error, e:
> +except locale.Error:
> +    e = sys.exc_info()[1]
> 
> which is clearly worse, and clearly also the kind of thing that can be
> easily mechanically translated by 2to3.  I wouldn't like to see that
> kind of thing merged until we've decided we really want a single
> codebase that works on both with no translation.I think this post and all subsequent issues regarding this code fragment
are based on the wrong assumption and so much of it is effectively
wrong.  The recommended transformation is:

except locale.Error , e :

->

except locale.Error as e :

which is clearly more or less a no-op.  If 2-3 generates anything else
then it is broken.

Clearly the issue of supporting pre-2.6 Python have an impact here but
only if you are trying to use a single code base to deal with Pythons
1.5 -> 2.6.  The SCons folk seem to be the real masters at doing this,
but they have just given up on Pythons 1.5, 2.0->2.3 and now have 2.4 as
a floor.  The issue of Python 3 is now arising but not being addressed
just yet.  They do have a very interesting approach to using a single
code base for all supported Pythons, that might be useful input to the
debate -- or not.  Just a thought.
John Barstow 1277245993Tue, 22 Jun 2010 22:33:13 +0000 (UTC)
On Wed, Jun 23, 2010 at 9:47 AM, Robert Collins
 wrote:
> I've done a couple of trivial patches moving in the direction of being
> able to play with Python 3 and bzr.
>Hmmm. These all seem to ignore the porting guidelines
[http://docs.python.org/py3k/whatsnew/3.0.html...]
There's actually very little we would need to do if we followed those
guidelines.> I'd like to document some changes to our coding style, to make
> eventual Python 3 migration easier - but they aren't all no-brainers
> so I thought I'd raise stuff here and we can discuss it.
>
> The no brainers (simple, not terribly ugly, not used a lot in our code):
> octal 0666 ->0o666
> print foo, bar -> print("%s %s" % (foo, bar))
> exec foo, locals() -> exec(foo, locals())
>These are handled by the numliterals, print, and exec fixers. If we
change our coding style we will need to disable these fixers when
using 2to3.> Somewhat ugly - we're going to see a lot of these:
> except Foo, e:
> ->
> except Foo:
>    e = sys.exc_info()[1]
>Shouldn't that be "except Foo as e:"? Handled by the except fixer.> bytestrings:
> This has the potential to slow load times slightly: in 3 to get a
> bytestring one says b'foo', but you can't do that in 2, so for
> single-source compatibility it gets written as:
>  _b('foo')
>
> which on 2 is a no-op, and on 3 reencodes using latin-1 (so everything
> works). Or we could split out separate files to import on 2 and 3, but
> I think the extra seek would make it a wash perf wise.Well, Python 2.6 supports bytestring literals, so you really only need
to do this if you're committed to supporting 2.4/2.5 going forward.
Personally I think it would make sense to simply drop support for
anything older than 2.6 when we decide we want to start supporting
3.x, then you could just use bytestring literals.>
> really ugly, but very few occurences:
>    raise type, val, tb
> ->
>    bzrlib.util.py._builtin._reraise(type, val, tb)
>The raises fixer will rewrite this as "raise
type(val).with_traceback(tb)"; personally I find your proposal much
uglier.

So I guess that makes my counterproposal to adopt a minimum
requirement of Python 2.6 and use 2to3 to until such time as we drop
2.x support.
Robert Collins 1277247272Tue, 22 Jun 2010 22:54:32 +0000 (UTC)
We could - barely - consider dropping 2.4 (note that the oldest
supported RHEL, as I understand it, is still python 2.3 only).

I'm very sure we would be shooting ourselves in the foot to drop 2.5.

As for the extension guidelines, 'meh'. For reference they are:--
For porting existing Python 2.5 or 2.6 source code to Python 3.0, the
best strategy is the following:

(Prerequisite:) Start with excellent test coverage.
Port to Python 2.6. This should be no more work than the average port
from Python 2.x to Python 2.(x+1). Make sure all your tests pass.
(Still using 2.6:) Turn on the -3 command line switch. This enables
warnings about features that will be removed (or change) in 3.0. Run
your test suite again, and fix code that you get warnings about until
there are no warnings left, and all your tests still pass.
Run the 2to3 source-to-source translator over your source code tree.
(See 2to3 - Automated Python 2 to 3 code translation for more on this
tool.) Run the result of the translation under Python 3.0. Manually
fix up any remaining issues, fixing problems until all tests pass
again.
It is not recommended to try to write source code that runs unchanged
under both Python 2.6 and 3.0; you’d have to use a very contorted
coding style, e.g. avoiding print statements, metaclasses, and much
more. If you are maintaining a library that needs to support both
Python 2.6 and Python 3.0, the best approach is to modify step 3 above
by editing the 2.6 version of the source code and running the 2to3
translator again, rather than editing the 3.0 version of the source
code.
John Arbash Meinel 1277247462Tue, 22 Jun 2010 22:57:42 +0000 (UTC)
Robert Collins wrote:
> We could - barely - consider dropping 2.4 (note that the oldest
> supported RHEL, as I understand it, is still python 2.3 only).The problem isn't as much that the *oldest* is so old, but that the
*newest* actual release (RHEL5) only has 2.4. RHEL6 isn't due out for a
while yet.

John
=:->
John Arbash Meinel 1277247750Tue, 22 Jun 2010 23:02:30 +0000 (UTC)
> As for the extension guidelines, 'meh'. For reference they are:
> --...> It is not recommended to try to write source code that runs unchanged
> under both Python 2.6 and 3.0; you’d have to use a very contorted
> coding style, e.g. avoiding print statements, metaclasses, and much
> more. If you are maintaining a library that needs to support both
> Python 2.6 and Python 3.0, the best approach is to modify step 3 above
> by editing the 2.6 version of the source code and running the 2to3
> translator again, rather than editing the 3.0 version of the source
> code.
> --a) We actually try to avoid 'print' since we really want to be writing
to specific file output (and print >>foo is pretty ugly anyway)

b) we don't really use metaclasses. I know we use __new__ in one or two
places (revspec), but really only because we wanted to compatibly raise
an error if someone used __init__ directly.> 
> We haven't really ignored them - we are ported to 2.6, we have
> fantastic test coverage. Folk who are doing Python 3 work can use the
> -3 switch; I'm interested here in how we enable those folk to have an
> easy life, not in the mechanics of actually doing the work - that is a
> separate, albeit related, problem.
> 2to3 is a problem for us. Here's the problem:
>  - folk will want 3 before there are no platforms that have python2.x
> and not python3.x support.
>  - 2to3 means an indeterministic program is actually run
>  - I like to be able to properly support things
>  - distros would prefer to ship a single 'bzr' package rather than two
> separate ones - and because they run with precompiled files not
> writable by the user...
> 
> python2to3 is essentially useless to us. IMNSHO. Its a great tool for
> understanding *the first pass* of what needs to be changed, but not
> for actually having a supported result.
> 
> As for the 'not recommended' clause - the *Python 3* release manager
> has submitted patches to another project - testtools - that does
> precisely that, a single source that works. And its actually pretty
> nice.
> 
> -RobI'll just note that PyQt has some v3 support, nad it causes some real
headaches for the Windows installer. Because the code is listed in a
conditional import (something like: if sys.? > (2, 5, ...) import py3).
It means that py2exe wants to include that code, but can't even import
it without getting an exception.

I don't know what that would mean for us, but splitting the codebase
there wasn't nice for some 3rd party tools.

John
=:->
John Arbash Meinel 1277247068Tue, 22 Jun 2010 22:51:08 +0000 (UTC)
...> I think that ideally, in a year or so we'd be in a position to make a
> concerted push to make 3 a first class citizen (because 3 is getting
> considerable upstream and in-distribution attention).Would you really be trying to do it by making the source code dual
compatible, rather than using either something like 2to3, or by
converting and using something like 3to2?> 
> If the consensus from this thread is that this is ok, I'll update
> HACKING docs appropriately.
> 
> -Rob
> 
>John
=:->
Robert Collins 1277247618Tue, 22 Jun 2010 23:00:18 +0000 (UTC)
On Wed, Jun 23, 2010 at 10:50 AM, John Arbash Meinel
 wrote:>
>
> ...
>
>> I think that ideally, in a year or so we'd be in a position to make a
>> concerted push to make 3 a first class citizen (because 3 is getting
>> considerable upstream and in-distribution attention).
>
> Would you really be trying to do it by making the source code dual
> compatible, rather than using either something like 2to3, or by
> converting and using something like 3to2?There are a bunch of ways we could do it. We haven't chosen one as a
project - and I think thats fine. Python3 is over the horizon for us.

However, there is interest in it now from various places, and that has
been growing over the last couple of years.

As I see it we have 2 major issues to address, which 2to3 and a 3to2
totally bail on.

Ignore syntax, syntax is easy and largely irrelevant - its noise.

C extensions are going to need an overhaul. We're going to want to
keep fixing two versions for a while; ifdefs and macros and pyrex are
good tools here; python2to3 doesn't help *at all*. See
http://docs.python.org/py3k/howto/cporting.ht...

Secondly, we have the *huge*, appallingly huge, mess that is 'all
strings are unicode' in Python3. I expect a majority of the work is
going to be identifying which are which, annotating them appropriately
and dealing with the fallout of things like listdir eliding damaged
filenames (for some versions of 3.x). We may need to set a minimum
version of 3 to use - though I don't claim to have any idea here :).
Adding a _u() decorator - like Benjamins patch in testtools - to make
that explicit would be a fine step, and one that slipped my mind at
the start of this thread. (Such a decorator is a no-op on 3, like _b()
is on 2).

-Rob
Ben Finney 1277250810Tue, 22 Jun 2010 23:53:30 +0000 (UTC)
Robert Collins  writes:

> Adding a _u() decorator - like Benjamins patch in testtools - to make
> that explicit would be a fine step, and one that slipped my mind at
> the start of this thread.Why not simply use Python 2's “u'foo'” syntax?

This doesn't entail maintaining two separate code bases: rather the
Python 2 code base is what gets maintained. A run of ‘2to3’ followed by
the full test suite run under Python 3 can be an indicator of how ready
the code base is for Python 3.

(This doesn't cover C modules, but they're a separate headache either
way.)

Eventually, the decision is made to switch to Python 3. This is only
done some time after the ‘2to3’-output passes the full test suite under
Python 3.

So why would this ‘_u()’ shim be necessary or desirable?
Robert Collins 1277251745Wed, 23 Jun 2010 00:09:05 +0000 (UTC)
On Wed, Jun 23, 2010 at 11:52 AM, Ben Finney  wrote:
> Robert Collins  writes:
>
>> Adding a _u() decorator - like Benjamins patch in testtools - to make
>> that explicit would be a fine step, and one that slipped my mind at
>> the start of this thread.
>
> Why not simply use Python 2's “u'foo'” syntax?It is incompatible with the Python 3 parser.

robertc@lifeless-64:~$ python3.1
Python 3.1.2 (r312:79147, Apr 15 2010, 15:35:48)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> u'\u1234'
  File "<stdin>", line 1
    u'\u1234'
            ^
SyntaxError: invalid syntax


This is the docstring from testtools.compat for its _u function (just
tidied up a few minutes ago):
+"""A function version of the 'u' prefix.
+
+This is needed becayse the u prefix is not usable in Python 3 but is required
+in Python 2 to get a unicode object.
+
+To migrate code that was written as u'\u1234' in Python 2 to 2+3 change
+it to be _u('\u1234'). The Python 3 interpreter will decode it
+appropriately and the no-op _u for Python 3 lets it through, in Python
+2 we then call unicode-escape in the _u function.
+"""> This doesn't entail maintaining two separate code bases: rather the
> Python 2 code base is what gets maintained. A run of ‘2to3’ followed by
> the full test suite run under Python 3 can be an indicator of how ready
> the code base is for Python 3.Modulo:
 - 2to3 limits and issues
 - users with unknown versions of 2to3
 - every distro out there> (This doesn't cover C modules, but they're a separate headache either
> way.)And bytestrings.> Eventually, the decision is made to switch to Python 3. This is only
> done some time after the ‘2to3’-output passes the full test suite under
> Python 3.
>
> So why would this ‘_u()’ shim be necessary or desirable?Because 2to3 is not complete, and structuring 'work with Python3' as a
cutover event - 'switch to 3 only when its ready' does not fit in with
the evolutionary approach we've taken to supporting Python 2.5, then
2.6, and 2.7 now.

Cutover transitions are harsh, slower than expected, dilute community
support, leave users behind unnecessarily.

-Rob
Ben Finney 1277254004Wed, 23 Jun 2010 00:46:44 +0000 (UTC)
Robert Collins  writes:

> On Wed, Jun 23, 2010 at 11:52 AM, Ben Finney  wrote:
> > Why not simply use Python 2's “u'foo'” syntax?
>
> It is incompatible with the Python 3 parser.I thought the agreement was that the code base shouldn't try to target
Python 2 and Python 3 simultaneously? That's certainly the
recommendation from the Python folks.> > This doesn't entail maintaining two separate code bases: rather the
> > Python 2 code base is what gets maintained. A run of ‘2to3’ followed by
> > the full test suite run under Python 3 can be an indicator of how ready
> > the code base is for Python 3.
>
> Modulo:
>  - 2to3 limits and issues
>  - users with unknown versions of 2to3
>  - every distro out thereI'm not suggesting that Bazaar users do the conversion. Rather, that
those automated tools are a way for the Bazaar developers to measure how
close the code base is to being ready for the switch to Python 3.> Because 2to3 is not complete, and structuring 'work with Python3' as a
> cutover event - 'switch to 3 only when its ready' does not fit in with
> the evolutionary approach we've taken to supporting Python 2.5, then
> 2.6, and 2.7 now.
>
> Cutover transitions are harsh, slower than expected, dilute community
> support, leave users behind unnecessarily.Well, I disagree with the direction, but I respect that it's not my
workload being increased.
Stephen J. Turnbull 1277261538Wed, 23 Jun 2010 02:52:18 +0000 (UTC)
Robert Collins writes:

> Cutover transitions are harsh, slower than expected, dilute community
> support, leave users behind unnecessarily.Maybe the net work increase would be less if you cut over but provide
a bundled installation with Python 3 of appropriate version (optional,
of course)?  At least for early versions of bzr/py3.  I understand why
this hasn't been done for 2.x, but Python 3 is a horse of a different
color.

Of course that might not be feasible due to IT policy of RHEL users
etc.
Vincent Ladeuil 1277279266Wed, 23 Jun 2010 07:47:46 +0000 (UTC)
>>>>> Robert Collins  writes:

<snip/>> This is the docstring from testtools.compat for its _u function (just
> tidied up a few minutes ago):
> +"""A function version of the 'u' prefix.
> +
> +This is needed becayse the u prefix is not usable in Python 3 but is required
> +in Python 2 to get a unicode object.
> +
> +To migrate code that was written as u'\u1234' in Python 2 to 2+3 change
> +it to be _u('\u1234'). The Python 3 interpreter will decode its/decode it/decode '\u1234'/ ?

I had to re-read to come to this conclusion ("Python3 will decode" and
"Python 3 lets it through" are contradictory otherwise).> +appropriately and the no-op _u for Python 3 lets it through, in Python
> +2 we then call unicode-escape in the _u function.
> +"""
Robert Collins 1277279348Wed, 23 Jun 2010 07:49:08 +0000 (UTC)
On Wed, Jun 23, 2010 at 7:47 PM, Vincent Ladeuil  wrote:
>>>>>> Robert Collins  writes:
>
> <snip/>
>
>    > This is the docstring from testtools.compat for its _u function (just
>    > tidied up a few minutes ago):
>    > +"""A function version of the 'u' prefix.
>    > +
>    > +This is needed becayse the u prefix is not usable in Python 3 but is required
>    > +in Python 2 to get a unicode object.
>    > +
>    > +To migrate code that was written as u'\u1234' in Python 2 to 2+3 change
>    > +it to be _u('\u1234'). The Python 3 interpreter will decode it
>
> s/decode it/decode '\u1234'/ ?
>
> I had to re-read to come to this conclusion ("Python3 will decode" and
> "Python 3 lets it through" are contradictory otherwise).Please put a patch forward to testtools - it can probably be improved better.

There are two function definitions, one for 2.x and one for 3, and the
explanation needs to explain how you end up with the right stuff on
both versions, and why its needed.

-Rob
Jelmer Vernooij 1277254717Wed, 23 Jun 2010 00:58:37 +0000 (UTC)
On Wed, 2010-06-23 at 09:47 +1200, Robert Collins wrote:
> I've done a couple of trivial patches moving in the direction of being
> able to play with Python 3 and bzr.
> 
> I'd like to document some changes to our coding style, to make
> eventual Python 3 migration easier - but they aren't all no-brainers
> so I thought I'd raise stuff here and we can discuss it.
> 
> The no brainers (simple, not terribly ugly, not used a lot in our code):
> octal 0666 ->0o666
> print foo, bar -> print("%s %s" % (foo, bar))
> exec foo, locals() -> exec(foo, locals())
> 
> Somewhat ugly - we're going to see a lot of these:
> except Foo, e:
> ->
> except Foo:
>     e = sys.exc_info()[1]
> 
> bytestrings:
> This has the potential to slow load times slightly: in 3 to get a
> bytestring one says b'foo', but you can't do that in 2, so for
> single-source compatibility it gets written as:
>  _b('foo')
> 
> which on 2 is a no-op, and on 3 reencodes using latin-1 (so everything
> works). Or we could split out separate files to import on 2 and 3, but
> I think the extra seek would make it a wash perf wise.
> 
> really ugly, but very few occurences:
>     raise type, val, tb
> ->
>     bzrlib.util.py._builtin._reraise(type, val, tb)
> 
> Now, there are a lot of other things that we will have to solve and
> talk about, this is really top level mechanical stuff and should not
> be taken as the whole list or a magic bullet.
> 
> However, I think changing our coding style this and not much more will
> be enough to let interested people slowly push forward and get things
> like:
>  - the test suite
>  - C / pyrex modules
> and so forth working on 3.
> 
> I think that ideally, in a year or so we'd be in a position to make a
> concerted push to make 3 a first class citizen (because 3 is getting
> considerable upstream and in-distribution attention).
> 
> If the consensus from this thread is that this is ok, I'll update
> HACKING docs appropriately.Is supporting the two versions (that are quite different, at least where
Bazaar is concerned) from the same code base really a good idea ? 

I'm wondering if having a separate "python 3" branch of Bazaar to which
newer revisions are explicitly ported is perhaps less of a strain on
development than having to fix up portability issues in the main branch
all the time and by introducing various wrappers around standard
functions.

Cheers,

Jelmer
Andrew Bennetts 1277259336Wed, 23 Jun 2010 02:15:36 +0000 (UTC)
Robert Collins wrote:
[...]
> I think that ideally, in a year or so we'd be in a position to make a
> concerted push to make 3 a first class citizen (because 3 is getting
> considerable upstream and in-distribution attention).To be honest, I'd be inclined to wait.  I don't see much benefit to us
pushing towards Python 3 support at the moment, and conversely I don't
think it'll be any harder to start doing that later — in fact it may get
easier as tools like 2to3 (or 3to2) improve, and it gets easier for us
to consider dropping Python 2.4, etc.

My suspicion is that we simply don't have the resources to make 3 a
first class citizen without a negative impact on our 2.x support, if you
include “no significant performance hit” in the meaning of “first class
citizen”.  The change to bytes and unicode handling seems like something
that will be difficult to support in 2.5 and 3.x from the same code base
without a performance hit, because b"" literals were only added in 2.6.

What benefits do you see?  As far as I can find with Google we've seen a
single inquiry about bzr on Python 3, so there doesn't seem to be much
demand for a port so far.

-Andrew.
Martin Pool 1277261066Wed, 23 Jun 2010 02:44:26 +0000 (UTC)
On 23 June 2010 12:15, Andrew Bennetts  wrote:
> Robert Collins wrote:
> [...]
>> I think that ideally, in a year or so we'd be in a position to make a
>> concerted push to make 3 a first class citizen (because 3 is getting
>> considerable upstream and in-distribution attention).
>
> To be honest, I'd be inclined to wait.  I don't see much benefit to us
> pushing towards Python 3 support at the moment, and conversely I don't
> think it'll be any harder to start doing that later — in fact it may get
> easier as tools like 2to3 (or 3to2) improve, and it gets easier for us
> to consider dropping Python 2.4, etc.I don't see any rush to make 3 the default platform.  I see some
benefit in testing on 3 so that we get earlier warning of any issues
and just because good testing across more environments can make some
hidden bugs easier to fix.  I like the idea of gradually getting the
test suite to pass on 2to3 by making only clean changes to the main
trunk because it will not hurt people not using py3.
Stephen J. Turnbull 1277270101Wed, 23 Jun 2010 05:15:01 +0000 (UTC)
Andrew Bennetts writes:

> The change to bytes and unicode handling seems like something that
> will be difficult to support in 2.5 and 3.x from the same code base
> without a performance hit, because b"" literals were only added in
> 2.6.I think the _b() device should handle literals.  If the 2.x compiler
doesn't already optimize _b('Andrew') to 'Andrew', some sort of
preprocessing step would be ugly but not impossible, or a monkey patch
to the compiler.

The issue that worries me (as is being discussed in a different
context on python-dev) is that some modules in the stdlib now expect
to be passed str = unicode instead of str = bytes.  To the extent that
such a module is on the critical path, you could take a big
performance hit there, or face the need to rewrite some of the stdlib
to provide a parallel bytes API.

OTOH, this would only affect Python 3, and you have plenty of time to
wait for or design performance improvements there.  N.B. Guido has
expressed support for "polymorphism", which he defines to mean that
functions which process strings (bytes or unicode) return the type fed
to them (and error on inconsistency).  I don't know if that applies to
Python 3.2, though (FLUFL, WDYT?)
Barry Warsaw 1277297779Wed, 23 Jun 2010 12:56:19 +0000 (UTC)
On Jun 23, 2010, at 02:11 PM, Stephen J. Turnbull wrote:

>OTOH, this would only affect Python 3, and you have plenty of time to
>wait for or design performance improvements there.  N.B. Guido has
>expressed support for "polymorphism", which he defines to mean that
>functions which process strings (bytes or unicode) return the type fed
>to them (and error on inconsistency).  I don't know if that applies to
>Python 3.2, though (FLUFL, WDYT?)I'm not sure what there would be to do for Python 3.2, i.e. on the
enforcement/code change side.  It sounds like Guido's polymorphism
recommendation is little more than just that, but TBH, lack of time is forcing
me to tune out on that particular thread for now. :/

-Barry
Russel Winder 1277273793Wed, 23 Jun 2010 06:16:33 +0000 (UTC)
Andrew,On Wed, 2010-06-23 at 12:15 +1000, Andrew Bennetts wrote:
[ . . . ]
> What benefits do you see?  As far as I can find with Google we've seen a
> single inquiry about bzr on Python 3, so there doesn't seem to be much
> demand for a port so far.Not entirely true, some people haven't asked because they know the
answer already and they just install 2.6 in order to use Bazaar, SCons,
etc. whilst using 3.1 for their own code.

There is a "chicken and egg" situation here, no-one is moving to 3.1
because no-one is moving to 3.1.  The fact that major distros still
haven't caught up to 2.6 is probably the major factor.

If pre-2.6 is an important factor for Bazaar then thinking about 3.1 is
something to work on in the background.  The question is whether pre-2.6
Pythons are used by users of Bazaar.  Are there statistics or is this
just an assumption?  I have no data, I am just asking the question.
John Szakmeister 1277277271Wed, 23 Jun 2010 07:14:31 +0000 (UTC)
On Wed, Jun 23, 2010 at 2:16 AM, Russel Winder  wrote:
[snip]
> If pre-2.6 is an important factor for Bazaar then thinking about 3.1 is
> something to work on in the background.  The question is whether pre-2.6
> Pythons are used by users of Bazaar.  Are there statistics or is this
> just an assumption?  I have no data, I am just asking the question.I'm not sure I can offer much here, but I a number of dev systems I
see are running with Python < 2.6.  Python 2.5 is really pervasive,
and we still see 2.4 on the server side quite a bit.  That's just one
example, but hopefully you see that the concerns are real.

-John
Russel Winder 1277272397Wed, 23 Jun 2010 05:53:17 +0000 (UTC)
Robert,On Wed, 2010-06-23 at 09:47 +1200, Robert Collins wrote:
[ . . . ]
> print foo, bar -> print("%s %s" % (foo, bar))[ . . . ]

Just to protect against creation of misinformation:


print foo , bar  ->  print ( foo , bar )

Also the "%s" and % operator are deprecated in Python 3, so:

print ( "%s %s" % ( foo , bar ) ) -> print ( "{} {}".format ( foo ,
bar ) )> except Foo, e:
> ->
> except Foo:
>     e = sys.exc_info()[1]Why?

except Foo , e : ->  except Foo as e :


I find that the following document is very helpful:

http://ptgmedia.pearsoncmg.com/imprint_downlo...
Robert Collins 1277272573Wed, 23 Jun 2010 05:56:13 +0000 (UTC)
On Wed, Jun 23, 2010 at 5:52 PM, Russel Winder  wrote:
> Robert,
>
> On Wed, 2010-06-23 at 09:47 +1200, Robert Collins wrote:
> [ . . . ]
>> print foo, bar -> print("%s %s" % (foo, bar))
> [ . . . ]
>
> Just to protect against creation of misinformation:
>
>
> print foo , bar  ->  print ( foo , bar )
>
> Also the "%s" and % operator are deprecated in Python 3, so:
>
> print ( "%s %s" % ( foo , bar ) ) -> print ( "{} {}".format ( foo ,
> bar ) )Oh yes; still, in 3.1 they still work, which is good enough for now :).

> except Foo , e : ->  except Foo as e :

working$ python2.5
Python 2.5.5 (r255:77872, Jan 31 2010, 21:34:29)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
...   1/0
... except Exception as e:
<stdin>:3: Warning: 'as' will become a reserved keyword in Python 2.6
  File "<stdin>", line 3
    except Exception as e:
                      ^
SyntaxError: invalid syntax> I find that the following document is very helpful:
>
> http://ptgmedia.pearsoncmg.com/imprint_downlo...Thanks.

-Rob
Russel Winder 1277273396Wed, 23 Jun 2010 06:09:56 +0000 (UTC)
On Wed, 2010-06-23 at 17:55 +1200, Robert Collins wrote:
[ . . . ]
> > except Foo , e : ->  except Foo as e :
> 
> working$ python2.5
> Python 2.5.5 (r255:77872, Jan 31 2010, 21:34:29)
> [GCC 4.4.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> try:
> ...   1/0
> ... except Exception as e:
> <stdin>:3: Warning: 'as' will become a reserved keyword in Python 2.6
>   File "<stdin>", line 3
>     except Exception as e:
>                       ^
> SyntaxError: invalid syntaxIndeed, and true for all pre2.6 Pythons.  All the 2.x -> 3.1 guides
effectively assume 2.6 as a floor.  I have the luxury of treating 2.6 as
a floor and tend to forget sometimes the hassles of using anything that
doesn't have the multiprocessing package as standard (*).

Sorry for slipping in a googly (**).

(*) Insert long spiel about multicore, processor bound computing, use of
Python, the GIL, process-orientation, and CSP.

(**) Cricket metaphor nothing to do with anything else that might have
come to mind.
Martin Pool 1277273511Wed, 23 Jun 2010 06:11:51 +0000 (UTC)
On 23 June 2010 15:52, Russel Winder  wrote:
> Robert,The message Russel is responding to here is Robert's proposal that we
make certain changes (shown by the arrows) to our coding style so that
we can have a single codebase that works on 2.4  .. 3.1.>
> On Wed, 2010-06-23 at 09:47 +1200, Robert Collins wrote:
> [ . . . ]
>> print foo, bar -> print("%s %s" % (foo, bar))
> [ . . . ]
>
> Just to protect against creation of misinformation:
>
>
> print foo , bar  ->  print ( foo , bar )Those two statements do different things in 2.x so no, the second is
not ok as a coding guideline for a single codebase.>
> Also the "%s" and % operator are deprecated in Python 3, so:
>
> print ( "%s %s" % ( foo , bar ) ) -> print ( "{} {}".format ( foo ,
> bar ) )
>There's no .format in python2.4.>
>> except Foo, e:
>> ->
>> except Foo:
>>     e = sys.exc_info()[1]
>
> Why?
>
> except Foo , e : ->  except Foo as e :What Robert proposes will work for a single codebase but there is no
"as e" on 2.4.

One of the major points of this thread and previous ones is that we
are not prepared to give up 2.4 support yet.  I do realize that it
would be easier to support 3.x if we dropped 2.4 but we're not going
there yet.
Russel Winder 1277274316Wed, 23 Jun 2010 06:25:16 +0000 (UTC)
Martin,On Wed, 2010-06-23 at 16:11 +1000, Martin Pool wrote:
> On 23 June 2010 15:52, Russel Winder  wrote:On Wed, 2010-06-23 at 16:11 +1000, Martin Pool wrote:
> On 23 June 2010 15:52, Russel Winder  wrote:
[ . . . ]
> > print foo , bar  ->  print ( foo , bar )
> 
> Those two statements do different things in 2.x so no, the second is
> not ok as a coding guideline for a single codebase.Indeed.  I was thinking of transformation rules rather than common code.
For common code, you might have to do something like:

print ( str ( foo ) + ' ' + str ( bar ) )


[ . . . ]> One of the major points of this thread and previous ones is that we
> are not prepared to give up 2.4 support yet.  I do realize that it
> would be easier to support 3.x if we dropped 2.4 but we're not going
> there yet.As far as I am aware you have to go to 2.6 and drop all pre-2.6 idioms
to have any chance of 2to3 and 3to2 being useful.  With 2.4 and 2.5
factors in the equation, then looking at 3 is for the future --
certainly this is the view in the SCons team.
Ben Finney 1277277403Wed, 23 Jun 2010 07:16:43 +0000 (UTC)
Russel Winder  writes:

> On Wed, 2010-06-23 at 16:11 +1000, Martin Pool wrote:
> > One of the major points of this thread and previous ones is that we
> > are not prepared to give up 2.4 support yet. I do realize that it
> > would be easier to support 3.x if we dropped 2.4 but we're not going
> > there yet.
>
> As far as I am aware you have to go to 2.6 and drop all pre-2.6 idioms
> to have any chance of 2to3 and 3to2 being useful.  With 2.4 and 2.5
> factors in the equation, then looking at 3 is for the future --
> certainly this is the view in the SCons team.+1.

I think Python 2.4 ↔ 3.1 is too broad a range to try supporting from a
single code base. This is exacerbated by the fact that Python 2.4 and
2.5 are not receiving upstream support, and Python 2.6 will soon be
receiving very little attention upstream (once Python 2.7 is released).
Danny van Heumen 1277403626Thu, 24 Jun 2010 18:20:26 +0000 (UTC)
(Small disclaimer: I'm not that familiar with python, but am interested
to learn and I think this thread is particularly interesting. The
following question came to my mind. Okay, so it might be rubbish - if so
just say it and I'll shut up :) ...)

I'm wondering. If 2.4 <--> 3.1 is too broad to support, and 2.6 is
better suited to convert to 3.x, since it supports some of the 3.x
constructs.

Would it be feasible to write code in a 2.6 compatible style and
automagically convert it to 2.4?

I mean, I think the most issues I've seen raised are issues like:
'except Foo as e' is not recognized in python 2.4, and 'with x as f' is
not supported.
But, if I understood it correctly, all of these constructs can be
expanded to a form that is understable by 2.4.

So, suppose that you should avoid (complex) modules that were introduced
in python 2.6, would it be feasible to just convert the basic constructs
to 2.4? (That way you can build to a more 3.x compatible code base.)On 06/23/2010 09:16 AM, Ben Finney wrote:
> Russel Winder  writes:
> 
>> On Wed, 2010-06-23 at 16:11 +1000, Martin Pool wrote:
>>> One of the major points of this thread and previous ones is that we
>>> are not prepared to give up 2.4 support yet. I do realize that it
>>> would be easier to support 3.x if we dropped 2.4 but we're not going
>>> there yet.
>>
>> As far as I am aware you have to go to 2.6 and drop all pre-2.6 idioms
>> to have any chance of 2to3 and 3to2 being useful.  With 2.4 and 2.5
>> factors in the equation, then looking at 3 is for the future --
>> certainly this is the view in the SCons team.
> 
> +1.
> 
> I think Python 2.4 ↔ 3.1 is too broad a range to try supporting from a
> single code base. This is exacerbated by the fact that Python 2.4 and
> 2.5 are not receiving upstream support, and Python 2.6 will soon be
> receiving very little attention upstream (once Python 2.7 is released).
>
John Barstow 1277417543Thu, 24 Jun 2010 22:12:23 +0000 (UTC)
On Fri, Jun 25, 2010 at 6:19 AM, Danny van Heumen
 wrote:
> (Small disclaimer: I'm not that familiar with python, but am interested
> to learn and I think this thread is particularly interesting. The
> following question came to my mind. Okay, so it might be rubbish - if so
> just say it and I'll shut up :) ...)
>
> I'm wondering. If 2.4 <--> 3.1 is too broad to support, and 2.6 is
> better suited to convert to 3.x, since it supports some of the 3.x
> constructs.
>Really what this tends to point up is the need to branch the code when
we decide we want to start supporting 3.x

If I were a theoretical release manager doing hypothetical releases, I
would imagine something like this (using inflated release numbers to
drive home the point this is just an example!):

bzr 4.4 - last release supporting python 2.4/2.5 - you won't be able
to upgrade anymore until you upgrade your python version.
bzr 4.6 - first release to support python 3.x using 2to3. Requires
minimum version of python 2.6
bzr 4.4.1 - point release of legacy branch where someone was paid to
backport a feature from 4.6
bzr 5.10 - last release with python 2.x support - you won't be able to
upgrade anymore until you upgrade your python version.
bzr 5.12 - first release without python 2.x support *at all*...

Honestly, there's no downside to saying that release X is going to
require Python 2.6; people who depend on an older version of python
just don't upgrade their installation of bazaar. Anyone sufficiently
motivated can backport code to the legacy branch (say, to accommodate
a new wire protocol), upgrade python without official distribution
support (you *can* compile python on a RHEL box when you need a newer
version!), or petition for a compatibility mode (please provide a bzr
2.2 server on legacy.launchpad.net, here's a bundle of money...)

Frankly, things like native NTLM support can't be integrated cleanly
while you support Python 2.4 (you need the MD4 support in the newer
hashlib module for the pure python implementation). As it is, last
time I checked Bazaar 1.6 was the only version available in any RHEL5
repository, so I'm not sure waiting for RHEL6 is really going to make
a difference.

So are we reluctant to drop Python 2.4/2.5 support because we have no
guarantee of backwards compatibility in the wire protocol and/or
branch format?  I don't really see us dropping 2a support just because
we make a move to Python 2.6. Or would we lose major contributors who
are unable to upgrade their version of Python?
John Arbash Meinel 1277418215Thu, 24 Jun 2010 22:23:35 +0000 (UTC)
> Honestly, there's no downside to saying that release X is going to
> require Python 2.6; people who depend on an older version of python
> just don't upgrade their installation of bazaar. Anyone sufficiently
> motivated can backport code to the legacy branch (say, to accommodate
> a new wire protocol), upgrade python without official distribution
> support (you *can* compile python on a RHEL box when you need a newer
> version!), or petition for a compatibility mode (please provide a bzr
> 2.2 server on legacy.launchpad.net, here's a bundle of money...)
> 
> Frankly, things like native NTLM support can't be integrated cleanly
> while you support Python 2.4 (you need the MD4 support in the newer
> hashlib module for the pure python implementation). As it is, last
> time I checked Bazaar 1.6 was the only version available in any RHEL5
> repository, so I'm not sure waiting for RHEL6 is really going to make
> a difference.Toshio maintains an rpm for RHEL of bzr. So while it isn't in the
official repo, it is still trivially available.

I've often felt that "if you can upgrade bzr, you can upgrade python",
but apparently that feeling is not held. (One is considered a user-space
command, the other a system-wide infrastructure.)

My personal feeling is that we can do something like:

1) 2.2 will be python2.4 compatible
2) 2.3 will drop 2.4 (maybe 2.5) support.
3) We continue with our current methodology of supporting old releases
(2.0/2.1/soon to be 2.2) for some time. I don't know whether we'll drop
2.0 as a supported stable series now that 2.2 will be released.
Regardless, though, we'll certainly still support 2.1, and *that* will
stay 2.4 compatible.

The big concern with having a break at 2.3 is how difficult it will be
to do the fixes in the 2.1/2.2 branch, and have them still apply to the
2.3 branch.

At the moment, most code is not very diverged, so it isn't very hard.
I'm guessing that will mostly stay true. Martin has mentioned that he
likes "except Exception as e" syntax, so there will be some effort
there. If we start using a lot of context objects, that would also make
the code a fair bit different.> 
> So are we reluctant to drop Python 2.4/2.5 support because we have no
> guarantee of backwards compatibility in the wire protocol and/or
> branch format?  I don't really see us dropping 2a support just because
> we make a move to Python 2.6. Or would we lose major contributors who
> are unable to upgrade their version of Python?I think it is mostly that we are likely to introduce new RPCs where we
find edge cases that can perform better. Those won't be backported to
older releases (because it is not a bugfix/security fix).

That said, we are unlikely (though it has happened) to remove RPCs. In
which case older clients talking to newer servers will work like they
always have. And newer clients talking to older servers will know to
fall back to the older code path.

We've had some stress points there, and haven't always gotten it 100%
correct, but generally I think we've done ok.

John
=:->
David Cournapeau 1277773610Tue, 29 Jun 2010 01:06:50 +0000 (UTC)
On Wed, Jun 23, 2010 at 4:16 PM, Ben Finney  wrote:
> Russel Winder  writes:
>
>> On Wed, 2010-06-23 at 16:11 +1000, Martin Pool wrote:
>> > One of the major points of this thread and previous ones is that we
>> > are not prepared to give up 2.4 support yet. I do realize that it
>> > would be easier to support 3.x if we dropped 2.4 but we're not going
>> > there yet.
>>
>> As far as I am aware you have to go to 2.6 and drop all pre-2.6 idioms
>> to have any chance of 2to3 and 3to2 being useful.  With 2.4 and 2.5
>> factors in the equation, then looking at 3 is for the future --
>> certainly this is the view in the SCons team.
>
> +1.
>
> I think Python 2.4 ↔ 3.1 is too broad a range to try supporting from a
> single code base. This is exacerbated by the fact that Python 2.4 and
> 2.5 are not receiving upstream support, and Python 2.6 will soon be
> receiving very little attention upstream (once Python 2.7 is released).I am not claiming any strong relevance to bzr, but in numpy, we intend
to support python 2.4 -> python 3.1 (and we have a lot of C code :) ),
and it seems we will manage to do it.

Yes, it is quite painful - and I would guess not very useful for an
application like bzr. To be honest, I don't see the point in python 3.
the benefits seem ridiculous compared to the backward incompatibility
pain and the only reason why I am contributing to the py3 port of
Numpy/Scipy is to be good "python citizen".

David>
> --
>  \         “If we don't believe in freedom of expression for people we |
>  `\           despise, we don't believe in it at all.” —Noam Chomsky, |
> _o__)                                                       1992-11-25 |
> Ben Finney
>
Alexander Belchenko 1277281130Wed, 23 Jun 2010 08:18:50 +0000 (UTC)
What is the real benefit of Python 3 over Python 2.5 for Bazaar?
Faster execution speed? Faster import speed?

Does it will give bzr more bonus point to increase its adoption?

2.6 is already headache to create windows standalone installer. I 
suppose moving to 3 only increase the pain.
Robert Collins 1277281555Wed, 23 Jun 2010 08:25:55 +0000 (UTC)
On Wed, Jun 23, 2010 at 8:18 PM, Alexander Belchenko  wrote:
> What is the real benefit of Python 3 over Python 2.5 for Bazaar?
> Faster execution speed? Faster import speed?
>
> Does it will give bzr more bonus point to increase its adoption?
>
> 2.6 is already headache to create windows standalone installer. I suppose
> moving to 3 only increase the pain.Moving to 3 would be a long way off - years+.

3 has a few interesting things for bzr;
Firstly, things like Pynie only run three - and it specifically claims
to be *much* faster at running code than CPython.
Secondly, If (I'm being generous :)) there are significant issues in
Python 3 that affect us, we stand a much better chance of having a
smooth transition if we file bugs - which requires us to actually
compile on the platform.
Lastly, eventually, Python 2 is going to fade away, and we should be
ready for the early stages of that, which is platforms such as Ubuntu
or RHEL that want to ship only Python 3, or Python 3 as default.

I'm not, nor do I think anyone was, talking about making installers in
Python 3 until it really is better for us. But right now its a big
unknown, and it would be nice to reduce the risk involved in getting
across it.

-Rob
Toshio Kuratomi 1277315493Wed, 23 Jun 2010 17:51:33 +0000 (UTC)
On Wed, Jun 23, 2010 at 09:47:36AM +1200, Robert Collins wrote:
> I've done a couple of trivial patches moving in the direction of being
> able to play with Python 3 and bzr.
> 
> I'd like to document some changes to our coding style, to make
> eventual Python 3 migration easier - but they aren't all no-brainers
> so I thought I'd raise stuff here and we can discuss it.
> 
> The no brainers (simple, not terribly ugly, not used a lot in our code):
> octal 0666 ->0o666
>This isn't python-2.4 compatible

-Toshio
John Arbash Meinel 1277316319Wed, 23 Jun 2010 18:05:19 +0000 (UTC)
Toshio Kuratomi wrote:
> On Wed, Jun 23, 2010 at 09:47:36AM +1200, Robert Collins wrote:
>> I've done a couple of trivial patches moving in the direction of being
>> able to play with Python 3 and bzr.
>>
>> I'd like to document some changes to our coding style, to make
>> eventual Python 3 migration easier - but they aren't all no-brainers
>> so I thought I'd raise stuff here and we can discuss it.
>>
>> The no brainers (simple, not terribly ugly, not used a lot in our code):
>> octal 0666 ->0o666
>>
> This isn't python-2.4 compatible
> 
> -ToshioNor is it python2.5 compatible. So even when we decide we can drop 2.4,
I don't think that means we can drop 2.5 until even later.

John
=:->
John Arbash Meinel 1277928843Wed, 30 Jun 2010 20:14:03 +0000 (UTC)
Robert Collins wrote:
> I've done a couple of trivial patches moving in the direction of being
> able to play with Python 3 and bzr.In another thread Barry made reference to the 'six' package from
Benjamin Peterson, available at lp:python-six.

Something you might want to look at. It is specifically focused on
tools/constants/constructs that are meant to help write source code that
is 2 and 3 compatible.

Documentation here:
  http://packages.python.org/six/

Also, I don't really know if he is going for all-of-2 (2.1,2.2,2.3 or
just 2.6+3.x compatibility.)

John
=:->
John Arbash Meinel 1277933365Wed, 30 Jun 2010 21:29:25 +0000 (UTC)
...> Documentation here:
>   http://packages.python.org/six/
> 
> Also, I don't really know if he is going for all-of-2 (2.1,2.2,2.3 or
> just 2.6+3.x compatibility.)I just looked at the code, and it is pretty clear it isn't meant for all
versions of 2.x. It does stuff like:

  from . import _util

Which seems to work under 2.5, but doesn't work on python2.4.

Though if we decided we could drop 2.4 support (and not 2.5) it might
work for us.

John
=:->
Max Kanat-Alexander 1277934599Wed, 30 Jun 2010 21:49:59 +0000 (UTC)
On 06/30/2010 02:28 PM, John Arbash Meinel wrote:
> Though if we decided we could drop 2.4 support (and not 2.5) it might
> work for us.Dropping 2.4 support makes the baby sysadmin messiah cry.

	(Or, more clearly--there are a lot of long-term support distros out
there that shipped with 2.4, and on which it is really complex to
install a separate or newer python.)

	-Max
Barry Warsaw 1277936711Wed, 30 Jun 2010 22:25:11 +0000 (UTC)
On Jun 30, 2010, at 02:49 PM, Max Kanat-Alexander wrote:

>On 06/30/2010 02:28 PM, John Arbash Meinel wrote:
>> Though if we decided we could drop 2.4 support (and not 2.5) it might
>> work for us.
>
>	Dropping 2.4 support makes the baby sysadmin messiah cry.
>
>	(Or, more clearly--there are a lot of long-term support
>	distros out
>there that shipped with 2.4, and on which it is really complex to
>install a separate or newer python.)cx_freeze ftw?

-Barry
Martin Pool 1277938583Wed, 30 Jun 2010 22:56:23 +0000 (UTC)
On 1 July 2010 07:49, Max Kanat-Alexander  wrote:
> On 06/30/2010 02:28 PM, John Arbash Meinel wrote:
>> Though if we decided we could drop 2.4 support (and not 2.5) it might
>> work for us.
>
>        Dropping 2.4 support makes the baby sysadmin messiah cry.This is really not on the cards for the foreseeable future.  There are
clear benefits to supporting 2.4, and only minor benefits at present
to supporting 3 now.  I think supporting 3 would be good but I
certainly wouldn't give up 2.4 to get there.

Perhaps we can send patches to six to make it 2.4 compatible.
Scott Aubrey 1277975021Thu, 01 Jul 2010 09:03:41 +0000 (UTC)
On 30 Jun 2010, at 23:55, Martin Pool wrote:

> On 1 July 2010 07:49, Max Kanat-Alexander  wrote:
>> On 06/30/2010 02:28 PM, John Arbash Meinel wrote:
>>> Though if we decided we could drop 2.4 support (and not 2.5) it might
>>> work for us.
>> 
>>        Dropping 2.4 support makes the baby sysadmin messiah cry.
> 
> This is really not on the cards for the foreseeable future.  There are
> clear benefits to supporting 2.4, and only minor benefits at present
> to supporting 3 now.  I think supporting 3 would be good but I
> certainly wouldn't give up 2.4 to get there.
> [snip]
> -- 
> Martin
>Why is that?

Not being a python developer myself, nor highly involved in the bazaar community, I feel I may be over stepping my boundary here, but here goes.

from a redhat 5 person perspective: I've got my bug fix, no new features version of an OS that's stable, and bazaar has a bug fix only version that works with this python version. The newer version of bazaar doesn't work, but that's OK.

Why is that idea of dropping Python 2.4 (and Redhat 5 etc) such a bad thing? Even putting aside python 3 support, supporting newer features and versions of python must have some benefit -- at least it's less platforms to make sure new code works on? combined with the fully, bug fix supported versions of bazaar available for those platforms, that will be supported for at least [insert timeframe]? Isn't that what these people expect from Redhat, why expect more from bazaar?

- Scott
Stephen J. Turnbull 1277977264Thu, 01 Jul 2010 09:41:04 +0000 (UTC)
Scott Aubrey writes:

> Why is that idea of dropping Python 2.4 (and Redhat 5 etc) such a
> bad thing?The short answer is that some customers want that support.  And
(surprise) these are Customers, excuse me, *Mr.* Customers *Sirs* with
a capital C (C for "cash in hand").  (That's a joke; I don't know what
the real reason is, that's a plausible guess, though.)> Isn't that what these people expect from Redhat, why expect more
> from bazaar?Because they are different cases.  Redhat is expected to leave you
with a stable, bugfix only version because their job is to provide a
platform on which *everything* works.  This allows you (FSVO "you")
to choose (a) to upgrade nothing and have a system that becomes
monotonically more stable over time, with exactly the same features,
or (b) to upgrade only locally-developed software, which presumably
you understand pretty well, and you can restrict bug hunts to that
well-understood software, or (c) to upgrade a select few mission-
critical applications, and restrict bug hunts to that relatively well-
understood mission-critical set.  It's your choice, Red Hat enables it
by providing that rock-solid platform.

Bazaar is an application, and while many people are perfectly happy
with Bazaar 0.9 still, others consider Bazaar a mission-critical
support tool, they greatly appreciate the new features and performance
improvements, and they want to upgrade to more modern version once it
becomes stable.  However, they may also demand a rock-solid platform
in other respects.  It's perfectly reasonable for them to *want* to
run Bazaar 2.2 on Red Hat 5.  The question for Bazaar devs is "should
they satisfy that desire?"

Martin's answer is clearly "yes", and you know what, I know he knows a
lot more about it than me, so there you go.
John Arbash Meinel 1277992809Thu, 01 Jul 2010 14:00:09 +0000 (UTC)
Martin Pool wrote:
> On 1 July 2010 07:49, Max Kanat-Alexander  wrote:
>> On 06/30/2010 02:28 PM, John Arbash Meinel wrote:
>>> Though if we decided we could drop 2.4 support (and not 2.5) it might
>>> work for us.
>>        Dropping 2.4 support makes the baby sysadmin messiah cry.
> 
> This is really not on the cards for the foreseeable future.  There are
> clear benefits to supporting 2.4, and only minor benefits at present
> to supporting 3 now.  I think supporting 3 would be good but I
> certainly wouldn't give up 2.4 to get there.
> 
> Perhaps we can send patches to six to make it 2.4 compatible.
>Certainly I don't see us doing it for any of bzr 2.0, 2.1 or 2.2. I was
thinking more along the lines of 2.3 (or 3.0 :). As you say, supporting
3.x is lower on the list right now than supporting python 2.4. However,
I think this will change at some point in the future, and we should be
aware of that, and of tech that might help us transition.

John
=:->
Martin Pool 1278031870Fri, 02 Jul 2010 00:51:10 +0000 (UTC)
On 1 July 2010 23:59, John Arbash Meinel  wrote:>
> Martin Pool wrote:
>> On 1 July 2010 07:49, Max Kanat-Alexander  wrote:
>>> On 06/30/2010 02:28 PM, John Arbash Meinel wrote:
>>>> Though if we decided we could drop 2.4 support (and not 2.5) it might
>>>> work for us.
>>>        Dropping 2.4 support makes the baby sysadmin messiah cry.
>>
>> This is really not on the cards for the foreseeable future.  There are
>> clear benefits to supporting 2.4, and only minor benefits at present
>> to supporting 3 now.  I think supporting 3 would be good but I
>> certainly wouldn't give up 2.4 to get there.
>>
>> Perhaps we can send patches to six to make it 2.4 compatible.
>>
>
> Certainly I don't see us doing it for any of bzr 2.0, 2.1 or 2.2. I was
> thinking more along the lines of 2.3 (or 3.0 :). As you say, supporting
> 3.x is lower on the list right now than supporting python 2.4. However,
> I think this will change at some point in the future, and we should be
> aware of that, and of tech that might help us transition.+1

I'm delighted to take patches that make things work more cleanly with
-3 warnings or through 2to3 or directly on 3, as long as they don't
break 2.4 or make the code excessively ugly.  Conversely if someone
feels they want to develop a parallel branch that does work on 3 that
might be an interesting experiment.

Every time the topic of dropping 2.4 support comes up several people
post that they still want it supported.  I run into machines running
multi-year-old OS releases fairly often so I sympathize.

The byte vs unicode transition may be hard enough that we want to
start chewing on it.

We're going to branch off 2.2 and bump the trunk to 2.3 in about a
month so it's possible that we could say the 2.2 series will be the
last one supported on 2.4.  I feel like the right time to do that is
when somebody says "I really want to do X and it's just not feasible
in 2.4" - whether that is using 2to3 or six or something else.  We've
had some "it would be nice" cases like context managers but nothing
really compelling yet.
John Arbash Meinel 1278032407Fri, 02 Jul 2010 01:00:07 +0000 (UTC)
...> We're going to branch off 2.2 and bump the trunk to 2.3 in about a
> month so it's possible that we could say the 2.2 series will be the
> last one supported on 2.4.  I feel like the right time to do that is
> when somebody says "I really want to do X and it's just not feasible
> in 2.4" - whether that is using 2to3 or six or something else.  We've
> had some "it would be nice" cases like context managers but nothing
> really compelling yet.
>I think most of the 2.4 vs newer comes down to code clarity sorts of
things. from . import foo, context managers, except X as e, defaultdict,
etc. I haven't seen anything that is 'must have', but quite a few things
that 'would be nice'.

As I understand it, that is the good/bad of python 3 as well. Apparently
it is much cleaner as a language 'from the beginning', but there really
isn't anything you can do that you couldn't before. So everyone that has
worked out python-2 quirks doesn't benefit very much.

Anyway, I see your point. From my pov, there is always the
chicken-and-the-egg stuff. I would probably look more closely at making
sure we work with python-2.7 today, and once that is out, we'd probably
be want to be close to ready for python3 around 1-year after python-2.7
is released (it will be supported for a long time, but after about a
year I would guess missing features would start to be noticeable.)

John
=:->
Stephen J. Turnbull 1278040810Fri, 02 Jul 2010 03:20:10 +0000 (UTC)
John Arbash Meinel writes:

> Anyway, I see your point. From my pov, there is always the
> chicken-and-the-egg stuff. I would probably look more closely at making
> sure we work with python-2.7 today, and once that is out, we'd probably
> be want to be close to ready for python3 around 1-year after python-2.7
> is released (it will be supported for a long time, but after about a
> year I would guess missing features would start to be noticeable.)I would not bet on that.  Currently Python 3 (the language) is in a
long-term feature freeze (for 3.2 at least; possible 3.3 as well), as
it is in catch-up mode for the stdlib even today (email needs a
complete rewrite because of the peculiar nature of email as a
bytes-oriented protocol which is nominally ASCII text but in fact is
only slightly less random than an MS-Word core dump, and there are
several other modules which need a fair amount of work).  And large
3rd party frameworks like numpy and Twisted are not there yet (real
soon now, really, for numpy; Real Soon Now (uh, right, yup, not!) for
Twisted.

Python 3 is very attractive for new development because it's cleaner
and because it has a usable approach to Unicode.  (All my own stuff
got converted a long time ago, and I've never missed Python 2 at all.)
Once people get their minds wrapped around it, I think the mail and
web-oriented stdlib modules are going to see quantum improvements.  If
you have other interests in Python 3, sure, you can get started on
porting Bazaar to it now, I think, which wasn't true at all for Python
3.0, and was only marginally true for 3.1.  But for general
programming and ongoing development of applications with a lot of
history (in which I would include Bazaar), there really is no hurry
AFAICS.
Russel Winder 1278054949Fri, 02 Jul 2010 07:15:49 +0000 (UTC)
Martin,> I'm delighted to take patches that make things work more cleanly with
> -3 warnings or through 2to3 or directly on 3, as long as they don't
> break 2.4 or make the code excessively ugly.  Conversely if someone
> feels they want to develop a parallel branch that does work on 3 that
> might be an interesting experiment.From the unmetricated stories from the field section, i.e. anecdotal
evidence to be treated as such.

SCons, like Bazaar, is sticking with 2.4 as a floor for the short- to
medium- term. So there is a consistency of approach amongst Python-based
frameworks that worry about how far back to go.

Waf 1.5.x requires Python 2.4 -> 2.6 to create the project specific
build code, but that generated code runs with any of Pythons 2.3 -> 3.1.
Thomas Nagy has written a document about plans for Waf 1.6.x and from
that document:

        Python 3 is now the default syntax. Waf now runs unmodified for
        2.6, 2.7, 3.0 and 3.1. Upon execution, a detection is performed
        and the code is then converted for python 2.3, 2.4 and 2.5 if
        necessary. It is far easier to modify Python3 code to run on
        Python2 than porting Python 2 code to Python 3.

Which seems to show there is a consistency of approach tot he Python 2 /
Python 3 issue across the board.
Ad
Home | About | Privacy