Public bug reported: Binary package hint: bzr bzr 2.1.1 fails through a HTTP proxy, both on http and lp URLs: johan@transwarp:/tmp/jox$ bzr branch http://bzr.savannah.gnu.org/r/gnash bzr: ERROR: Connection error: while sending POST /r/gnash/.bzr/smart: [Errno 113] No route to host johan@transwarp:/tmp/jox$ bzr branch lp:sl bzr: ERROR: Connection error: while sending POST /bazaar/: [Errno 110] Connection timed out Here are the proxy-related environment variables provided through GNOME's "system wide" network proxy settings: johan@transwarp:/tmp/jox$ set|grep -i http HTTPS_PROXY=https://www-proxy.se.company.com:80/ http_proxy=http://www-proxy.se.company.com:80/ https_proxy=https://www-proxy.se.company.com:80/ Note that everything needs to go through the proxy, there's no direct Internet access. Here's my bzr version: johan@transwarp:/tmp/jox$ bzr version Bazaar (bzr) 2.1.1 Python interpreter: /usr/bin/python 2.6.5 Python standard library: /usr/lib/python2.6 Platform: Linux-2.6.32-22-generic-i686-with-Ubuntu-10.04-lucid bzrlib: /usr/lib/python2.6/dist-packages/bzrlib Bazaar configuration: /home/johan/.bazaar Bazaar log file: /home/johan/.bzr.log ProblemType: Bug DistroRelease: Ubuntu 10.04 Package: bzr 2.1.1-1 ProcVersionSignature: Ubuntu 2.6.32-22.33-generic 220.127.116.11+drm33.2 Uname: Linux 2.6.32-22-generic i686 NonfreeKernelModules: fglrx Architecture: i386 Date: Thu May 27 14:30:47 2010 ProcEnviron: LANGUAGE=sv_SE:sv:en_GB:en PATH=(custom, user) LANG=sv_SE.utf8 SHELL=/bin/bash SourcePackage: bzr ** Affects: bzr (Ubuntu) Importance: Undecided Status: New ** Tags: apport-bug i386 lucid
** Attachment added: "Dependencies.txt" http://launchpadlibrarian.net/49203571/Depend...
So, there are two quite different things happening here. 2.2b3 and newer include a fix for the lp: urls to work through https_proxy (though usually that would be https_proxy=http://host:post/ - we haven't tested with an https listening proxy. Nevertheless, that part is fixed and has a separate bug. So lets focus on the gnash issue. Can you get a backtrace for it please - run bzr with -Derror. thanks, Rob
<snip/> > Can you get a backtrace for it please - run bzr with -Derror. and -Dhttp
I got different output in different runs. Here's the first run, it was done with a Swedish locale and the error message... [Errno 110] Förbindelsen dog ut (timeout) ... means... [Errno 110] The connection timed out (timeout) ** Attachment added: "Another output from bzr branch -Derror http://bzr.savannah.gnu.org/r/gnash" http://launchpadlibrarian.net/49257406/Derror...
Yes, the proxy is correctly configured :-). Curl works for example: johan@transwarp:/tmp/ech$ curl http://ftp.sunet.se/pub/FreeBSD/CERT/README | wc -l % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 101 3258 101 3258 0 0 45965 0 --:--:-- --:--:-- --:--:-- 93085 92 johan@transwarp:/tmp/ech$ Attaching output with "-Dhttp" added. I can't see that I got anything that I didn't see before though. Note that I don't have any direct Internet access, so *everything* has to go through the proxy. DNS requests will fail for example, you have to pass URLs with host names to the proxy. ** Attachment added: "Output from "bzr branch -Derror -Dhttp http://bzr.savannah.gnu.org/r/gnash"" http://launchpadlibrarian.net/49422930/bzr-dh...
Sorry. I've been unclear. What we need here is the relevant part of ~/.bzr.log as -Dhttp dumps the headers for all the requests there and that's where I expect to get relevant information to diagnose the problem.
Oh, and be sure to inspect these headers for auth info, we don't need them (we just need to make sure they are present when needed) but they may contain sensitive data (like poorly encrypted user/passwords), look for 'Authorization' headers in particular.
Here's the .bzr.log after running with -Dhttp. I don't see anything about any proxy in there; how does bzr detect it should go via a proxy? Regards /J ** Attachment added: ".bzr.log with -Dhttp" http://launchpadlibrarian.net/49437462/.bzr.l...
Huh ? bzr checks the *_proxy environment variables to decide whether a proxy should be used... ... urgh, of course the proxy handling tracing doesn't use -Dhttp :-( Since we don't connect, -Dhttp can't output anything (the 'About to connect' lines are unconditional). Hmm, that needs to be fixed, the only solution I can think of for you is to edit the bzrlib/transport/http/_urllib2_wrappers.py and set the DEBUG variable to 3 (instead of 0). It's nearly at the top of the file after the docstring. Sorry about that.
** Attachment added: "~/.bzr.log from running with DEBUG=3 in _urllib2_wrappers.py" http://launchpadlibrarian.net/49482846/bzr-de...
Thanks for the detailed feedback. So, AFAICT from these output, the proxy variables are not seen and bzr try to connect to 0.105 * About to connect() to bzr.savannah.gnu.org:80 yet, Will bind http_request for 'http://www-proxy.se.oracle.com:80/' Will bind http_request for 'https://www-proxy.se.oracle.com:80/' means that we've seen the http_proxy and https_proxy variables (typo in the second line, should be https_request, not http_request). That seem to leave only the all_proxy or no_proxy variables playing tricks. Can you check those ? What we should see is: 0.105 * About to connect() to www-proxy.se.oracle.com:80 (proxy for bzr.savannah.gnu.org:80) In any case, I'll mark this bug as confirmed if only to cleanup this part of the tracing. ** Changed in: bzr (Ubuntu) Status: New => Confirmed ** Package changed: bzr (Ubuntu) => bazaar (Ubuntu)
johan@transwarp:/tmp$ echo $all_proxy socks://www-proxy.se.oracle.com:80/ johan@transwarp:/tmp$ echo $no_proxy localhost,127.0.0.0/8,*.local,*.oracle.com,*.oraclecorp.com,infraworklist,wiki,trat,tendril,infradb01,bkstage,*.jrpg.bea.com,*.bea.com,reporting,qareport,devportal,monitor, Unsetting both variables doesn't change anything, I still get exactly the same symptoms. The "About to connect() to" lines still don't mention anything about the proxy.
Marking as private because of list of private hosts. Editing them out of comment 15 would probably work as well but I don't think that's possible for me.
No, we can't edit the comments so I suppose marking it private is the only alternative :-/ So, if unsetting the no_proxy variable doesn't work, the last resort is that python's urllib.proxy_bypass() is telling us to avoid the proxy for bzr.savannah.gnu.org ? That sounds quite suspicious and hard to believe... >From there, the alternatives are: - you try single-stepping with pdb if you have a basic python knowledge, the focus is on ProxyHandler.set_proxy method (which should be called for all requests whether or not a proxy exists) or just had some print statements there to better locate the root cause (the method is defined at line ~930 in _urllib2_wrappers.py), - wait for me to provide an instrumented branch for you to try (this will certainly be delayed a bit as I'm swamped with other bugs right now) I can't yet figure out what is happening by just reading the code :-/ None of the domains mentioned in the no_proxy variable should match bzr.savannah.gnu.org at least.
I think I found it. My NO_PROXY environment variable had a list of hosts ending in ",": johan@transwarp:/tmp/bzr-2.1.1$ echo $NO_PROXY localhost,127.0.0.0/8,*.local,*.oracle.com,*.oraclecorp.com,infraworklist,wiki,trat,tendril,infradb01,bkstage,*.jrpg.bea.com,*.bea.com,reporting,qareport,devportal,monitor, If I remove the trailing comma from $NO_PROXY I can create the branch. So the issue is that bzr needs to handle trailing commas (and probably consecutive commas as well) in *PROXY and *proxy environment variables.
hmpf Well done !>>>>> Johan Walles writes: > I think I found it. > My NO_PROXY environment variable had a list of hosts ending in ",":branch. Pfew, what a journey. I'm glad you're unblocked.> johan@transwarp:/tmp/bzr-2.1.1$ echo $NO_PROXY > localhost,127.0.0.0/8,*.local,*.oracle.com,*.oraclecorp.com,infraworklist,wiki,trat,tendril,infradb01,bkstage,*.jrpg.bea.com,*.bea.com,reporting,qareport,devportal,monitor, > If I remove the trailing comma from $NO_PROXY I can create the
** Summary changed: - Bzr 2.1.1 broken through HTTP proxy + NO_PROXY env variable ending with ',' tricks bzr into never proxying any host
** Package changed: bazaar (Ubuntu) => bzr (Ubuntu) ** Also affects: bzr Importance: Undecided Status: New ** Changed in: bzr Importance: Undecided => Medium ** Changed in: bzr Status: New => Confirmed
** Changed in: bzr Assignee: (unassigned) => Vincent Ladeuil (vila)
** Changed in: bzr Assignee: Vincent Ladeuil (vila) => Martin Pool (mbp) ** Changed in: bzr Status: Confirmed => In Progress
This bug is present in the Ubuntu bzr package, but I don't think it should be explicitly fixed there. It will be fixed when Ubuntu gets bzr 2.4. ** Changed in: bzr (Ubuntu) Status: Confirmed => Won't Fix
** Changed in: bzr Status: In Progress => Fix Released