...a clever blog name here...

I just fired off yet another request to get access to the Coverity Scan information for the Tk project. I fully expect another round of resounding silence for this request as that will match every other attempt I've made. It seems that if you missed making such a request just after Coverity's original public announcement of the Open Source scan results you just get ignored forever. Thats not that suprising really as managing such access must have some cost and they are providing this for free. On the other hand it look pretty shoddy from where I sit. They made the offer to permit maintainers to access these results as a PR exercise and then ignore anyone that tries to follow up on this. The Tcl project got access and fixed a few issues and moved to 'Rung 2' along with a small number of other fairly high profile projects. But I also know that the Tcl maintainer who had access to this no longer does so and his requests have since been ignored too. I cannot say this encourages me to recommend this service for use in my commercial development. An all-round PR fail as far as I am concerned.

Following a lot of discussion on an msysGit issue and having researched this quite a bit myself I decided the state of proxying git on Windows could use some summarizing. In brief, it can be done for some circumstances and not for others. We will need some git patches to let everyone be able to operate over proxies.

Proxying the git protocol over an http proxy

The git protocol is a simple transport used over stream connections that is used when the repository URL begins with git://. This transport can be made to use a proxy by providing a command to handle the proxy connection. This is done by setting GIT_PROXY_CONNECTION or the core.gitproxy variable to point to some program that will make a connection to the git server on the target site. The connect program can be used for this purpose and creating a Windows command script like that below will use your http_proxy environment variable settings to open a connection through your http proxy and operate the git protocol over this connection.

@connect -h %*

There is a problem with this however. The way to make a direct connection through an http proxy is to issue the CONNECT command with the hostname and port of the target site. In this case we will be issuing CONNECT git.example.com:9418. Generally http proxies are configured to deny making connections to anything other than port 443 (the secure http port) and perhaps a limited set of additional ports (imaps is also common). So there is a good chance that your proxy will refuse this request and quite often it seems it will not bother to tell you either. If this is the case then you are restricted to using some other transport method unless your proxy admins will permit the git port.

Incidentally, it seems that curl cannot be used to make such a proxy tunnel. Curl provides good support for using proxies with and without authentication and even has a --proxytunnel option that will make it use the CONNECT method with the proxy to open a tunnel to the remote site. However it then always issues a GET request or some other request as specified by the --request option. It cannot be instructed just to silently pass data over this link once it is connected. For this reason we use the connect program as described above.

Proxying the git protocol over a SOCKS proxy

See the previous article for this but the essentials are that you set the SOCKS_SERVER environment variable to point to your SOCKS5 server and then use connect -s %* instead of using the -h option as above.

Proxying git using the http protocol

Git supports using the http protocol when the repository url uses the http:// scheme identifier. In this case it uses standard HTTP for the transport. HTTP can obviously be proxied so all that is necessary to use a proxy is to set the git variables or http_proxy environment variable. If authentication is required then this can be included in the proxy url (eg: http://userid:password@proxy.example.com:3128/. Alternatively curl supports the use of a .netrc file where authentication details can be recorded on a per hostname basis. On Windows this is called _netrc and needs to live in your HOME directory (a HOME environment variable may need to be created). The layout of the file is as below.

machine git.example.com login username password secret

Unfortunately it appears that curl will always use Basic authentication with the proxy. If your proxy needs something else, perhaps NTLM for a Windows network, then you have a problem. Curl is used to handle all the http transport details and this does support the NTLM authentication method but I know of no method to pass the necessary options to curl. Git makes use of curl via its library binding so it is not enought just to replace the curl executable with a wrapper script. It might be sufficient to compile the curl library with SSPI support. On windows this can be used to pass the logged in user authentication details to a program and usually works for NT domains with proxies that use NTLM. Otherwise it seems the http transport in git needs to have some variables added to control this.

One of the problems with running services on a unix box is that port numbers below 1024 are restricted. A process cannot bind to a low numbered port unless it is owned by the super-user. In the context of tclhttpd this means we have to launch the tcl process using the root account and then drop privileges as soon as we have bound our listening sockets by switching to another user identity. This can be done quite handily using the TclX extension and tclhttpd has support for this. But I have an aversion to running script interpreters as root.

One solution that is often used with scripted services like tclhttpd is to operate on a non-restricted port and have firewall rules rewrite the incoming traffic to send it from port 80 to your service port (perhaps 8080). With firewalls supporting NAT redirection this is a simple solution but once we consider ipv6 we have a problem. IPv6 has no need for NAT and the Linux ip6tables firewall does not provide a means to redirect packets to a different port.

An ideal solution is offered by the new Linux capabilities that is now available since kernel 2.6.24 and can be found in the latest Karmic Koala Ubuntu Server versions. Capabilities permits much more finely grained permissions assignments than the traditional mechanism of 'setuid root'. In particular we can assign our process exactly the permission required: permission to bind restricted ports - and no other permissions. So now we can set this capability and run the executable image using the correct user id from start to finish.

As a specific example, I use a tclkit-cli executable to run my tclhttpd service. I copied it into the tclhttpd tree and named it tclhttpd as this helps the init.d daemon tools to manage the process. I then used the setcap utility from the libcap2-bin package to set the correct capability. In this case we want to be able to bind the restricted ports and that is the CAP_NET_BIND_SERVICE capability.

cp /opt/bin/tclkit-cli tclhttpd
setcap 'cap_net_bind_service=+ep' tclhttpd

The capability flags above raise the effective and permitted flags for the cap_net_bind_service capability. capabilities(7) explains what all this means but basically as we did not raise the inherited flag any child processes will not have this capability.

Now I have my server listening on ipv4 and ipv6 without messing with the firewall and without requiring to be started by root. With the installation is a suitable file to automate the restart of the service it is all up and running.

It is interesting to note that this method can be used to remove the setuid bit from other programs. For instance, the ping utility requires the CAP_NET_RAW capability and does not need root access if this is provided. Once capabilities spread through the linux distributions this should help make systems just that little bit more secure.

git-gui is a graphical interface to git written in Tk that is particularly useful as a commit tool. However, it doesn't look all that native when used on Windows XP or Vista. So I have spent a few hours applying ttk magic to the application to bring it up to date for Tk 8.5/8.6. Mostly this just means the usual selection of ttk widgets over the standard ones or replacing tk_optionMenu with ttk::combobox. I left in some of the coloured backgrounds that are being used as headers but did the appropriate ttk::style stuff to make it work with the style engine. I also added support for the 8.6 tk fontchooser which shows the stock font selection dialog on MacOS and Windows. So now it looks like this on Windows 7

I actually did most of this on linux testing out the tile-qt theme so on Ubuntu it looks pretty native too if that theme is available. Under X11 there is no 'native' theme to select but you can choose one by defining TkTheme in the X resource database. Something like echo '*TkTheme: tileqt' | xrdb -merge - will select the Qt theming engine. Another option is clam which is built in to Tk 8.5.

There are still some issues to deal with. The menus and tooltips ping a bit with Ubuntu's compiz window manager. I've done some work on this recently to get the correct extended window manager hints applied. We have patched 8.5.8+ already for the menu hint but Tk needs a generic way to let scripts specify the window type using EMWH attributes. At the moment I have a wm attributes $win -type command option to do this and this works pretty well. At least the tooltips animate the right way with this and the combobox dropdown acts more normally.

I recently had cause to try and merge two branches from separate git repositories that had related content but disconnected ancestry. The situation was caused by creating a git repository from a CVS conversion. I then wanted to join work from a separate git repository that had been started from the same CVS tree. However, the developer who made this git repository had begun his git tree having already made changes to the content.

So I ran git cvsimport to convert all the CVS history into one git repository. I can now pull in the new work by adding another git repository as a new remote and fetching the master branch from this remote. However, what I now have is a disconnected chain of commits. There are now two separate chains in my repository. What I need to do is merge the two together. I can do this if I can take the first commit of the new work and inject it into the cvs converted repository as the next commit. I don't want to apply a patch, I need the state of the repository for this commit to be exactly that of the first commit of the new work. For this there is git read-tree. This command takes a git tree and reads it into the index. This is where git stages your work when you do git add and git rm and where it assembles the new tree to be committed by git commit. So by setting the index to a given tree, we are setting up a for a new commit that is exactly the tree provided. Just what I need. Having staged this tree, I can go ahead and commit it and then cherry-pick or merge the rest of the tree. In my case I cherry-picked so that I could fix the author name and tidy up the commit messages.

Below are the commands I used to fix up the tile-qt repository. First adding the remote and fetching its commits. Checking the correct tree and loading it into the index and then updating the working files from the index.

git remote add jdc file:///opt/src/tile-qt-jdc
git fetch jdc
git ls-tree 1519481 
git read-tree -m 1519481
git checkout-index -f -u -a

The final results can then be pushed to some public location like github or in this case the tktable sourceforge project.

If you have to use git from behind a firewall you may have encountered problems using the default git protocol to communicate with remote repositories. The git protocol usies port 9418 which is not normally open on corporate network setups. Sometimes however, you can use a SOCKS proxy which permits forwarding connections at the tcp level. Or you can arrange to tunnel connections over SSH using SOCKS on the client end.

OpenSSH has a -D option which enables dynamic tunnels. What this means it that your SSH session can act as a SOCKS4 or SOCKS5 proxy, accepting connection requests on a local port and actually making the connection from the remote machine.

So how do we make use of this with Git? Git permits the user to define a proxy command using either variables in the config file or with an override via an environment variable. So setting GIT_PROXY_COMMAND to some command that will use the SOCKS proxy will enable proxying git. I use the connect program to do this. I've previously used this with SSH to use http proxies but it also supports SOCKS. We need a script that will take two arguments, the host and port, and make a network connection over socks.

#!/bin/sh
connect -s $*

will have connect read the SOCKS_SERVER environment variable for the proxy address (eg: export SOCKS_SERVER=localhost:1080). Given this script we can now set

GIT_PROXY_COMMAND=$HOME/bin/socks_connect

and git will begin tunnelling over the SOCKS proxy.

This method works with Windows using msysGit as well. However, on windows it is more troublesome to use a script like this so instead I build a version of the connect program with the default option set to -s and called it socks_connect.exe. Otherwise nearly everything works the same using plink or putty for the SSH connection and enabling a dynamic tunnel on the putty configuration page.

Ever since I enabled the advanced desktop effects on Ubuntu I have had problems with repaints of various windows. This has been an issue on two different laptops for multiple versions of Ubuntu. It is due to some problem between the nVidia drivers I have been using to enable the fancy effects and the compositing window manager. If the effects are disabled the repaint issues go away.

Today I decided to google for this and encountered the ubuntu bug 269904 which covers this. From this, buried in among 300 comments is a suggestion to enable a workaround that solves the problem for me.

In the CompizConfig Settings Manager, under the Utilities section select the Workarounds page and check the "Force synchronization between X and GLX" button. This forces a more extensive repaint and fixes all the trouble I see in Emacs and other heavily scrolled windows.

We received a report from a user that in Tk 8.5.8 the treeview widget is not displaying selected rows properly on Windows. The user reported that this was fine in the 8.5.7 release. We now know that the bug was introduced somewhere between these two points. To refine this I created a small test script and switched to the 8.5 development branch.

We can begin the bisection by letting git know the references for the bad and the known good commits. These can be provided as SHA1 ids or as tags. In this case we have release tags.

C:\opt\tcl\src\tk.git>git bisect start core-8-5-8 core-8-5-7
Bisecting: 30 revisions left to test after this (roughly 5 steps)
[b2905899c223a5b9d98dd53577267ba9f26781ea] Fix [Bug 2824378] by correcting expected output.

We then build this version and run the test script and find that this commit has the bug we are interested in. So we tell bisect this is a bad commit. It then chooses a new commit and switches the working tree to that point in time.

C:\opt\tcl\src\tk.git>git bisect bad
Bisecting: 14 revisions left to test after this (roughly 4 steps)
[efc88ec4463f836d3e2a65a01034205ae0a66ee3] avoid panic for CFStrings longer than INT_MAX

Now we repeat the build and test cycle 4 more times and on eacy cycle tell git if the test passed (good) or failed (bad). Eventually it works down to a single commit and lets us know about the commit that introduced this bug.

C:\opt\tcl\src\tk.git>git bisect good
5a4d9fdad0040ecb889e37f71852ad5841716647 is the first bad commit
commit 5a4d9fdad0040ecb889e37f71852ad5841716647
Author: patthoyts <patthoyts>
Date:   Thu May 14 00:53:04 2009 +0000

    Backported support for the Vista theme.
    This requires the vsapi element engine, the hover state and the theme script
 definition.

:100644 100644 f7b29f475f368228a42832783753bd9e0155d46b 28ff725bfb329f6702ce2713f42711167165fd28 M      ChangeLog
:040000 040000 d1720dc0d00d10c9fb3e3d50b852164276942b42 9ed214bee06fd297e88e1fdeca77bc9319fcc386 M      doc
:040000 040000 24c258f8c1068b82ff3f9d7850611ee897c06ade 5a5179809754c654bcc4fb76e49fd9c476c1acfc M      generic
:040000 040000 c56aa9dadb0e2b02808b9fc34515481290091768 e0776e27ae03e709e2cc6da95ce1bf9ef43ad8a7 M      library
:040000 040000 76e04b7d2c6f7e851d68d420bab551382818b7bf 08bb26a478f72b8e92baebf0213e1eb0b4d0ce92 M      tests
:040000 040000 2ffb214a52d608f8801c7a4cab210f082528c398 0178942ec72d67f4bddc3a8a88870dcd83869ff1 M      win

Now I can use gitk to examine the details of this commit and see that when I backported this from the 8.6 branch it removed some entries for the XP theme that are still required on 8.5 even though the elements have been removed from 8.6. A quick patch and test and the fix is ready.

The final thing to remember is to tell git you have finished using git bisect reset so that it can clean up the internal tags it uses to tract the bisect progress and move your working tree back to where it began.