...a clever blog name here...

A recently merged commit to the CyanogenMod tree caught my eye recently - Change I5be9bd4b: bluetooth networking (PAN). Bluetooth Personal Area Networking permits tethering over bluetooth. Now this is something that will allow me to tether a Wifi Xoom to my phone when I'm someplace without wifi. Unfortunately, while this is in the CM tree - we need some additional kernel support for the Nexus S. The prebuilt kernel provided with CM 7.1.0 RC1 doesn't include BNEP support which we need to complete this feature. So lets build a kernel module.

The Nexus S kernel is maintained at android.git.kernel.org but I notice there is also a cyanogenMod fork on github as well. So to begin we can clone these and see what differences exist between the stock kernel and the CM version. To setup the repository:

mkdir kernel && cd kernel
git clone git://android.git.kernel.org/kernel/samsung.git samsung
cd samsung
git remote add cm git://github.com/CyanogenMod/samsung-kernel-crespo.git
git remote update
git log --oneline origin/android-samsung-2.6.35..cm/android-samsung-2.6.35 
f288739 update herring_defconfig

The only changes between the samsung repository and the CM repository are a single commit changing the configuration. This has evidently been done using the kernel configuration utility so its a bit hard to work out the changes by comparing the config files directly. However, if I use each config in turn and regenerate a new one via the kernel configuration utility I can then extract just the changes.

git cat-file blob cm/android-samsung-2.6.35^:arch/arm/configs/herring_defconfig > x_prev.config
git cat-file blob cm/android-samsung-2.6.35:arch/arm/configs/herring_defconfig > x_cm.config
Then make gconfig and load each one, saving to a new file.
diff -u z_prev.config z_cm.config

and we can see that the new settings are just:

CONFIG_SLOW_WORK=y
CONFIG_TUN=y
CONFIG_CIFS=y
CONFIG_CIFS_STATS=y
CONFIG_CIFS_STATS2=y
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y

In current versions of Linux the modules retain version information that includes the git commit id of the kernel source used to build them. This is also present in the kernel and the kernel will reject a module with the wrong id. So to make a module that will load into the currently running kernel I need to checkout that version - simple enough as it is the current head-but-one of the cm-samsung-kernel repository (e382d80). Adding CONFIG_BNEP=m to the kernel config file enables building BNEP support as a module and taking the HEAD herring_defconfig and building HEAD^ results in a compatible bnep.ko module.

To test this I copied the module onto the device and restarted the PAN service.

% adb push bnep.ko /data/local/tmp/bnep.ko
% adb shell
# insmod /data/local/tmp/bnep.ko
# ndc pan stop
# ndc pan start
# exit

With this done we can try it out. The Blueman app on ubuntu lets me rescan the services on a device and following the above changes the context menu for my device now shows Network Access Point. Selecting this results in a bluetooth tethering icon on the Nexus S and we are away. Further checking on the Xoom with Wifi disabled proves that it all works. Routing is properly configured and the Xoom can access the internet via the phone.

To make this permanent, I could just remount the phone system partition read-write and edit /system/etc/init.d/04modules to add my new module on restart. That works ok. However, I may as well add the configuration changes from above to the current samsung stock kernel and change the kernel in use when I re-build a CyanogenMod image. So that is what I am running now.

Moving Python packages out of the Windows Roaming profile

On my work machine it has recently been taking quite a while to start-up when it is first turned on. This tends to mean there is too much data in the corporate roaming profile so I started to look into this. I recently installed ActivePython and added a few packages using the python package manager (pypm). This has dumped 150MB into my Roaming profile. This is the wrong place for this stuff on Windows 7. It should be in LOCALAPPDATA which would restrict it to just this machine (where I have Python installed) and not get copied around when logging in and out.

A search points up PEP 370 as being responsible for this design and this document suggests how to move the location. In the specific case of ActivePython the packages are stored in %APPDATA%\Python so we need to set the PYTHONUSERBASE environment variable to %LOCALAPPDATA%\Python. Editing the environment using the Windows environment dialog and using %LOCALAPPDATA% doesn't work correctly as the variable does not get expanded. However, we can set it to %USERPROFILE%\Local\Python which is expanded properly and produces right result.

Once the environment is setup we can move the package directory from %APPDATA%\Python to %LOCALAPPDATA%\Python and check everything is still available:

C:\Users\pat> pypm info
PyPM 1.3.1 (ActivePython 2.7.1.4)
Installation target: "~\AppData\Local\Python" (2.7)
(type "pypm info --full" for detailed information)

C:\Users\pat>pypm list
numpy        1.6.0    NumPy: array processing for numbers, strings, records, a
pil          1.1.7~2  Python Imaging Library
xmpppy       0.5.0rc1 XMPP-IM-compliant library for jabber instant messaging.

I have been running various versions of CyanogenMod on my HTC Magic for a long time now. Ever since I decided that Vodafone were never going to bother to release any updates after the Donut (Android 1.6) update. However, I recently noticed that in December 2010 they finally release an update to upgrade these phones to Android 2.2.1 (FroYo). The files for this can be downloaded from Google's servers just like the last OTA update. This time there are two files - one to update the bootloader and one for the operating system update.

Before going any further I'll just point out that when I started this the fastboot screen reports the following settings and the final settings after completing the update in column 2:

Before	After
SAPPHIRE PVT 32B SHIP S-ON G	SAPPHIRE PVT 32B SHIP S_OK G
HBOOT-1.33.0004 (SAPP10000)	HBOOT-1.33.0013 (SAPP10000)
CPLD-10	CPLD-10
RADIO-2.22.19.26I	RADIO-2.22.28.25
Apr 9 2009,23:30:40	Oct 21 2009,22:33:27

Official OTA updates are written to expect a known configuration as a starting point. So if we want to use this it is time to revert back to the original image shipped with this phone. Fortunately I have my nandroid backup from the 1.5 days, so using the Amon RA recovery image I have installed as part of the CyanogenMod ROM I can revert back and begin applying first the update to Donut and then the new one to FroYo. Part of the reason for this is that the lastest update includes a new radio image which apparently leaves a bit more RAM available for the running system. As memory contention is the most significant problem with the HTC Magic, this has to be good. We shall see.

I downloaded 3 update files. The update from Cupcake to Donut, the bootloader update and the update from Donut to FroYo. Each file in turn needs to be copied to the sdcard and called update.zip. (adb push filename /sdcard/update.zip). Then reboot the phone holding down the Home key to restart in recovery mode. Once the recovery image is shown, pressing Home and Power shows a menu and you can select apply update.zip to flash the image. The hboot image looks quite scary as it works - it reboots the phone 3 times but eventually boots the operating system once again. Just wait for it patiently.

So now I have the official Vodafone released Android 2.2.1. So lets see about making a backup. Boot the phone to fastboot mode and try running the Amon RA recovery image: fastboot boot recovery-RA-sapphire-1.7.0G.img. Access denied. I had a suspicion this might happen given the bootloader update. This is a shame but not really a problem. We can use various rooting methods but I simply copied psneuter to /data/local/tmp and ran that from an adb shell. With that done I have a root shell and can remount the system partition readwrite and copy su, buzybox and the Superuser.apk file from my CyanogenMod build tree. This fixes root access.

Completing the job requires changing the recovery image so that I can boot that and make complete nandroid backups. The Vodafone release includes a script that checks for a valid image in the recovery partition and replaces it if it no longer matches the known version. This script is in /etc/install-recovery.sh and this is called from the Android init process. It updates the recovery partition by patching a copy of the boot partition using the binary patch file in /system/recovery-from-boot.p. So to make our recovery stick around we need to rename the install-recovery.sh script and the /system/recovery-from-boot.p files and place our own copy of Amon-RA 1.7.0G at /system/recovery.img. Then we can flash this using flash_image recovery /system/recovery.img.

The reason Vodafone use a patch file is to keep the size of the recovery copy small. Possibly we could so the same thing however I used the imgdiff program from the Android build to generate a patch to from the current boot image to the new recovery image but it just makes a patch containing the whole image. Clearly there is insufficient commonality between the two to make this worthwhile.

Now at last I can make a nandroid backup of the updated system by booting to the new recovery screen. And I've got root access for anything that might require it. And after all this: Was it worth it? Actually yes. The phone seems to be running more smoothly with this over CyanogenMod 6. Watching logcat there are less messages about processes being killed due to low memory. So far it does appear to be worth doing. But I'm noticing all sorts of little CyanogenMod tweaks that are no loger with me. So how long it will stay this way I'm not sure.

The Tcl developers have now switched from CVS to and new and shiny distributed version control system. There are an number of these to select from with various advantages and disadvantages. Git is perhaps the most famous and possibly the most powerful although it can be unintuitive for Windows users. Mercurial is getting extremely popular as it works fairly smoothly on Windows as well as unix platforms. Bazaar appears to be another contender.

Tcl has chosen fossil. This is a rather less well known DVCS written by the guy who did sqlite. It seems to do what all the others do. So we are going to be using fossil.

Fossil holds the repository files in an sqlite file so we create a clone database and then 'open' a working tree from this repository file.

fossil clone http://mirror1.tcl.tk/tcl /opt/repos/tcl.fossil
mkdir /opt/src/tcl
cd /opt/src/tcl
fossil open /opt/repos/tcl.fossil

Following this procedure we now have a local repository that we can share and a working tree that we can build. To pull in more changes from the remote we can to fossil pull and then use fossil update when we want to update the working tree or switch to another branch (eg: fossil update core-8-5-branch).

% fossil update
Autosync:  http://core.tcl.tk/tk
                Bytes      Cards  Artifacts     Deltas
Sent:             130          1          0          0

Huh? I was only trying to update from the local repository. Why is it talking to the network? This might not be a problem but at the time I was offline with no WiFi link. It turns out that fossil has per-repository settings (fossil settings) and one of these (autosync) makes it work like CVS. We need to disable this using fossil settings autosync 0. And I'll need to do that each time I make a clone. At least it now works like a typical DVCS.

Now to setup on the Windows machine. To save bandwidth I'll use a copy of the repo I obtained earlier on the linux machine and update it. This is partly because cloning the tcl core repository is really damn slow - apparently because the Tcl core machine is bandwidth limited.

C:\opt\tcl\src\tk>fossil open /opt/repos/tk.fossil
c:\opt\bin\fossil.exe: incorrect repository schema version
c:\opt\bin\fossil.exe: you have version "2011-02-25 14:52" but you need version
"2011-01-28"
c:optinossil.exe: run "fossil rebuild" to fix this problem

Hmm. We seem to be exposing our guts a bit here. So a given repository would only seem to work with a given version of fossil? A sign of an immature system I think. At least it told me how to fix this.

C:\opt\tcl\src\tk>fossil rebuild -R /opt/repos/tk.fossil
  100.0% complete...

C:\opt\tcl\src\tk>fossil open /opt/repos/tk.fossil
c:\opt\bin\fossil.exe: already within an open tree rooted at C:/opt/tcl/src/tk/

Huh!

C:\opt\tcl\src\tk>dir
 Volume in drive C has no label.
 Volume Serial Number is 56FF-5C9C

 Directory of C:opt	clsrc	k

10/03/2011  12:15    <DIR>          .
10/03/2011  12:15    <DIR>          ..
10/03/2011  12:15             7,168 _FOSSIL_
               1 File(s)          7,168 bytes
               2 Dir(s)  163,961,884,672 bytes free

Delete and repeat.

C:\opt\tcl\src\tk>fossil open /opt/repos/tk.fossil

C:\opt\tcl\src\tk>fossil pull
Server:    http://core.tcl.tk/tk/
via proxy: http://uknml1869:3129/
                Bytes      Cards  Artifacts     Deltas
Sent:             146          2          0          0
Received:        2854         62          0          0
Sent:             741         15          0          0
Received:        6766         74          5          8
Total network traffic: 876 bytes sent, 2983 bytes received

C:\opt\tcl\src\tk>fossil update
UPDATE ChangeLog
REMOVE doc/.cvsignore
REMOVE unix/.cvsignore
UPDATE unix/configure.in
REMOVE win/.cvsignore
--------------
updated-to:   53d9debe536c57e0f54b7ab88dc941e57cf21edb 2011-03-09 17:03:34 UTC
tags:         trunk
comment:      Fix libXft check (user: rmax)
"fossil undo" is available to undo changes to the working checkout.

C:\opt\tcl\src\tk>fossil branch
  core-8-4-branch
  core-8-5-branch
* trunk

Great. We now have a current checkout of Tk and I can go and commit a patch I've been nursing in my git repository.

I recently decided that it might be nice to provide some Windows shell customizations for handling a new file format that I am working on. Making some fields available to Windows Search might be nice and customizing the default view when a user examines a directory of our files. Also customizing the icon shown so that it reflects the file content. All these are possible using Shell plugins and there is even a nice wizard for ATL in Visual Studion 2010 that will get things started.

So I began with the default ATL filehandler extension that is provided by the ATL wizard and started to add some code to the preview window and implemented the thumbnail handler. For this we can see if the file has some data we could draw and then generate an image. In our case, sometimes the file contains an image - if so, we can draw this in the preview and also use it as the thumbnail.

Now the preview view was working fine but the images fail to paint properly in the thumbnail. I reduced the code there to just draw some lines and it started to work in that the line was present but the color was always black.

So here is the code used by ATL to prepare the drawing context. It creates a memory display context and selects a bitmap into that and when we later draw on this memory DC the result ends up in this bitmap.

BOOL GetThumbnail(
		_In_ UINT cx,
		_Out_ HBITMAP* phbmp,
		_In_opt_ WTS_ALPHATYPE* /* pdwAlpha */)
	{
		HDC hdc = ::GetDC(NULL);
		RECT rcBounds;

		SetRect(&rcBounds, 0, 0, cx, cx);

		HDC hDrawDC = CreateCompatibleDC(hdc);
		if (hDrawDC == NULL)
		{
			ReleaseDC(NULL, hdc);
			return FALSE;
		}

		HBITMAP hBmp = CreateCompatibleBitmap(hDrawDC, cx, cx);
		if (hBmp == NULL)
		{
			ReleaseDC(NULL, hdc);
			DeleteDC(hDrawDC);
			return FALSE;
		}

		HBITMAP hOldBitmap = (HBITMAP) SelectObject(hDrawDC, hBmp);

		// Here you need to draw the document's data
		OnDrawThumbnail(hDrawDC, &rcBounds);

		SelectObject(hDrawDC, hOldBitmap);

		DeleteDC(hDrawDC);
		ReleaseDC(NULL, hdc);

		*phbmp = hBmp;
		return TRUE;
	}

There are two problems here. When a memory DC is created it has by default a monochrome bitmap. Here is what MSDN has to say:

A memory DC exists only in memory. When the memory DC is created, its display surface is exactly one monochrome pixel wide and one monochrome pixel high. Before an application can use a memory DC for drawing operations, it must select a bitmap of the correct width and height into the DC.

So when we then create a compatible bitmap from this, we get a monochrome bitmap. So the first fix is to use the window display context so that we can support a color bitmap: CreateCompatibleBitmap(hdc, cx, cx);.

The second problem shows up when reading the documentation for IThumbnailProvider::GetThumbnail.

phbmp

[out] When this method returns, contains a pointer to the thumbnail image handle. The image must be a device-independent bitmap (DIB) section and 32 bits per pixel.

Oops. CreateCompatibleBitmap creates device dependent bitmaps. We need to be using CreateDIBSection to get a device independent bitmap. If we create a DIB and select that into the memory display context then all should be well. So to fix this the default GetThumbnail() function must be overridden to prepare a proper surface for drawing.

BOOL CDemoDocument::
GetThumbnail(_In_ UINT cx, _Out_ HBITMAP* phbmp, _In_opt_ WTS_ALPHATYPE* /* pdwAlpha */)
{
    BOOL br = FALSE;
    HDC hdc = ::GetDC(NULL);
    HDC hDrawDC = CreateCompatibleDC(hdc);
    if (hDrawDC != NULL)
    {
        void *bits = 0;
        RECT rcBounds;
        SetRect(&rcBounds, 0, 0, cx, cx);

        BITMAPINFO bi = {0};
        bi.bmiHeader.biWidth = cx;
        bi.bmiHeader.biHeight = cx;
        bi.bmiHeader.biPlanes = 1;
        bi.bmiHeader.biBitCount = 32;
        bi.bmiHeader.biSizeImage = 0;
        bi.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
        bi.bmiHeader.biClrUsed = 0;
        bi.bmiHeader.biClrImportant = 0;

        HBITMAP hBmp = CreateDIBSection(hdc, &bi, DIB_RGB_COLORS, &bits, NULL, 0);
        if (hBmp != NULL)
        {
            HBITMAP hOldBitmap = (HBITMAP)SelectObject(hDrawDC, hBmp);
            OnDrawThumbnail(hDrawDC, &rcBounds);
            SelectObject(hDrawDC, hOldBitmap);
            *phbmp = hBmp;
            br = TRUE;
        }
        DeleteDC(hDrawDC);
    }
    ReleaseDC(NULL, hdc);
    return br;
}

Now it works!

I have 64 bit Ubuntu 10.10 installed on my laptop. When using the default Gnome desktop the volume wheel on the side of the laptop correctly controls the audio volume. However I quite like using XFCE so I have been booting into an XFCE desktop instead recently and I noticed that the volume wheel doesn't work for this desktop. I turns out there is an additional package to install that is not included when you apt-get install xfce4 and this is xfce4-volumed. As soon as this daemon was running, the volume wheel operated as expected. Excellent!

I installed the 64 bit version of Ubuntu 10.10 a few days ago. I had an 32 bit version installed but the new Android build requires 64 bit linux so it was time for a complete re-install. Following the re-install I found the pointer keeps halting for random amounts of time which gets really tiresome very quickly. As I had installed the proprietary nVidia drivers the first thing to do was to remove those. Unfortunately that didn't change anything so I added the XFCE desktop and booted that instead of the default Gnome one. I also tried Gnome without compiz but none of these solved the fault. I vaguely remembered having some trouble with this before so looked for kernel interrrupt issues. Some articles suggest the ACPI system can be a problem so we can reboot and add acpi=off as a boot parameter or a more restricted pnpacpi=off. This has worked fine. I've since re-enabled the current nVidia driver and the mouse has been behaving itself ever since.

The grub boot system has moved around since previous versions of ubuntu. It is no longer in /boot/grub/menu.lst but /etc/default/grub. It is quite obvious what to change and I just appended pnpacpi=off to the GRUB_CMDLINE_LINUX_DEFAULT parameter to resolve things permanently.

While I was poking about in grub2 I took the opportunity to change the grub background. The /etc/grub.d/05_debian_theme file contains options to change the background image and the colors and there is a handy grub2-splashimages package to get a few pre-made images to use.

Tcl 8.5.9 was released today after a fairly long gestation. So I've uploaded TclKits for 32bit Windows and Intel Linux already.

The Windows builds use MSVC6 and the Linux ones were build on Debian Etch so in both cases these should be usable on a wide range of systems ancient and modern.

I was admiring a co-workers new Kindle device recently and noticed that there is a Kindle application for Android available in the Android Market. This application works quite well. The e-books are reasonably easy to read although without the e-ink screen technology used by the real Kindle there is no reading in bright sunlight on a phone. However, buying a book (even a free one) from Amazon makes it pretty much immediately available on the phone to be read.

However, I'm interested to try it out with Project Gutenberg e-book. This project has for years been colecting electronic versions of free and out-of-copyright texts. They are now available in various e-book formats in addition to the usual plain text that Project Gutenberg prefers. It turns out that you can use these texts with the Android Kindle application. You download the Kindle version of your chosen e-book and it arrives as a .mobi file. This needs to be moved to the kindle directory on your sdcard. Once there, the next time the Kindle app is started it will notice the new files and show them in the list. They won't have cover images but otherwise they work great.