Ubuntu Intrepid Ibex is Five Kinds of Fail

Posted to linux on 2008-11-16 08:31:00

I've been a Ubuntu user since before it was called Ubuntu, back in the days when the domain name was no-name-yet.com. It wasn't a hard switch from Fedora Core, which at the time was Core 1 or Core 2. Ubuntu was like entering a magical world where stuff just worked. At the time I had an IBM ThinkPad A31 -- it worked perfectly, wifi and all. Now, things weren't quite as good as they are now, the magic of NetworkManager had not yet arrived, and DBUS was just a thought in j5's mind.

The various revisions of Ubuntu made it better and better. Improvements to multi-monitor support were wonderful -- suddenly presentations were easy. With Ubuntu Hardy basically everything on my laptop worked. Wifi, suspend, sound, bluetooth. It was really nice. Unfortunately, it appears that Intrepid is a step backward -- numerous features that were supported in Hardy no longer function properly. So, here I present my Intrepid Ibex Five Kinds of Fail.

  1. NetworkManager periodically disables networking (291062)
  2. Atheros drivers are no longer properly supported (259157)
  3. PulseAudio sometimes hangs after suspend/resume (292129)
  4. My thinkpad occasionally fails to suspend (298683)
  5. NetworkManager forgets my WPA saved key (276578)

I'm not intending this to be another "Linux Sux0rs" post or anything like that, but here are five major issues that worked just fine in the last version of Ubuntu and now are total fail making the Ubuntu experience of today remind me of the Ubuntu experience from Dapper days. There is certainly some irony that several of the features relate to wifi which Ubuntu brags about having great support for in Intrepid. Maybe I should have heeded jwz's advice when he lamented about upgrading three years ago. Sigh...Here's hoping Jaunty won't have the same problems.


Goodbye OpenBSD

Posted to computer on 2008-01-03 00:37:00

My servers have run OpenBSD since sometime in 1998. I still have a few really ancient official CD sets laying around for OpenBSD 2.3 (May 1998), 2.5 (May 1999), and 2.7 (June 2000). My name can be found on the OpenBSD Donations page, and I think that PF is one of the sweetest pieces of software I've seen. I've ran OpenBSD on x86, PPC, and Sparc on both physical and virtual hardware. I've used it as a mail server, spam filter, firewall, web server, database server, and even an X11 terminal. OpenBSD is just a well engineered piece of software that works well and doesn't need much maintenance.

Last year, in an effort to reduce clutter in my house and ease my power bill, I changed from a physical to virtual OpenBSD box as my main Internet server computer. For the most part I had some good luck, it worked pretty well once I changed the virtual NIC from a PCNet to an E1000. Back in October I decided that this holiday season I would upgrade to the 4.2 release of OpenBSD, which contained a lot of features I was looking forward to such as a port for Python 2.5 and the new vic VMWare network driver. About a week after I made this decision, my Apache server process began to randomly segfault when using gzip compression to serve web pages -- as someone who runs their website off a cable modem, this was a hit that I really didn't want to take. I disabled gzip compression and decided to look at it again over the holidays.

The process of bringing up a new virtual machine with VMWare server is nearly brain dead. I can create multiple virtual machines in a matter of minutes. For this upgrade the choice was made to start fresh to avoid some of the botched installs from previous versions of tools in the original virtual machine. My strategy was to install the basics for the machine, then compile the ports I needed -- apache2, subversion (with apache support), python 2.5, screen, and irrsi -- were at the top of my list. Wait, you may say, OpenBSD offers compiled packages for these -- no need to compile. This is true, but the Python setup includes the tkinter libraries, which no ones uses anymore and forces X on my system. Subversion also links to some packages I'll never use again.

After some fighting the build process got running, but I almost immediately hit a snag. Programs were complaining about expat not being present, and the port kept on telling me it was included as of OpenBSD 4.2. As I had OpenBSD 4.2 installed, I could tell that it was not included. My first strategy was to lie and build the package regardless. While this allowed me to build some software, it quickly caused problems. I was soon modifying my FLAVOR environment variable with reckless abandon. In the end, I broke enough things that I needed to start over. Luckily, VMWare makes that simple.

A little sleuthing discovered that for some reason in OpenBSD, expat is currently a part of the xbase package, and if I wanted the development headers I also needed the xshare package too. What? Why I do need X for everything all the sudden. Whatever, I installed those packages, rebuilt the software, but still needed to fudge around with lots of environment variables to keep it from installing TCL with anything that had optional TCL bindings. Don't even get me started on the headache of keeping Ruby off my system. After about six hours of actual work at fighting with it, much more if you count compilation, I had the software I thought I needed installed. A little testing revealed that all of my web services still worked. Great! Then I turned on gzip and the segmentation faults returned. I returned to the default configuration, and the segmentation faults continued. I debugged the program, realized I need to do some nasty patching, and concluded that this wasn't going to be the proper choice.

It was time to consider something else. I really wanted to run a BSD system still, I figured I'd give FreeBSD a shot again. For some reason their automatically discovered mirror for me was ungodly slow, worse than dialup. Going through tor actually managed to speed things up. Wow, that's strange isn't it? I also noticed they were close to a release -- I don't want to run pre-release software and I didn't want to have to upgrade, and I'm really impatient. Crap. By this point it was the 27th, and I was supposed to leave for a week in just a few days.

Suck it up, install Ubuntu Server. I really didn't want to run Linux, and Ubuntu Server may have a bit too much stuff on it in the default install. I was already starting to feel dirty. A little more digging revealed a nifty project, Ubuntu JeOS (Just Enough Operating System), a specialized and stripped down version of Ubuntu that has the core software and an optimized VMWare kernel. The download was pretty small, at about 150MB or so. Installation was a breeze and incredibly straightforward. As an added bonus, VMWare tools compiles for Linux, to further improve performance. A few apt-get commands later and my system was running. What's better is that the packaging system didn't require me to install X11, TCL, Ruby, PHP, or MySQL for my server setup. I was able to get just the tools I needed. Total installation time for everything, 41 minutes. That includes the time to copy all my web files over, port the httpd.conf file to the Ubuntu style of multiple configuration files, and update some absolute paths in scripts.

Next up was to see how fast it was. I always loved how few processes I'd see running in my OpenBSD machine. Hammering the machines over and over an testing the speed of response with YSlow showed the Linux box to complete requests in about 40% of the time of the OpenBSD box when requested from the local LAN. Furthermore, the responses were compressed, which means a great saving of time over the line. I was hooked -- sorry OpenBSD, you've been replaced.

Of course, there is a potential cost here. What about the security of my new server? What about stability? What about updates? As far as security goes, I'm already being fairly smart by running it in as a virtual machine. The main filesystems are backed up nightly to both local and remote systems, and about once a month, a backupninja script shuts down the machine, creates a snapshot of the full machine, then restarts it. That gives me about 10 minutes of downtime every month -- I'm sure the spiders will understand. I also am not running any unnecessary servers -- just apache2, an instance of CherryPy for PennAve, and very restricted OpenSSH for remote access -- that's it. I'm hoping I'm in good shape with that, but security is something you can never guarantee unless the machine is off, has no drives, and lacks any sort of persistent memory (of course, I'm sure that even then there may be a way to find some sort of electromagnetic residue in the RAM). Stability wise, I'm not entirely certain. Ubuntu Dapper on my MythTV machine has been excellent, while Gutsy on my laptop is sometimes flaky -- but that may also be because of a long upgrade chain going back up to March of 2005 on this machine. Finally, I'm pretty sure that using APT makes upgrades, especially security related upgrades, even easier. I don't need to worry about seeing that a patch was released and installing it by hand. It just happens -- yes I'm placing trust in a remote entity, but I can handle that. I just want my machine to work.

So, after two days of mucking around with OpenBSD, I finally got Ubuntu JeOS to just work, and I'm very happy with it. I'm sorry to see OpenBSD go, but it was just too difficult to work with. In particular, here's some issues that really need to be addressed to make it competitive with what JeOS offers:

  • If you say you include expat, really include it, don't place it in two different packages for X. I don't need X on my server.
  • Finer grain package control. Some packages are getting pretty good at creating subpackages for optional components -- subversion is a good example of this. Other packages, such as Python, could really use some help here.
  • Inheritance of the FLAVOR environment variable. If I compile subversion as NO_TCL or NO_RUBY, that should cascade down to swig. I had to literally watch every dependency as it was compiled to make sure that it was going to decide I needed TCL, TK, and Ruby.
  • Automated checking for security patches. I'm not sure if OpenBSD has this, but it wasn't obvious how to do it. But, then again, it's not immediately obvious how to use apt either.

For the time being, OpenBSD and I have parted ways. In a few years when it comes time to upgrade my server again, we may again join forces. Of course, with the continual increase of software as a service, in a few years, there may be no need for me to run my own server anymore.


Things They Don't Teach You In Tip-Of-The-Day: Environment Variables in GNOME

Posted to linux on 2007-10-22 10:36:00

Ubuntu has been incredibly successful in large part thanks to its ease of use and great packaging. However, one thing they don't teach you is how to force your GNOME session to export environment variables to processes. When a process is launched from a terminal, you set these environment variables through .bash_profile, .bashrc, .login or other similar files. You can't simply do this in GNOME.

The problem I'm facing is that I need a way to set a JAVA_HOME environment variable in emacs. The naïve method is to create a launcher with the command JAVA_HOME=/usr/local/java/jdk1.6.0 emacs, after all, such syntax works from the shell and launches Emacs with the variable set. However, the launcher mechanism in GNOME does not recognize this and will fail to launch.

The solution lies in a file that no one ever talks about, .gnomerc. If you add the following lines to it, you should be able to export that variable to all applications launching in GNOME:

JAVA_HOME=/usr/local/java/jdk1.6.0
export JAVA_HOME

This is the same syntax you'll expect in bash, so if you use bash, you may be able to get away with symlinking .gnomerc from .bashrc.


DPKG eats your bandwidth and rots your hard drive -- a plea for incremental packages

Posted to linux on 2007-10-09 11:02:00

Most modern packaged based operating systems have a facility for easily allowing remote systems to pull down updates. In the Windows world this is primarily used for distributing bug fixes and security patches through the Windows Update system, although major service packs may introduce new features. In Linux, depending on distribution and configuration, the system may download security and bug fixes, or it may download new versions of software running on the system. In particular, this is useful for running the development versions of distributions.

On Windows the patches typically come with cryptic names and descriptions and don't provide much of a clear description what they do. A patch is incremental against a known state of the operating system and includes logic to update to the new state. On Linux the updates come as complete new packages that can be independently installed. While this removes some of the dependency issues and allows one to install the updates as a complete package, it is often wasteful, especially when the system is going through massive updates as is the case right before major system releases. For example, here's what was presented to me this morning:

[patrick@hedgehog] sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages have been kept back:
  linux-headers-generic linux-image-generic linux-restricted-modules-generic
The following packages will be upgraded:
  alsa-base apport apport-gtk apturl auctex avahi-autoipd avahi-daemon base-files bash bind9-host bzip2 capplets-data
  command-not-found command-not-found-data compiz compiz-core compiz-gnome compiz-plugins cups-pdf cupsys cupsys-bsd
  cupsys-client cupsys-common cupsys-driver-gutenprint dbus deskbar-applet displayconfig-gtk dnsutils esound esound-common
  evince evolution evolution-common evolution-data-server evolution-data-server-common evolution-plugins flex fontconfig
  fontconfig-config fortune-mod fortunes-min gdebi gdebi-core gimp gimp-data gimp-print gimp-python gnome-app-install
  gnome-control-center gnome-keyring gnome-panel gnome-panel-data gnome-session gnome-user-guide gstreamer0.10-tools
  gtk2-engines-pixbuf gtk2-engines-ubuntulooks guidance-backends hal hal-device-manager hal-info hpijs hplip hplip-data
  hplip-doc initramfs-tools iproute irb1.8 iso-codes language-selector language-selector-common libavahi-client-dev
  libavahi-client3 libavahi-common-data libavahi-common-dev libavahi-common3 libavahi-compat-libdnssd1 libavahi-core5
  libavahi-glib-dev libavahi-glib1 libavahi-qt3-1 libbind9-30 libbz2-1.0 libbz2-dev libcamel1.2-10
  libcompizconfig-backend-gconf libcupsimage2 libcupsys2 libcupsys2-dev libdb4.3 libdb4.4 libdbus-1-3 libdbus-1-dev
  libdecoration0 libdns32 libebook1.2-9 libecal1.2-7 libedata-book1.2-2 libedata-cal1.2-6 libedataserver1.2-9
  libedataserverui1.2-8 libegroupwise1.2-13 libesd-alsa0 libesd0-dev libexchange-storage1.2-3 libfltk1.1 libfontconfig1
  libfontconfig1-dev libgda3-3 libgda3-common libgimp2.0 libgimp2.0-dev libglade2.0-cil libglib2.0-cil
  libgnome-keyring-dev libgnome-keyring0 libgnome-window-settings1 libgnome2-0 libgnome2-common libgnome2-dev
  libgnomevfs2-0 libgnomevfs2-bin libgnomevfs2-common libgnomevfs2-dev libgnomevfs2-extra libgstreamer0.10-0
  libgstreamer0.10-dev libgtk2.0-0 libgtk2.0-bin libgtk2.0-cil libgtk2.0-common libgtk2.0-dev libgtk2.0-doc libgutenprint2
  libgutenprintui2-1 libhal-dev libhal-storage-dev libhal-storage1 libhal1 libisc32 libisccc30 libisccfg30 liblwres30
  libmetacity0 libmono-accessibility2.0-cil libmono-cairo1.0-cil libmono-cairo2.0-cil libmono-corlib1.0-cil
  libmono-corlib2.0-cil libmono-data-tds1.0-cil libmono-data-tds2.0-cil libmono-dev libmono-microsoft-build2.0-cil
  libmono-peapi1.0-cil libmono-peapi2.0-cil libmono-relaxng1.0-cil libmono-security1.0-cil libmono-security2.0-cil
  libmono-sharpzip0.84-cil libmono-sharpzip2.84-cil libmono-sqlite1.0-cil libmono-sqlite2.0-cil libmono-system-data1.0-cil
  libmono-system-data2.0-cil libmono-system-runtime1.0-cil libmono-system-runtime2.0-cil libmono-system-web1.0-cil
  libmono-system-web2.0-cil libmono-system1.0-cil libmono-system2.0-cil libmono-winforms2.0-cil libmono0 libmono1.0-cil
  libmono2.0-cil libnautilus-extension1 libncurses5 libncurses5-dev libncursesw5 libnotify1 libopenal0a
  libpam-gnome-keyring libpanel-applet2-0 libpanel-applet2-dev libpcap0.8 libpng12-0 libpng12-dev libpng3 libpoppler-dev
  libpoppler-glib-dev libpoppler-glib2 libpoppler2 libreadline-ruby1.8 libruby1.8 libsane libsane-dev libservlet2.4-java
  libsnmp-base libsnmp10 libssl-dev libssl0.9.8 libtiff4 libtiff4-dev libtiffxx0c2 libtotem-plparser7 libuniconf4.3
  libvolume-id0 libvte-common libvte9 libwvstreams4.3-base libwvstreams4.3-extras libx11-6 libx11-data libx11-dev libxml2
  libxml2-dev libxml2-utils liferea linux-generic linux-kernel-devel linux-libc-dev linux-restricted-modules-common
  linux-sound-base lsb-base lsb-release m4 mencoder metacity metacity-common module-init-tools mono mono-common mono-devel
  mono-gac mono-gmcs mono-jay mono-jit mono-mcs mono-runtime mono-utils mplayer nautilus nautilus-data ncurses-base
  ncurses-bin ncurses-term ntp ntpdate openssh-client openssl poppler-utils ppp preview-latex-style python-apport
  python-avahi python-central python-gnome2-extras python-gtkhtml2 python-launchpad-bugs python-libxml2
  python-problem-report python-vte python-xml python2.4 python2.4-minimal python2.5 python2.5-dev python2.5-doc
  python2.5-minimal ruby1.8 ssh-askpass-gnome synaptic system-config-printer tasksel task
  xserver-xorg xserver-xorg-input-all xserver-xorg-vide
Do you want to continue [Y/n]?

This update was going to download 295 packages that were around 180MB. It actually claimed to free up some space on my hard drive when it was done. Luckily, I was on campus so the download actually went pretty fast. But, really, what was being accomplished? Was it really necessary to download 180MB of updates for changes that aren't even perceptible to me? I decided to find out.

Before performing the system update I got a list of all the files to be updated and saved a copy of the information. I also calculated the md5sums of the files and saved the file size for each file in each package to be updated. Next, I ran the system update and got a list of the files from the new packages, md5sums, and size of the files. I compared the two sets of information to see what files were actually updated, what files were new, and what files were removed. Finally, I made a "dummy" archive of the updated and new files by copying them into a directory and calculating the size of a tarball of those files. This is a rough estimate for the size of a incremental package.

The total size of all packages downloaded was 160.33MB, when unpacked these files took up 452.37MB, versus 476.14MB before the update; about what apt indicated I would save. However, this is not the complete story. apt caches the previously downloaded files in /var/cache/apt/archive. In reality, my drive now has an extra 136.61MB of data stored on it because of the saved packages. Although, it is safe to delete those packages in most cases.

While drive storage is a concern, it's actually pretty minor. Drive space is really cheap and a couple of hundred megs here or there don't make much of a difference. However, transferring all that data is what consumes my time. Surely it's not the case that everything in each one of those packages was new. To examine this, I plotted the number of files in the package versus the number of files the incremental update modified. The correlation is 0.43, which is neither high nor low -- some correlation is to be expected because the number of modified files can never go higher than the number of files in the package. The fact that the correlation isn't higher means that many packages are sending files that don't need to be updated.

total files vs modified files plot

For many people, correlations and log-log plots may not be the most helpful in understanding what is going on, so lets visualize this another way. Below is a histogram showing how many packages had what percentage of files modified. It's pretty clear that most packages had only a small fraction of files modified. In many cases, 80% of the files transmitted had no changes at all. Multiply that across multiple incremental updates, and that's a lot of wasted bandwidth and disk space.

histogram of percentage of files modified

After seeing the histogram, some folks will argue that updates are more likely to larger files, so we're not really wasting as much bandwidth. After all, looking at many packages, such as bash, the largest files are often binaries and libraries -- those that are most likely to be changed in the incremental updates. To understand this, I wrote a script to create dummy incremental packages. Basically, I copied all of the files that were modified into a new directory, created a tarball of the files, and examined the size of the executable. This is only a rough estimate of the size as I did not create a dpkg, so it's missing the associated overhead there and estimates of bandwidth saved may be a bit optimistic.

The total size of all the dummy packages was 76.35MB, a savings of almost 84MB, or 53% of the original size. It's like a double speedup in your download speeds. In addition, we save a bit a disk space, and the bandwidth requirements for update sites are diminished significantly. Not all updates compressed the same amount, as shown below.

histogram of percentage of incremental package size

About a sixth of the smaller packages would be under 10% of the downloaded package size, while another sixth of the packages would have little change in their size. The rest of the packages have a pretty uniform distribution.

What I'm proposing here is a form of incremental packaging -- it's apparently one of the key features of the Foresight Linux. Their system is built on top a pretty cool packaging system called conary, but I don't think that Ubuntu will adopt it any time soon. Debian folks will consider it to be some kind of conspiracy and either choose not to implement a change, or it will take them 10 years to do it.

There may be a way to accomplish this with the only slight modifications to the current dpkg infrastructure by taking some hints from digital video compression. In video compression, the entire frame is not encoded from one frame to another, rather only the incremental changes are stored. Once in a while, depending on configuration usually from every 0.5 - 3 seconds a key frame comes along. This key frame is a complete re-encoding of the image and the next frame is based off the changes from the key frame. The process repeats until the next key frame comes along.

This results in some great compression especially when the changes in a scene are small, however can take a hit when changes are significant. That's why if you look closely at HDTV signals in action films you can see some little blocks. Basically the encoding is basing it off a key frame and there is a lot of data to re-encode at a low bitrate. I'm simplifying here, but hopefully you get the picture.

As far as dpkg goes, a similar situation could be created. Periodically package managers could roll a new "key frame" version of a package, such as when a new official release of a package is made. When security updates, or bug patches are made, a incremental package would be generated with only the changes from the "key frame" -- this differs from digital video in that video works from the last incremental frame. The reason to differ from the "key frame" is because we can't assume that everyone has obtained the entire stream of packages, like we usually can with video. The information on the "key frame" package name can be stored in the package information and it could be automatically downloaded from the same source.

Here's where it could get interesting. When first installing a package, you'll end up taking a bit of a hit if you're not installing a "key frame" because you'll need both the incremental package and the key frame. However, when upgrading you'll save some nice bandwidth.

The astute reader may note that this requires you to keep a copy of the "key frame" package around, which makes some of the savings a little moot. There are a couple of different ways to address this. One is to create an archive cache on the system. When an incremental package makes modifications to a file from a "key frame" package, the original "key frame" version would be saved and probably compressed somewhere. Then when upgrading to a new incremental package, dpkg would roll back the last package and apply the new one. The roll back configuration means we won't have to save files from an incremental package that another incremental package modified. Another option is to keep a listing of the files changed somewhere and download the "key frame" package if there are enough changes that the incremental package can't deal with it.

As far as distributing the operating system on physical media, this change shouldn't make any difference -- all of the packages on the CD/DVD would be "key frames" that work independently. When upgrading and applying security fixes we'd save a lot of bandwidth. However, a down side is that such a technology is not very well suited for locations with very low bandwidth as an incremental package without a "key frame" is rather useless. I haven't quite figured out how this could work with platforms like the OLPC, but hopefully it's a start.


What you should love about Ubuntu Gutsy

Posted to linux on 2007-09-25 16:28:00

The newest releast of Ubuntu, codenamed Gutsy Gibbon, is less than a month away. In preparation for manning the GNOME booth at Ohio LinuxFest, I recently upgraded my laptop to the latest development release of Ubuntu Gutsy. This newest release features GNOME 2.20, kernel 2.6.22, and OpenOffice.org 2.3, amongst other things. This isn't meant to be a complete review of Gutsy, rather, its what you should like about it -- mainly because it's what I like about Gutsy.

One of the more controversial moves in Gutsy was enabling desktop effects by default. No longer is XGL-based-composited-desktop-bling limited to people with too much time on their hands and access to apt repositories. If you want to disable the effects, go to System->Preferences->Appearance, and select the Desktop Effects tab. Uncheck it to make the effects go away and return to your plain, boring, desktop experience. It's also nice to see that Compiz has gotten smart enough to realize that it can't run on desktops where the maximum texture size is less than the desktop resolution. This means that on my T43p, which has an ATI card with a 2048x2048 maximum texture size, Compiz will safely back to metacity if I'm running multiple monitors.

Along with compositing, Gutsy supports the new xrandr 1.2 protocol. While xrandr 1.2 breaks some older xorg.conf files, the improvement is well worth it. In my old setup, I had a customized xorg.conf that took advantage of a few ATI settings to automatically detect the monitors connected and size the output accordingly. Unfortunately, if I wanted the external display to be active, it needed to be connected when X started. This caused many unnecessary restarts of X and loss of session information. Xrandr 1.2 fixes a lot of these problems by allowing for dynamic addition of monitors and reconfiguration of those monitors. This means that while running I can plug in a new monitor, have xrandr read the EDID information for the capabilities and configure the new display without having to restart X. As someone who gives lots of presentations, this is a life saver. There's even a graphical tool, grandr, that allows you to drag and drop your monitor configuration. Grandr isn't feature complete -- I've found some bugs in grandr that cause it to crash and die. Look for me to post my git repository with fixes soon.

Another great feature for laptop users is the improved power management. Putting my laptop to sleep and then waking it up again is much faster and more reliable. Before I was never certain what key I needed to press to get the laptop to come back to life, and even then it was unreliable. Now I just tap the "Fn" key and the laptop comes right back to life. Great support that just works.

There's also the usual batch of incremental changes to the system, such as tweaked graphics, new versions of minor programs, and additional utilities to keep you happy about your shiny new Linux desktop. GNOME 2.20 has come together as one of the most solid releases yet. Add in a few companion programs like F-Spot and Banshee for Media and Conduit for synching your data and you'll have a great and open desktop experience for far less than what the other guys want.


Compiling a custom kernel can save you more than $60 a year!

Posted to linux on 2006-08-19 13:56:00

Hey you! Yes you! What does your computer do most of the time? If it's like most computers, it just sits idle. Sure, some of us may have reasons to leave our computers on all day, such as a web server, or a mythtv box, but even then they sit idle. Fortunately, CPU manufacturers have noticed this fact, and developed technologies that allow processors to slow down when not being fully utilized. On Intel chips this is called SpeedStep, and on AMD chips it's called Cool'n'Quiet.

Here's the dirty little secret you may have not known. Altough the chip you bought may support this wonderful technology, your motherboard may not be able to enable it. I previously had an AMD Athlon 64 3500+ in an MSI K8N Neo4 Platinum motherboard. It was really a nice combination, worked pretty zippy. Then, when Conroe dropped at the end of July, I decided to take advantage of the price drops and install a nice new AMD Athlon 64x2 4400+. After all, it was just a matter of dropping the chip into the board. When I booted up though, I was greated by something I didn't expect. The new system was drawing 65 watts more power than the old system was. Ouch. A look at the dmesg logs showed exactly what the problem was.

Aug  2 13:06:39 spongebob kernel: [4294696.356000] powernow-k8: Found 1 AMD Athlon 64 / Opteron processors (version 1.50.4)
Aug  2 13:06:39 spongebob kernel: [4294696.360000] powernow-k8: BIOS error - no PSB or ACPI _PSS objects

That's odd, that can't be normal. I looked back further in my logs and found this result from my Athlon 64 3500+:

Aug  1 07:10:00 spongebob kernel: [4294691.444000] powernow-k8: Found 1 AMD Athlon 64 / Opteron processors (version 1.50.4)
Aug  1 07:10:00 spongebob kernel: [4294691.444000] powernow-k8:    0 : fid 0xc (2000 MHz), vid 0x6 (1400 mV)
Aug  1 07:10:00 spongebob kernel: [4294691.444000] powernow-k8:    1 : fid 0xa (1800 MHz), vid 0x8 (1350 mV)
Aug  1 07:10:00 spongebob kernel: [4294691.444000] powernow-k8:    2 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV)

Time for some fun hacking away at stuff. Apparently, this means that my motherboard wasn't providing the correct data in the BIOS for the operating system to know how to sleep the processor. In other words, while my processor could be put to sleep, the motherboard didn't know how to. The specific item that Linux was looking for is called the Differentiated System Description Table, or DSDT for short. Lucky for us, there is a feature of Linux that lets you override the motherboard DSDT with one you specify yourself. Wonderfuly, just hack the DSDT and you've got powersaving! Wait, you didn't say we're going to hack the firmware of a critical system did you? Of course I did! It's not that hard, so let's jump right in!

First things you'll need the Intel ASL compiler/decompiler to build get your DSDT to a readable format. Under ubuntu this is just a matter of running sudo apt-get install iasl. Now we'll dump the DSDT to a file and then decode it to make it human readable. Linux is nice in that it makes the DSDT availble at /proc/acpi/dsdt

cat /proc/acpi/dsdt > custom.aml
iasl -d custom.aml

This should have created a file called custom.dsl. Open it up with your favorite text editor and have a look around. If you're like me and lacking support for power saving, see a block for your processors that looks a little like this at the top:

    Scope (\_PR)
    {
        Processor (\_PR.CPU0, 0x00, 0x00000000, 0x00) {}
        Processor (\_PR.CPU1, 0x01, 0x00000000, 0x00) {}
    }

The \_PR scope defines ACPI information for your processors. As you can see here, it's completely empty. There are a couple of ways to populate it, you can either trust the snippet of code that I'm going to paste below, or you can try to find your own information from the ACPI4Linux DSDT Repository. You'll need to find a DSDT for a motherboard that is reasonably close to yours, thus if you've got a socket 939 motherboard, like I do, you'll need to find a DSDT for a different socket 939 motherboard. Unfortunately, their site is very difficult to navigate, so if you trust me, you can paste the information here.

Scope (\_PR)
    {
        Processor (\_PR.CPU0, 0x00, 0x00000000, 0x00) 
        {
            Name (_PCT, Package (0x02)
            {
                ResourceTemplate ()
                {
                    Register (FFixedHW, 
                        0x00,               // Register Bit Width
                        0x00,               // Register Bit Offset
                        0x0000000000000000, // Register Address
                        )
                }, 

                ResourceTemplate ()
                {
                    Register (FFixedHW, 
                        0x00,               // Register Bit Width
                        0x00,               // Register Bit Offset
                        0x0000000000000000, // Register Address
                        )
                }
            })
            Name (_PSS, Package (0x04) // number of p-states
            {
                Package (0x06)
                {
                    0x0898, // 2200 MHz
                    0x000105B8, 
                    0x64, 
                    0x07, 
                    0xE020298E, 
                    0x018E
                },

                Package (0x06)
                {
                    0x07D0, // 2000 MHz
                    0xFCBC, 
                    0x64, 
                    0x07, 
                    0xE0202A0C, 
                    0x020C
                }, 

                Package (0x06)
                {
                    0x0708, // 1800 MHz
                    0xD610, 
                    0x64, 
                    0x07, 
                    0xE0202A8A, 
                    0x028A
                }, 

                Package (0x06)
                {
                    0x03E8, // 1000 MHz
                    0x6B6C, 
                    0x64, 
                    0x07, 
                    0xE0202C82, 
                    0x0482
                }
            })
            Name (PSXG, Buffer (0x18)
            {
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
            })
            Name (PSXF, Buffer (0x18)
            {
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
            })
            Name (PSXE, Buffer (0x18)
            {
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
            })
            Name (PSXD, Buffer (0x18)
            {
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
            })
            Name (PSXC, Buffer (0x18)
            {
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
            })
            Name (PSXB, Buffer (0x18)
            {
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
            })
            Method (_PPC, 0, NotSerialized)
            {
                Return (Zero)
            }

            Name (PSSC, 0x0A)
        }
        Processor (\_PR.CPU1, 0x01, 0x00000000, 0x00) 
        {
            Name (APCT, Package (0x02)
            {
                ResourceTemplate ()
                {
                    Register (FFixedHW, 
                        0x00,               // Register Bit Width
                        0x00,               // Register Bit Offset
                        0x0000000000000000, // Register Address
                        )
                }, 

                ResourceTemplate ()
                {
                    Register (FFixedHW, 
                        0x00,               // Register Bit Width
                        0x00,               // Register Bit Offset
                        0x0000000000000000, // Register Address
                        )
                }
            })
            Name (APSS, Package (0x0A)
            {
                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }, 

                Package (0x06)
                {
                    0x09999999, 
                    0x00099999, 
                    0x00999999, 
                    0x00999999, 
                    0x99999999, 
                    0x99999999
                }
            })
            Method (APPC, 0, NotSerialized)
            {
                Return (Zero)
            }

            Name (PSSC, 0x0A)
        }
    }

Whew, that's a lot of data. Don't ask me what it all means either, because I'm not an ACPI expert. I had to dig around to find a DSDT that worked for me. You'll notice that I have 4 different processor power states, which correspond to the states for the AMD Athlon 64x2 4400+. If your processor doesn't have four states, such as with an Athlon 64 3500+, then you'll want to comment out the fastest state and change the line that says number of p-states to 3 instead of 4. Anyway, your mileage may very. You'll also notice that the first value for each p-state corresponds to the speed in MHz. I'll leave it as an exercise to the reader what sort of damage you can do with that.

Great, now you've got your spiffy new dsl file -- time to compile it and have the kernel load it. Running the command iasl -tc custom.dsl will produce a file called custom.hex, this is your new compiled DSDT table for inclusion in the kernel. If you're like me you might have had errors at this point. If you're smart, you can fix them, if you're lazy you can ignore them. I chose the latter.

Because you're running Ubuntu (right?), the kernel source tree already has support for Cool'n'Quiet by default. But it doesn't enable drivers that require additional firmware. Unselect Device Drivers-> Generic Driver Options-> Select only drivers that don't need compile-time external firmware. Then go to Power management options-> ACPI Support -> Include Custom DSDT and give it the path of your custom.hex file you just created. Finally, rebuild your kernel with make-kpkg clean && make --append-to-version=-customdsdt kernel_image modules_image kernel_headers initrd. This will dump a few new .deb files in /usr/src for you. Install them with dpkg -i and reboot!

After the reboot your system will take lots of juice initially, that's because a system is usually pegged during the boot sequence, but afterwards it should drop precipitously. You can verify it's working by looking at the contents of your dmesg output and see the following:

[17179571.504000] powernow-k8: Found 2 AMD Athlon 64 / Opteron processors (version 1.50.4)
[17179571.508000] powernow-k8:    0 : fid 0xe (2200 MHz), vid 0x6 (1400 mV)
[17179571.508000] powernow-k8:    1 : fid 0xc (2000 MHz), vid 0x8 (1350 mV)
[17179571.508000] powernow-k8:    2 : fid 0xa (1800 MHz), vid 0xa (1300 mV)
[17179571.508000] powernow-k8:    3 : fid 0x2 (1000 MHz), vid 0x12 (1100 mV)

If you see something like this, congratulations, it's working quite well. It's not perfect, I still get a few little error, but at least Cool'n'Quiet works now. My system idle power dropped from 195W to 134W. Going off the rough estimate the 1W of power costs about $1/yr, that's a saving of $61! Nice. Of course, it took me about four hours to figure this out, which means it may have not been worth it, espeically if I apply a discount rate to my future savings, but I digress. If you use external proprietary drivers, it might be worth your time to install module-assistant so those get compiled too. Otherwise you'll boot your nice new kernel without support for 3d acceleration and all that jazz.

After all of this, I found a nice package that allows for the placement of the DSDT in the initrd. Using this method, you don't need to recompile your kernel every time you change your DSDT -- instead you just rebuild your initrd. Very nice for situations where the drivers aren't always built as part of the kernel build process, such as IVTV.

Happy Hacking!


Fives about Ubuntu Dapper

Posted to linux on 2006-03-20 08:00:00

Ubuntu 06.06 (note the delay from the original 06.04), aka Dapper Drake, is well into development, and being as I have a shiny new laptop, I decided to give it a shot. So, here's a list of five things that I like about Dapper and five things that could use some work. Many of these aren't necessary Dapper specific, but they're things that I didn't do with Breezy (or Hoary, or Warty...).

Five Things That Rock
  1. deskbar applet - totally changes the way you use your computer. Alt-F3 is your friend.
  2. NetworkManager
  3. Improved startup time
  4. Banshee and iPod transcoding
  5. SSHFS doesn't crash in Nautilus
Five things that need help
  1. gtkmozembed needs some serious help - breaks Blam! and Liferea
  2. Sleep/Hibernate not so hot on my T43p
  3. xgl halts my T43p
  4. No built in support for fingerprint scanner (but it does work)
  5. NetworkManager tends to halt sometimes - especially if I unplug the network cable

Overall I'm really impressed. I'm hoping that by the May WPLUG installfest I can have xgl working and all bugs hammered out. Of course, my dream for a Ubuntu focused installfest is dead because Dapper is delayed.


Faster Transcoding on AMD Athlon 64's

Posted to linux/mythtv on 2006-01-13 11:18:00

If you're running MythTV on an Athlon 64 platform, you've probably noticed that the CPU throttles itself down when transcoding and cutting commercials despite the fact that your processor may show as being 100% utilized. This results in much slower transcoding than one would normally expect, but saves a small amount of power.

However, if you record a lot of stuff, or even a moderate amount of stuff in HD, this property and increased delay can be quite annoying. It can take several extra hours to cut and transcode an hour long HDTV program when running at 1.2GHz instead of 2.4GHz (my analysis shows a greater then 2.5x slowdown - which is weird in itself). First, let's cover the reason you system slow down, and then we'll cover what to do about it.

Processors take a lot of power - most of that power is dissipated as heat because of the resistance in the CPU. Slowly, AMD and Intel are realizing that people don't want extra heaters in their living room/office, that powering all of these computers puts a real dent in your wallet, and that most of the energy is wasted because the processor is idle anyway. The solution? Create not only more effecient processors, but also allow processors that throttle back to conserve power when they're not being fully utilized. On the Athlon 64 platform this is called "Cool'n'Quiet" and has been available since the Winchester core chips.

Cool'n'Quiet works with a couple of different parts: a Cool'n'Quiet enabled processor, a Cool'n'Quiet enabled motherboard, and a software program that manages it all. If you're running a socket 939 Athlon 64, you've almost certainly got the first two. Under Linux, you can directly control the CPU power state through the proc file system, or you can have a daemon process, such as powernowd, manage it for you. Ubuntu installs powernowd automatically and starts it for you, so out of the box you're getting some of the benefits of this technology.

However, powernowd is configured to not take into account nice processes. Normally, this is fine, nice processes can afford to run a bit slower, that's why they're nice in the first place. Unfortunately, it is only set up to consider nice processes as a group and can't differentiate between a process that can't take advantage of increase speed and one that can, such as transcoding. And, as you've probably guessed by now, to minimize interference with other programs, MythTV runs transcoding and commercial flagging processes as nice. To speed them up all we need to do is tell powernowd to take into account nice processes.

In Ubuntu, this is really straight forward. Most startup scripts for daemons look in the /etc/default directory for local configuration overrides. By default powernow is run with only the -q flag, telling it to be quiet. If we also pass the -n flag, it will take nice processes into account when deciding whether or not to throttle the processor. The solution, create a file /etc/default/powernowd with a single line in it:

OPTIONS="-q -n"

Then issue the command /etc/init.d/powernowd restart and your computer will no longer throttle itself when transcoding and commercial flagging. Enjoy the increased speed of transcoding and happy MythTVing.


Context Switches on Ubuntu

Posted to linux/ubuntu on 2005-12-07 14:08:00

After trying a new motherboard and all that jazz to get my HDTV tuner card working properly I decided to try a new operating system. The thing that prompted this was the notice that some people thought it might be a buffer related issue that was causing some of the reception problems with the card. I also noticed that my system was consistently above 2000 context switches a second even when idle. There was usually around 1100 interrupts a second. I compared this to my RHEL AS box which was around 30/s when idle and running a lot more stuff.

So, today I set out to test this theory some. I dropped in a FC4 x86_64 dvd and installed it on a blank partition on my hard disk. Sure enough, preliminary tests indicate that I'm not loosing data and reception remains clear even when I'm under heavy IO load, such as copying a file over.

So here's my theory. Ubuntu uses a kernel that switches tasks more often to create a more interactive computing experience. However, for devices that produce large amounts of streaming bandwidth, such as HDTV tuner cards (and to a lesser degree SDTV tuner cards), the tasks switch too fast and the buffers never can be completely emptied. By switching to a higher minimum time slice the processor has enough time to empty out the buffers.

Now, the question is this: is there a simple way to change the minimum time slice for tasks under Ubuntu? I remember seeing something where I could set the timer at 10, 100, or 1000 and seeing that it was set to 1000, but I can't find it in the kernel configuration anymore.

Anyway, if you're having HDTV reception problems, I strongly suggest looking at this as an issue. I'm not 100% certain, but the preliminary tests are good. Unfortunately, I can't do the full battery yet because I don't have time and I don't even have video working on MythTV box under Fedora.


MythTV, Ubuntu, Athlon64, WinTVPVR, Comcast Digital Cable, and a pcHDTV HD-3000 in a pear tree!

Posted to linux/mythtv on 2005-11-17 11:08:00

I've been writing a nice tutorial on setting up MythTV on Ubuntu x86_64 over the course of the past few weeks. Please feel free to browse it and give feedback. It still is a work in progress and I could certainly use some help in some spots. I should clarify this by explaining that I've got my system more or less working, with the exception of static on my pcHDTV which I understand is an electro-magnetic interference issue that I can't do anything about. When I move that card into another computer, I'll add details about that too.

For those that are concerned, I'll eventually put up a TRAC site for managing the project and and providing access to the repository of revisions. My goal is to provide a slightly more up-to-date and Ubuntu centric version of Jarod Wilson's Fedora MythTV Guide. I know, that's a lot of hoping. Come on magic google juice, lead people to me.