My Delusional Dream

My servers have run OpenBSD since sometime in 1998. I still have a few really ancient official CD sets laying around for OpenBSD 2.3 (May 1998), 2.5 (May 1999), and 2.7 (June 2000). My name can be found on the OpenBSD Donations page, and I think that PF is one of the sweetest pieces of software I've seen. I've ran OpenBSD on x86, PPC, and Sparc on both physical and virtual hardware. I've used it as a mail server, spam filter, firewall, web server, database server, and even an X11 terminal. OpenBSD is just a well engineered piece of software that works well and doesn't need much maintenance.

Last year, in an effort to reduce clutter in my house and ease my power bill, I changed from a physical to virtual OpenBSD box as my main Internet server computer. For the most part I had some good luck, it worked pretty well once I changed the virtual NIC from a PCNet to an E1000. Back in October I decided that this holiday season I would upgrade to the 4.2 release of OpenBSD, which contained a lot of features I was looking forward to such as a port for Python 2.5 and the new vic VMWare network driver. About a week after I made this decision, my Apache server process began to randomly segfault when using gzip compression to serve web pages -- as someone who runs their website off a cable modem, this was a hit that I really didn't want to take. I disabled gzip compression and decided to look at it again over the holidays.

The process of bringing up a new virtual machine with VMWare server is nearly brain dead. I can create multiple virtual machines in a matter of minutes. For this upgrade the choice was made to start fresh to avoid some of the botched installs from previous versions of tools in the original virtual machine. My strategy was to install the basics for the machine, then compile the ports I needed -- apache2, subversion (with apache support), python 2.5, screen, and irrsi -- were at the top of my list. Wait, you may say, OpenBSD offers compiled packages for these -- no need to compile. This is true, but the Python setup includes the tkinter libraries, which no ones uses anymore and forces X on my system. Subversion also links to some packages I'll never use again.

After some fighting the build process got running, but I almost immediately hit a snag. Programs were complaining about expat not being present, and the port kept on telling me it was included as of OpenBSD 4.2. As I had OpenBSD 4.2 installed, I could tell that it was not included. My first strategy was to lie and build the package regardless. While this allowed me to build some software, it quickly caused problems. I was soon modifying my FLAVOR environment variable with reckless abandon. In the end, I broke enough things that I needed to start over. Luckily, VMWare makes that simple.

A little sleuthing discovered that for some reason in OpenBSD, expat is currently a part of the xbase package, and if I wanted the development headers I also needed the xshare package too. What? Why I do need X for everything all the sudden. Whatever, I installed those packages, rebuilt the software, but still needed to fudge around with lots of environment variables to keep it from installing TCL with anything that had optional TCL bindings. Don't even get me started on the headache of keeping Ruby off my system. After about six hours of actual work at fighting with it, much more if you count compilation, I had the software I thought I needed installed. A little testing revealed that all of my web services still worked. Great! Then I turned on gzip and the segmentation faults returned. I returned to the default configuration, and the segmentation faults continued. I debugged the program, realized I need to do some nasty patching, and concluded that this wasn't going to be the proper choice.

It was time to consider something else. I really wanted to run a BSD system still, I figured I'd give FreeBSD a shot again. For some reason their automatically discovered mirror for me was ungodly slow, worse than dialup. Going through tor actually managed to speed things up. Wow, that's strange isn't it? I also noticed they were close to a release -- I don't want to run pre-release software and I didn't want to have to upgrade, and I'm really impatient. Crap. By this point it was the 27th, and I was supposed to leave for a week in just a few days.

Suck it up, install Ubuntu Server. I really didn't want to run Linux, and Ubuntu Server may have a bit too much stuff on it in the default install. I was already starting to feel dirty. A little more digging revealed a nifty project, Ubuntu JeOS (Just Enough Operating System), a specialized and stripped down version of Ubuntu that has the core software and an optimized VMWare kernel. The download was pretty small, at about 150MB or so. Installation was a breeze and incredibly straightforward. As an added bonus, VMWare tools compiles for Linux, to further improve performance. A few apt-get commands later and my system was running. What's better is that the packaging system didn't require me to install X11, TCL, Ruby, PHP, or MySQL for my server setup. I was able to get just the tools I needed. Total installation time for everything, 41 minutes. That includes the time to copy all my web files over, port the httpd.conf file to the Ubuntu style of multiple configuration files, and update some absolute paths in scripts.

Next up was to see how fast it was. I always loved how few processes I'd see running in my OpenBSD machine. Hammering the machines over and over an testing the speed of response with YSlow showed the Linux box to complete requests in about 40% of the time of the OpenBSD box when requested from the local LAN. Furthermore, the responses were compressed, which means a great saving of time over the line. I was hooked -- sorry OpenBSD, you've been replaced.

Of course, there is a potential cost here. What about the security of my new server? What about stability? What about updates? As far as security goes, I'm already being fairly smart by running it in as a virtual machine. The main filesystems are backed up nightly to both local and remote systems, and about once a month, a backupninja script shuts down the machine, creates a snapshot of the full machine, then restarts it. That gives me about 10 minutes of downtime every month -- I'm sure the spiders will understand. I also am not running any unnecessary servers -- just apache2, an instance of CherryPy for PennAve, and very restricted OpenSSH for remote access -- that's it. I'm hoping I'm in good shape with that, but security is something you can never guarantee unless the machine is off, has no drives, and lacks any sort of persistent memory (of course, I'm sure that even then there may be a way to find some sort of electromagnetic residue in the RAM). Stability wise, I'm not entirely certain. Ubuntu Dapper on my MythTV machine has been excellent, while Gutsy on my laptop is sometimes flaky -- but that may also be because of a long upgrade chain going back up to March of 2005 on this machine. Finally, I'm pretty sure that using APT makes upgrades, especially security related upgrades, even easier. I don't need to worry about seeing that a patch was released and installing it by hand. It just happens -- yes I'm placing trust in a remote entity, but I can handle that. I just want my machine to work.

So, after two days of mucking around with OpenBSD, I finally got Ubuntu JeOS to just work, and I'm very happy with it. I'm sorry to see OpenBSD go, but it was just too difficult to work with. In particular, here's some issues that really need to be addressed to make it competitive with what JeOS offers:

• If you say you include expat, really include it, don't place it in two different packages for X. I don't need X on my server.
• Finer grain package control. Some packages are getting pretty good at creating subpackages for optional components -- subversion is a good example of this. Other packages, such as Python, could really use some help here.
• Inheritance of the FLAVOR environment variable. If I compile subversion as NO_TCL or NO_RUBY, that should cascade down to swig. I had to literally watch every dependency as it was compiled to make sure that it was going to decide I needed TCL, TK, and Ruby.
• Automated checking for security patches. I'm not sure if OpenBSD has this, but it wasn't obvious how to do it. But, then again, it's not immediately obvious how to use apt either.

For the time being, OpenBSD and I have parted ways. In a few years when it comes time to upgrade my server again, we may again join forces. Of course, with the continual increase of software as a service, in a few years, there may be no need for me to run my own server anymore.

To my loyal reader (or perhaps readers, who knows) -- you probably noticed some issued with the accessibility of this site over the past few days. It seems like two different things were conspiring to take down my site. The first is that my uplink connection has been a little flunky -- I've got a WRT54Gv1 that I use for my net access and that's been acting really strange. The second is that my server, scissors has been acting a bit flunky. Before I go too far, a little description of the box is necessary.

The machine is a circa 1997 AMD K6-200 with 96MB of RAM and Quantum Fireball 4.3GB hard disk -- this was pretty hot stuff when I went to college in 1997. Yes, that's right, it's my first computer from undergraduate. The machine first started to have problems in the fall of 1999. At the time I thought it was disk issues, so I bought a new disk. That didn't fix the problems, in fact, I think I threw out at perfectly good 8GB disk at the time because of the issues. Really, it was just problem with the IDE cables in the machine, but I digress. That year I was working more or less full time and going to school, so I had money to burn. I plunked down some cash for an Athlon 700 machine, and that machine became my backup server machine (replacting a Pentium 60). This is also where the machine got the name scissors, for scissors.dtd.iit.edu. I used some of the original parts in my newer machines, this left scissors with a black and with text only video card, and a couple of ISA ethernet cards (at least they're 3com). Yup, 10mbps is the max ethernet rate on the machine.

This point was also the first time that I installed OpenBSD on it. I believe it was OpenBSD version 2.5 or thereabouts - I actually paid for the CDs and have them laying around. Since then, the machine has been upgraded to newer operating systems twice, first to OpenBSD 2.9, then again to OpenBSD 3.3. It's served as a mail server, online game server, and most importantly my personal web server for the last few years. At some point, probably in early 2003 the fan on the CPU died, but it didn't seem to affect performance, so I didn't care. At some point later the fan on the power supply died -- once again it didn't seem to affect performance so I didn't care. I now had a basically entirely silent machine. Cool. As an added bonus, according to my Kill-a-watt device, it sips power at a consumption of 33 watts. This makes it nice and affordable to run.

However, the machine is now approaching it's ninth birthday. It's had a good run -- but all good things must come to an end. I've been getting rid of my old stuff and well scissors, it's time for you to go. You've been replaced by the forward marching tide of technology. However, it's a real pain to reinstall web servers all the time. My solution? Virtualization.

Thanks to VMWare Server and my copy of VMWare Workstation, I've created a very nice little OpenBSD virtual machine. Best of all, I've got all the services that previously were running between a couple of different servers consolidated onto this single box. That means this virtual machine now acts as my trac, subversion, dav, and person web server.

Virtualization has quite a few benefits. One of the biggest ones is that I can just copy the virtual machine from one physical computer to another. This makes upgrades a breeze. (As an aside, it also makes me wonder why I don't just install VMWare Server on my MythTV box in the living room and drop down to a single computer that is on 24/7. Probably has something to do with the printer we never use I guess.) Also, being virtualized, it let's me test other configuration changes easier. I can take a snapshot of the server in workstation and try an upgrade or change in software with no risk to the live machine. The VMWare Server console lets me manage everything from my laptop too. Wow.

I know that someone out there will believe that I'm a sellout because VMWare is closed source. Which is probably partially true. However, I've been using VMWare since it first came out. Back at LEC Ltd, we used VMWare on the developer workstations to see how the web pages would look on different platforms. Of course, we never thought about using it for server use -- but now it seems that's what the cool kids are doing. An astute reader might wonder why I didn't choose Xen or QEMU. With regards to Xen, I've heard that Theo considers Xen to be more of a toy, however he might be coming around to it (this was a year or more ago now). I did notice that there is a Summer of Code request to make OpenBSD a native Guest OS. That might be really interesting, but it's not here yet. For QEMU, the big issue is that if you want native performance you need to the unfree accelerator program. That, and it's not as feature rich as VMWare. Given that I have extensive VMWare experience too, it seemed like this was the fastest way to get running again.

So what is all this saying? Basically, if you're reading this weblog entry you're connected to my VMWare server instance, now called "donkey" running OpenBSD 3.9. Well, if this post is recent that's what happening. In the future I should be able to upgrade pretty easy. For now, the host machine for Donkey is a Pentium 3/900 with 640MB of RAM. So far, so good. I took extensive notes about the whole process and will be putting together a nice writeup soon. If you've got questions about the process, feel free to email me in the mean time.