My Delusional Dream

One of the major issues with utilizing a virtual machine for a server is that of time synchronization. VMWare normally has access to a real time clock handler that helps to synchronize time, but even that causes time to skew. This issue becomes more prominent with modern processors that support CPU frequency scaling. However, there are many cases where even the VMWare custom kernel module can't manage the time skew properly. Such was the case with my updated VMWare virtual machine -- to put it simply, time had stopped.

Looking around, it appears that some of the problem may be to the new tickless features in the linux kernel. Basically, this feature allows the system to stop waking up periodically if there is nothing to do. On desktop and laptop machines this saves a lot of power and is a key feature that has helped the OLPC be such a power miser. However, it also really screws with the clocks in virtual machines, and may be cause of other issues. I've also read that there are issues with heavy disk I/O and Gutsy in VMWare Server -- which is my virtualization platform of choice.

Previously, with the time issue, I would fix problems by adding in a cron job to periodically synchronize the date with a remote system. It's important to note, that ntp won't work because it tries to gradually synchronize the time -- so it will never catch up. However, using a cron job on the virtualized system relies on the virtualized system eventually hitting even minute marks. Last night in the course of 12 hours, my time advanced 32 seconds; cron is not an option. The solution is to have the host operating system, which can keep time, periodically SSH into the virtualized machine and synchronize the time. Getting started, you'll need to activate the time service in xinetd on the host operating system. Open up /etc/xinetd.d/time and change the lines that say

disable = yes

to say "no". Restart xinetd by running /etc/init.d/xinetd restart and you'll now be able to rdate to your host machine.

The next step is to create the ssh key and setup the cron job to connect to guest operating system. These commands should do it for you. Hit enter to leave it without a passphrase. Then, copy the key to the guest operating system.

root@host:~# ssh-keygen -t dsa -C "automated RDATE ssh key" -f id_dsa.rdate
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in id_dsa.rdate.
Your public key has been saved in id_dsa.rdate.pub.
The key fingerprint is:
d4:5f:01:83:ee:df:e0:e4:ea:ff:0a:6e:50:d1:e2:d3 automated RDATE ssh key
root@host:~# scp id_dsa.rdate.pub USERACCOUNT@GUEST:

Now that the key is on the guest, it's time to enable a passwordless login. First, login to the guest machine, and open up a root shell using sudo -s. Then run the following command to add the new key to your ~/root/.ssh/authorized_keys.

echo 'command="rdate -s HOST"' $(cat id_dsa.rdate.pub) >> ~root/.ssh/authorized_keys

The final step is to edit your /etc/crontab on your host system, and make the time synchronization happen every five minutes or so.

*/5 * * * * root ssh -i ~root/.ssh/id_dsa.rdate root@GUEST "rdate -s HOST"

Now, you should have a nice infrastructure set up where your VMWare virtual machine will never be more than five minutes away from the actual time. It's important to note that there are a few alternative ways of doing this. For example, one could bypass the usage of rdate entirely through some clever shell scripting that passes the time on the host system to the time command on the remote system. In my experience, this works in a homogeneous environment, but not every system can interpret times in the same way, so sticking with rdate seems like a good compromise.

My servers have run OpenBSD since sometime in 1998. I still have a few really ancient official CD sets laying around for OpenBSD 2.3 (May 1998), 2.5 (May 1999), and 2.7 (June 2000). My name can be found on the OpenBSD Donations page, and I think that PF is one of the sweetest pieces of software I've seen. I've ran OpenBSD on x86, PPC, and Sparc on both physical and virtual hardware. I've used it as a mail server, spam filter, firewall, web server, database server, and even an X11 terminal. OpenBSD is just a well engineered piece of software that works well and doesn't need much maintenance.

Last year, in an effort to reduce clutter in my house and ease my power bill, I changed from a physical to virtual OpenBSD box as my main Internet server computer. For the most part I had some good luck, it worked pretty well once I changed the virtual NIC from a PCNet to an E1000. Back in October I decided that this holiday season I would upgrade to the 4.2 release of OpenBSD, which contained a lot of features I was looking forward to such as a port for Python 2.5 and the new vic VMWare network driver. About a week after I made this decision, my Apache server process began to randomly segfault when using gzip compression to serve web pages -- as someone who runs their website off a cable modem, this was a hit that I really didn't want to take. I disabled gzip compression and decided to look at it again over the holidays.

The process of bringing up a new virtual machine with VMWare server is nearly brain dead. I can create multiple virtual machines in a matter of minutes. For this upgrade the choice was made to start fresh to avoid some of the botched installs from previous versions of tools in the original virtual machine. My strategy was to install the basics for the machine, then compile the ports I needed -- apache2, subversion (with apache support), python 2.5, screen, and irrsi -- were at the top of my list. Wait, you may say, OpenBSD offers compiled packages for these -- no need to compile. This is true, but the Python setup includes the tkinter libraries, which no ones uses anymore and forces X on my system. Subversion also links to some packages I'll never use again.

After some fighting the build process got running, but I almost immediately hit a snag. Programs were complaining about expat not being present, and the port kept on telling me it was included as of OpenBSD 4.2. As I had OpenBSD 4.2 installed, I could tell that it was not included. My first strategy was to lie and build the package regardless. While this allowed me to build some software, it quickly caused problems. I was soon modifying my FLAVOR environment variable with reckless abandon. In the end, I broke enough things that I needed to start over. Luckily, VMWare makes that simple.

A little sleuthing discovered that for some reason in OpenBSD, expat is currently a part of the xbase package, and if I wanted the development headers I also needed the xshare package too. What? Why I do need X for everything all the sudden. Whatever, I installed those packages, rebuilt the software, but still needed to fudge around with lots of environment variables to keep it from installing TCL with anything that had optional TCL bindings. Don't even get me started on the headache of keeping Ruby off my system. After about six hours of actual work at fighting with it, much more if you count compilation, I had the software I thought I needed installed. A little testing revealed that all of my web services still worked. Great! Then I turned on gzip and the segmentation faults returned. I returned to the default configuration, and the segmentation faults continued. I debugged the program, realized I need to do some nasty patching, and concluded that this wasn't going to be the proper choice.

It was time to consider something else. I really wanted to run a BSD system still, I figured I'd give FreeBSD a shot again. For some reason their automatically discovered mirror for me was ungodly slow, worse than dialup. Going through tor actually managed to speed things up. Wow, that's strange isn't it? I also noticed they were close to a release -- I don't want to run pre-release software and I didn't want to have to upgrade, and I'm really impatient. Crap. By this point it was the 27th, and I was supposed to leave for a week in just a few days.

Suck it up, install Ubuntu Server. I really didn't want to run Linux, and Ubuntu Server may have a bit too much stuff on it in the default install. I was already starting to feel dirty. A little more digging revealed a nifty project, Ubuntu JeOS (Just Enough Operating System), a specialized and stripped down version of Ubuntu that has the core software and an optimized VMWare kernel. The download was pretty small, at about 150MB or so. Installation was a breeze and incredibly straightforward. As an added bonus, VMWare tools compiles for Linux, to further improve performance. A few apt-get commands later and my system was running. What's better is that the packaging system didn't require me to install X11, TCL, Ruby, PHP, or MySQL for my server setup. I was able to get just the tools I needed. Total installation time for everything, 41 minutes. That includes the time to copy all my web files over, port the httpd.conf file to the Ubuntu style of multiple configuration files, and update some absolute paths in scripts.

Next up was to see how fast it was. I always loved how few processes I'd see running in my OpenBSD machine. Hammering the machines over and over an testing the speed of response with YSlow showed the Linux box to complete requests in about 40% of the time of the OpenBSD box when requested from the local LAN. Furthermore, the responses were compressed, which means a great saving of time over the line. I was hooked -- sorry OpenBSD, you've been replaced.

Of course, there is a potential cost here. What about the security of my new server? What about stability? What about updates? As far as security goes, I'm already being fairly smart by running it in as a virtual machine. The main filesystems are backed up nightly to both local and remote systems, and about once a month, a backupninja script shuts down the machine, creates a snapshot of the full machine, then restarts it. That gives me about 10 minutes of downtime every month -- I'm sure the spiders will understand. I also am not running any unnecessary servers -- just apache2, an instance of CherryPy for PennAve, and very restricted OpenSSH for remote access -- that's it. I'm hoping I'm in good shape with that, but security is something you can never guarantee unless the machine is off, has no drives, and lacks any sort of persistent memory (of course, I'm sure that even then there may be a way to find some sort of electromagnetic residue in the RAM). Stability wise, I'm not entirely certain. Ubuntu Dapper on my MythTV machine has been excellent, while Gutsy on my laptop is sometimes flaky -- but that may also be because of a long upgrade chain going back up to March of 2005 on this machine. Finally, I'm pretty sure that using APT makes upgrades, especially security related upgrades, even easier. I don't need to worry about seeing that a patch was released and installing it by hand. It just happens -- yes I'm placing trust in a remote entity, but I can handle that. I just want my machine to work.

So, after two days of mucking around with OpenBSD, I finally got Ubuntu JeOS to just work, and I'm very happy with it. I'm sorry to see OpenBSD go, but it was just too difficult to work with. In particular, here's some issues that really need to be addressed to make it competitive with what JeOS offers:

• If you say you include expat, really include it, don't place it in two different packages for X. I don't need X on my server.
• Finer grain package control. Some packages are getting pretty good at creating subpackages for optional components -- subversion is a good example of this. Other packages, such as Python, could really use some help here.
• Inheritance of the FLAVOR environment variable. If I compile subversion as NO_TCL or NO_RUBY, that should cascade down to swig. I had to literally watch every dependency as it was compiled to make sure that it was going to decide I needed TCL, TK, and Ruby.
• Automated checking for security patches. I'm not sure if OpenBSD has this, but it wasn't obvious how to do it. But, then again, it's not immediately obvious how to use apt either.

For the time being, OpenBSD and I have parted ways. In a few years when it comes time to upgrade my server again, we may again join forces. Of course, with the continual increase of software as a service, in a few years, there may be no need for me to run my own server anymore.

To my loyal reader (or perhaps readers, who knows) -- you probably noticed some issued with the accessibility of this site over the past few days. It seems like two different things were conspiring to take down my site. The first is that my uplink connection has been a little flunky -- I've got a WRT54Gv1 that I use for my net access and that's been acting really strange. The second is that my server, scissors has been acting a bit flunky. Before I go too far, a little description of the box is necessary.

The machine is a circa 1997 AMD K6-200 with 96MB of RAM and Quantum Fireball 4.3GB hard disk -- this was pretty hot stuff when I went to college in 1997. Yes, that's right, it's my first computer from undergraduate. The machine first started to have problems in the fall of 1999. At the time I thought it was disk issues, so I bought a new disk. That didn't fix the problems, in fact, I think I threw out at perfectly good 8GB disk at the time because of the issues. Really, it was just problem with the IDE cables in the machine, but I digress. That year I was working more or less full time and going to school, so I had money to burn. I plunked down some cash for an Athlon 700 machine, and that machine became my backup server machine (replacting a Pentium 60). This is also where the machine got the name scissors, for scissors.dtd.iit.edu. I used some of the original parts in my newer machines, this left scissors with a black and with text only video card, and a couple of ISA ethernet cards (at least they're 3com). Yup, 10mbps is the max ethernet rate on the machine.

This point was also the first time that I installed OpenBSD on it. I believe it was OpenBSD version 2.5 or thereabouts - I actually paid for the CDs and have them laying around. Since then, the machine has been upgraded to newer operating systems twice, first to OpenBSD 2.9, then again to OpenBSD 3.3. It's served as a mail server, online game server, and most importantly my personal web server for the last few years. At some point, probably in early 2003 the fan on the CPU died, but it didn't seem to affect performance, so I didn't care. At some point later the fan on the power supply died -- once again it didn't seem to affect performance so I didn't care. I now had a basically entirely silent machine. Cool. As an added bonus, according to my Kill-a-watt device, it sips power at a consumption of 33 watts. This makes it nice and affordable to run.

However, the machine is now approaching it's ninth birthday. It's had a good run -- but all good things must come to an end. I've been getting rid of my old stuff and well scissors, it's time for you to go. You've been replaced by the forward marching tide of technology. However, it's a real pain to reinstall web servers all the time. My solution? Virtualization.

Thanks to VMWare Server and my copy of VMWare Workstation, I've created a very nice little OpenBSD virtual machine. Best of all, I've got all the services that previously were running between a couple of different servers consolidated onto this single box. That means this virtual machine now acts as my trac, subversion, dav, and person web server.

Virtualization has quite a few benefits. One of the biggest ones is that I can just copy the virtual machine from one physical computer to another. This makes upgrades a breeze. (As an aside, it also makes me wonder why I don't just install VMWare Server on my MythTV box in the living room and drop down to a single computer that is on 24/7. Probably has something to do with the printer we never use I guess.) Also, being virtualized, it let's me test other configuration changes easier. I can take a snapshot of the server in workstation and try an upgrade or change in software with no risk to the live machine. The VMWare Server console lets me manage everything from my laptop too. Wow.

I know that someone out there will believe that I'm a sellout because VMWare is closed source. Which is probably partially true. However, I've been using VMWare since it first came out. Back at LEC Ltd, we used VMWare on the developer workstations to see how the web pages would look on different platforms. Of course, we never thought about using it for server use -- but now it seems that's what the cool kids are doing. An astute reader might wonder why I didn't choose Xen or QEMU. With regards to Xen, I've heard that Theo considers Xen to be more of a toy, however he might be coming around to it (this was a year or more ago now). I did notice that there is a Summer of Code request to make OpenBSD a native Guest OS. That might be really interesting, but it's not here yet. For QEMU, the big issue is that if you want native performance you need to the unfree accelerator program. That, and it's not as feature rich as VMWare. Given that I have extensive VMWare experience too, it seemed like this was the fastest way to get running again.

So what is all this saying? Basically, if you're reading this weblog entry you're connected to my VMWare server instance, now called "donkey" running OpenBSD 3.9. Well, if this post is recent that's what happening. In the future I should be able to upgrade pretty easy. For now, the host machine for Donkey is a Pentium 3/900 with 640MB of RAM. So far, so good. I took extensive notes about the whole process and will be putting together a nice writeup soon. If you've got questions about the process, feel free to email me in the mean time.

This week I managed to unpack the T2000 server. It's a pretty large box, but not overly huge. The hard disks are nice little 73GB SAS drives that pull right out of the box. The CD/DVD drive is a sleek slot loading drive. I do have to say, Sun did a good job making this box look really nice and sleek. A datacenter of these things would be quite cool (literally and figuratively).

My initial hope was to commandeer a monitor, plug in a USB keyboard, and go at it. One major proble, there is no video card installed on the box, although if I had an extra PCI-E card, it would be interesting to see what happens with one. Initial system boot-up and installation is done via the serial console. Okay, find a null modem and a laptop with a serial port (my new T43p does not have a serial port). Wrong. Even if I could have found a null modem, the regular serial port is not the serial console, instead you need to use an RJ45->DB9 dongle to get a serial terminal. Of course, these things were no where to be found around Carnegie Mellon.

Luckily, an email went off, and a schmeatic of the circuit came back. Thinking for a bit, I remembered that Make Magazine vol 1 had an article on how to build a 5-in-1 network cable. Off to Radio Shack for some parts. Unfortunately, Radio Shack has slowly turned itself into "crappy overpriced cell-phone shack". Not a single RJ45-DB9 connector was to be found. The only DB9 connectors I could find were just a single snap in port. In fact, they didn't even have a coupler for RJ45. So I picked up two of them. The next step was to mangle an ethernet cable, and have some breadboard fun. After about 30 minutes, I had my circuit.

The Ghetto RJ45->DB9 Converter

Luckily, after a little bit of tweaking the circuit, and reseating some wires on my breadboard, the ALOM on the T2000 started up and spit out data to the Hyper Terminal running on Kristina's laptop. It was pretty straight forward to get the network console running after this point, and yes, it can DHCP an address, but be very careful about putting the network admin port on a public network because the VxWorks OS that runs ALOM does NOT support SSH. Suckage. Here's another shot of getting everything going. I've got the serial terminal up on the HP laptop and the network terminal up on the T43p. After booting into Solaris, it's much better to use SSH to connect to the OS than go over the network terminal.

Getting the T2000 Up and Running

After a long series of delays and communication problems, the T2000 server from Sun has finally arrived. Apparently there was an issue with my original submission back in February. Once I resubmitted my request I was approved in fairly short order and received some pretty good responses from folks at Sun. Kudos to them for making the most out of an annoying situation.

I do have a few critiques about the process. First of all, the sales people were, for lack of a better term, sales people. I started to get random messages from people I had no knowledge of and seemed to have no relation to me. I was sent an invoice with a subject line of "INVOICE", which of course usually goes right to the spam filter. Finally, I got some helpful documents under a subject line of "[Fwd: [Fwd: cool tool FYI]]", with no explanation of why I was getting it. So let me provide a few suggestions to make the process nicer to people like me:

• Use informative subject lines that are not all capitalized. Remove all those "Fwd" headers too. Instead write "Helpful Sun T2000 Documents".
• If you're going to send me an email from an address that looks real, make sure you reply to it when I ask a question about it. This goes for you, Rebecca Lukens. I received an Invoice which said I needed to take some action regarding it, I replied to it with a few questions and never got a response. I tried again, and never got a response. There was also no contact information in the message, which I've reproduced in verbatim here:

  Try & Buy quote attached.  Thanks!

• Take some time to say why you're emailing me. For example Instead of the above message, I would have appreciated something like this:

  Mr. Wagstrom,
  
  Attached you'll find a quote for your recent Sun Microsystems T2000 Try and Buy request.
  Should you choose to keep the machine beyond 60 days, you'll be required to pay the full
  amount of the invoice unless other arrangements have been made.  For right now, please
  enjoy the machine and we hope it sits your needs well.
  
  If you have questions, you can contact me at: 
  [CONTACT INFORMATION HERE]

  You'll notice it's a form letter. They don't even have to think more than pasting in the current text, but it would be very helpful to me. Also, the change would give it a chance of evading the Carnegie Mellon spam filters.
• When sending out unsolicited attachments via email, also include a web link to a site on sun.com where I can retrieve these documents. I trust documents coming from sun.com more than I do general email that is sent to me from unknown individuals.

But like I said, the process was actually relatively painless. A few frustrations, but I'm happy with the opportunity to test out this technology. We'll start benchmarking it soon.

Last week I submitted a try and buy request to Sun for some equipment to us on some research projects at school. In particular, I'm excited about using it to scan the network for spyware. I was a little worried that my information was not processed properly as I got a brief error message upon submission of the request. No problem, email my personal Sun representive (aka my brother), and see what's going on. After a few messages were passed around, I got an phone call from Sun today about the project.

As I thought, the machines are way backed up because of the massive response to their admittedly very cool offer. I explained our situation to their educational sales person for the Pittsburgh area (note to Sun web folks, I could not easily find this information on your web site. Instead Phil had to contact the sales person for the University of Minnesota who forwarded my information to the nice person here in Pittsburgh) and she seemed quite receptive and reassured me that Sun was processing the stuff and that our purpose seemed pretty novel and interesting. So, we're still in the queue, but it should be all good from here.

She could have just left it at that, but she didn't. She also was helpful enough to provide us information about Sun matching grants for hardware. This drops the price of a 4 core box down to a more reasonable $3800 or so. Pretty nice. However, we still don't have that much money, at least not yet. Anyway, if this can run on $3800 hardware, I'd think that it makes things a whole lot more attractive for doing some sustained testing on the system. We'll just have to see how well everything actually runs on it.

Once this thing shows up I'll be sure to blog some more and provide some photos of the system along with preliminary benchmarks.

Last week, Phil sent me an email about Sun Microsystem's new T2000 series of servers, based on their Niagara core. Basically, this is a 4, 6, or 8 core system, each of which can execute up to four threads in parallel. It's supposed to be great for highly threaded, low FPU applications -- like web serving and database serving.

As part of their promotion, they're doing this cool Try before you Buy thing. Basically, you given them your information, they send you a server free for 60 days -- they even cover postage. You evaluate it, if you like it, you pay for it. If it doesn't work well for your field, send it back -- once again, on Sun's dime. Pretty cool stuff. What's cooler, is that according to Jonathan Schwartz, they're looking for people to benchmark these guys. Specifically, if you do a good enough job on the writeup, they may let you keep it.

So, being the graduate student I am, I immediately realized a couple of different applications that I'd like to work on with this little guy. Firstly, we're going to see how well he works as a spyware monitoring box -- run multiple instances of snort on him and a database to log everything to. Should be kinda interesting to see. Secondly, I'm going to try and use it for some of my network analysis stuff. Specifically, seeing if it's faster to run ORA on it, or at the very least what's the overall throughput.

I'm a little skeptical about it's performance on some of the measures in ORA because the cores share an FPU. However, the integer measures should run quite nicely. I'll also put it through the mixer by running Automap on a large amount of text obtained from some of my open source reserach to evaluate how it's able to build a network based on all that data.

So if folks from Sun are reading this, please let me get the machine. I've got real research to be doing here, and I'm even willing put down that I got cool hardware from Sun in my papers. It'll be even cooler if you let me keep the box. Here's hoping for some interesting results once it ships!

When all the cool kids were raving about Skype, I went and signed up for an account. I haven't used it much. I think I got maybe two calls on it, and both times I didn't have a microphone handy to actually take the call. Something about Skype always made me a little uneasy. Maybe it was the fact that they didn't reveal just how it worked. What encryption was in use? Could they drop in on your calls? Well, now that eBay bought Skype, I don't think I'll use them much anymore. eBay is known for not being a very privacy friendly company, and let's face it, their customer support sucks.

So what's a guy like me to do? Well, luckily Michael Robertson (the same guy behind MP3.com and Linspire), has started a cool toy called Gizmo, which is based on standard SIP technology. That's nice. They also have clients available for Windows, Mac, and Linux (linspire based debs, but whatever, it works fine on Ubuntu).

What makes this really nice is that first, you get a free quarter to make phone calls. At 1.8 cents a minute to the US this won't go really far, but it should be enough for me to make a phone call or two. Second, because it's SIP based, you can dial into your client from any regular phone. There's a list of access numbers for a large number of cities. Just pick an access number, press 1, and then type the SIP Phone number you'd like to dial (my SIP number is 1-747-617-4120). With free long distance dialing on cell phones, this means that it's no problem to call my computer from your cell phone. That's cool.

They've also got a slew of other nifty features, mainly made possible by using a real standard. For example, if I get a voice mail, it gets emailed to my GMail account as a wav file. Then I can download the file and listen to the message at my leisure. I would also think that in theory that it should be compatable with MythPhone, so you can easily integrate a phone into your MythTV box. I'm just in a state of wow right now. Maybe one of these days I'll need to find a way to hook up Asterisk too.

MindBending Software produces nifty products that inserts subliminal messages into games. This is accomplished by intercepting GL calls and layering new graphics on top of it. They also can perform some trippy transformations to get your zen on. Unfortunately, it's really expensive, most modules will run you between $700 and $1100. Ouch.

In particular, take a look at some of the videos they have online. I dunno if putting images of Jesus in a kids game of Grand Theft Auto is really going to help them. Interesting technology none the less.

I've just finished working CSCW at the Chicago Hilton hotel. My role there was as both as student volunteer and as the computing chair. As computer chair my role was to ensure that the attendees had network access and were able to connect to the Internet during the main sessions to participate in back channel conversations. My role as a student volunteer was to help out whenever possible. I didn't need to do a whole lot in that respect because most of my time was taken up by working on the network issues.

So let's talk the original vision for the network. Originally the conference was going to provide wireless access in the Marquette room on the third floor of the hotel for use as a email room. We had a few computers that we rented and were connected by a wired network to the internet. Downstairs the main sessions for the conference were to take place in the Continental ballrooms. This was one large room that could be separated into three different rooms. The original plan called for us to use the hotels wireless at their rather exoborant rate.

However, that would have been far too easy. A week before I was supposed to arrive in Chicago, I received an email informing me that the hotel would be unable to provide the wireless access we requested. Furthermore the contact for the hotel would not be in for the next few weeks (the email was conventiently sent at 7pm on a Friday evening). Furthermore, I was also informed that we now had to plan wireless access for another large ballroom on Wednesday. This was due to another failure on the part of the hotel as they had "accidentally" given rooms we were contracted for to another group. At first the thought was that I could probably just ignore the other room and not provide access, then I was told that Lessig's talk would be in the new room. Joy.

So, it went from being a very easy job to a rather difficult job. I now needed to plan a full network for 500 people in about four days. Did I mention I was really annoyed with the hotel by this point in time? Well, if not, I was.

So, I got a hold of Jim on Saturday afternoon to discuss what we should do. The decision was made to give me the go ahead to purchase whatever else I needed. I estimated that on the low end I would need about $700 worth of equipment to do the job and that on the high end I would need about $2000. It's hard when you can't say what the power of the wireless signals will be or if you're going need to buy a computer to serve as a DHCP server or not.

I ended up just upping my original order a little bit. I had originally planned to order about 1000 feet of cable to wire whatever I really needed to wire, along with a crimper and some heads. I also had already planned on getting four WRT54Gs and using those for the bandwidth upstairs. I just upped the order by three and picked up a total of seven WRT54Gs. In addition I ordered a 16 port wired switch for the Internet room. Total bill for all the equipment (cable, crimper, heads, WRT54Gs, and switch) was just under $800. I went with CDW because they were local and if stuff started blowing up they had same day delivery in downtown Chicago. Plus, it wasn't my money so I wasn't concerned about saving $3 a router by going with some obscure website I had never heard of.

I arrived on Saturday morning and found that the equipment had arrived without any difficulties. Everything was there in nice little boxes. Shortly after lunch time the tech from the hotel met me and we went to turn on all of the internet jacks in each of the rooms. Brian, the AV chair went with me to make sure everythin went smoothly and to ensure I remembered where everything was. That afternoon I set up the internet room. I used two WRT54Gs for the room. One was the primary connection and the other just served as a switch/access point. They were put on two different frequencies, away from the Hilton's channels. I also made the decision to use the custom Sveasoft firmware. I'm not big on the fact that the guy behind it seems to be a dick, but I run an old version on my router and it seems to work well. This just worked, no problem at all, it seems like everyone was able to utilize everything just fine.

The next day I set up everything in the ballrooms downstairs. I had originally planned on doing one wireless network with a central DHCP server that I would have to purchase. However, due to Chicago fire code, I was unable to do this as we'd have ethernet cable running across a doorway or two, which is verboten in Chicago. Instead I made three different networks, each with a different SSID and on a different channel. Fearing that I would run out of IP addresses, I limited the leases to 45 minutes. Later this would come back and bite me in the ass.

On Monday the conference began. It was not pretty. While some people could get on the network just fine, others were having a horrible time. There were a couple of reasons that first day. One reason was just bandwidth. I hadn't blocked any services. I did a little bit of sniffing on the network and found that there were people doing streaming video and BitTorrent. At a conference you just can't have this sort of thing going on. Luckily, the firmware has a layer 7 packet classifier that is able to block just those services without me blocking the ports. This helped out considerably.

But we still had the problem with people not being able to get IP addresses. It seemed like going up to the machines and rebooting them "fixed" the problem. In the sense that it allowed people to get IP addresses, but it also sometimes caused address collisions. People seemed okay with the network at this point, but they weren't happy. I decided to lower the lease time on the addresses, thinking that may have had something to do with it. So I dropped the time to 15 minutes, this only made the problem worse. Sigh.

I didn't get the problem figured out on Monday. I could still associate with the access points if I gave myself a static address, but this was not practical for everyone. Just to make sure that Messyboard kept running, I ran a 200 foot ethernet cable to Adam's display and gave him a couple of static addresses.

Tuesday rolled around and I was able to pinpoint the problem down to the DHCP server on the WRT54Gs dying. There was pretty simple solution to this, just put something in cron to automatically restart the DHCP server. The problem is where to put it. Entries that were put in crontab just seemed to disappear at random, they were not reliable. I found I could write a script and place the cron entry in /etc/cron.d and that would do the trick. However, if the machine was rebooted then I would loose the script. I made a copy of the script on my disk and proceeded to put it on each of the machines. It worked like a charm. I also realized that to minimize traffic on the DHCP server I should put the lease time at 2 hours, as the is the amount of time between sessions. Important note for conference planners, put your lease times at the length of time of the sessions and no less.

Wednesday proved to be my biggest challenge. There was session on backchannel communication in the morning that Joe asked me to attend to make sure everything worked fine. I arrived down there early in the morning to drop another access point in the room. Both were set to restart the DHCP server every 5 minutes. Overall the backchannel session worked like a charm. The presenters were all in the backchannel as were many people in the audience. The two access points were serving 120 different wireless devices at the same time (70 on one, 50 on the other). Yes, these little boxes kept going after I fixed the DHCP issues.

The backchannel session was the most stressful, but also the most rewarding. After a while, I just stopped watching the routers because I become comfortable in knowing that they were going to keep on working. Instead I just started searching for naughty pictures to make Liz Lawley's instance of EtherPeg show funny stuff on the projector.

If I had it to do over, I would make a few changes. First, I would definately find a way to wire all the access points together. Originally, this had been a possibility for us if I would have gotten one of the wedge things that your put in a doorway and run cables under. This would have made everything much more seemless. This would have also required a computer to manage all the addresses. A little box from Soekris would probably do the trick. This would also allow me to create a larger subnet, so I wouldn't have to worry about running out of addresses. However, because we had to move upstairs on Wednesday, and thus the conference was split between floors, this turned out to be a non-option anyway.

The other thing I would do is give more thought to backchannel. Most importantly, mandate compensation for FreeNode as part of the deal. They are great folks and really helped us out a lot. Secondly, get a bot that is able to post the logs and an archive of the links on a web page. Lots of people have asked for the logs, which have been posted to my webpage. I'm not entirely sure about the need for a wiki during the conference, but post conference, it seems like it would have been helpful. During the conference it was great to have Messyboard running in a highly visible location. I can't stress that enough.

Finally, one last thought is that it might be nice to have a Planet style weblog aggregator for people who are blogging at the conference.