Looking around, it appears that some of the problem may be to the new tickless features in the linux kernel. Basically, this feature allows the system to stop waking up periodically if there is nothing to do. On desktop and laptop machines this saves a lot of power and is a key feature that has helped the OLPC be such a power miser. However, it also really screws with the clocks in virtual machines, and may be cause of other issues. I’ve also read that there are issues with heavy disk I/O and Gutsy in VMWare Server — which is my virtualization platform of choice.

Previously, with the time issue, I would fix problems by adding in a cron job to periodically synchronize the date with a remote system. It’s important to note, that ntp won’t work because it tries to gradually synchronize the time — so it will never catch up. However, using a cron job on the virtualized system relies on the virtualized system eventually hitting even minute marks. Last night in the course of 12 hours, my time advanced 32 seconds; cron is not an option. The solution is to have the host operating system, which can keep time, periodically SSH into the virtualized machine and synchronize the time. Getting started, you’ll need to activate the time service in xinetd on the host operating system. Open up /etc/xinetd.d/time and change the lines that say

disable = yes

to say “no”. Restart xinetd by running /etc/init.d/xinetd restart and you’ll now be able to rdate to your host machine.

The next step is to create the ssh key and setup the cron job to connect to guest operating system. These commands should do it for you. Hit enter to leave it without a passphrase. Then, copy the key to the guest operating system.

root@host:~# ssh-keygen -t dsa -C "automated RDATE ssh key" -f id_dsa.rdate
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_dsa.rdate.
Your public key has been saved in id_dsa.rdate.pub.
The key fingerprint is:
d4:5f:01:83:ee:df:e0:e4:ea:ff:0a:6e:50:d1:e2:d3 automated RDATE ssh key
root@host:~# scp id_dsa.rdate.pub USERACCOUNT@GUEST:

Now that the key is on the guest, it’s time to enable a passwordless login. First, login to the guest machine, and open up a root shell using sudo -s. Then run the following command to add the new key to your ~/root/.ssh/authorized_keys.

echo 'command="rdate -s HOST"' $(cat id_dsa.rdate.pub) >> ~root/.ssh/authorized_keys

The final step is to edit your /etc/crontab on your host system, and make the time synchronization happen every five minutes or so.

*/5 * * * * root ssh -i ~root/.ssh/id_dsa.rdate root@GUEST "rdate -s HOST"

Now, you should have a nice infrastructure set up where your VMWare virtual machine will never be more than five minutes away from the actual time. It’s important to note that there are a few alternative ways of doing this. For example, one could bypass the usage of rdate entirely through some clever shell scripting that passes the time on the host system to the time command on the remote system. In my experience, this works in a homogeneous environment, but not every system can interpret times in the same way, so sticking with rdate seems like a good compromise.

Last year, in an effort to reduce clutter in my house and ease my power bill, I changed from a physical to virtual OpenBSD box as my main Internet server computer. For the most part I had some good luck, it worked pretty well once I changed the virtual NIC from a PCNet to an E1000. Back in October I decided that this holiday season I would upgrade to the 4.2 release of OpenBSD, which contained a lot of features I was looking forward to such as a port for Python 2.5 and the new vic VMWare network driver. About a week after I made this decision, my Apache server process began to randomly segfault when using gzip compression to serve web pages — as someone who runs their website off a cable modem, this was a hit that I really didn’t want to take. I disabled gzip compression and decided to look at it again over the holidays.

The process of bringing up a new virtual machine with VMWare server is nearly brain dead. I can create multiple virtual machines in a matter of minutes. For this upgrade the choice was made to start fresh to avoid some of the botched installs from previous versions of tools in the original virtual machine. My strategy was to install the basics for the machine, then compile the ports I needed — apache2, subversion (with apache support), python 2.5, screen, and irrsi — were at the top of my list. Wait, you may say, OpenBSD offers compiled packages for these — no need to compile. This is true, but the Python setup includes the tkinter libraries, which no ones uses anymore and forces X on my system. Subversion also links to some packages I’ll never use again.

After some fighting the build process got running, but I almost immediately hit a snag. Programs were complaining about expat not being present, and the port kept on telling me it was included as of OpenBSD 4.2. As I had OpenBSD 4.2 installed, I could tell that it was not included. My first strategy was to lie and build the package regardless. While this allowed me to build some software, it quickly caused problems. I was soon modifying my FLAVOR environment variable with reckless abandon. In the end, I broke enough things that I needed to start over. Luckily, VMWare makes that simple.

A little sleuthing discovered that for some reason in OpenBSD, expat is currently a part of the xbase package, and if I wanted the development headers I also needed the xshare package too. What? Why I do need X for everything all the sudden. Whatever, I installed those packages, rebuilt the software, but still needed to fudge around with lots of environment variables to keep it from installing TCL with anything that had optional TCL bindings. Don’t even get me started on the headache of keeping Ruby off my system. After about six hours of actual work at fighting with it, much more if you count compilation, I had the software I thought I needed installed. A little testing revealed that all of my web services still worked. Great! Then I turned on gzip and the segmentation faults returned. I returned to the default configuration, and the segmentation faults continued. I debugged the program, realized I need to do some nasty patching, and concluded that this wasn’t going to be the proper choice.

It was time to consider something else. I really wanted to run a BSD system still, I figured I’d give FreeBSD a shot again. For some reason their automatically discovered mirror for me was ungodly slow, worse than dialup. Going through tor actually managed to speed things up. Wow, that’s strange isn’t it? I also noticed they were close to a release — I don’t want to run pre-release software and I didn’t want to have to upgrade, and I’m really impatient. Crap. By this point it was the 27th, and I was supposed to leave for a week in just a few days.

Suck it up, install Ubuntu Server. I really didn’t want to run Linux, and Ubuntu Server may have a bit too much stuff on it in the default install. I was already starting to feel dirty. A little more digging revealed a nifty project, Ubuntu JeOS (Just Enough Operating System), a specialized and stripped down version of Ubuntu that has the core software and an optimized VMWare kernel. The download was pretty small, at about  50MB or so. Installation was a breeze and incredibly straightforward. As an added bonus, VMWare tools compiles for Linux, to further improve performance. A few apt-get commands later and my system was running. What’s better is that the packaging system didn’t require me to install X11, TCL, Ruby, PHP, or MySQL for my server setup. I was able to get just the tools I needed. Total installation time for everything, 41 minutes. That includes the time to copy all my web files over, port the httpd.conf file to the  Ubuntu style of multiple configuration files, and update some absolute paths in scripts.

Next up was to see how fast it was. I always loved how few processes I’d see running in my OpenBSD machine. Hammering the machines over and over an testing the speed of response with YSlow showed the Linux box to complete requests in about 40% of the time of the OpenBSD box when requested from the local LAN. Furthermore, the responses were compressed, which means a great saving of time over the line. I was hooked — sorry OpenBSD, you’ve been replaced.

Of course, there is a potential cost here. What about the security of my new server? What about stability? What about updates? As far as security goes, I’m already being fairly smart by running it in as a virtual machine. The main filesystems are backed up nightly to both local and remote systems, and about once a month, a backupninja script shuts down the machine, creates a snapshot of the full machine, then restarts it. That gives me about 10 minutes of downtime every month — I’m sure the spiders will understand. I also am not running any unnecessary servers — just apache2, an instance of CherryPy for PennAve, and very restricted OpenSSH for remote access — that’s it. I’m hoping I’m in good shape with that, but security is something you can never guarantee unless the machine is off, has no drives, and lacks any sort of persistent memory (of course, I’m sure that even then there may be a way to find some sort of electromagnetic residue in the RAM). Stability wise, I’m not entirely certain. Ubuntu Dapper on my MythTV machine has been excellent, while Gutsy on my laptop is sometimes flaky — but that may also be because of a long upgrade chain going back up to March of 2005 on this machine. Finally, I’m pretty sure that using APT makes upgrades, especially security related upgrades, even easier. I don’t need to worry about seeing that a patch was released and installing it by hand. It just happens — yes I’m placing trust in a remote entity, but I can handle that. I just want my machine to work.

So, after two days of mucking around with OpenBSD, I finally got Ubuntu JeOS to just work, and I’m very happy with it. I’m sorry to see OpenBSD go, but it was just too difficult to work with. In particular, here’s some issues that really need to be addressed to make it competitive with what JeOS offers:

• If you say you include expat, really include it, don’t place it in two different packages for X. I don’t need X on my server.
• Finer grain package control. Some packages are getting pretty good at creating subpackages for optional components — subversion is a good example of this. Other packages, such as Python, could really use some help here.
• Inheritance of the FLAVOR environment variable. If I compile subversion as NO_TCL or NO_RUBY, that should cascade down to swig. I had to literally watch every dependency as it was compiled to make sure that it was going to decide I needed TCL, TK, and Ruby.
• Automated checking for security patches. I’m not sure if OpenBSD has this, but it wasn’t obvious how to do it. But, then again, it’s not immediately obvious how to use apt either.

For the time being, OpenBSD and I have parted ways. In a few years when it comes time to upgrade my server again, we may again join forces. Of course, with the continual increase of software as a service, in a few years, there may be no need for me to run my own server anymore

The machine is a circa 1997 AMD K6-200 with 96MB of RAM and Quantum Fireball 4.3GB hard disk — this was pretty hot stuff when I went to college in 1997. Yes, that’s right, it’s my first computer from undergraduate. The machine first started to have problems in the fall of 1999. At the time I thought it was disk issues, so I bought a new disk. That didn’t fix the problems, in fact, I think I threw out at perfectly good 8GB disk at the time because of the issues. Really, it was just problem with the IDE cables in the machine, but I digress. That year I was working more or less full time and going to school, so I had money to burn. I plunked down some cash for an Athlon 700 machine, and that machine became my backup server machine (replacting a Pentium 60). This is also where the machine got the name scissors, for scissors.dtd.iit.edu. I used some of the original parts in my newer machines, this left scissors with a black and with text only video card, and a couple of ISA ethernet cards (at least they’re 3com). Yup, 10mbps is the max ethernet rate on the machine.

This point was also the first time that I installed OpenBSD on it. I believe it was OpenBSD version 2.5 or thereabouts - I actually paid for the CDs and have them laying around. Since then, the machine has been upgraded to newer operating systems twice, first to OpenBSD 2.9, then again to OpenBSD 3.3. It’s served as a mail server, online game server, and most importantly my personal web server for the last few years. At some point, probably in early 2003 the fan on the CPU died, but it didn’t seem to affect performance, so I didn’t care. At some point later the fan on the power supply died — once again it didn’t seem to affect performance so I didn’t care. I now had a basically entirely silent machine. Cool. As an added bonus, according to my Kill-a-watt device, it sips power at a consumption of 33 watts. This makes it nice and affordable to run.

However, the machine is now approaching it’s ninth birthday. It’s had a good run — but all good things must come to an end. I’ve been getting rid of my old stuff and well scissors, it’s time for you to go. You’ve been replaced by the forward marching tide of technology. However, it’s a real pain to reinstall web servers all the time. My solution? Virtualization.

Thanks to VMWare Server and my copy of VMWare Workstation, I’ve created a very nice little OpenBSD virtual machine. Best of all, I’ve got all the services that previously were running between a couple of different servers consolidated onto this single box. That means this virtual machine now acts as my trac, subversion, dav, and person web server.

Virtualization has quite a few benefits. One of the biggest ones is that I can just copy the virtual machine from one physical computer to another. This makes upgrades a breeze. (As an aside, it also makes me wonder why I don’t just install VMWare Server on my MythTV box in the living room and drop down to a single computer that is on 24/7. Probably has something to do with the printer we never use I guess.) Also, being virtualized, it let’s me test other configuration changes easier. I can take a snapshot of the server in workstation and try an upgrade or change in software with no risk to the live machine. The VMWare Server console lets me manage everything from my laptop too. Wow.

I know that someone out there will believe that I’m a sellout because VMWare is closed source. Which is probably partially true. However, I’ve been using VMWare since it first came out. Back at LEC Ltd, we used VMWare on the developer workstations to see how the web pages would look on different platforms. Of course, we never thought about using it for server use — but now it seems that’s what the cool kids are doing. An astute reader might wonder why I didn’t choose Xen or QEMU. With regards to Xen, I’ve heard that Theo considers Xen to be more of a toy, however he might be coming around to it (this was a year or more ago now). I did notice that there is a Summer of Code request to make OpenBSD a native Guest OS. That might be really interesting, but it’s not here yet. For QEMU, the big issue is that if you want native performance you need to the unfree accelerator program. That, and it’s not as feature rich as VMWare. Given that I have extensive VMWare experience too, it seemed like this was the fastest way to get running again.

So what is all this saying? Basically, if you’re reading this weblog entry you’re connected to my VMWare server instance, now called “donkey” running OpenBSD 3.9. Well, if this post is recent that’s what happening. In the future I should be able to upgrade pretty easy. For now, the host machine for Donkey is a Pentium 3/900 with 640MB of RAM. So far, so good. I took extensive notes about the whole process and will be putting together a nice writeup soon. If you’ve got questions about the process, feel free to email me in the mean time.

My initial hope was to commandeer a monitor, plug in a USB keyboard, and go
at it. One major proble, there is no video card installed on the box, although
if I had an extra PCI-E card, it would be interesting to see what happens with
one. Initial system boot-up and installation is done via the serial console.
Okay, find a null modem and a laptop with a serial port (my new T43p does not
have a serial port). Wrong. Even if I could have found a null modem, the
regular serial port is not the serial console, instead you need to use an
RJ45->DB9 dongle to get a serial terminal. Of course, these things were no
where to be found around Carnegie Mellon.

Luckily, an email went off, and a schmeatic of the circuit came back. Thinking
for a bit, I remembered that Make Magazine
vol 1 had an article on how to build a
5-in-1 network cable. Off to
Radio Shack for some parts. Unfortunately, Radio Shack has slowly turned
itself into “crappy overpriced cell-phone shack”. Not a single RJ45-DB9
connector was to be found. The only DB9 connectors I could find were just
a single snap in port. In fact, they didn’t even have a coupler for RJ45. So I picked up two of them. The next step was to
mangle an ethernet cable, and have some breadboard fun. After about 30 minutes,
I had my circuit.

The Ghetto RJ45->DB9 Converter

Luckily, after a little bit of tweaking the circuit, and reseating some wires
on my breadboard, the ALOM on the T2000 started up and spit out data to the
Hyper Terminal running on Kristina’s
laptop. It was pretty straight forward to get the network console running
after this point, and yes, it can DHCP an address, but be very careful about
putting the network admin port on a public network because the VxWorks OS that
runs ALOM does NOT support SSH. Suckage. Here’s another shot of getting
everything going. I’ve got the serial terminal up on the HP laptop and the
network terminal up on the T43p. After booting into Solaris, it’s much better
to use SSH to connect to the OS than go over the network terminal.

Getting the T2000 Up and Running

I do have a few critiques about the process. First of all, the sales people were, for lack of a better term, sales people. I started to get
random messages from people I had no knowledge of and seemed to have no relation to me. I was sent an invoice with a subject line of “INVOICE”,
which of course usually goes right to the spam filter. Finally, I got some helpful documents under a subject line of “[Fwd: [Fwd: cool tool FYI]]”,
with no explanation of why I was getting it. So let me provide a few suggestions to make the process nicer to people like me:

• Use informative subject lines that are not all capitalized. Remove all those “Fwd” headers too. Instead write “Helpful Sun T2000 Documents”.
• If you’re going to send me an email from an address that looks real, make sure you reply to it when I ask a question about it. This goes for
  you, Rebecca Lukens. I received an Invoice which said I needed to take some action regarding it, I replied to it with a few questions and never
  got a response. I tried again, and never got a response. There was also no contact information in the message, which I’ve reproduced in verbatim here:

  Try & Buy quote attached.  Thanks!

• Take some time to say why you’re emailing me. For example Instead of the above message, I would have appreciated something like this:

  Mr. Wagstrom,
  
  Attached you'll find a quote for your recent Sun Microsystems T2000 Try and Buy request.
  Should you choose to keep the machine beyond 60 days, you'll be required to pay the full
  amount of the invoice unless other arrangements have been made.  For right now, please
  enjoy the machine and we hope it sits your needs well.
  
  If you have questions, you can contact me at:
  [CONTACT INFORMATION HERE]

  You’ll notice it’s a form letter. They don’t even have to think more than pasting in the current text, but it would be very helpful to me. Also, the change would give it a chance of evading the Carnegie Mellon spam filters.

• When sending out unsolicited attachments via email, also include a web link to a site on sun.com where I can retrieve these documents.
  I trust documents coming from sun.com more than I do general email that is sent to me from unknown individuals.

But like I said, the process was actually relatively painless. A few frustrations, but I’m happy with the opportunity to test out this technology.
We’ll start benchmarking it soon.

As I thought, the machines are way backed up because of the massive response
to their admittedly very cool offer. I explained our situation to their
educational sales person for the Pittsburgh area (note to Sun web folks, I
could not easily find this information on your web site. Instead Phil had to
contact the sales person for the University of Minnesota who forwarded my
information to the nice person here in Pittsburgh) and she seemed quite
receptive and reassured me that Sun was processing the stuff and that our
purpose seemed pretty novel and interesting. So, we’re still in the queue,
but it should be all good from here.

She could have just left it at that, but she didn’t. She also was helpful
enough to provide us information about Sun matching grants for hardware. This
drops the price of a 4 core box down to a more reasonable $3800 or so. Pretty
nice. However, we still don’t have that much money, at least not yet. Anyway,
if this can run on $3800 hardware, I’d think that it makes things a whole lot
more attractive for doing some sustained testing on the system. We’ll just have
to see how well everything actually runs on it.

Once this thing shows up I’ll be sure to blog some more and provide some
photos of the system along with preliminary benchmarks.

As part of their promotion, they’re doing this cool Try before you Buy thing. Basically,
you given them your information, they send you a server free for 60 days — they
even cover postage. You evaluate it, if you like it, you pay for it. If it
doesn’t work well for your field, send it back — once again, on Sun’s dime. Pretty cool stuff. What’s cooler, is that according to Jonathan Schwartz, they’re looking for people to benchmark these guys. Specifically, if you
do a good enough job on the writeup, they may let you keep it.

So, being the graduate student I am, I immediately realized a couple of
different applications that I’d like to work on with this little guy. Firstly,
we’re going to see how well he works as a spyware monitoring box — run multiple
instances of snort on him and a database to log everything to. Should be
kinda interesting to see. Secondly, I’m going to try and use it for some of
my network analysis stuff. Specifically, seeing if it’s faster to run ORA on
it, or at the very least what’s the overall throughput.

I’m a little skeptical about it’s performance on some of the measures in ORA
because the cores share an FPU. However, the integer measures should run quite
nicely. I’ll also put it through the mixer by running Automap on a large amount
of text obtained from some of my open source reserach to evaluate how it’s able
to build a network based on all that data.

So if folks from Sun are reading this, please
let me get the machine. I’ve got real research to be doing here, and I’m even
willing put down that I got cool hardware from Sun in my papers. It’ll be even
cooler if you let me keep the box. Here’s hoping for some interesting results
once it ships!

So what’s a guy like me to do? Well, luckily Michael Robertson (the same
guy behind MP3.com and Linspire), has
started a cool toy called Gizmo,
which is based on standard SIP technology. That’s nice. They also have clients
available for Windows, Mac, and Linux (linspire based debs, but whatever, it
works fine on Ubuntu).

What makes this really nice is that first, you get a free quarter to make
phone calls. At 1.8 cents a minute to the US this won’t go really far, but
it should be enough for me to make a phone call or two. Second, because it’s
SIP based, you can dial into your client from any regular phone. There’s a
list of access
numbers for a large number of cities. Just pick an access number, press 1, and then
type the SIP Phone number you’d like to dial (my SIP number is 1-747-617-4120). With free long distance dialing
on cell phones, this means that it’s no problem to call my computer from
your cell phone. That’s cool.

They’ve also got a slew of other nifty features, mainly made possible by using
a real standard. For example, if I get a voice mail, it gets emailed to my
GMail account as a wav file. Then I can download the file and listen to the
message at my leisure. I would also think that in theory that it should be
compatable with MythPhone, so you can easily integrate a phone into your MythTV box. I’m just
in a state of wow right now. Maybe one of these days I’ll need to find a way
to hook up Asterisk too.

In particular, take a look at some of the videos they have online. I dunno if putting images of Jesus in
a kids game of Grand Theft Auto is really going to help them. Interesting
technology none the less.

So let’s talk the original vision for the network. Originally the conference
was going to provide wireless access in the Marquette room on the third floor
of the hotel for use as a email room. We had a few computers that we rented
and were connected by a wired network to the internet. Downstairs the main
sessions for the conference were to take place in the Continental ballrooms.
This was one large room that could be separated into three different rooms.
The original plan called for us to use the hotels wireless at their rather
exoborant rate.

However, that would have been far too easy. A week before I was supposed to
arrive in Chicago, I received an email informing me that the hotel would be
unable to provide the wireless access we requested. Furthermore the contact
for the hotel would not be in for the next few weeks (the email was
conventiently sent at 7pm on a Friday evening). Furthermore, I was also
informed that we now had to plan wireless access for another large ballroom
on Wednesday. This was due to another failure on the part of the hotel as
they had “accidentally” given rooms we were contracted for to another group.
At first the thought was that I could probably just ignore the other room
and not provide access, then I was told that Lessig’s
talk would be in the new room. Joy.

So, it went from being a very easy job to a rather difficult job. I now needed
to plan a full network for 500 people in about four days. Did I
mention I was really annoyed with the hotel by this point in time?
Well, if not, I was.

So, I got a hold of Jim on Saturday afternoon to discuss what we
should do. The decision was made to give me the go ahead to purchase
whatever else I needed. I estimated that on the low end I would need
about $700 worth of equipment to do the job and that on the high end I
would need about $2000. It’s hard when you can’t say what the power
of the wireless signals will be or if you’re going need to buy a
computer to serve as a DHCP server or not.

I ended up just upping my original order a little bit. I had
originally planned to order about 1000 feet of cable to wire whatever
I really needed to wire, along with a crimper and some heads. I also
had already planned on getting four WRT54Gs and using those for the
bandwidth upstairs. I just upped the order by three and picked up a
total of seven WRT54Gs. In addition I ordered a 16 port wired switch
for the Internet room. Total bill for all the equipment (cable,
crimper, heads, WRT54Gs, and switch) was just under $800. I went with
CDW because they were local and if stuff started blowing up they had
same day delivery in downtown Chicago. Plus, it wasn’t my money so I
wasn’t concerned about saving $3 a router by going with some obscure
website I had never heard of.

I arrived on Saturday morning and found that the equipment had arrived
without any difficulties. Everything was there in nice little boxes.
Shortly after lunch time the tech from the hotel met me and we went to
turn on all of the internet jacks in each of the rooms. Brian, the AV
chair went with me to make sure everythin went smoothly and to ensure
I remembered where everything was. That afternoon I set up the
internet room. I used two WRT54Gs for the room. One was the primary
connection and the other just served as a switch/access point. They
were put on two different frequencies, away from the Hilton’s
channels. I also made the decision to use the custom Sveasoft firmware. I’m not big on
the fact that the guy behind it seems to be a dick, but I run an old
version on my router and it seems to work well. This just worked, no problem at all, it seems like everyone
was able to utilize everything just fine.

The next day I set up everything in the ballrooms downstairs. I had
originally planned on doing one wireless network with a central DHCP
server that I would have to purchase. However, due to Chicago fire
code, I was unable to do this as we’d have ethernet cable running
across a doorway or two, which is verboten in Chicago. Instead I made
three different networks, each with a different SSID and on a
different channel. Fearing that I would run out of IP addresses, I
limited the leases to 45 minutes. Later this would come back and bite
me in the ass.

On Monday the conference began. It was not pretty. While some people
could get on the network just fine, others were having a horrible
time. There were a couple of reasons that first day. One reason was
just bandwidth. I hadn’t blocked any services. I did a little bit of
sniffing on the network and found that there were people doing
streaming video and BitTorrent. At a conference you just can’t have
this sort of thing going on. Luckily, the firmware has a layer 7
packet classifier that is able to block just those services without me
blocking the ports. This helped out considerably.

But we still had the problem with people not being able to get IP
addresses. It seemed like going up to the machines and rebooting them
“fixed” the problem. In the sense that it allowed people to get IP
addresses, but it also sometimes caused address collisions. People
seemed okay with the network at this point, but they weren’t happy. I
decided to lower the lease time on the addresses, thinking that may
have had something to do with it. So I dropped the time to 15
minutes, this only made the problem worse. Sigh.

I didn’t get the problem figured out on Monday. I could still
associate with the access points if I gave myself a static address,
but this was not practical for everyone. Just to make sure that Messyboard kept running, I ran a
200 foot ethernet cable to Adam’s display and gave him a couple of
static addresses.

Tuesday rolled around and I was able to pinpoint the problem down to
the DHCP server on the WRT54Gs dying. There was pretty simple
solution to this, just put something in cron to automatically restart
the DHCP server. The problem is where to put it. Entries that were
put in crontab just seemed to disappear at random, they were not
reliable. I found I could write a script and place the cron entry in
/etc/cron.d and that would do the trick. However, if the machine was
rebooted then I would loose the script. I made a copy of the script
on my disk and proceeded to put it on each of the machines. It
worked like a charm. I also realized that to minimize traffic on the
DHCP server I should put the lease time at 2 hours, as the is the
amount of time between sessions. Important note for conference
planners, put your lease times at the length of time of the sessions
and no less.

Wednesday proved to be my biggest challenge. There was session on
backchannel communication in the morning that Joe asked me to attend to
make sure everything worked fine. I arrived down there early in the
morning to drop another access point in the room. Both were set to
restart the DHCP server every 5 minutes. Overall the backchannel
session worked like a charm. The presenters were all in the
backchannel as were many people in the audience. The two access
points were serving 120 different wireless devices at the same time
(70 on one, 50 on the other). Yes, these little boxes kept going
after I fixed the DHCP issues.

The backchannel session was the most stressful, but also the most
rewarding. After a while, I just stopped watching the routers because
I become comfortable in knowing that they were going to keep on
working. Instead I just started searching for naughty pictures to
make Liz Lawley’s instance of
EtherPeg show funny stuff on the projector.

If I had it to do over, I would make a few changes. First, I would
definately find a way to wire all the access points together.
Originally, this had been a possibility for us if I would have gotten
one of the wedge things that your put in a doorway and run cables
under. This would have made everything much more seemless. This
would have also required a computer to manage all the addresses. A
little box from Soekris would
probably do the trick. This would also allow me to create a larger
subnet, so I wouldn’t have to worry about running out of addresses.
However, because we had to move upstairs on Wednesday, and thus the
conference was split between floors, this turned out to be a
non-option anyway.

The other thing I would do is give more thought to backchannel. Most
importantly, mandate compensation for FreeNode as part of the deal.
They are great folks and really helped us out a lot. Secondly, get a
bot that is able to post the logs and an archive of the links on a web
page. Lots of people have asked for the logs, which have been posted to
my webpage. I’m not entirely sure about the need for a wiki during
the conference, but post conference, it seems like it would have been
helpful. During the conference it was great to have Messyboard
running in a highly visible location. I can’t stress that enough.

Finally, one last thought is that it might be nice to have a Planet style weblog aggregator for
people who are blogging at the conference.