Problematic Update on Ubuntu 6.06 Server

I had a client with a Ubuntu 6.06 server install primarily running just samba and a few other apps. I had it set to automatically update with all security updates at 6am each morning.

Generally, things had run problem free for over a year, but last week an update came thru for initramfs-tools that boinked the system. To complicate things, this wasn't noticed until a reboot was done a few days later by the user because of unrelated network issues. The system would start up, but never fully startup.

The system ran headless, so I pulled it onto the bench, hooked it up to a monitor, keyboard, and mouse, and set it up to boot. It would boot until it got to load initrd.img and then it would report:

RAMDISK: Compressed image found at block 0
VFS: Cannot open root device or unknown-block(253,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(253,0)

That isn't the exact message, but pretty close.

The problem seems to be with initramfs-tools not building the boot image correctly. The date/time stamp on the initrd.img was the same as when the security update ran. I believe lilo didn't or wasn't run after the initrd.img file changed, and this was preventing it from booting.

My ultimate solution, after a significant investment in time, was to roll back the initramfs-tools update and install a fresh kernel package (they were a little behind anyway). I don't normally recommend updating the kernel in the middle of a mess like this, but some of my attempts to recover the system appeared to mess up my kernel. The actual culprit was that lilo wouldn't run and update the system. That also seemed to be the problem with the original update of the initramfs-tools package (that lilo didn't run).

Here's the steps I did that mattered:

• Boot the system with the install CD and select the "rescue" option
• Get to the command prompt
• I had to manually mount the other partitions so that I could use them in the chroot'ed rescue mode:
  #mount /var
  #mount /opt
  
• Very important: I also had to mount the /proc system. Failure to do this prevented lilo from running properly:
  #mount /proc
  
• Roll back to a previous version of initramfs-tools:
  #apt-get install initramfs-tools=<version-number-here>
  This probably wasn't necessary as I now believe the problem was that lilo didn't get run.
• I couldn't get the network connection to work right with the network cards in the system, so I had to use the CD archive to update the kernel with apt-get. I renamed /etc/apt/sources.list, inserted the cdrom with the archive (it was already there from booting with it) and pointed apt-get to just to the archive:
  #apt-get cdrom
• Now I could update the kernel:
  #apt-get install linux-server
  
• Next I ran LILO to update everything and allow booting:
  #lilo
  

That was pretty much the high points. The system is now humming along nicely and I have disabled automatic updates.

UPDATE: 2008/02/18
Happened when I rebooted another Ubuntu server. I was able to get this one up by booting up as described above, but all I did was mount my partitions and run lilo. That fixed it.