NETCONSOLE to the rescue

On a headless server, when you get a kernel panic and the system is totally locked up, and there is no sync of kernel log, its extremely frustrating trying to debug.

Discovered a wonderful kernel module/feature called NETCONSOLE.  From netconsole.txt - "This module logs kernel printk messages over UDP allowing debugging of problem where disk logging fails and serial consoles are impractical."

Using this usefull howto as a basis I got this working on ubuntu 10.04, which already has netconsole built as a module.

On the source system where you want to trap the messages:

modprobe netconsole netconsole=<sport>@<saddr>/<dev>,<dport>@<daddr>/<dmac>

where sport is the source port, saddr is the source address, dev is interface, dport is destination port, daddr is destination address and dmac is destination mac address.

On the remote end, install netcat, and do:

netcat -ul <dport>  >> /var/log/netconsole

So easy.. and so useful..

Note: This is different than remote syslogd - which will capture kernel messages over the network - since netconsole is at the kernel level and thus more capable of capturing panics.