Discovered a wonderful kernel module/feature called NETCONSOLE. From netconsole.txt - "This module logs kernel printk messages over UDP allowing debugging of problem where disk logging fails and serial consoles are impractical."
Using this usefull howto as a basis I got this working on ubuntu 10.04, which already has netconsole built as a module.
On the source system where you want to trap the messages:
modprobe netconsole netconsole=<sport>@<saddr>/<dev>,<dport>@<daddr>/<dmac>
where sport is the source port, saddr is the source address, dev is interface, dport is destination port, daddr is destination address and dmac is destination mac address.
On the remote end, install netcat, and do:
netcat -ul <dport> >> /var/log/netconsole
So easy.. and so useful..
Note: This is different than remote syslogd - which will capture kernel messages over the network - since netconsole is at the kernel level and thus more capable of capturing panics.
