Installed a WinTV 1600 video capture card, and after that, the VirtualBox WinXP guest would not boot.. got the message:
Failed to load VMMR0.r0 VERR_NO_MEMORY
After digging thru the logs, and noticing vmalloc errors - I decided to blacklist the cx18 modules (for capture card), and reboot. VirtualBox worked now, but then if I removed the blacklist on the cx18 modules, and tried to load the module - I got the same vmalloc errors.
Determined its a shortage of virtual memory in the linux kernel. Default is 128M.
Certain device drivers (modules) - in my case, a video capture card - require chunks of this virtual memory for device mapping.
Virtualbox obviously uses the same mechanism. You can have 16GB of RAM, but if you only have 128M of vm allocated, you are going to run into the same issue.
Solution is to pass the "vmalloc=<size>" parameter to your kernel on boot. Preferably in 64M chunks.
In ubuntu, edit /etc/default/grub, and add "vmalloc=192M" to the GRUB_CMDLINE_LINUX_DEFAULT list.
eg. GRUB_CMDLINE_LINUX_DEFAULT="vmalloc=192M quiet splash"
Now they play nicely together - and the overall system performance is much snappier.