Ok, I'll give you a few tricks I learned about video and Linux. Direct Rendering is always better. With VLC, enable all the Hurry Up and Speed Tricks you can find. If you hover over something and it says it needs lots of CPU to do, then disable it or use a simpler setting. Drop frames where possible, but avoid hard frame dropping. This normally optimises and balances out the CPU and Read rate from discs so they work a little better. The trick is to under-run the buffer the whole time, while rendering as much as possible.

Anyway, start up VLC and press Ctrl+P. Click ALL for Show settings. Go to Input/Codecs ---> Other Codecs ---> FFmpeg. Enable Direct rendering, both Hurry Up's, Allow speed tricks and MPEG4 quantization matrix. Disable Interlaced Encoding and set Quality to Simple. Click Save and you'll be done. Not sure if this will work better, but the driver that handles FFmpeg has been optimised to be way faster and not eat up your buffer as quickly as it used to.

So I finally found the solution under advanced > Input/Codecs > Access Modules > File is the cache setting I needed. Setting the cache in the simple mode or in a line command didn't do it.
Just for anyones information I tested VLC,gxine and mplayer for cpu usage with the same file on the same laptop (a 366mhz Thinkpad 600).

1. VLC
2. gxine (but it is slow loading and kind of unstable)
3. Mplayer (I love the gui but it has limited options and highest cpu usage)

Using VLC is kind of like Slitaz, it takes awhile to get it configured but works great once your done.