I recently experienced some trouble with VLC and heavy mp4 files; although “normal” HD files would play fine, VLC would not play ball with my GPU when playing some heavy 1080p mp4 files. Here’s how I fixed it on my Samsung Ativ Book 9 Lite.
I am using Elementary OS Luna and the ATI Radeon driver from Jockey (the post-release update from the “Additional Drivers”). The FOSS driver for my GPU (ATI Radeon HD 8250) does not work well for now, so I use the proprietary driver until the FOSS driver is fixed.
In VLC, I checked the “Use GPU accelerated decoding” box in the Input & Codec settings, but 1080p mp4 files still failed to play properly. I searched and found a solution that worked for me in an AskUbuntu page. So apparently, installing the proprietary GPU driver and ticking a box in VLC is not enough to make VLC use the GPU. You also have to install a bit of software blob so that VLC can use the GPU for decoding. Weird.
Anyway, I installed these packages (please note that these commands worked for me but might not for you, please make sure you understand them before copying/pasting in your terminal):
sudo apt-get install xvba-va-driver libva-glx1 vainfo
To check that everything worked and that VLC is able to pick up your GPU, use the vainfo command. My output looks like this:
libva: VA-API version 0.32.0
Xlib: extension "XFree86-DRI" missing on display ":0".
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/x86_64-linux-gnu/dri/fglrx_drv_video.so
libva: va_openDriver() returns 0
vainfo: VA-API version: 0.32 (libva 1.0.15)
vainfo: Driver version: Splitted-Desktop Systems XvBA backend for VA-API - 0.7.8
vainfo: Supported profile and entrypoints
VAProfileH264High : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
I had no problem ever since with VLC and large H264 files, it works like a charm again, and playback is silky smooth. I really wish I could have the same experience with the FOSS driver, but unfortunately the nice people that make the Linux kernel have not picked on it yet.
N.B.: This solution works with the 3.11.0-26-generic kernel. I tried the 3.13 kernel but my laptop boots to tty. Not funny.