I've been experimenting a bit with the puppies bath in 4k video off 4ksamples.com as well as a few others. I got a combination that worked with my test video. I note reasonably fast storage is essential for quality playback. Considering some trial and error is needed, do backup the current settings. These settings work on a 3770, 16gb of ram and a gforce 980TI which I know is 4k capable
I've changed quite a few advanced settings to get it to work. In theory I want DXVA to be enabled but that causes VLC to crash.
Under simple mode in preferences go to Tools -> Preferences and Input / Codecs and in the drop-down box for Skip H.264 in-loop deblocking filter change it to All. This is recommended by VLC
Switch to show all under preferences and try these changes
I turned off all filters under Video Filters. This was the last thing I did, and seemed to help the most so I'd start here
Under "Video Codecs" - > FFMPEG I set my threads to 3. You might want to experiment with higher or lower numbers.
Make sure everything under output modules/DirectX is unticked.
These changes seem to have worked.
For reference, once I stripped out the commented lines, my vlcrc looks like
[directdraw]
directx-hw-yuv=0
directx-3buffering=0
[direct3d]
direct3d-shader=Disabled
[swscale]
swscale-mode=9
[qt4]
qt-privacy-ask=0
[x264]
sout-x264-profile=main
sout-x264-preset=veryfast
sout-x264-tune=film
[qsv]
sout-qsv-software=1
sout-qsv-h264-profile=decide
sout-qsv-h264-level=decide
sout-qsv-mpeg2-profile=decide
sout-qsv-mpeg2-level=decide
[avcodec]
avcodec-skiploopfilter=4
avcodec-hw=any
avcodec-threads=3
[waveout]
waveout-volume=0.700000
[directsound]
directx-volume=0.700000
[filesystem]
filetypes=m3u,db,nfo,ini,jpg,jpeg,ljpg,gif,png,pgm,pgmyuv,pbm,pam,tga,bmp,pnm,xpm,xcf,pcx,tif,tiff,lbm,sfv,txt,sub,idx,srt,cue,ssa
[core]
aout=any
video-deco=0
vout=direct3d
video-filter=swscale
vod-server=any
I have a gist with my full vlcrc for 4k here