- Windows 8 can play x264-encoded video as long as it's in an MP4 or 3GP file (reference)
- AVI files might work, but don't have to, because AVI doesn't support everything x264 offers.
For further info, I'll point you to the answer I've posted here: What is a Codec (e.g. DivX?), and how does it differ from a File Format (e.g. MPG)? This goes way beyond what I'm writing here, but for this question, we can narrow it down a little.
What is x264?
So, first of all,
libx264 is an FFmpeg library that allows FFmpeg to encode video using the x264 encoder. x264 again is just one encoder that encodes video in the H.264 codec standard – also known as MPEG-4 Part 10 AVC / Advanced Video Coding. The codec ("encoder/decoder") simply specifies a standard that encoders and decoders have to adhere to.
Because there's just a codec standard, there are many other H.264 encoders apart from x264 such as Mainconcept or even a built-in one from FFmpeg. But x264 is free, open source, very efficient and delivers good quality, which is why it's used so often.
Why do we need a container?
x264 can only encode actual video. You could theoretically just store it in a
.264 file, and it'd be a valid video bit/bytestream. No audio, no subtitles, etc. But often, you don't just want to watch video without audio. This is why we need to merge video, audio and subtitles in one file. This file is called the container, as it contains all the individual media tracks.
A container will synchronize video and audio frames according to their Presentation Time Stamp (PTS), which makes sure they are displayed at exactly the same time.
Popular containers these days include …
MP4, short for MPEG-4 Part 14, not to be confused with video codec standards such as MPEG-4 Part 2 video (e.g. XviD-encoded) or MPEG-4 Part 10 video (the H.264 we've talked about above, e.g. x264-encoded).
MKV, short for Matroska, a container that can include almost everything you feed it.
AVI, a really old container format that is supported almost anywhere, but doesn't support a lot itself. Don't use it unless you really have to.
To sum it up: Codecs are stored inside of a container. Codecs represent the actual media, and the containers just bundle them into one file. You could call them both "formats", but don't use that term because it's ambiguous.