What is a profile?
A H.264 profile more or less defines what "bells and whistles" the encoder can use when compressing your video – and there are lots of H.264 features that the encoder can enable. Which ones it's allowed to enable is defined by the profile. Profiles ensure compatibility between devices that have different decoding capabilities. With profiles, the encoder and decoder agree on a feature set that they can both handle.
What do the different profiles do?
For a detailed list, see H.264 Profiles on Wikipedia.
Generally, the Baseline profile restricts the encoder to certain basic features only. Videos encoded with baseline profile can be easily played back, even on devices with lower computational power, such as smartphones. Android phones, for example, only play video encoded with the baseline profile, but not High or even Main:
Primarily for low-cost applications, this profile is most typically used in videoconferencing and mobile applications. It corresponds to the subset of features that are in common between the Baseline, Main, and High Profiles
Similarly, Main and High just add features to that. Especially the high profile is often used in broadcasting:
The primary profile for broadcast and disc storage applications, particularly for high-definition television applications (for example, this is the profile adopted by the Blu-ray Disc storage format and the DVB HDTV broadcast service).
B slices are for example only allowed in the Main profile and above. They can be used to save on bandwidth, but are harder to decode, which is why some devices might not support them.
What does that have to do with quality?
The profile only indirectly influences the quality. Some features of higher profiles may enable you to get the same quality with lower file sizes as compared to lower profiles.
For example, CABAC entropy coding (Main and High) is more efficient than CAVLC (Baseline). It is also computationally more intensive. Thus, if you give the encoder a certain bit rate to spend, it'll be able to create a better quality video with CABAC than with CAVLC because it achieves much better compression.
This also explains why you achieved smaller file size with the High profile — obviously, you somehow set a constant quality level and the encoder could use more advanced compression techniques to create a video file that has the same quality as the Baseline profile, but with smaller size.
So… which one should you use?
Some basic rules:
- Baseline profile if you're targeting mobile devices, e.g. even when encoding for web streaming
- Main or High profile if you're targeting long-term storage, PCs or Macs, Blu-ray authoring, etc.