Other answers talk about quantization and noise. HiFiBerry did a test with software vs hardware and found noise levels ranging between -88 to -98dB, which would be inaudible (lower than 0 dB) even if there was no music to accompany the noise unless your music is as loud as a food blender.
There is another issue at play: modern audio codecs use floating-point samples. If you look at an mp3 file with a program like ffprobe, using
ffprobe music.mp3, you'll see it uses 's16p' samples. AAC, found in mp4 files, and Opus and Vorbis, found in webm or other container formats like mkv, use 'fltp'. (
ffmpeg -sample_fmts also lists 'flt' as distinct from 'fltp', but I don't know what the difference is other than that it's sometimes responsible for audio encoding being slower due to extra format conversions.)
This has resulted in a sort of arms race, or perhaps a better analogy would be plants competing for sunlight. People react positively to loud music, but music that's too loud is painful, so they set their volume low because they prefer to sometimes have to turn the volume up than to sometimes experience pain. Result, music has gotten louder. On YouTube, this effect is moderated by the player limiting loud music, though this isn't always effective.
As a result of this volume competition, you can find songs on YouTube where over 10% of samples clip, being higher than 1 or lower than -1, and the loudest samples are over twice the nominal maximum volume. (Have seen extreme cases from recorded videos where RMS volume was around +1dB, about four times the amplitude or 16 times the power of a typical loud song.)
How software deals with samples that exceed 1 varies. From my tests, ffplay clips samples regardless of changes to volume during playback with hotkeys, though it would be easy to specify a volume filter (or other filters like
dynaudnorm=40:5:m=2 or a traditional compressor) to prevent this clipping. Totem, the default video player for Gnome, clips even with a low volume. Vlc stops clipping if the volume is turned down.
So if you're able to discern the slight change in audio quality from a mere 1% or 0.1% of samples clipping as found in a typical loud audio stream, and your software does clipping after applying its volume control instead of before, then you may want to leave your software application's volume at a moderate level and increase the OS or hardware volume. But if you're only concerned with one particular song being too quiet, and you don't mind adjusting the volume for each song, then all the evidence says it doesn't matter at all, unless your hardware is amplifying noise to an audible level.
If you have or install ffmpeg, you can easily test how your software handles clipping by creating a loud file:
ffmpeg -i loudmusic.mp4 -vn -af volume=10dB,ebur128,astats -t 10 superloud.mp4. This should use AAC audio with fltp, astats should report peak level of around 10dB, and ebur128 should report an integrated loudness of around 0 (typical music is -8 to -15). If they don't, increase the volume boost. The volumedetect filter will tell you exactly how many samples clip, but it converts to a non-floating point format so you would avoid it during encoding or use the asplit filter and make sure the automatic conversions don't get messed up:
ffmpeg -i video.webm -vn -af volume=10dB,ebur128,astats,asplit[tmp],volumedetect,anullsink,[tmp]acopy -t 10 loud.webm
If a software layer's volume control can boost volume instead of just reducing it, potentially leading to clipping, this would be a reason to avoid using software to increase volume. But this is probably obvious. My environment, Gnome on Ubuntu, used to have a sound option to "allow volume to go above 100%" in the OS controls, though it has since disappeared, and I had to use it when I first got this computer since the maximum volume was too quiet for some reason. Vlc also lets you increase volume above 100%, making it obvious what is happening.