You can install what is called a software preamp that will automatically change the volume, which will only apply to the devices you want.
Credit goes to JasSmith87 for explaining how:
You can use Equalizer APO.
Download it, run the installer.
It will ask you to select which devices you want to use the equalizer on. Select your headphones only.
You will have to reboot after the installer has run.
After the reboot edit
Preamp: -12 dB
Make sure you delete the
Include: example.txt line as it does change some equalization settings.
Play around with the -12, the volume changes as soon as you save the file.