You are asking too many things. You will also have to give more information on which operating system you are running from, but I think you should be able to open two instances of VLC at a time and have one record the webcam and the other, your monitor.
To output in JPG, taken from the VLC Command-line help:
--scene-format=<string> Image format
Format of the output images (png, jpeg, ...).
However, I think this only works on Unix-based machines. It's too bad for Windows users because there is a plethora of options all documented in that link.
There is no VLC "service" for Windows. You'll just have to minimize it. Setting up the recording from Windows is quite easy however. See their very good documentation:
Also, if you are going to do this recording from the terminal on a Unix based machine, you will need to figure out what your devices are so that you can set it up accordingly. Because I don't know your hardware, I can't figure out the commands exactly. Look at the two links above if you are bent on this approach. They have a multitude of good information.