I don't know which version of VLC you are using, but, from a long time, VLC supports the feature that you are asking for. Here I am providing the steps for VLC version 2.2.6.
Basically there are three options in VLC settings that you need to look:
Under the interface tab of VLC settings, you will find an area called "Playlist and Instances" (as shown in this image). Ensure the following tick boxes are as mentioned.
Tick box corresponding to "Allow only one instance" should be ticked on.
Tick box corresponding to "Use only one instance when started from file manager" should be ticked on.
Tick box corresponding to "Enqueue items into playlist in one instance mode" should be ticked off.
Other than this, if you prefer to turn off the playlist every time VLC is opened, open VLC, go to View, and click Playlist. This will turn playlist off. Now close VLC in order to save the setting.