Yes, there will be an increase in the time taken for data packets (of any type) to be processed across the VPN link. This does not mean that the voice will sound slower, just that the audio delay on the link will be milliseconds longer so there will be a bit more 'long distance' delay in the conversation flow. In practical terms, provided the VPN link is solid and reliable, you may not even notice the effect but the only way to find out is to try it.
In my last job, I ran a VoIP (Asterisk) phone system at Head Office, linked to a similar setup in another office about 150 miles away. The inter-site link was via standard 8Mbit ADSL broadband with an IPSec VPN tunnel created between a pair of Draytek 2800-series ADSL routers and there was no significant sound delay on calls made across the link.
I once did a test where I made a call from my desk, though the VPN and back to my colleague opposite me and the speech delay was comparable to calling your desk phone from your mobile and speaking to yourself.