I want to restrict the bandwidth on YouTube in pfsense firewall. I try to setup layer7 with set the limit on http video/audio,flash, swf protocol and created the separate rule for YouTube but did not work. Please let me know which streaming protocol used on YouTube?
First of you are kind of doing "reverse engineering":
You are trying to understand how YouTube works and then "adapt" it to your needs.
IF you will get that to work it could for a short period of time. Services like YouTube are actively changed to improve it.
On the other hand the method that YouTube uses to function will most likely be used by many other programs (so blocking them too!).
What you could do (not sure if that is possible in pfsense) is use the hostname [youtube.com] (or even use an IP range) to restrict that in amount of data used or speed or whatever.