Can you create your own options?
If by create, you mean write a Linux Kernel module, then yes. You can build all the modules and new options. There are also many non-standard netfilter modules for obscure or new protocols and filtering that haven't made it into the mainline kernel.
Modules occasionally get added to the kernel. So that
policy module you mention seems to available on my 12.04 Ubuntu system and an older Debian system. But it isn't installed on an even older Debian system I have.
I searched these options for iptables but wasn't able to find anything.
It seems to be documented on by Ubuntu system in the iptables(8) man page. It also seems to be documented in the die.net iptables man page. It many not be available on your system because of the age, or because your system was built differently.
Here is an excerpt from my man page.
This modules matches the policy used by IPsec for handling a packet.
Used to select whether to match the policy used for decapsula‐
tion or the policy that will be used for encapsulation. in is
valid in the PREROUTING, INPUT and FORWARD chains, out is valid
in the POSTROUTING, OUTPUT and FORWARD chains.
Matches if the packet is subject to IPsec processing. --pol none
cannot be combined with --strict.
If you have the .config file used to build your kernel you can see if the option was set to build that as a module
# grep -i policy /boot/config-2.6.26-2-686