As ultrasawblade said, there should be no difference, but there is, at least in the transition period.
The main problem are misconfigured DNS servers which may take forever to respond to queries requesting AAAA records for sites which hold the IPv6 addresses. This is the reason why there are so many people who claim that disabling IPv6 will increase web browser performance. Standards say that IPv6 is to be preferred over IPv4. That means that browser will first check if the site is IPv6 capable and if it isn't, it will go to IPv4. If everything works normally, the check shouldn't be noticeable at all, but some badly configured DNS servers will not send proper response and connection will have to "time out". The main reason for that is that people are generally lazy. Very large number of people don't see any benefits in IPv6 and will have to be forced to use it when Internet starts stagnating because of lack of IPv4 addresses. That will require software updates on servers, routers and the rest of the networking equipment or replacement, but the IPv6 is still far away and there's no pressing need to do the updates now or that's what large number of people thinks. This creates the chicken and egg problem because people won't use IPv6 until IPv6 equipment is available and problems are fixed and on the other hand equipment manufacturers don't want to pay too much attention to IPv6 capable equipment until there's need to do so and the server maintainers don't want to fix IPv6 problems until there's need to do so.
Also, a part which sometimes gets forgotten are devices that aren't network related but use network like IP phones, scanners, printers and so on. They won't require IPv6 support immediately because of dual stack and will probably have no problems during their lifetime, but some misinformed managers think that they do and in their minds this increases the cost of migration, since all of them would have to be replaced.
Another point mentioned here is NAT. Some people feel safe behind NAT thinking that it increases security. In IPv6, it isn't needed. Instead correctly configured firewalls should be used, but people oppose the change. There was also the "MAC" conspiracy. Basically IPv6 address is divided into two parts. The prefix which goes to the router and number of the device connected to the router. In some situations, MAC address can be used to determine the device's number and the story goes that this will enable tracking of devices across networks. However the feature can be disabled and works when first 64 bits are assigned to router and the second 64 bits of the address are assigned to the devices. The downside of this is that in some circles this built mistrust of IPv6, especially when that is combined with lack of NAT.