No, you won't damage those devices. There are occasional posts to the contrary online but for the most part it seems that PoE is only activated after an initialization request is sent. A better discussion can be found here: http://community.spiceworks.com/topic/187505-mixing-poe-and-non-poe-devices
PoE Detection occurs like this "In order to avoid damage to non-PoE devices, a key component of the 802.3af PoE standard is detection. PSEs must be able to detect the presence of a PoE-enabled device before sending power, and must be able to detect when the powered device is no longer present and remove power. To achieve accurate detection, a low-level detection current is sent from the PSE through the cable and the response is analyzed to determine if a compliant device is attached."
The rest of the explanation of PoE and the standard can be found here: http://www.network-1.com/poe/poe_ovw.htm