As already suggested, DDR3 is incompatible with DDR2 slots.
Upgrading RAM in Macs can be tricky. I recently upgraded a MacMini of the same vintage (2009), and initially bought RAM that "matched" Apple's suggestion, only to discover that if I used both of the new DIMMs the system wouldn't boot. It needed a "good" chip it could pull timing data from (or some other piece of voodoo that I haven't fully researched) that the others didn't supply. Ordered another DIMM, this time from Crucial, and all was well.
You may check with Crucial first. I'm not absolutely recommending buying from them, but they do a very good job of researching and matching the RAM to the system.