As answered above, DDR3 is incompatible with DDR2 slots.
Upgrading RAM in Macs can be tricky. I recently upgraded a MacMini of the same vintage (2009), and initially bought RAM that "matched" Apple's suggestion, only to discover that if I used both of the new DIMMs the system wouldn't boot. It needed a "good" chip it could pull timing data from (or some other piece of voodoo that I haven't fully researched) that the others didn't supply. Ordered another DIMM, this time from Crucial, and all was well.
So, from now on, I'll check with Crucial first. I won't absolutely buy from them, but they do a very good job of researching and matching the RAM to the system.
So, I suggest you check here: http://www.crucial.com/mac/ before tou commit to upgrading your MB.