AMD crammed 64-bit support into the Sempron without a redesign. This necessarily entailed a lot of compromises as a design optimized for 64-bit operation would have a lot of fundamental differences at a very low level.
As a result, 64-bit operations in 64-bit mode are about 35% slower on the Sempron than similar 32-bit operations in 32-bit mode. Of course if you would need two 32-bit operations instead of one 64-bit operation, that's still a win for 64-bit mode. But if you replace 64-bit operations with 32-bit operations, as the kernel does for pointers, you take that performance penalty.
There are still a lot of benefits to 64-bit operation. 64-bit operations are significantly faster for those times when you need them. And you have twice as many general purpose registers.
Unfortunately, 64-bit programs tend to require more memory bandwidth than 32-bit programs, and the Sempron has a memory controller with just a single channel. It relies on caches to hide this cost, but 64-bit programs move more data and so tend to have higher cache miss rates for the same cache size.
So your results aren't really all that surprising.