Is it possible to change the radius of a rounded rectangle after its created in Photoshop?
No, but you can work around this by creating 4 circles and 2 rectangles in the vector mask. The circles work as the rounded corners and the rectangles fill out the horizontal and vertical space. From there, all you have to do is transform the circles and the rectangles as you see fit.
You can download a CS5 .PSD of this at http://imsky.org/files/CustomRoundedRectangle.psd