Yes, it's okay to just copy them over: Git for Windows (what you called "msysgit") ships with a build of an OpenSSH client, and since Ubuntu ships OpenSSH as its SSH client as well, the keys require no conversion.
Note that both keys should be placed under the
~/.ssh directory on the Ubuntu machine (note that if this directory does not yet exist, and you're about to create it by hand, it must be owned by your primary user and group and have
0600 access mode). Then the
id_rsa file — the private key — must be set to have
0600 access mode. The public key (
id_rsa.pub) should use
0644 access mode — that is, write access for you only and read access for everyone.
Note that it's okay to lose
id_rsa.pub as it can be regenerated from your private key at any time.