This is probably caused by one of your login scripts on the Ubuntu machine that is generating output to places it should never output to. (It's like using the sink instead of the bathroom... I bet your room mate will run out of memory tolerance pretty soon.) PuTTY has a FAQ section about it. Read it here.
Summary:
A.7.6 When attempting a file transfer, either PSCP or PSFTP says ‘Out of memory’ and dies.
This is almost always caused by your
login scripts on the server generating
output. PSCP or PSFTP will receive
that output when they were expecting
to see the start of a file transfer
protocol, and they will attempt to
interpret the output as file-transfer
protocol. This will usually lead to an
‘out of memory’ error for much the
same reasons as given in question
A.7.5.
This is a setup problem in your
account on your server, not a
PSCP/PSFTP bug. Your login scripts
should never generate output during
non-interactive sessions; secure file
transfer is not the only form of
remote access that will break if they
do.
On Unix, a simple fix is to ensure
that all the parts of your login
script that might generate output are
in .profile (if you use a Bourne shell
derivative) or .login (if you use a C
shell). Putting them in more general
files such as .bashrc or .cshrc is
liable to lead to problems.