On recent versions of Linux (since 2.6.36), you can use the prlimit command and system call to set resource limits on an arbitrary process (given appropriate permissions):
$ prlimit --core=unlimited: --pid $$
$ prlimit --core --pid $$
RESOURCE DESCRIPTION SOFT HARD UNITS
CORE max core file size unlimited unlimited blocks
You need util-linux-2.21 for the prlimit command, but you should be able to throw together a quick program to invoke the prlimit system call otherwise:
int prlimit(pid_t pid, int resource, const struct rlimit *new_limit, struct rlimit *old_limit);
If you don't have a new enough version of Linux (or another OS) the only fix I'm aware of is to connect to the process with gdb and issue setrlimit from the debugger:
$ gdb -p $PID
...
(gdb) set $rlim = &{0ll, 0ll}
(gdb) print getrlimit(9, $rlim)
$1 = 0
(gdb) print *$rlim
$2 = {-1, -1}
(gdb) set *$rlim[0] = 1024*1024
(gdb) print setrlimit(9, $rlim)
$3 = 0
This is for setting ulimit -m, RLIMIT_AS = 9; exactly the same applies for ulimit -c (RLIMIT_CORE, numeric value 4 on Linux on x86-64). For "unlimited", use RLIM_INFINITY, usually -1. You should check in /usr/include/bits/types.h what the size of rlim_t is; I'm assuming long long (it's actually unsigned, but using a signed type makes "unlimited" -1 easier to read).
