First of all, you want to use ls -1 (number 1, not lowercase L) if you want to use grep on it's output.
But the main problem is, that you list the contents of a subdirectory, but the rm command is executed in the current directory.
For example see this directory tree (. and .. entries omitted)
a/
a/x
a/y
a/z
Let's say you want to remove all from directory a except file y and use the command as you tried:
$ ls -1 a | grep -v 'y' | xargs rm -rf
ls -1 a gives:
x
y
z
grep filters out the line with y:
x
z
so xargs excutes the following command:
rm x z
But it executes the command in the top-level directory and sees only directory a, not the files x and z.
In order to make it work, you need the path of the files - easily achieved by applying a minor change to the ls command. Also we refine the grep pattern a bit:
ls -1d a/* | grep -v '/y$' | xargs rm -rf
So this time ls produces the following output:
a/x
a/y
a/z
and the final command executed would be rm -rf a/x a/z
So now let's go back to your command line and apply the discussed changes:
$ ls -1d domain.com/wp-content/themes/* | grep -v -E '/twentyeleven$|/index.php$' | xargs rm -rf
One last note: Using xargs rm -rf is always venturesome (although sometimes neccessary) - be sure to check the output of the grep command first!