This is a GREAT question and I find the responses fascinating. I am going to comment on this as an Oracle DBA and my answers are SPECIFIC to the oracle database. This is a major mistake alot of people make when working with Oracle. I am not sure if this applies to other applications as well. This is not meant to be off topic, but is meant as a specialized answer.
When you tune performance with oracle, you are really looking to clear up bottlenecks. Though most of us don't say it, it is based on the Theory of Constraints: http://en.wikipedia.org/wiki/Theory_of_constraints
Memory may not be your bottleneck. Oracle has complex mechanisms for managing memory and just increasing memory can actually slow things down if other areas are where the bottleneck is. Let me give you one example that is VERY common.
Queries seem to be slow. The consensus is if we increase RAM, we should increase the response time of queries since memory is faster than disc. Well... This is how Oracle handles memory manage for data. Oracle has a variety of memory locations that are allocated to specific duties. So you can increase these memories. The area used for data is called the 'buffer cache'. This is a series of linked lists(the number of them tend to increase with each version). Everytime a block is found on disc during a query a hash algorithm is run on it to determine which list to stick it in. Where to put it in the list is based on a touch count algorithm (explain on the oracle support site, so you have to pay to get it... its not real important).
HOWEVER, when you run a query oracle takes out a latch on the buffer chain you search at the time. This LATCH (note this is not a lock. google latch if you dont know the different) blocks all other operations on that chain for the duration of your read. So it blocks reads AND writes (this is entirely different than oracle claiming locks dont block reads).
This is necessary because as you read the block in the chain, oracle moves it around based on how often it is 'requested'. More frequently requested blocks are moved to the top and less frequently requested blocks are left at the bottom and aged out. You cannot have 2 sessions reading a linked list and moving blocks around or you will hit pointers that point to non-existent locations.
When you increase the size of memory, you increase the size of each linked list. This increasing the time it takes to read the list. A single poor query or complex query can do 10s of thousand or even millions of reads down linked lists. Each read is fast, but the number of them leads to latches taken and these will block other sessions. Oracle calls this a 'logical IO'(or buffer get or some other stuff. This lingo is specific to oracle and may mean something else in other parts of IT).
So if the list is longer and you have really bad SQL than the SQL statements will hold their latches longer. Increasing memory can occassionally REDUCE performance. Most of the time this won't happen. People will spend alot of money and see no benefit. That being said there are times when you need more memory in the buffer cache, but you have to properly identify the bottleneck to know whether this is appropriate. I can't discuss how to analyze this in this post. See the DBA forums. Some people discuss it there. It is rather complex.
Does anyone have specific examples with other pieces of software where this can happen? There is a terrific business book called 'The Goal' that discusses alleviating constraints in a factory. This process is very similiar to what Oracle DBAs do when assessing performance issues. It is often standard reading in MBA programs. It is very valuable to read for IT professions.