Looking at the top of your second image it looks to me that it is quite probably CoreTemp that is the problem. While is only using some 20MB of working memory it has a severe virtual memory leak and is using up 24GB of virtual memory.
As a result it has probably absorbed all of the VM available to your system and so the memory manager is struggling to give any memory to new applications.
First task: kill CoreTemp.
The "modified" memory is usually memory pages that are waiting to be flushed to the swap file, this would be consistent with a program that has allocated an insane amount of virtual memory and the system is stuck paging it out.
To explain the commit charge we resort to Wikipedia:
In computing, commit charge is a term used in Microsoft Windows operating systems to describe the total amount of pageable virtual address space for which no backing store is assigned other than the pagefile. On systems with a pagefile, it may be thought of as the maximum potential pagefile usage. On systems with no pagefile, it is still counted, but all such virtual address space must remain in physical memory (RAM) at all times.
Essentially the maximum commit charge of a system is the total amount of memory that can be backed by the swap file. The commit charge of any process is the amount of memory then that program has requested access to.
The problem there though is that a program can request more memory than is physically installed in the system, or more than it actually needs, and essentially starve out other processes. I suspect that this is what has happened in this case. CoreTemp has requested every last piece of allocatable memory that your system has and so the memory manager is having difficulty forcing pages out to disk so that it can load new tasks in.
It's probably also having trouble allocating pages at all, at that kind of size I'd suspect that that is all of your physical and virtual (swapfile) memory allocated.
