It sounds like you have already made some design choices.
we know various objects required,
interactions required between various
objects and IPC communication between
tasks.
you need a multi-threaded OS and a uC that will run it, usually meaning you want to aim for a processor with an MMU. OS options are things like Quadros, QNX, linux, wince, etc.
Based on what your "objects" (modules is better term for C :) ) are doing you can likely determine what type of architecture you need. Is a 16bit arch enough? need more memory or work with larger numbers then is 32bit the right answer? lots of floating point work? then you likely need a processor with a FPU. Doing lots of DSP like work? maybe you need a DSP or a uC with DSP like instructions or a co-proc. Running a graphical display? need a SoC with a built in LCD controller or expect to do it externally. Doing heavy 2D graphics? need a SoC with some graphics acceleration.
Make a list of the features you need and estimate how much of your code falls into the categories like integer operations, looping, floating point ops, graphical operations, DSP operations, etc.
This should allow you to class the level of device you need. For some architectures you could cross compile some code using GCC and emulate it using qemu on top of linux. This probably is only worth it if you need to test the performance of a critical algorithm on a particular architecture. This can help you scale the speed you need for your application.
The second consideration should be power usage and support for power management. Combined with the required performance you can make the choice of DSP, uC, application processor, etc.
As others have said i would not worry about memory usage, just aim big, often different ram sizes are pin compatible so you can just cut the ram down for production. The only real questions to answer up front here are:
*How big of an address space do i need? 16bit? 32bit? etc
*Do i need external ram or will a uC internal ram be enough? <--answer this after you choose an architecture and can go SoC hunting.
For the most part choosing among processors in the same class is a "holy war" aka in the 32bit risc market some will back ARM, some will back coldfire, some may even back PIC32. At the end of the day likely any would work. You have to choose based on available SoCs with the required peripherals, ease of development (how good is the tool chain) and cost.
The same can be true of OS choice, linux vs QNX vs quadros is a toss up for most applications, usually the best answer is the one you have the most experience with. Even if it ends up being slightly more expensive the reduction in development time often offsets the build cost. Do make sure the OS has the required features, shared libraries, inter-process communication, pipes, whatever you need.
As a general rule i would select your architecture first. This will have a much higher impact on the performance of your device than the operating system will. In addition operating systems in this space are often supported on many architectures. Also the better OS's are POSIX compliant, written correctly the bulk of your code should be able to run on multiple OS's.
Don't feel bad if you have to make several attempts to make the correct choice. You may find a core that perfectly fits your code needs but find after some research that it doesn't support some minor feature you need, or there isn't an available SoC with the peripherals you need, or even that the thing is back ordered for 6 months. Just make sure after making the initial choice that you research how the design will come together based on that part so that you see the stumbling block now rather than half way through development.