How does colormatrix work and what does "similar to Adobe Flash" mean? I'm looking for a plain mathematical definition. How are the input vector(s), matrix and output vector composed? How is the matrix applied?
Plain mathematical definition is it uses matrix multiplication. Here is an explanation with some examples: http://msdn.microsoft.com/enus/library/windows/desktop/ms533875(v=vs.85).aspx RGBA uses 5x5 matrix and CMYKA uses 6x6, but the idea is the same.
? is ignored. Here is the live matrix to experiment with: http://www.adobetutorialz.com/articles/1987/1/ColorMatrix 

