Depends on the processor and, by inference, what is needed to represent the task state. Also depends, to a degree, on the OS.
In the old original (pre-virtual-memory) Unix the registers would be saved into a fixed location in memory, then the entire user memory written to disk and a new user memory image read in. (Unix "fork" was accomplished by simply skipping the "read it in" step.) This was very quickly supplanted by a virtual swap scheme when CPUs with TLBs became available ("Berkley Unix").
In a Burroughs-style stack architecture all that need be swapped (in theory) is the stack pointer and the task ID. Memory addressing (in the original) is via "capabilities" and "segments", vs using a TLB.
Older register architectures with TLB-based virtual memory required that the TLBs (and sometimes cache) at least be invalidated on swap, in addition to swapping the program registers (including IAR, condition code, etc). Newer TLB-based architectures finesse this issue various ways, avoiding flushing so that a fairly quick switch back does not need to reload everything. (For this reason, on multiprocessor system, tasks are often given an "affinity" of a given processor, to minimize the amount of TLB/cache reloading.)