Zsh on its own starts up in around 0.1 second for me, which is plenty good enough. I just noticed as I got near the 50,000 command history mark that it became more like 3 seconds to load up the first prompt.
I dunno how you guys are finding all these other reasons for slow startup, but mine was exactly what my first guess was. I did a
mv ~/.zsh_history zsh_history_backup and bam, 3 second startup is now 0.1 second startup. Curiously,
/usr/bin/time /bin/zsh -i -c exit fails to capture the time it takes to load in the history.
If you don't have tens of thousands of commands in your zsh history, though, then this isn't it. My
~/.zsh_history measured 1.8MB. Its very possible to accidentally paste a large chunk of stuff as a command into a terminal, this will also bloat history up right quick (though this is something very much to avoid doing as it can obviously be extremely destructive).