I know that I can use lastlog to read logs that are already in memory, but if I want to read older stuff that won't work. And on windows the log files are locked and I can't read them unless I close IRSSI. Any ideas?
Irssi's logging isn't really made with Windows' limitations in mind. You can make irssi close the logs by setting logging off and on again. If you're using autologging, you can do this without missing any message in between like this:
I have no Windows running here to test it on, but that should make irssi close the logfiles until it has to write a line to it again. So it gives you a short time to open them in some viewer. You should use a viewer that doesn't keep the files open (otherwise irssi won't be able to log to them anymore), but I expect that should be most text file editors.