This is complicated issue due to the fact that Illustrator performs zoom based on the document, not comparisons to the real world. Zooming to 100% will always be an approximation, as will Fit to Screen. Since the resolution of the screen can vary on each computer as well as the size of the screen and the size of the document, Adobe makes no secret that actual size on the a screen will never be exact.
Your best solution is to perform periodic printouts. If you are concerning yourself with inches and actual size then that tells me that you are working on something that will end up in print (otherwise you should be working in pixels). If your final medium is print, then perform periodic printouts and see how it will look. Your screen can't reproduce the resolution of a printer, so print be your litmus test for if it looks good.
If you are still interested in having it match the real world actual size, spend a few minutes finding the exact zoom that matches up. IE: grab a ruler and make a box 5" wide and then change the zoom until the box is 5" wide. Write the zoom down and either stay at that zoom or just go back to it when you need to compare.
A little trick: To avoid spending all day finding the zoom, put the cursor in the zoom percentage box (it's in the bottom left corner of the screen on the mac, not sure about Windows) and hit the up/down buttons, it will change the screen and allow you to quickly find the right size.