I would suggest you should be worried about such a large file size (for example search this site for ‘large Word file’). It would be a lot of work to lose if it gets corrupted! (the likelihood of which seems greater for large files than for a similar total volume of small files). Remember also that while editing the file size may be significantly larger than after a full save (because of undo history). Also, repaginating very many pages can take a long time (during which further edits may not be accepted).
Occasionally some books have no chapters but very large documents are almost always broken up into ‘sections’ (Preface, Chapters, Appendices etc) and that applies as much to hard copy as to e-format – so presumably for a good reason (that they are simply more convenient that way).
Once more or less ‘finished’ I personally might reassemble the sections into a single file but I’d still keep the components and use only the components while doing any heavy editing. And I would aim for about 5MB max per section (though that would be after compression should that make much difference), in part because I often need to e-mail what I edit.
Per kb211489 (applies to Word2007 & Word 2010): The maximum file size is limited to 32 MB for the total document text only and does not include graphics, regardless of how the graphics image is inserted (Link to file, Save with document, or Wrapping style) into the document. Therefore, if the file contains graphics, the maximum file size can be larger than 32 MB.
The maximum that can be opened (ibid) is 512MB so in theory you have some headroom but as already over 32MB I'd say you are on dangerous ground. Also, I suspect if broken up into about half a dozen pieces your other problem (speed - or lack of it!) would be ameliorated.