I have a project involving around 4000 flat text files which represent legal documents. I need to parse them out by named section for a text analysis project I'm starting. The structure of these documents is somewhat standard, but parsing them out into standardized sections will involve some trial and error, some python scripting, and a lot of brute force "look at the file and edit it" type work.
My questions is the following. I'm not a very advanced programmer with respect to text analysis. I do most of my work on structured data (you know, numbers and stuff) using statistical languages. What is the simplest way to set these files up so that I can pull individual sections, and categorize them in the future?
Should I mark them up with some kind of XML like structure? If so, what's the right toolset? Should I put the parts in a database, or is that overkill?