(I have no knowledge in the field of NLP, so if the following question is inappropriate, I’ll delete it).
I remember skimming the sentence segmentation section from the NLTK site a long time ago.
I use a crude text replacement of “period” “space” with “period” “manual line break” to achieve sentence segmentation, instead of something like the Punkt tokenizer of NLTK.
I segment to help me more easily locate and reread sentences, which can sometimes help with reading comprehension.
What about independent clause boundary disambiguation, and independent clause segmentation? Are there any tools that attempt to do this?
Below is some example text. If an independent clause can be identified within a sentence, there’s a split. Starting from the end of a sentence, it moves left, and greedily splits:
Sentence boundary disambiguation (SBD), also known as sentence breaking, is the problem in natural language processing of deciding where
sentences begin and end.
Often, natural language processing tools
require their input to be divided into sentences for a number of reasons.
However, sentence boundary identification is challenging because punctuation
marks are often ambiguous.
For example, a period may
denote an abbreviation, decimal point, an ellipsis, or an email address - not the end of a sentence.
About 47% of the periods in the Wall Street Journal corpus
As well, question marks and exclamation marks may
appear in embedded quotations, emoticons, computer code, and slang.
Another approach is to automatically
learn a set of rules from a set of documents where the sentence
breaks are pre-marked.
Languages like Japanese and Chinese
have unambiguous sentence-ending markers.
The standard 'vanilla' approach to
locate the end of a sentence:
it's a period,
it ends a sentence.
(b) If the preceding
token is on my hand-compiled list of abbreviations, then
it doesn't end a sentence.
(c) If the next
token is capitalized, then
it ends a sentence.
strategy gets about 95% of sentences correct.
Solutions have been based on a maximum entropy model.
The SATZ architecture uses a neural network to
disambiguate sentence boundaries and achieves 98.5% accuracy.
(I’m not sure if I split it properly.)
If there are no means to segment independent clauses, are there any search terms that I can use to further explore this topic?