Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

(I have no knowledge in the field of NLP, so if the following question is inappropriate, I’ll delete it).

I remember skimming the sentence segmentation section from the NLTK site a long time ago.

I use a crude text replacement of “period” “space” with “period” “manual line break” to achieve sentence segmentation, instead of something like the Punkt tokenizer of NLTK.

I segment to help me more easily locate and reread sentences, which can sometimes help with reading comprehension.

What about independent clause boundary disambiguation, and independent clause segmentation? Are there any tools that attempt to do this?

Below is some example text. If an independent clause can be identified within a sentence, there’s a split. Starting from the end of a sentence, it moves left, and greedily splits:


Sentence boundary disambiguation (SBD), also known as sentence breaking, is the problem in natural language processing of deciding where

sentences begin and end.

Often, natural language processing tools

require their input to be divided into sentences for a number of reasons.

However, sentence boundary identification is challenging because punctuation

marks are often ambiguous.

For example, a period may

denote an abbreviation, decimal point, an ellipsis, or an email address - not the end of a sentence.

About 47% of the periods in the Wall Street Journal corpus

denote abbreviations.[1]

As well, question marks and exclamation marks may

appear in embedded quotations, emoticons, computer code, and slang.

Another approach is to automatically

learn a set of rules from a set of documents where the sentence

breaks are pre-marked.

Languages like Japanese and Chinese

have unambiguous sentence-ending markers.

The standard 'vanilla' approach to

locate the end of a sentence:

(a) If

it's a period,

it ends a sentence.

(b) If the preceding

token is on my hand-compiled list of abbreviations, then

it doesn't end a sentence.

(c) If the next

token is capitalized, then

it ends a sentence.


strategy gets about 95% of sentences correct.[2]

Solutions have been based on a maximum entropy model.[3]

The SATZ architecture uses a neural network to

disambiguate sentence boundaries and achieves 98.5% accuracy.

(I’m not sure if I split it properly.)

If there are no means to segment independent clauses, are there any search terms that I can use to further explore this topic?


share|improve this question

closed as off-topic by and31415, Tog, Breakthrough, random May 28 at 4:03

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question is not about computer hardware or software, within the scope defined in the help center." – and31415, Breakthrough, random
If this question can be reworded to fit the rules in the help center, please edit the question.

This does not appear to be about computer hardware or software. Perhaps English Language and Usage would be a better site. –  jww May 24 at 21:45
"I use a crude text replacement [...] instead of something like the Punkt tokenizer of NLTK." Uh, why not? Unless you're doing natural language parsing of the text, how would you expect to be able to detect such a thing if you're just looking at ASCII characters? I've found different classifiers using the NPS Chat Corups training set to be very effective for a similar application. –  Breakthrough May 27 at 22:28
I should be using it. I’m just getting into natural language processing now. Text replacement was easy to apply, and it was suitable enough for my sentence segmentation requirements. I’ll read about the NPS Chat corpus. Thanks. –  Jeff Kang May 28 at 20:04

2 Answers 2

Via user YourWelcomeOrMine from the subreddit /r/LanguageTechnology/:

“I would check out Stanford's CoreNLP. I believe you can customize how a sentence is broken up.”

share|improve this answer
up vote 0 down vote accepted

Via Chthonic Project from Stackoverflow:

To the best of my knowledge, there is no readily available tool to solve this exact problem. Usually, NLP systems do not get into the problem of identifying different types of sentences and clauses as defined by English grammar. There is one paper published in EMNLP which provides an algorithm which uses the SBAR tag in parse trees to identify independent and dependent clauses in a sentence.

You should find section 3 of this paper useful. It talks about English language syntax in some details, but I don't think the entire paper is relevant to your question.

Note that they have used the Berkeley parser (demo available here), but you can obviously any other constituency parsing tool (e.g. the Stanford parser demo available here).

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.