Why does an en-dash count as a word?

[AdrianSingfield]

I'm posting some drabbles I wrote in nW to AO3 and just noticed that some of them AO3 says are less than 100 words. It took me a while to find the reason: nW counts my en-dashes (–) as words. I'm wondering what is the reason behind this and is there any other punctuation I should be on the lookout for that might skew the word count?

[vkbo]

En and em dashes are replaced with spaces before counting, so no, they are not counted as words. They do count as word separators though. I just double-checked this in the app, but the code that pre-processes the text before the word count is anyway clear:

novelWriter/novelwriter/text/counting.py

Lines 37 to 73 in 60d7bd3

	def preProcessText(text: str, keepHeaders: bool = True) -> list[str]:
	"""Strip formatting codes from the text and split into lines."""
	if not isinstance(text, str):
	return []
	# We need to treat dashes as word separators for counting words.
	# The check+replace approach is much faster than direct replace for
	# large texts, and a bit slower for small texts, but in the latter
	# case it doesn't really matter.
	if nwUnicode.U_ENDASH in text:
	text = text.replace(nwUnicode.U_ENDASH, " ")
	if nwUnicode.U_EMDASH in text:
	text = text.replace(nwUnicode.U_EMDASH, " ")
	ignore = "%@" if keepHeaders else "%@#"
	result = []
	for line in text.splitlines():
	line = line.rstrip()
	if line:
	if line[0] in ignore:
	continue
	if line[0] == ">":
	line = line.lstrip(">").lstrip(" ")
	if line: # Above block can return empty line (Issue #1816)
	if line[-1] == "<":
	line = line.rstrip("<").rstrip(" ")
	if "[" in line:
	# Strip shortcodes and special formatting
	# RegEx is slow, so we do this only when necessary
	line = RX_SC.sub("", line)
	line = RX_SV.sub("", line)
	line = RX_LO.sub("", line)
	result.append(line)
	return result

[vkbo]

I assumed from your profile that reading the code above is ok, but the documentation also explains the process in text form: https://novelwriter.io/docs/more/counting.html

Could you post some example text that gives the wrong word count? Maybe there's something else going on.

Edit: You posted an example the same instant, so I'll check 🙂

[AdrianSingfield]

Oh, okay. But for some reason this bit of interrupted dialogue is 3 words: „Bibi, ich–“

If I put an en-dash in-between two words, the word count doesn't change.

[vkbo]

Ah, that clears it up. It is counting the closing quote as a word in this specific case.

The algorithm splits the text by (consecutive) white spaces and considers en and em dash as a white space in this context. Anything else is counted. If you have a "word" of pure punctuation, that too counts as a word.

For the record, LibreOffice Writer does the same thing. It was used as a reference for the word counter in novelWriter. Here are a few examples:

[vkbo]

During the manuscript build process, a new word count is run on the final text (including all substitutions and such). It also provides a character count that excludes white spaces. It wouldn't be a problem to also add a word count that ignores all punctuation. But this count would require a character-by-character check, which is a little slow, so it shouldn't be run in the editor at least.

[AdrianSingfield]

That is.... not how I would have expected a word to be defined. Sometimes a punctuation mark is a word and sometimes it isn't? Honestly, I don't agree at all that this is correct. "Hello!" being one word, but "Hello–" being two just seems very illogical to me.

Maybe this is a case of "LibreOffice might perhaps not be 100% correct with their word count" or maybe it's more of a "the design follows the implementation so this is 'as designed'". What do you think?

EDIT because I typed too slowly :D
Interesting option with the char-by-char check, but I'm not sure how useful it would be. I'm also not usually too concerned about absolutely correct word counts, since different programs always just handle that differently. I only noticed because I was writing something with a specific word count goal. If no-one else is bothered by this, I don't think that extra functionality is necessary. Having a different word count in the editor vs the build tool would most likely also lead to more confusion :)

[vkbo]

There isn't really a way to get a correct word count without considering each language separately. In French there are spaces between words and some punctuation, which makes those extra words. In English, dashes can be used with and without surrounding spaces (which is why the dashes count as spaces) and in numbers, some languages inject punctuation as thousand separators while others use spaces.

I expect the implementation in LibreOffice Writer is a mix of simple and somewhere in the middle. I followed a similar philosophy. Any generalised word counting approach will be a little off here and there. For instance, if I were to ignore punctuation, should I consider a punctuation a word separator or not? There just isn't a clear cut answer here. Is "2,000.00" one, two or three words? How about the same number formatted in another locale: "2 000,00"?

[vkbo]

EDIT because I typed too slowly :D Interesting option with the char-by-char check, but I'm not sure how useful it would be. [...] If no-one else is bothered by this, I don't think that extra functionality is necessary.

I added the char count without white spaces option because someone asked for it because some publications count it that way. It's a pretty good count to use if you're evaluating the amount of content (and not printing it into newspaper columns).

I'm also not usually too concerned about absolutely correct word counts, since different programs always just handle that differently. I only noticed because I was writing something with a specific word count goal.

Yeah, I've written for print media and magazines before, where they have strict limits, but they recognise that word counts are implementation-specific, so there is usually a tolerance margin.

Having a different word count in the editor vs the build tool would most likely also lead to more confusion :)

That's already the case because the full word count isn't known until the manuscript is generated.