Smartening punctuation

[DiapDealer]

Originally Posted by eschwartz

@DiapDealer,
But the whole point of my suggestion is using the diff feature to tell what changed, so there is no need to proofread.

Would reviewing all the differences in a dumbly punctuated book after smartening be that much different from proofing the whole book?

I still recommend small, complex chunks be smartened and reviewed--individually--in order to understand exactly what to expect from calibre's smartening routines before tackling whole books.

[eschwartz]

Originally Posted by DiapDealer

Would reviewing all the differences in a dumbly punctuated book after smartening be that much different from proofing the whole book?

Hmm, proof the whole book or just proof the (punctuation) differences. Which would I rather do...

At least the difference skips the step of wading through and tracking down all the punctuation before checking if it should be changed.

[DiapDealer]

Originally Posted by eschwartz

Hmm, proof the whole book or just proof the (punctuation) differences. Which would I rather do...

At least the difference skips the step of wading through and tracking down all the punctuation before checking if it should be changed.

Hmmm, now take the next logical step: would you rather face the task of proofing "just" the punctuation differences of whole books (quite possibly many times)? Or proof the results of a few carefully crafted test paragraphs that will teach you exactly what you can expect from the algorithm and be done with punctuation proofing altogether?

Roger wants to learn what can be expected from the smartening algorithm. I believe studying small test cases (that he crafts specifically for this purpose) is going to help him achieve that goal more easily (and systematically) than eyeballing entire books' worth of random punctuation differences will. But you're free to disagree.

[mrmikel]

It depends on whether a native speaker wrote the algorithm, I think.

[roger64]

Originally Posted by DiapDealer

Roger wants to learn what can be expected from the smartening algorithm. I believe studying small test cases (that he crafts specifically for this purpose) is going to help him achieve that goal more easily (and systematically) than eyeballing entire books' worth of random punctuation differences will. But you're free to disagree.

I followed the first course of action (comparing a whole book) which was disappointing. The syntax was poor and the collect was the same.
- straight apostrophes were changed to curly ones
- three points to horizontal ellipsis
- emdash changed to endash
The last one is not really convenient for my use. For dialogues we can use the former or the latter, but I prefer using emdash.

And that's it. So, it appears either to be a very basic tool or a very poor book.

I will now follow DiapDealer advice to get forcefully more information.

[DiapDealer]

That last one doesn't sound right. The algorithm should leave existing emdash (or endash) characters alone. It should only alter double (and triple) regular dash/hyphen instances. Under no circumstances have I witnessed the smartening algorithm change existing emdash characters (u2014) to endash characters (u2013).

Is there some confusion about what constitutes an em|en dash?

[DoctorOhh]

Originally Posted by DiapDealer

That last one doesn't sound right. The algorithm should leave existing emdash (or endash) characters alone. It should only alter double (and triple) regular dash/hyphen instances. Under no circumstances have I witnessed the smartening algorithm change existing emdash characters (u2014) to endash characters (u2013).

Is there some confusion about what constitutes an em|en dash?

This is how I understand the situation too. Read for further info.

[roger64]

Hi

No confusion. One of the good things wiith the calibre editor is that we can read the name of the sign on the lower right. And we can see the difference too. All my emdashes have consistently been replaced by endashes.

However, all my tirets de dialogue (this is when I only use an emdash), according to French "arcane" rules, are followed by a no-break space (here an utf-8 sign): maybe this induced SP to take this unusual behaviour?

I'll try to replicate with a shorter text and I'll post a test EPUB.

— 2014
– 2013

Edit: I can't reproduce. I try again...

[roger64]

Apologies for my mistake.

Yesterday I really got ENdashes (there are still here today) but as the corrected book was not mine, the only explanation I can find today is that it had previously EN-dashes (which is slightly unusual).

[JLius]

I tried it on a Dutch book, and every ' becomes a ‘.
This is fine for the conversations, but stuff like 't becomes ‘t. So smarten is quite useless for me.

[Toxaris]

@Jlius, sent me a PM. I might have something to help.

Sent from my Nexus 5 using Forum Fiend v1.2.5.

[DiapDealer]

Originally Posted by JLius

I tried it on a Dutch book, and every ' becomes a ‘.
This is fine for the conversations, but stuff like 't becomes ‘t. So smarten is quite useless for me.

I would hope that at least a few of them (') became closing single-quotes, rather than all opening ones.

[roger64]

A little trial, to make myself forgiven. Don't pay any attention to the meaning. There is none. The joint EPUB is uncorrected. You are supposed to smarten its pants.

This is what I got from this trial:

- modified

nnbsp &#_8239; entity was converted to its utf-8 counterpart (invisible \_u202F)
three points to ellipsis as said before
--- and -- were both converted to EN dash which is a little surprising

- unmodified

EM and EN dashes were maintained as such
& ditto
the trailing spaces (after AR and ligne) were left untouched
the forward space (before Centrage) ditto
the PARAGRAH SEPARATOR (before nombre) was left untouched
the untouched section sign stopped the Compare algorithm (unrelated to SP)

[DiapDealer]

Originally Posted by roger64

nnbsp &#_8239; entity was converted to its utf-8 counterpart (invisible \_u202F)

This surprises me a bit. I experience the same thing with your test file, to be sure, but that's not part of the "smartening" algorithm that calibre uses. Seems like something from "Beautify" might be leaking over.

three points to ellipsis as said before

Yep, no surprise there.

--- and -- were both converted to EN dash which is a little surprising

That's not what I'm experiencing at all (using your test epub) -- is converted to EM dash, --- is converted to EN dash. Consistently.

[roger64]

Well, I'll really need to buy magnifying glasses.

What about the "section" sign?