[Conversion Output] KFX Output

[jhowell]

Originally Posted by Walnut356

If it's not too much trouble, would it be possible to automatically handle "incorrect table structure" via this plugin?

Moving tfoot to the end of a table to prevent this conversion failure will be included in the next release of the plugin. It was a simple change.

[Ruskie_it]

By the way @Jhowell, I can't find where I posted my original request about KFX output plugin in Linux, I just wanted to let you know that I have been able to make it work on Fedora Linux 39.
I could successfully install the Kindle Previewer that Amazon is currently offering for download, 3.74.0, and it runs successfully under wine (only command line, as usual)

[jhowell]

Originally Posted by Ruskie_it

By the way @Jhowell, I can't find where I posted my original request about KFX output plugin in Linux, I just wanted to let you know that I have been able to make it work on Fedora Linux 39.
I could successfully install the Kindle Previewer that Amazon is currently offering for download, 3.74.0, and it runs successfully under wine (only command line, as usual)

That is good news. Do you have any tips to share on getting it to run properly?

[cajhin]

Originally Posted by jhowell

That is good news. Do you have any tips to share on getting it to run properly?

@Ruskie_it : Thanks for the encouragement, I expected a hackfest, but it was actually really smooth and easy.

How to install kfx output on Fedora 39:
1. sudo dnf install wine
2. download the Kindle Previewer for Windows (https://www.amazon.com/Kindle-Previe...de=21381691011)
3. right-click the .exe > Open with Wine
4. click through the installer, keep all defaults
5. in Calibre, install the kfx output plugin (Preferences > Plugins > Get New Plugins > (find kfx output) > Install
6. convert your books...

@jhowell: thanks a LOT for this really cool piece of software!
Rebooting to Win11 is pain, the typography nerd in me says 'no' to AZW, the Telegram "eBook Converter Bot" requires 10 steps.
Without your plugin I would probably have to buy a Libre 2, but I love my Oasis1.

[jhowell]

Thanks for the feedback!

[jackm8]

What exactly happens to book covers .jpg file as you convert epub to kfx?

I believe that they get compressed at one point during the process, and I'd like to know if it's possible to avoid it, so that kfx would use the same exact quality as the one present in .epub format.

I could be wrong, I could also be doing something incorrectly, but here's how I got to this conclusion:

If I don't tick "Use cover from source file" in Calibre, cover resolution will be reduced quite a bit, and differences will be plainly visible. If I tick it, cover resolution will be preserved, but file will still be compressed somewhat. I tried to convert files both in Calibre and via Command Line Interface, and the results were almost identical, but file sizes did decrease in both cases compared to original .epub, making me suspect that cover image was compressed. I then converted that .kfx back to .epub. File size decreased again. When I compared that cover.jpg to my source cover.jpg, compression artefacts were visible under higher magnifications.

Here things got interesting. I then went and used uncompressed .png format as a cover (I had to open .epub and change metadata.opf and titlepage.xhtml). I converted this .epub to .kfx via Command Line Interface. I again did .kfx to .epub to get a viewable file back. It converted cover.png to cover.jpg, but this time jpeg compression quality was much higher compared to previous attempt. It was practically identical to my source file. File sizes even increased.


File sizes:

'original cover present in epub.jpg' = 2.34mb
'cover converted to kfx then back to epub.jpg' = 1.57mb

'cover as png.png' = 2.22.mb (lossless compressed png)
'cover as png' converted to kfx then back to jpg.jpg' = 2,63mb

'original epub' = 2.65mb
'kfx made in calibre with keep source file' = 2.16mb
'kfx made with command line' = 2.15mb
'epub converted from 'kfx made in calibre...'' = 1.80mb

'epub using .png file for cover' = 2.50mb
'kfx made from 'epub using .png as cover'' = 3.13mb
'epub converted from 'kfx made from 'epub using .png as cover'' = 2.78mb

Comparison; right original jpeg, left as converted via kfx output. Dithered image, dithering goes to shambles when jpeg is compressed:

[federoni]

I just want to say, for those who are having errors in Calibre when trying to output any format to Kfx or vice versa. All you need to do is install Kindle previewer on the computer you are using Calibre in! Totally fixed my issue and now exporting KFX files perfectly.

[jhowell]

Originally Posted by jackm8

What exactly happens to book covers .jpg file as you convert epub to kfx?

I believe that they get compressed at one point during the process, and I'd like to know if it's possible to avoid it, so that kfx would use the same exact quality as the one present in .epub format.

There are several factors that influence the handing of cover images during conversion.

Originally Posted by jackm8

If I don't tick "Use cover from source file" in Calibre, cover resolution will be reduced quite a bit, and differences will be plainly visible.

If you are doing conversion in calibre instead of using the KFX Output plugin CLI make sure to select "Use cover from source file" in the metadata tab and set the Output profile in the Page Setup tab to "Tablet" in order to avoid replacement or degradation of images by calibre's conversion.

Originally Posted by jackm8

If I tick it, cover resolution will be preserved, but file will still be compressed somewhat.

The plugin will regenerate a JPEG cover image, potentially reducing the quality slightly, if it is not in JPEG File Interchange Format (JFIF) since that format is required for covers to appear on the lock screen of Kindle devices. That might be what is happening in your case.

Originally Posted by jackm8

I then converted that .kfx back to .epub. File size decreased again. When I compared that cover.jpg to my source cover.jpg, compression artefacts were visible under higher magnifications.

There should be no change in cover images when converting KFX to EPUB as long as the proper calibre conversion settings or the KFX Input plugin CLI is used. Those artifacts are most likely from the conversion of EPUB to KFX.

Originally Posted by jackm8

Here things got interesting. I then went and used uncompressed .png format as a cover (I had to open .epub and change metadata.opf and titlepage.xhtml). I converted this .epub to .kfx via Command Line Interface. I again did .kfx to .epub to get a viewable file back. It converted cover.png to cover.jpg, but this time jpeg compression quality was much higher compared to previous attempt. It was practically identical to my source file. File sizes even increased.

The Kindle Previewer, which does most of the work in converting EPUB to KFX, will convert PNG images to JPEG, often resulting in a larger size than the original image.

Originally Posted by jackm8

Comparison; left original jpeg, right as converted via kfx output. Dithered image, dithering goes to shambles when jpeg is compressed:

Dithered images are a bad idea in e-books. Images often need to be converted or rescaled during rendering, resulting in unpleasant artifacts.


Perhaps there is something else going on with the high resolution cover images that you are using. Feel free to post (public domain) sample files that demonstrate the problem if you want me to look into this further.

[Ruskie_it]

Originally Posted by jhowell

That is good news. Do you have any tips to share on getting it to run properly?

Well, not really. As cajhin implicitly suggested: do not overcomplicate it. At first I tried I should use winetricks and I somehow tried to create a "Windows environment" (it probably isn't the right wording, but I am not very familiar with wine yet), but it really isn't needed.
Second time around I installed wine from Discover package manager (I use KDE Fedora) and then installed the Kindle Previewer by command line with "wine KindlePreviewerInstaller.exe"
I let it run and voilà!
Next time I ran Calibre (that also, installed with the command line provided in the website, and with a library imported from Windows that also imported plugins), KFX output worked like a charm.

[Ruskie_it]

Originally Posted by cajhin

@Ruskie_it : Thanks for the encouragement, I expected a hackfest, but it was actually really smooth and easy.

How to install kfx output on Fedora 39:
1. sudo dnf install wine
2. download the Kindle Previewer for Windows (https://www.amazon.com/Kindle-Previe...de=21381691011)
3. right-click the .exe > Open with Wine
4. click through the installer, keep all defaults
5. in Calibre, install the kfx output plugin (Preferences > Plugins > Get New Plugins > (find kfx output) > Install
6. convert your books...

@jhowell: thanks a LOT for this really cool piece of software!
Rebooting to Win11 is pain, the typography nerd in me says 'no' to AZW, the Telegram "eBook Converter Bot" requires 10 steps.
Without your plugin I would probably have to buy a Libre 2, but I love my Oasis1.

Yeah, actually when I tried "too hard" it was more difficult (and it didn't work). Keeping it simple was actually better!
And I agree on the typography nerd thing.
Expecially because I hate azw3 that are not hypenated automatically; I had to run them through Hypenate this!, which works fine but does not provide the same quality

[jackm8]

Originally Posted by jhowell

Dithered images are a bad idea in e-books. Images often need to be converted or rescaled during rendering, resulting in unpleasant artifacts.

Perhaps there is something else going on with the high resolution cover images that you are using. Feel free to post (public domain) sample files that demonstrate the problem if you want me to look into this further.

I'll go and do some proper kfx samples later. I guess that it's best to ignore converting kfx back to epub for this comparison, and just look at kfx on e-ink device for results visible in the end. But this will involve talking photos and a bit of work.

Regarding dithering. You're right that generally dithering images is not ideal, but it is when you do dithering in exactly the same resolution as target device. Then dithering provides best possible results, at least in my experience. The problem with it is that it is very sensitive to jpeg compression, as jpeg compression basically does it's own compression that is similar to dithering, not overwriting dithered results, but adding to it, ultimately causing bad quality.
When image gets saved into jpeg to compress file sizes, some of the data is lost no matter the quality selected when saving it. Jpeg is lossy format, after all. Saving dithered files in .png format is really ideal way, but it's not supported by many readers, so saving it with highest possible .jpeg quality is an alternative. I always save them at max quality in image editors. Even going down to 95% will cause visible jpeg artefacts, whats even worse, it'll change number of colours in the file. It'll do that even when saving at max quality, but quality will still be high enough not to spot differences.

Exaggerated example on what happens when images are saved into .jpeg:

Cover source: https://archive.org/details/vilebodies0000evel_p2d8

Cover is low resolution jpeg. Opened in Photoshop. Converted into black and white. Quickly adjusted with curves for more contrast. Then dithered with ximagic grey dithering plugin (https://www.ximagic.com/) 5 levels, clustered dot 16x16 (exagerated settings). When saved as PNG this file has 5 colours, and only 135kb. When saves as jpeg in photoshop with maximum quality, resulting jpeg has 22 colours and 858kb size, but differences are still hard to spot. When saved in photoshop with '8 quality', I get 241 colours and 349kb. This time differences are easily observed, and are detrimental to the look.

I used "FastStone Image Viewer" to compare the files using it's compare function. It's important that smoothing is turned off when zooming in on images. I used it to count colours after conversions. It can also be used as a free alternative to compress posted .png file to get similar results I got using Photoshop.

Files:
"Original cover.jpg": Cover from internet archive saved as jpeg. Source is highly compressed jpeg as well.
"PNG Vile Bodies.png": Dithered image generated from "Original cover.jpg" in Photoshop via ximagic plugin.
"sample 1.png": Comparisons at 100% magnification, saved without resizing. Histograms enabled, showing that jpegs contain more colour than png. (it's in jpeg format now, so not as valid for comparison, best is saving "png vile bodies.png" as png and converting that into jpeg to get verifiable results)
"sample 2.png": Comparisons at 500% magnification in png format.

[edit]
Sample 1 really isn't quite accurate, it got downscaled by the site into .jpeg, probably for file size reasons.

[jhowell]

Originally Posted by jackm8

I'll go and do some proper kfx samples later. I guess that it's best to ignore converting kfx back to epub for this comparison, and just look at kfx on e-ink device for results visible in the end. But this will involve talking photos and a bit of work.

My interest is seeing that the cover image makes it into the KFX file without a loss of quality. What happens after that in rendering the image on a Kindle device is not under my control so I do not see any point in posting Kindle screen images.

What I meant by asking for an example is any case where the original cover image does not make it into the KFX file unchanged. The easiest way to check for that is by converting from KFX back to EPUB using the command line interface (CLI) of the KFX input plugin. Using that avoids potential image manipulation by calibre itself. If a case can be found I would need to be able to duplicate it myself using your source file to determine the cause.

I previously mentioned one known issue. If the original cover image is not in JPEG File Interchange Format (JFIF) then it will be regenerated by the plugin with a possible loss of quality. I will look into improving that.

In addition calibre's conversion and metadata updating system can impact cover images. I recommend using the KFX Output plugin CLI to avoid those possibilities.

There may still be other cases that I am not aware of. The bulk of the KFX conversion process is performed using Amazon's Kindle Previewer and that might potentially do further image manipulation that only comes into play under certain circumstances.

Originally Posted by jackm8

Regarding dithering. You're right that generally dithering images is not ideal, but it is when you do dithering in exactly the same resolution as target device. Then dithering provides best possible results, at least in my experience. The problem with it is that it is very sensitive to jpeg compression, as jpeg compression basically does it's own compression that is similar to dithering, not overwriting dithered results, but adding to it, ultimately causing bad quality.

I agree that tailoring images to a specific device can make them fragile if any unexpected manipulations occur. I will try to improve what I can but some things may not be under my control.

[jackm8]

Originally Posted by jhowell

My interest is seeing that the cover image makes it into the KFX file without a loss of quality. What happens after that in rendering the image on a Kindle device is not under my control so I do not see any point in posting Kindle screen images.

What I meant by asking for an example is any case where the original cover image does not make it into the KFX file unchanged. The easiest way to check for that is by converting from KFX back to EPUB using the command line interface (CLI) of the KFX input plugin. Using that avoids potential image manipulation by calibre itself. If a case can be found I would need to be able to duplicate it myself using your source file to determine the cause.

I previously mentioned one known issue. If the original cover image is not in JPEG File Interchange Format (JFIF) then it will be regenerated by the plugin with a possible loss of quality. I will look into improving that.

In addition calibre's conversion and metadata updating system can impact cover images. I recommend using the KFX Output plugin CLI to avoid those possibilities.

There may still be other cases that I am not aware of. The bulk of the KFX conversion process is performed using Amazon's Kindle Previewer and that might potentially do further image manipulation that only comes into play under certain circumstances.

Ah, I see. I haven't even considered that. I did most of my own conversions to kfx using command line, but I converted them back to epub using gui in Calibre (laziness). I'll go and do it both ways using CLI asap.


Test 1:

Ebook that includes plenty of low resolution illustrations:
Three Men in a Boat: EPUB3 version: "pg308-images-3.epub" https://www.gutenberg.org/ebooks/308
I converted this epub into kfx, then back into epub using CLI:

Spoiler:

calibre-debug -r "KFX Output" -- "pg308-images-3.epub" "pg308-images-3.kfx"
calibre-debug -r "KFX Input" -- -e "pg308-images-3.kfx" "pg308-images-3 back to kfx.epub"

I opened both epub files, and extracted images. Cover images appear to be identical. Same file sizes. Under extreme pixel peeping, I could not spot a single difference. Illustration images are changed slightly. File sizes in "pg308-images-3 back to kfx.epub" can be slightly smaller or larger compared to files in "pg308-images-3.epub". Image quality isn't worse, but it's possible to spot differences in this case.

Test 2:

Small resolution cover: From above "PNG Vile Bodies.png". Opened in FastStone, then saved as "PNG to JPEG Vile Bodies.jpg" jpeg at "Quality 100, GrayScale"

I replaced cover present in "pg308-images-3.epub" with "PNG to JPEG Vile Bodies.jpg". I just manually opened .epub in file explorer, and replaced it. Exactly the same conversion as before. Results are not identical. New file is slightly smaller, Faststone reports that it has 21 colours compared to 13 of the original file, but differences are practically impossible to spot.

Test 3:

High resolution cover. Sample used in attachment to this post. "2col.png". I only used 2 channels while dithering this png, so file size remains low enough for this forum to leave it as png. It was converted to jpeg in Faststone at max quality. File size increases to about 20mb, colours from two to five. That jpeg was then used as a cover in the same .epub file. Same procedure as "Test 2". Results this time are identical. File size of original jpeg and the one that went through conversion is identical. I can't spot any differences, even while pixel peeping. Illustrations are changed.

Test 4:

Same as before, but I used Calibre to convert file back from kfx to epub. Tablet output, epub3, use cover from source file. Identical results as in Test 3.

Notes:
- I believe that incorrect Calibre setting in the test carried in the previous post caused big differences between results, and got me to believe that something off was going on.
- I'm still not exactly sure what causes files to be untouched in certain cases, while adjusted in others. It may be that it's something to do with the way original .jpeg files are saved. "PNG Vile Bodies.png" was saved as jpeg in colour profile, while I was careful and saved 2col.png in black and white profile. Also, as I converted it from .png to .jpg, I used grayscale option and made sure to turn every possible compression setting off. Coincidently, doing this in Photoshop, gives less options regarding this.
- I'll test this a bit more to figure exactly what's going on. As far as quality is concerned, doing everything by the book, appears to leave everything untouched, or at least in so high quality that differences are very hard to spot, but I'm still wondering what exactly happens. Especially, why are covers images untouched and illustrations changed.

More notes:
- So I did a few more tests, and there is quite a bit of difference in files saved in photoshop and ones saved in Faststone. So there really must be something with the way .jpeg files are saved. I used same 2col.png as source. I opened it in Photoshop, changed it from grayscale to rgb, then saved it as jpeg at max possible quality. File size was 15mb. I packed that file into .epub. I again did .epub to .kfx and back to .epub using CLI. And resulting jpeg file was 5mb smaller, and went from 8 colours to 22 colours. I also did the same but kept the file in grayscale. Resulting file was of slightly different size. In both cases it was practically impossible to spot differences compared to original .jpeg that was generated in Photoshop, but it wasn't identical. Going by the Faststone procedure, files were identical. Puzzling.

[jhowell]

Originally Posted by jackm8

Test 1:

Ebook that includes plenty of low resolution illustrations:
Three Men in a Boat: EPUB3 version: "pg308-images-3.epub" https://www.gutenberg.org/ebooks/308
I converted this epub into kfx, then back into epub using CLI:

I opened both epub files, and extracted images. Cover images appear to be identical. Same file sizes. Under extreme pixel peeping, I could not spot a single difference. Illustration images are changed slightly. File sizes in "pg308-images-3 back to kfx.epub" can be slightly smaller or larger compared to files in "pg308-images-3.epub". Image quality isn't worse, but it's possible to spot differences in this case.

I repeated your test and observed basically the same result. When converting to KFX and back to EPUB the cover image (1704462598893215816_cover.jpg) is identical but the first interior image (1704462598893215816_p0s.jpg) is re-encoded, changing the file size and pixel values slightly.

I then changed the original file so that the contents of the first interior image and cover image were swapped. In that case the interior image converts unchanged and the cover is modified. That shows there is something about the content of some images and not others that triggers the re-encoding process.

I opened the original EPUB using the Kindle Previewer, saved the converted KPF file and examined its contents. The images there were identical to the KFX file that this plugin produces, showing that all of the image manipulations in this case are due to the processing by the Previewer and not this plugin.

It is unclear why this image manipulation is being done in some cases and not others. The Kindle Previewer's conversion process is completely undocumented.

[jackm8]

Originally Posted by jhowell

I repeated your test and observed basically the same result. When converting to KFX and back to EPUB the cover image (1704462598893215816_cover.jpg) is identical but the first interior image (1704462598893215816_p0s.jpg) is re-encoded, changing the file size and pixel values slightly.

I then changed the original file so that the contents of the first interior image and cover image were swapped. In that case the interior image converts unchanged and the cover is modified. That shows there is something about the content of some images and not others that triggers the re-encoding process.

I opened the original EPUB using the Kindle Previewer, saved the converted KPF file and examined its contents. The images there were identical to the KFX file that this plugin produces, showing that all of the image manipulations in this case are due to the processing by the Previewer and not this plugin.

It is unclear why this image manipulation is being done in some cases and not others. The Kindle Previewer's conversion process is completely undocumented.

Yeah, I thought there was something on that side. One part of the problem is undoubtedly just how aged jpeg standard it. It really wasn't designed for high resolutions, certain compressions, and even display medium present today. The other is Kindle Previewer. There are workarounds. Saving everything images in using just the right settings, is one of them. But I think that one of the main problems is how everything clashes on the side of Calibre that most of the people use. Tools do everything as well as possible, and when used using CLI, they provide great results even when used with imperfectly formatted images. Problems start when used in Calibre, where there's a lot of code on top of it, with default settings that are set so that heavy compressions happen.

For my end, I've found at least two fool proof methods. One is export images as black and white profile png, and convert them in faststone to jpeg. There is something about Photoshops jpeg export engine that probably clashes with Kindle Previewer later in the process. The other appears to be skipping Adobe software and using Capture One instead.