calibre 6 beta: Full text searching

[ownedbycats]

Originally Posted by kovidgoyal

Once you turn on indexing for a library, any future books added to it will be automatically indexed in the background.

As a fanfiction reader: does 'future books' include updates to existing ones? (I'll test it myself once I get the initial indexing + some fics to update...)

EDIT: I also came across a failure in connected_device_name(). It appears to be a template issue so I posted it on the bugtracker for assigning.

[kovidgoyal]

yes, as long as you make the update through calibre and not through windows explorer or some other non-calibre mechanism.

[ownedbycats]

The updates are done with FanFicFare, which should count as 'through Calibre.'

[theengine]

Hi,

When adding an epub by dragging and dropping on a Mac, I get the following warning:

calibre, version 5.99.10
ERROR: Cannot add books: Failed to add some books, click "Show details" for more information.

Worker process crashed while executing job
Traceback (most recent call last):
File "calibre/utils/ipc/pool.py", line 108, in recv
File "calibre/utils/ipc/__init__.py", line 24, in eintr_retry_call
File "multiprocessing/connection.py", line 221, in recv_bytes
File "multiprocessing/connection.py", line 419, in _recv_bytes
File "multiprocessing/connection.py", line 388, in _recv
EOFError

It adds the same book in mobi format ok. The epub adds ok in 5.44.

EDIT: by disabling a plugin whose name can't be mentioned here, all worked well!


I also noted in the beta that in Edit Mode, the preset background and foreground colours in the Preview Pane did not honour the setting already in 5.44. They had to be reentered. No big issue, but I t thought it worth reporting.

Many thanks for the wonderful program and all your work Govid.

[kovidgoyal]

Originally Posted by theengine

I also noted in the beta that in Edit Mode, the preset background and foreground colours in the Preview Pane did not honour the setting already in 5.44. They had to be reentered. No big issue, but I t thought it worth reporting.

Many thanks for the wonderful program and all your work Govid.

You're welcome! and do you mean the book details panel on the right of the calibre window by preview pane?

[kovidgoyal]

Originally Posted by ownedbycats

The updates are done with FanFicFare, which should count as 'through Calibre.'

assuming fanficfare uses the calibre apis, which I see no reason to doubt it does, then yes.

[theengine]

Originally Posted by kovidgoyal

You're welcome! and do you mean the book details panel on the right of the calibre window by preview pane?

No, I mean Edit Book - File Preview.

[kovidgoyal]

Originally Posted by theengine

No, I mean Edit Book - File Preview.

I tried changing those settings to custom color under calibre 5 then switching to calibre 6 the custom colors remained unaffected.

[theengine]

Originally Posted by kovidgoyal

I tried changing those settings to custom color under calibre 5 then switching to calibre 6 the custom colors remained unaffected.

I seem to remember that I set the colour a different way in the previous Calibre version a long time ago, perhaps even before the way you used existed. I changed something deep in setting, with the help of folks on this forum.

But the main thing is all is well now.

[ownedbycats]

Left the indexing process running overnight on 'fast'. Surprisingly, it hadn't finished by the time I got up, and my system was choking up a bit. I looked in the task manager and found over a hundred 'pdftohtml' processes taking anywhere from 50-150 mb of ram each and pushing my computer into the swap memory (which probably stalled up the actual indexing...).

I suspect this wasn't supposed to happen, though I'm not sure how much control Calibre has over the actual indexing process.

I guess it reached the part of my library with all the game guides...

[thiago.eec]

Hi, Kovid.

Although not related to 'Full text searching', I got a problem with this new version.
Up until now, I was testing the beta versions with a portable installation. Now, I've decided to install the 5.99.10 over my main 5.44 version.

My last beta version was 5.99.7, which I used to test my plugins for calibre 6 compatibility check. Everything was working fine, including the new themes mechanism. When I changed to the full blown installation, the automatic theme changing stopped working for my plugins (Skoob Sync and Check Books).

To be sure, I tested the 5.99.10 portable and it works fine (just like the 5.99.7 portable).

Using the normal installation makes these plugin's icons to disappear when I change from Dark to Light theme (on Windows). I'm using calibre style. Also, changing to System default makes the icons disappear too.

I'm using Windows 10 x64 and I have no entries on ../resources/images/ for these plugins, so no override there.

Maybe I made a mistake on the code for the new theme mechanism, but this does not explain why would it work on the portable version.

[ownedbycats]

Also, is it safe to clear out Calibre's temp dir after indexing? Thanks to some Internet Archive layered PDFs (the indexer apparently extracted every layer as a separate image), it's bloated to over 60 GB.

I wonder if that's what resulted in the 100+ pdftohtml processes.

[kovidgoyal]

@ownedbycats: are you sure there were 100+ pdftohtml processes as opposed to threads? I dont see how the indexer would launch so many. In fast mode, IIRC it uses num of cpus + 1 worker processes. That said pdf text extraction is not optimized currently, I need to find the time to use a dedicated tool like pdftotext for it which should be much more efficient. And yes you can always delete the temp dir after shutting down calibre (although it should be auto deleted).

@thiago: I dont see why the portable and installed version would behave differently. There is no code difference in them as far as handling icons is concerned. Try cleaning out your calibre config folder temporarily and seeing if that changes things.

[ownedbycats]

They appeared as separate processes in the Task Manager's Details view.

I have a six-core CPU, though interestingly I noticed there were 13 calibre-parallels running.

Also, alongside the image extraction, it's copying every book in my library to the temporary folder. I'm a bit glad I took my temp folder off the system SSD.

[kovidgoyal]

It has to since it runs in the background it cant work on files in the calibre library folder, since they could be moved or deleted at any time.