[GUI Plugin] Action Chains

[Terisa de morgan]

Originally Posted by ownedbycats

I was thinking something about the amount of chains, or some issue with the JSON.

I don't know if you can validate the json (with some script or at some web like json lint or something like that).

[capink]

Originally Posted by ownedbycats

Is anyone else getting crashes from using the 'move rows up/down' buttons too many times in a row?

Yes. This is a bug with the new test version. I will release a fix for this soon.

[capink]

Originally Posted by ownedbycats

It happens with the current test version, yes. I didn't have any existing modifiers to test with, I was trying to create a new one but opening the settings triggered a freeze.

Very strange behaviour: I created an empty library that copied my existing structure, including saved searches. Everything worked here, creating a new modifier and then opening the settings again. When switching back to my main library, it also worked there.

But then when restarting Calibre, I get freezes until I switch to my test library and back.

Weird, but at least it provides a workaround.

Can you please test with the official version of the plugin and see whether the same behavior persists.

[ownedbycats]

I tested with the official version, then with the test version after you asked me to.

[capink]

OK, since I cannot replicate this, nor figure out the reason for it, we have one of two options:

• I keep doing some random tests, that are not guaranteed to fix the problem, and keep posting them here with you testing to see whether they have any effect.
• If you are comfortable with it, dm a metadata.db file of a library that have this problem. You can copy it to separate folder, open it with calibre, and delete all the book records before sending it.

It is your call.

[ownedbycats]

I pm'd you a link. I'm inclined to think it might be just something about the size of my database that's doing it, especially since I wasn't able to replicate on an empty library with the same structure.

[capink]

Originally Posted by ownedbycats

I pm'd you a link. I'm inclined to think it might be just something about the size of my database that's doing it, especially since I wasn't able to replicate on an empty library with the same structure.

If it happens only with the test version and not with the official version, the size of your library is not the culprit here. It might be a factor, but something else is going on here.

Will check tomorrow if I can spare some time.

[capink]

OK, I've tested with your library, and here what happened:

• At first I was not able to replicate the problem.
• I upgraded my calibre version from 7.7.0 to 7.12.0. After the upgrade I tested again and was able to replicate the issue on the new version. To pinpoint what is the earliest calibre version that causes this issue, I tested multiple versions. It turns out the issue starts with calibre version 7.8.0
• The problem is unrelated to test version of the plugin. It happens with previous versions of the plugin as well.
• The issue happens because Modifier Selection validates all saved searches before opening the dialog. It works perfectly on calibre version older than 7.8.0, because the plugin makes sure to add restriction of one book when validating the saved search.
• There is modification in calibre starting in version 7.8.0 that seemingly makes the one book restriction not work anymore.
• As a result, the dialog takes a long time to open on recent calibre versions until it completes validating all saved searches. But if you give it enough time, it will ultimately open.
• I can fix this at my end by only validating the saved search that the user will ultimately choose from the drop menu. This will not completely solve the issue, but will significantly mitigate the issue on most libraries. But I've seen users whose libraries have hundreds of thousands of books. Validating even one complicated search without being able to set the restriction might take some time.
• I also will have to turn off the validate as you type for regular searches, because they are also affected by this.
• I tried to pinpoint the commit in calibre base that made this change to no avail.

I will post a test version shortly.

[capink]

Here is a new test version

• Workaround for the issue reported by @ownedbycats regarding Selection Modifier.
• Fix issue in previous test version when moving chains up and down.
• Chains Dialog: If you press escape while in filter mode, the filter be removed and the dialog will scroll to the currently selected row. This allows the user to filter, select one of the filtered rows, exit the filter and apply some of the actions disabled in filter mode (e.g. copy chain).

[ownedbycats]

Yes, that fixes it! What a strange issue.

Out of curiosity, what was the cause of the crashing issue?

[chaley]

@capink: I think the slowdown is caused by commit "Template searches should never be cached.". This changes would cause a template search to be re-executed wherever it occurs in the list of saved searches. Before, the cached result would be used even if it was wrong. On the other hand, it would take millions of template searches for this change to cause the delays you are seeing. It is hard to believe that the saved search tree would cause this, but I suppose it is possible if there are thousands of saved searches and all of them run some template search(es) multiple times.

As for running the search on a single book, this should work, although I didn't test it in the actual plugin:

def is_search_valid(db, search_text):
    # search in a vl of only one book for performance
    res = True
    try:
        book_ids = set((db.all_ids()[0],))
    except:
        book_ids = None
    try:
        db.new_api.search(search_text, book_ids=book_ids)
    except ParseException:
        res = False
    return res

It will be faster than using the restriction because it avoids running "id=N" on every book in the library.

I don't see any reason why the old code wouldn't limit the query to the single book. The restriction was evaluated before the search, returning a set of candidates (1 book). The saved search query should be run on that one book. Of course, as noted above the "id=N" query is run on every book.

[ownedbycats]

@chaley - here's the saved_searches from the preferences table. I mentioned that the glitch didn't occur on an empty library copying structure (including the searches).

"{
  ""Books Purchased"": ""#purchasesource:true\nAND NOT #cost:\""=$0.00\"""",
  ""Books Purchased.Freebies"": ""#purchasesource:true\nAND  #cost:\""=$0.00\"""",
  ""Books Purchased.Purchased 2013"": ""#purchasesource:true \nAND date:=2013"",
  ""Books Purchased.Purchased 2014"": ""#purchasesource:true \nAND date:=2014"",
  ""Books Purchased.Purchased 2015"": ""#purchasesource:true \nAND date:=2015"",
  ""Books Purchased.Purchased 2016"": ""#purchasesource:true \nAND date:=2016"",
  ""Books Purchased.Purchased 2017"": ""#purchasesource:true \nAND date:=2017"",
  ""Books Purchased.Purchased 2018"": ""#purchasesource:true \nAND date:=2018"",
  ""Books Purchased.Purchased 2019"": ""#purchasesource:true \nAND date:=2019"",
  ""Books Purchased.Purchased 2020"": ""#purchasesource:true \nAND date:=2020"",
  ""Books Purchased.Purchased 2021"": ""#purchasesource:true \nAND date:=2021"",
  ""Books Purchased.Purchased 2022"": ""#purchasesource:true \nAND date:=2022"",
  ""Books Purchased.Purchased 2023"": ""#purchasesource:true \nAND date:=2023"",
  ""Books Purchased.Purchased 2024"": ""#purchasesource:true \nAND date:=2024"",
  ""Books Read.Read Multiple Times"": ""#timesread:>1"",
  ""Books Read.Read in 2020"": ""#datesread:\""^2020\""\nAND #readstatus:\""=Read\""\nAND NOT #fanficstatus:\""~(In-Progress|Dormant)\"""",
  ""Books Read.Read in 2021"": ""#datesread:\""^2021\""\nAND #readstatus:\""=Read\""\nAND NOT #fanficstatus:\""~(In-Progress|Dormant)\"""",
  ""Books Read.Read in 2022"": ""#datesread:\""^2022\""\nAND #readstatus:\""=Read\""\nAND NOT #fanficstatus:\""~(In-Progress|Dormant)\"""",
  ""Books Read.Read in 2023"": ""#datesread:\""^2023\""\nAND #readstatus:\""=Read\""\nAND NOT #fanficstatus:\""~(In-Progress|Dormant)\"""",
  ""Books Read.Read in 2024"": ""#datesread:\""^2024\""\nAND #readstatus:\""=Read\""\nAND NOT #fanficstatus:\""~(In-Progress|Dormant)\"""",
  ""Books Read.Reading Goal Multiple Years"": ""template:\""\""\""program: \n\nlist = list_re($#admintags, ',', 'readinggoal:', '');\n\nlist_count(list, ',')#@#:n:>1\""\""\"""",
  ""Books Read.Recently Finished Reading"": ""#readstatus:\""=Read\""\nAND (#lastread:>14daysago)\nAND NOT #fanficstatus:\""~(In-Progress|Dormant)\"""",
  ""Cleanup"": ""tags:\""=[Cleanup]\""\nOR search:\""=Cleanup.Missing Identifiers\"" \nOR search:\""=Cleanup.Missing Metadata\"" \nOR search:\""=Cleanup.Other Issues\"""",
  ""Cleanup.Missing Identifiers"": ""search:\""=Cleanup.Missing Identifiers.Missing amazon: identifier\""\nOR search:\""=Cleanup.Missing Identifiers.Missing ao3: identifier\""\nOR search:\""=Cleanup.Missing Identifiers.Missing ffnet: identifier\""\nOR search:\""=Cleanup.Missing Identifiers.Missing kobo: identifier\""\nOR search:\""=Cleanup.Missing Identifiers.Missing odid: identifier\""\nOR search:\""=Cleanup.Missing Identifiers.Missing standard: identifier\"""",
  ""Cleanup.Missing Identifiers.Missing amazon: identifier"": ""#purchasesource:\""=Amazon\""  \nAND NOT identifiers:\""=amazon:\"""",
  ""Cleanup.Missing Identifiers.Missing ao3: identifier"": ""identifiers:\""=url:\""\nAND NOT #fanficstatus:\""=Anthology\""\nAND publisher:\""=Archive of Our Own\"" \nAND NOT identifiers:\""=ao3:\"""",
  ""Cleanup.Missing Identifiers.Missing ffnet: identifier"": ""identifiers:\""=url:\""\nAND NOT #fanficstatus:\""=Anthology\""\nAND publisher:\""=FanFiction.net\"" \nAND NOT identifiers:\""=ffnet:\"""",
  ""Cleanup.Missing Identifiers.Missing kobo: identifier"": ""#purchasesource:\""=Kobo\""  \nAND NOT (#admintags:\""=removedfromsite:kobo\"" OR identifiers:\""=kobo:\"")"",
  ""Cleanup.Missing Identifiers.Missing odid: identifier"": ""formats:\""=OVERDRIVE\"" \nAND NOT identifiers:\""=odid:\"""",
  ""Cleanup.Missing Identifiers.Missing standard: identifier"": ""publisher:\""=Standard Ebooks\"" \nAND NOT identifiers:\""=standard:\"""",
  ""Cleanup.Missing Metadata"": ""search:\""=Cleanup.Missing Metadata.Missing booktype\""\nOR search:\""=Cleanup.Missing Metadata.Missing counts\""\nOR search:\""=Cleanup.Missing Metadata.Missing languages\""\nOR search:\""=Cleanup.Missing Metadata.Missing purchase cost\""\nOR search:\""=Cleanup.Missing Metadata.Missing fanfic metadata\"""",
  ""Cleanup.Missing Metadata.Missing booktype"": ""#booktype:false"",
  ""Cleanup.Missing Metadata.Missing counts"": ""#pagecount:false\nOR (#chaptercount:false AND NOT formats:paperbook)\nOR (#wordcount:false AND formats:~epub|azw3|mobi)"",
  ""Cleanup.Missing Metadata.Missing fanfic metadata"": ""#booktype:\""=Fanfiction\"" AND \n(#fanficrating:false)"",
  ""Cleanup.Missing Metadata.Missing languages"": ""languages:false"",
  ""Cleanup.Missing Metadata.Missing purchase cost"": ""#purchasesource:true \nAND #purchasecost:false"",
  ""Cleanup.Missing Metadata.No author link"": ""template:\""\""\""program: link =author_links('|',','); if link then 'true' else 'false' fi#@#:t:false\""\""\"""",
  ""Cleanup.Other Issues"": ""search:\""=Cleanup.Other Issues.Missing format\""\nOR search:\""=Cleanup.Other Issues.Dormant Fanfics\""\nOR search:\""=Cleanup.Other Issues.Errored Fanfics\""\nOR search:\""=Cleanup.Other Issues.Unwanted crossovers\""\nOR search:\""=Cleanup.Other Issues.Unwanted identifiers\"""",
  ""Cleanup.Other Issues.Dormant fanfics"": ""#fanficstatus:\""=In-Progress\"" \nAND #fanficupdated:<=365daysago"",
  ""Cleanup.Other Issues.Errored fanfics"": ""(#fanficerror:true \nAND #fanficstatus:~In-Progress|Dormant)\nOR\n(#fanficerror:\""Story does not exist\""\nAND NOT #fanficerror:\""=Story on Reject URLs list.\""\nAND NOT #admintags:\""removedfromsite:\"")"",
  ""Cleanup.Other Issues.Missing format"": ""formats:false"",
  ""Cleanup.Other Issues.Unwanted crossovers"": ""#fanficcat:#=3\nAND\n((#fanficcat:\""=Crossover\"" and #fanficcat:\""=Mass Effect Trilogy\"" and #fanficcat:\""=Mass Effect: Andromeda\"")\nOR (#fanficcat:\""=Crossover\"" and #fanficcat:\""=Half-Life\"" and #fanficcat:\""=Portal\""))"",
  ""Cleanup.Other Issues.Unwanted identifiers"": ""identifiers:\""=mobi-asin:\""\nOR (identifiers:\""=barnesnobke:\"" AND NOT #purchasesource:\""=Barnes & Noble\"" )\nOR (identifiers:\""=kobo:\"" AND NOT #purchasesource:\""=Kobo\"" )\nOR ((#fanficstatus:\""=Completed\"" OR #fanficstatus:\""=Oneshot\"") AND identifiers:\""=url:\"" AND (identifiers:\""=ao3:\"" OR identifiers:\""=ffnet:\""))"",
  ""Cleanup.Other Issues.Wrong ISBN type"": ""identifiers:\""=isbn:\"" AND not identifiers:isbn:~^.{13}$"",
  ""Documentation & Manuals"": ""#booktype:\""=Documentations & Manuals\"""",
  ""Fanfiction"": ""#booktype:\""=Fanfiction\"""",
  ""Fanfiction.Has unread chapters"": ""template:\""program: if fanfic_unreadchaps() >=#1 then 'True' fi#@#:t:True\"" \nAND  NOT #readstatus:\""=Did Not Finish\"""",
  ""Fanfiction.Incomplete Fics"": ""#fanficstatus:\""~(Abandoned|Dormant|In-Progress)\"""",
  ""Fanfiction.Recently Updated"": ""#fanficupdated:>14daysago \nAND NOT #fanficstatus:\""=Oneshot\"""",
  ""Fanfiction.Waiting for update"": ""template:\""program: fanfic_unreadchaps()#@#:n:0\""\nAND  NOT #readstatus:\""=Did Not Finish\"""",
  ""Has annotations"": ""template:\""program: \n\tannotation_count()#@#:n:>=1\"""",
  ""Has eBook"": ""NOT formats:\""~(overdrive|paperbook)\""\nOR NOT formats:#=1"",
  ""Has extra files"": ""template:\""\""\""program: has_extra_files()#@#:b:yes\""\""\"""",
  ""Has physical book"": ""formats:paperbook"",
  ""Has series"": ""series:true"",
  ""Has series notes"": ""template:\""\""\""program: get_note('series', $series, '')#@#:b:yes\""\""\"""",
  ""Is library book"": ""formats:#=1 AND formats:overdrive"",
  ""Is physical book"": ""formats:#=1 AND formats:paperbook"",
  ""On Device"": ""ondevice:true OR #kobostatus:true"",
  ""Partially read series"": ""template:\""\""\""python:\ndef evaluate(book, context):\n\tdb = context.db.new_api\n\tanswer = context.globals.get('answer', None)\n\tif answer is None:\n\t\tanswer = set()\n\t\tall_series = db.all_field_names('series')\n\t\tfor series in all_series:\n\t\t\tseries_id = db.get_item_id('series', series)\n\t\t\tbooks_in_series = db.books_for_field('series', series_id)\n\t\t\tvals = db.all_field_for('#readstatus', books_in_series)\n\t\t\tunique_vals = {v for v in vals.values()}\n\t\t\tif len(unique_vals) > 1:\n\t\t\t\tanswer.add(series)\n\t\tcontext.globals['answer'] = answer\n\treturn 'Yes' if book.series in answer else ''#@#:t:Yes\""\""\"""",
  ""Recently Added"": ""date:>14daysago"",
  ""Recently Modified"": ""#lastmodified:>14daysago"",
  ""Recently Read"": ""#lastread:>14daysago""
}"

fanfic_unreadchaps():

program:
input = $#kobobookmark;
	if 
		$#booktype == 'Fanfiction'
		&& substr(input, 0, 10) == 'OEBPS/file'
		&& !$#fanficstatus in 'Anthology,Oneshot'
	then
		currentchap = re(input, '.*\/file(\d+).*', '\1') - 1;
		totalchap = $#chaptercount;

		totalchap - currentchap
	fi

[capink]

Originally Posted by ownedbycats

Out of curiosity, what was the cause of the crashing issue?

Not notifying table model properly before moving rows, which related to the addition of a new proxyfiltermodel.

[capink]

Originally Posted by chaley

@capink: I think the slowdown is caused by commit "Template searches should never be cached.". This changes would cause a template search to be re-executed wherever it occurs in the list of saved searches. Before, the cached result would be used even if it was wrong. On the other hand, it would take millions of template searches for this change to cause the delays you are seeing. It is hard to believe that the saved search tree would cause this, but I suppose it is possible if there are thousands of saved searches and all of them run some template search(es) multiple times.

As for running the search on a single book, this should work, although I didn't test it in the actual plugin:

def is_search_valid(db, search_text):
    # search in a vl of only one book for performance
    res = True
    try:
        book_ids = set((db.all_ids()[0],))
    except:
        book_ids = None
    try:
        db.new_api.search(search_text, book_ids=book_ids)
    except ParseException:
        res = False
    return res

It will be faster than using the restriction because it avoids running "id=N" on every book in the library.

I don't see any reason why the old code wouldn't limit the query to the single book. The restriction was evaluated before the search, returning a set of candidates (1 book). The saved search query should be run on that one book. Of course, as noted above the "id=N" query is run on every book.

Thanks chaley. It works better and solves the problem. I will include it in the final release version.