[Metadata Source Plugin] Douban PY3 updated

[bgme]

This plugin can fetch metadata from 豆瓣读书.
This plugin is removed from core source on this commit.


Version 3.2.0 - 8 Oct 2020
Fix: Changes for Python 3 support in calibre.

[likekindle]

Hi, this plugin is down, maybe douban blocked it, do you have plan to fix it?

[kyalpha]

Originally Posted by likekindle

Hi, this plugin is down, maybe douban blocked it, do you have plan to fix it?

Error Message

calibre, version 5.14.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': '龙眠', 'authors': ['宫部美雪'], 'identifiers': {'mobi-asin': '8a4ea100-c28b-4d9a-9ef8-f2b3a7cb253c'}, 'timeout': 30}
Using plugins: Douban Books (3, 2, 0)
The log from individual plugins is below

****************************** Douban Books (3, 2, 0) ******************************
Found 0 results
Downloading from Douban Books took 0.13665342330932617
Failed to make identify query: 'https://api.douban.com/v2/book/search?count=10&q=%E9%BE%99%E7%9C%A0+%E5%AE%AB%E9% 83%A8%E7%BE%8E%E9%9B%AA&apikey=054022eaeae0b00e0fc 068c0c0a2102a'
Traceback (most recent call last):
File "calibre_plugins.dummy1.__init__", line 334, in identify
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 403: Forbidden

************************************************** ******************************
The identify phase took 0.20 seconds
The longest time (0.136653) was taken by: Douban Books
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds

[likekindle]

Originally Posted by kyalpha

Error Message

calibre, version 5.14.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': '龙眠', 'authors': ['宫部美雪'], 'identifiers': {'mobi-asin': '8a4ea100-c28b-4d9a-9ef8-f2b3a7cb253c'}, 'timeout': 30}
Using plugins: Douban Books (3, 2, 0)
The log from individual plugins is below

****************************** Douban Books (3, 2, 0) ******************************
Found 0 results
Downloading from Douban Books took 0.13665342330932617
Failed to make identify query: 'https://api.douban.com/v2/book/search?count=10&q=%E9%BE%99%E7%9C%A0+%E5%AE%AB%E9% 83%A8%E7%BE%8E%E9%9B%AA&apikey=054022eaeae0b00e0fc 068c0c0a2102a'
Traceback (most recent call last):
File "calibre_plugins.dummy1.__init__", line 334, in identify
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 403: Forbidden

************************************************** ******************************
The identify phase took 0.20 seconds
The longest time (0.136653) was taken by: Douban Books
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds

Hi, how do you get this log message?

[davidfor]

Originally Posted by likekindle

Hi, how do you get this log message?

When the results of the metadata download are displayed for you to choose the book, there is a button for viewing the log. Pressing that will show what @kyalpha has posted.

But, based on the error displayed, "HTTP Error 403: Forbidden" and visiting the search URL in the message, the API key used by the plugin is no longer valid. That could be because it has expired or because the use broke the usage rules. Someone needs to renew it.

[likekindle]

Originally Posted by davidfor

When the results of the metadata download are displayed for you to choose the book, there is a button for viewing the log. Pressing that will show what @kyalpha has posted.

But, based on the error displayed, "HTTP Error 403: Forbidden" and visiting the search URL in the message, the API key used by the plugin is no longer valid. That could be because it has expired or because the use broke the usage rules. Someone needs to renew it.

The message said:
Failed to download the metadata or cover for the selected book.
Unknown (Failed metadata)

By this message, I cannot get details why it failed.
(Sorry i cannot upload the screenshot cause the "Insert image" button is not working on my Chrome/Edge)

[davidfor]

Originally Posted by likekindle

The message said:
Failed to download the metadata or cover for the selected book.
Unknown (Failed metadata)

By this message, I cannot get details why it failed.
(Sorry i cannot upload the screenshot cause the "Insert image" button is not working on my Chrome/Edge)

The "View log" button will be displayed if there are any results. But, if there are no results and error message dialog is displayed, the buttons will be different. The "Details" will show the log in the same way as the "View log" button. And the "Copy to clipboard" will copy those contents to the clipboard and you can paste it into post.

[likekindle]

calibre, version 5.10.1
ERROR: Download failed: Failed to download metadata or cover for the selected book.

未知 (Failed metadata)

this is the message i copied, also no details, maybe the plugin not write the error log.

[likekindle]

hi, i've made a new version of this plugin, but it's not working for now,
when i install it, the error message pop up:

calibre 5.14* [64bit] embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: zh_CN
Traceback (most recent call last):
File "D:\code\calibre-5.14.0\src\calibre\gui2\preferences\plugins.py", line 316, in add_plugin
File "D:\code\calibre-5.14.0\src\calibre\customize\ui.py", line 472, in add_plugin
File "D:\code\calibre-5.14.0\src\calibre\customize\ui.py", line 61, in load_plugin
File "D:\code\calibre-5.14.0\src\calibre\customize\zipplugin.py", line 293, in load
File "importlib\__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "D:\code\calibre-5.14.0\src\calibre\customize\zipplugin.py", line 192, in exec_module
File "calibre_plugins.dummy1.__init__", line 23, in <module>
ModuleNotFoundError: No module named 'pandas'

i used pandas and pymongo in the plugin, but i've installed them using pip, why it still failed to run?

PS. this version is based on Li Fanxi's work, for more details, plz check the code.
PPS. this is the plugin file as attachment, but I'm not sure it's available for you.
https://www.mobileread.com/forums/at...1&d=1617454991

[davidfor]

Originally Posted by likekindle

hi, i've made a new version of this plugin, but it's not working for now,
when i install it, the error message pop up:



i used pandas and pymongo in the plugin, but i've installed them using pip, why it still failed to run?

PS. this version is based on Li Fanxi's work, for more details, plz check the code.
PPS. this is the plugin file as attachment, but I'm not sure it's available for you.
https://www.mobileread.com/forums/at...1&d=1617454991

Calibre includes its own Python installation. It does not use anything from the Python installed in the OS. If you want to use modules not included with calibre, you need to package them yourself.

But, how are you using Pandas and PyMongo? I would have thought Pandas was overkill. But, I haven't touched it enough to see what advantage it would have over other methods. But, you completely baffle me with PyMongo. As it is an interface for MongoDB, I could only see it being useful if Douban is allowing direct access to their database.

[likekindle]

Originally Posted by davidfor

Calibre includes its own Python installation. It does not use anything from the Python installed in the OS. If you want to use modules not included with calibre, you need to package them yourself.

But, how are you using Pandas and PyMongo? I would have thought Pandas was overkill. But, I haven't touched it enough to see what advantage it would have over other methods. But, you completely baffle me with PyMongo. As it is an interface for MongoDB, I could only see it being useful if Douban is allowing direct access to their database.

Yes, you're right, the pandas/pymongo is not used in the plugin(maybe i copied the code from somewhere), now i removed the dependencies, then the plugin is installed(using command "calibre-customize -b ." BTW)
But when i start the app using command "calibre-debug -g", then use the plugin, it's still not working, the error message is like:

calibre, version 5.14.0
ERROR: Download failed: Failed to download metadata or cover for the selected book.

Unknown (Failed metadata)

I've write some log message(using print statement), but they are not showed. i don't know why, and how to debug the plugin?

Thanks a lot for help.

[davidfor]

Looking at the code you posted, you are using print statements rather than writing to the log. The log is a parameter of the identify method. Writing anything to that will appear in the log that calibre displays.

And I cannot install the version you posted after removing the extra imports. The problem is the __init__ does not have the correct parameters. And there is no call to the parent class to initialize it. It isn't really needed. You should set those properties at the class level, or as constants.

[likekindle]

Just as you said, the __init__ is not needed, so i removed it, then use log to write log message.
But it still not working, I'll post the new code, and error message i got, could you see what's wrong in it?

Hello, calibre!
PS D:\douban> calibre-customize -b .
Hello, calibre!
Plugin updated: Douban Books v2 (4, 2, 1)
PS D:\douban> calibre-debug -g
Hello, calibre!
calibre 5.14* [64bit] embedded-python: True is64bit: True
Windows-10-10.0.19041 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19041')
Python 3.8.5
Windows: ('10', '10.0.19041', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Douban Books v2 (4, 2, 1)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 120.0 x 120.0
physicalDpi: 157.8252427184466 x 157.65517241379308
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.47] splash screen shown
[0.47] Initializing db...
[0.47] db initialized
[0.47] Constructing main UI...
[1.62] main UI initialized...
[1.62] Hiding splash screen
Starting QuickView
[1.80] splash screen hidden
[1.80] Started up in 1.80 seconds with 3 books
Metadata sources cache was recently updated not updating again
Job: 0 Download metadata for one book finished
Starting job: Download metadata for one book
Download complete, with 1 failures

[davidfor]

From that, you are using the batch metadata download. Don't. Do your debugging from the metadata editor. That will show the error and the log will show the full details of the attempt to download the metadata and any errors. That is what @kyalpha used and what I was referring to.

Using your code for a random book in my library shows:

calibre, version 5.14.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters: 
{'title': 'Alien Hunters', 'authors': ['Daniel Arenson'], 'identifiers': {'isbn': '9781505658200', 'amazon': '1505658209', 'ff': 'a/daniel-arenson/alien-hunters', 'fictiondb': 'daniel-arenson~alien-hunters~1197738~b', 'goodreads': '24533763', 'kobo': 'alien-hunters'}, 'timeout': 30} 
Using plugins: Douban Books v2 (4, 2, 1) 
The log from individual plugins is below 

****************************** Douban Books v2 (4, 2, 1) ****************************** 
Found 0 results 
Downloading from Douban Books v2 took 4.281980037689209 
start get metadata from douban...
{'isbn': '9781505658200', 'amazon': '1505658209', 'ff': 'a/daniel-arenson/alien-hunters', 'fictiondb': 'daniel-arenson~alien-hunters~1197738~b', 'goodreads': '24533763', 'kobo': 'alien-hunters'}
9781505658200
http://douban.com/isbn/9781505658200
Plugin Douban Books v2 failed
Traceback (most recent call last):
  File "E:\Development\GitHub\calibre\src\calibre\ebooks\metadata\sources\identify.py", line 47, in run
  File "calibre_plugins.dummy2.__init__", line 91, in identify
  File "calibre_plugins.dummy2.__init__", line 292, in get_book
  File "calibre_plugins.dummy2.__init__", line 377, in __get_author
IndexError: list index out of range 

******************************************************************************** 
The identify phase took 4.30 seconds 
The longest time (4.281980) was taken by: Douban Books v2 
Merging results from different sources 
We have 0 merged results, merging took: 0.00 seconds

[likekindle]

Do your debugging from the metadata editor. That will show the error and the log will show the full details of the attempt to download the metadata and any errors.

sorry, i don't know how to do that...

BTW, i've fixed the bugs, now it's working~~~
thanks for all your help. and i'll do some more tests on it, then i could post it here.