Bloomberg Businessweek fail

[darrenma]

calibre, version 6.24.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from Bloomberg Businessweek

Spoiler:

Fetch news from Bloomberg Businessweek
Conversion options changed from defaults:
verbose: 2
output_profile: 'kindle_oasis'
Resolved conversion options
calibre version: 6.24.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000002396C6CBF40>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleOasisOutput object at 0x000002396C6CA230>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:bloomberg-business-week
Trying to get latest version of recipe: bloomberg-business-week
Traceback (most recent call last):
File "runpy.py", line 196, in _run_module_as_main
File "runpy.py", line 86, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_inpu t.py ", line 138, in convert
File "calibre\web\feeds\news.py", line 1068, in download
File "calibre\web\feeds\news.py", line 1237, in build_index
File "<string>", line 80, in parse_index
File "calibre\web\feeds\news.py", line 718, in index_to_soup
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 307: s2s_high_score

[unkn0wn]

https://github.com/kovidgoyal/calibr...966c20472087c3

Maybe they are on to us!

simultaneous downloads don't work anymore, so, it will take longer to fetch content. If you still face s2s_high_score error, use vpn or new ip, or else we can increase the delay further.

[mkgtu]

Still not working. Have tried recipe with VPN, three different locations, IP's.

Get same error:
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 307: s2s_high_score

Could maybe increase delay? I don't see any "delay" in current recipe

[mkgtu]

In the latest recipe on github the original 7 second delay had been removed (#). The simultaneous downloads "3" had also been removed (#) and changed to "1".

You mention possibly "increasing the delay", but since there was no longer any delay in the recipe to increase, in calibre I "customized" the recipe by adding back "Delay 7" seconds right above the "Simultaneous downloads=1"

So lines 74 and 75 read:

delay = 7 # seconds
simultaneous_downloads = 1

With that change the recipe then downloaded successfully, though, with the 7 second delay, quite slowly (14mins, 38secs)

I should also note that I left the VPN "on".

I'll try it again at some future time with no VPN and maybe shortening the delay a bit.

[mkgtu]

Have tried the 7 sec, single simultaneous download changes I reference above several more times. Have changed IP with VPN each time and/or used my own IP. Have changed delay to 2, 3, and back to 7.

It only worked once, and failed every other time with the same error, regardless of the delay or IP.
Maybe I'll try again tomorrow. But it looks like Bloomberg guard dogs are pretty vigilant.

[unkn0wn]

download from my ip doesnt work anymore even after 2 weeks, website opens through browser but in calibre there's s2s error, I test with vpn.

to increase delay you have to increase values in time.sleep in pre_process, so remove the lower values in

pause = random.choice((0.25, 0.5, 0.75, 1))

and replace them with higher values like 0.5, 1, 1.5, 2 etc.

but the sleep time already ads up to 10 or more seconds for each article, sometimes it even takes a minute with the present values. And it takes like half an hour to fetch the whole thing, increasing delay further might be unnecessary.

if you still want to, you can just remove the lower value 0.25 and add 1.25 to the random.choice.

The idea was that with random delay instead of fixed delay, we could evade detection. (and this method worked for sometime with simultaneous downloads too).

I tested the present recipe with vpn and its still working without issues.

[unkn0wn]

Okay I tried bloomberg recipe which has a lot of articles and takes like 30 minutes and it fetched okay.

For Businessweek we can maybe increase the delay in time.sleep(3) part to 5 or something. don't touch the random.choice. which might be unnecessary.

[unkn0wn]

https://github.com/kovidgoyal/calibr...ca06ec9a57cb46

[Pistol@1204]

calibre, version 6.26.0 (win32, embedded-python: True)
ValueError: No articles found, aborting
Fetch news from Bloomberg Businessweek

Spoiler:

Conversion options changed from defaults:
output_profile: 'kindle_voyage'
verbose: 2
Resolved conversion options
calibre version: 6.26.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001877CB86800>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleVoyageOutput object at 0x000001877CB86E60>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1000
Using user agent: common_words/based
Downloading https://www.bloomberg.com/magazine/businessweek/23_39
Traceback (most recent call last):
File "runpy.py", line 196, in _run_module_as_main
File "runpy.py", line 86, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 138, in convert
File "calibre\web\feeds\news.py", line 1068, in download
File "calibre\web\feeds\news.py", line 1248, in build_index
ValueError: No articles found, aborting

[unkn0wn]

https://github.com/kovidgoyal/calibr...94c7b93fc7a6a7

[Pistol@1204]

This recipe worked ! Thanks a ton !