Skip to content

Commit

Permalink
tiktokscraper do not share proxy_map
Browse files Browse the repository at this point in the history
this seems to fail particularly on localhost. it seems one processor releases and then immediately reassigns localhost and so any other processors are stuck waiting. if we wanted to limit tiktok requests 4CAT wide, we could try to do that here, but we would need processors to expect to wait (instead of just failing after waiting for 600 seconds) or have some sort of in waiting queue for proxies.
  • Loading branch information
dale-wahl committed Jan 8, 2024
1 parent a56e731 commit 04faf23
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion datasources/tiktok_urls/search_tiktok_urls.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ def map_item(item):


class TikTokScraper:
proxy_map = {}
proxy_map = None
proxy_sleep = 1
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
Expand All @@ -149,6 +149,7 @@ def __init__(self, processor, config):
"""
:param Processor processor: The processor using this function and needing updates
"""
self.proxy_map = {}
self.processor = processor

def update_proxies(self):
Expand Down

0 comments on commit 04faf23

Please sign in to comment.