Skip to content

Prowlarr parallel requests and rank torrent name fix #362

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lucamaia9 wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Prowlarr parallel requests and rank torrent name fix #362

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9dac1ca
Update prowlarr.py to include timeout from the settings
lucamaia9 Oct 2, 2025
7f2b65a
Update prowlarr.py
lucamaia9 Oct 3, 2025
7fd7b8f
Update torrent.py
lucamaia9 Oct 3, 2025
ac4c095
Merge pull request #1 from g0ldyy/main
lucamaia9 Oct 10, 2025
0018ca1
Update pyproject.toml
lucamaia9 Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 39 additions & 8 deletions comet/scrapers/prowlarr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,16 +77,42 @@ async def process_torrent(
return torrents


async def fetch_prowlarr_results(
session: aiohttp.ClientSession, indexer_id: int, query: str
):
"""
Fetches results from a single Prowlarr indexer with a configured timeout.
"""
try:
search_url = f"{settings.INDEXER_MANAGER_URL}/api/v1/search?query={query}&indexerIds={indexer_id}&type=search"
response = await session.get(
search_url,
headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY},
# This is the key change, mirroring the Jackett implementation
timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT),
)
response.raise_for_status()
return await response.json()
except asyncio.TimeoutError:
logger.warning(f"Prowlarr search timed out for indexer ID {indexer_id}")
return []
except aiohttp.ClientError as e:
logger.warning(f"Prowlarr search failed for indexer ID {indexer_id}: {e}")
return []


async def get_prowlarr(manager, session: aiohttp.ClientSession, title: str, seen: set):
torrents = []
try:
# Get all configured indexer IDs from Prowlarr
indexers = [indexer.lower() for indexer in settings.INDEXER_MANAGER_INDEXERS]

get_indexers = await session.get(
get_indexers_response = await session.get(
f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer",
headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY},
timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT),
)
get_indexers = await get_indexers.json()
get_indexers = await get_indexers_response.json()
Comment on lines +110 to +115
Copy link

@coderabbitai coderabbitai bot Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add status check before parsing JSON response.

The code calls json() without verifying the HTTP response status. If the indexer list fetch fails with a non-2xx status, this could raise an exception.

Apply this diff to check the status:

 get_indexers_response = await session.get(
     f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer",
     headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY},
     timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT),
 )
+get_indexers_response.raise_for_status()
 get_indexers = await get_indexers_response.json()
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
get_indexers_response = await session.get(
f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer",
headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY},
timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT),
)
get_indexers = await get_indexers.json()
get_indexers = await get_indexers_response.json()
get_indexers_response = await session.get(
f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer",
headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY},
timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT),
)
get_indexers_response.raise_for_status()
get_indexers = await get_indexers_response.json()
🤖 Prompt for AI Agents 
In comet/scrapers/prowlarr.py around lines 110 to 115, the code calls
get_indexers_response.json() without checking the HTTP status; add a status
check on get_indexers_response.status and handle non-2xx responses before
parsing JSON (e.g., raise or log an error and return/raise an exception) so you
only call .json() when the response is successful; ensure you include the status
and response text/body in the error handling for debugging and use the same
timeout and headers already present.


indexers_id = []
for indexer in get_indexers:
Expand All @@ -96,12 +122,17 @@ async def get_prowlarr(manager, session: aiohttp.ClientSession, title: str, seen
):
indexers_id.append(indexer["id"])

response = await session.get(
f"{settings.INDEXER_MANAGER_URL}/api/v1/search?query={title}&indexerIds={'&indexerIds='.join(str(indexer_id) for indexer_id in indexers_id)}&type=search",
headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY},
)
response = await response.json()
# Create a search task for each indexer ID
tasks = [
fetch_prowlarr_results(session, indexer_id, title)
for indexer_id in indexers_id
]
all_results_lists = await asyncio.gather(*tasks)

# Flatten the list of lists into a single list of results
response = [result for sublist in all_results_lists for result in sublist]

Comment on lines +126 to 134
Copy link

@coderabbitai coderabbitai bot Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add return_exceptions=True to asyncio.gather for graceful degradation.

Without return_exceptions=True, an unexpected exception from any indexer task (not caught by fetch_prowlarr_results) will cancel all other tasks and lose partial results, defeating the purpose of parallel requests with timeout tolerance.

Apply this diff:

 tasks = [
     fetch_prowlarr_results(session, indexer_id, title)
     for indexer_id in indexers_id
 ]
-all_results_lists = await asyncio.gather(*tasks)
+all_results_lists = await asyncio.gather(*tasks, return_exceptions=True)

 # Flatten the list of lists into a single list of results
-response = [result for sublist in all_results_lists for result in sublist]
+response = [
+    result 
+    for sublist in all_results_lists 
+    if not isinstance(sublist, Exception)
+    for result in sublist
+]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
tasks = [
fetch_prowlarr_results(session, indexer_id, title)
for indexer_id in indexers_id
]
all_results_lists = await asyncio.gather(*tasks)
# Flatten the list of lists into a single list of results
response = [result for sublist in all_results_lists for result in sublist]
tasks = [
fetch_prowlarr_results(session, indexer_id, title)
for indexer_id in indexers_id
]
all_results_lists = await asyncio.gather(*tasks, return_exceptions=True)
# Flatten the list of lists into a single list of results
response = [
result
for sublist in all_results_lists
if not isinstance(sublist, Exception)
for result in sublist
]
🤖 Prompt for AI Agents 
In comet/scrapers/prowlarr.py around lines 126 to 134, asyncio.gather is called
without return_exceptions=True which will cancel all tasks if any task raises;
change the gather call to asyncio.gather(*tasks, return_exceptions=True) and
then post-process all_results_lists to filter out exceptions (e.g., iterate
results, if isinstance(item, Exception) log or ignore it, otherwise treat as the
expected list), finally flatten only the successful result lists into response
so partial results are preserved.

# Process the combined results
torrent_tasks = []
for result in response:
if result["infoUrl"] in seen:
Expand Down
4 changes: 4 additions & 0 deletions comet/utils/torrent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,10 @@ async def download_torrent(session: aiohttp.ClientSession, url: str):
).decode("utf-8")
return (None, info_hash, location)
return (None, None, None)
except asyncio.TimeoutError:
# Log the specific message for a timeout
logger.warning(f"Timeout while trying to download torrent from {url}")
return (None, None, None)
except Exception as e:
logger.warning(
f"Failed to download torrent from {url}: {e} (in most cases, you can ignore this error)"
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ dependencies = [
"orjson",
"pydantic-settings",
"python-multipart",
"rank-torrent-name",
"rank-torrent-name==1.9.0",
"uvicorn",
]