Python SDK 2.4.0 (research background, finance_research, source_control, output_schema)

[tyler5673]

Summary

Release Python SDK 2.4.0. Adds the new Finance Research endpoint, Research background-mode helpers (poll/stream), search_post and retrieve typed methods, structured output via output_schema, source_control domain constraints, and Content API cache freshness. Plus the post-generation hand-edits that close gaps between Speakeasy output and what 2.4.0 needs to ship as a clean, tested release.

What's new in 2.4.0

• Finance Research: you.finance_research(input=..., research_effort=FinanceResearchEffort.DEEP|EXHAUSTIVE) — finance-optimised index (earnings releases, SEC filings, analyst coverage).
• Research background mode: you.research(..., background=True) returns a typed TaskResponse directly. New you.get_research_task(task_id) for polling, you.stream_research_task(task_id) for SSE.
• Research source_control: include_domains, exclude_domains, boost_domains, freshness, country (beta).
• Research output_schema: structured JSON output in response.output.content (beta; lite returns 422).
• Search boost_domains: per-request domain boost that doesn't restrict (POST + GET via you.search.unified).
• Contents max_age: filter cached content freshness.
• New typed errors: per-endpoint *UnprocessableEntityError / *UnauthorizedError / etc.

Hand-maintained code (NOT regenerated by Speakeasy)

These need re-application after future regens until the overlay in overlays/python_overlay.yaml causes Speakeasy to bake them into the generated output:

• src/youdotcom/research_helpers.py — research_background[_async], poll_research_task[_async], research_and_wait[_async], stream_research_events_raw[_async]. Models background-mode research flow that the generated SDK can't model cleanly (typed TaskResponse narrowing + poll/stream helpers + tolerant SSE decoder).
• src/youdotcom/_hooks/registration.py — YDCUserAgentOverrideHook honors sdk_configuration.user_agent overrides so integrations (langchain-youdotcom, youdotcom-temporal, n8n-nodes-youdotcom) can identify their traffic.
• src/youdotcom/utils/security.py — get_security_from_env reads YDC_API_KEY (canonical per you.com/docs) with YOU_API_KEY_AUTH as fallback for 2.3.x users. Covered by tests/test_security_env.py.

Post-generation hand-edits to generated output

All justified in the skill's new Step 4i-9 (and the regeneratable ones will be re-applied after the next regen via the overlay):

• CHANGELOG.md — full 2.4.0 release notes.
• MIGRATION.md — 2.3.0 → 2.4.0 section: env var rename, Union[str, object] content, ResearchEffort enum rename (no rename here, child enums added), enhanced search domain constraints.
• examples/api-example-calls.py — every example fixed to compile + use real model names + return the right shapes (the Speakeasy-generated copy had ResearchResearchEffort typos, wrong isinstance, wrong boost_domains type, and a model_dump() call that prints nothing).
• README.md / USAGE.md — lead search examples in both stopped combining include_domains + exclude_domains + boost_domains (returned 422); now show only exclude_domains + boost_domains which is the only valid pair.
• Generated Content and Result models in src/youdotcom/models/researchresponse.py and taskdetail.py are class Content(BaseModel): pass / class Result(BaseModel): pass, which silently drop structured payload via pydantic extra="ignore". Workarounds documented in CHANGELOG, MIGRATION, helper docstring, examples, and tests. Durable fix staged in overlays/python_overlay.yaml (see below).

Durable fix staged for next regen

overlays/python_overlay.yaml (bumped to 1.1.0) now contains two actions verified against .speakeasy/out.openapi.yaml:

- target: $["components"]["schemas"]["ResearchResponse"]["properties"]["output"]["properties"]["content"]["oneOf"][1]
  update: { additionalProperties: true }
- target: $["components"]["schemas"]["TaskDetail"]["properties"]["result"]
  update: { additionalProperties: true }

Per Speakeasy's additionalProperties semantics, the next speakeasy run should turn these anonymous object schemas into extra="allow" Python models. Once we regen and verify with the check in the skill's Step 4i-9 (grep "extra=\"allow\"" src/youdotcom/models/researchresponse.py src/youdotcom/models/taskdetail.py), the workaround caveats in CHANGELOG / MIGRATION / examples / helper docstring / tests get swept up to the clean story in a follow-up commit.

Skill update

.agents/skills/generate-sdk-and-open-pr/SKILL.md Step 4i grew from 3 to 9 sub-steps to capture every post-generation fix we ended up applying this release:

• 4i-1 env var (YDC_API_KEY + YOU_API_KEY_AUTH fallback)
• 4i-2 verify search/contents URLs aren't accidentally rewritten (canonical ydc-index.io)
• 4i-3 preserve and verify hand-maintained files
• 4i-4 Speakeasy auto-version-bump check
• 4i-5 pyright/pylint fixes in hand-maintained code
• 4i-6 verify live test skip uses YDC_API_KEY
• 4i-7 run full validation suite (pytest + pylint + smoke)
• 4i-8 verify auto-generated Search examples are valid (catches include_domains+exclude+boost regression + RetryConfig positional regression)
• 4i-9 audit empty-type / open-ended model schemas (extra="ignore" data-loss risk)

Future maintainers regen'ing the SDK should walk through 4i-1 through 4i-9 before shipping.

Tests

• 69 unit tests pass (was 56 in 2.3.x). 13 new tests this release: finance_research happy path + 401, research(background=True) TaskResponse shape, get_research_task TaskDetail shape, output_schema content_type flip with caveat lock-in, source_control (include only + boost+exclude pair), helpers — async error paths for poll_research_task_async and stream_research_events_raw_async, from_id reconnection parameter, env-var precedence (primary + fallback + override), and background-mode progress streaming.
• pylint src/youdotcom/ clean (10.00/10).
• Smoke test at ~/Workspace/Temp/youdotcom-sdk-smoketest/ exercises every SDK call path against the local Go mockserver; all 13 checks pass.

Checklist

• Speakeasy generation ran successfully against front-end specs
• Version updated in pyproject.toml, gen.yaml, _version.py (= 2.4.0)
• CHANGELOG.md updated
• MIGRATION.md updated
• README.md and USAGE.md examples hand-fixed for known Speakeasy codegen issues
• Tests cover 2.4.0 additions (69 passed, 13 added)
• overlays/python_overlay.yaml has the additionalProperties: true overlay for the next regen
• Skill has all post-gen steps documented for next maintainer

[claude]

Review summary

Nicely scoped 2.4.0 release: Finance Research API, research background mode + task polling/SSE streaming, source_control/output_schema betas, Search boost_domains, and the YDC_API_KEY env-var rename. The hand-maintained surface (research_helpers.py, the env-var precedence in security.py, the User-Agent hook) is well tested, and the CHANGELOG/MIGRATION docs are unusually thorough — including honest documentation of the known Content()/Result empty-model unmarshal caveat and the overlay fix staged for the next regen. I verified the two overlay JSONPath targets resolve against .speakeasy/out.openapi.yaml, the removed error names are actually gone, ResearchResponse1 = Union[ResearchResponse, TaskResponse], and the migration anchor is valid. No hardcoded secrets.

Findings

[HIGH] 1. Async SSE helper crashes on exit — research_helpers.py:489 (confirmed bug). stream_research_events_raw_async wraps the stream in contextlib.aclosing(stream), whose __aexit__ calls await stream.aclose(). But EventStreamAsync only defines an async close() — there is no aclose(), so a clean exit raises AttributeError: EventStreamAsync object has no attribute aclose. This breaks stream_research_events_raw_async() and the cleanup path of research_and_wait_async(mode="stream"). The sync path is fine (EventStream implements __exit__). Fix suggested inline (try/finally: await stream.close()).

[MEDIUM] 2. Test-coverage gap on the async streaming path — tests/test_research_helpers.py. No test exercises stream_research_events_raw_async or research_and_wait_async(mode="stream") — which is exactly why finding 1 slipped through. Async mirrors of TestStreamResearchEventsTolerant / TestResearchAndWaitStreamMode (using httpx.MockTransport on an AsyncClient) would catch it. A cheap mode="bogus" -> ValueError test would also be worth adding.

[LOW] 3. SDK title mislabeled as Finance Research API — README.md summary / .speakeasy/out.openapi.yaml info.title. info.title changed from You.com API to You.com Finance Research API, so the regenerated README summary now presents the entire SDK as a Finance Research API when it actually spans Agents/Research/Search/Contents. Recommend restoring the top-level title in the source spec/overlay.

Minor / non-blocking

• research_and_wait(mode="poll") computes a deadline (line 235) it never uses; timeout_s semantics differ slightly between poll (excludes submit time) and stream (includes it). Harmless.
• is_custom in YDCUserAgentOverrideHook has redundant conditions — correct, just belt-and-suspenders.

Everything else (poll/terminal-state logic, _decode_raw_event matching the eventstreaming decoder contract, finance_research deliberately omitting source_control/output_schema, livecrawl_formats list migration, error-class consolidation) checks out.

[claude]

Bug: contextlib.aclosing calls aclose(), which EventStreamAsync does not implement — this raises AttributeError on every clean exit.

contextlib.aclosing(thing) calls await thing.aclose() in its __aexit__. But EventStreamAsync (in utils/eventstreaming.py) only defines an async close() method — there is no aclose(). So when this async with block exits (stream fully consumed, or consumer break/return), it will raise:

AttributeError: 'EventStreamAsync' object has no attribute 'aclose'

This breaks stream_research_events_raw_async() (the error surfaces to the caller once the stream ends) and the cleanup path of research_and_wait_async(mode="stream"). The sync counterpart (stream_research_events_raw, line 407) is fine because it uses with _open_raw_stream(...) and EventStream implements __exit__.

The comment on 483–485 claims aclosing "guarantees the underlying httpx SSE response is closed deterministically" — in practice it throws instead of closing.

Suggested fix — drop aclosing and close the stream explicitly via the method it actually exposes:

Suggested change

	# contextlib.aclosing guarantees the underlying httpx SSE response is
	# closed deterministically on consumer break/return/throw, rather than
	# relying on garbage-collected aclose() of the suspended async generator.
	stream = await _open_raw_stream_async(
	client, task_id, http_headers=http_headers, from_id=from_id,
	)
	async with contextlib.aclosing(stream) as closing_stream:
	async for evt in closing_stream:
	yield evt
	# Close the underlying httpx SSE response deterministically on consumer
	# break/return/throw. EventStreamAsync exposes async close() (not aclose),
	# so a try/finally is used rather than contextlib.aclosing.
	stream = await _open_raw_stream_async(
	client, task_id, http_headers=http_headers, from_id=from_id,
	)
	try:
	async for evt in stream:
	yield evt
	finally:
	await stream.close()

Please also add an async streaming test (see the coverage note in the summary) — this would have caught it.

[claude]

Test coverage gap on the async streaming path. There are async tests for research_background_async and poll_research_task_async (incl. error paths), but nothing exercises stream_research_events_raw_async or research_and_wait_async(mode="stream"). That's precisely the untested code that hides the contextlib.aclosing/aclose() AttributeError bug flagged in research_helpers.py. Please mirror TestStreamResearchEventsTolerant and TestResearchAndWaitStreamMode with async variants (an httpx.MockTransport on an httpx.AsyncClient works the same way). While here, a small test for the mode validation (research_and_wait(mode="bogus") → ValueError) would be cheap coverage too.

[claude]

Doc accuracy: the summary now leads with "You.com Finance Research API: Unified API for Express, Advanced, and Custom Agents…", which mislabels the whole SDK — it covers Agents, Research, Search, and Contents, not just Finance Research. This flows from info.title being changed to You.com Finance Research API in .speakeasy/out.openapi.yaml. Consider restoring the top-level title to You.com API (in the source spec / overlay) so the regenerated README summary stays accurate.

[tyler5673]

Thanks for the thorough second pass. All findings addressed in the latest commit:

[HIGH] 1. Async SSE aclosing bug — fixed. Replaced contextlib.aclosing(stream) with try/finally: await stream.close() in stream_research_events_raw_async. EventStreamAsync exposes close() (not aclose()), so aclosing.__aexit__ was raising AttributeError on every clean exit. Removed the now-unused import contextlib.

[MEDIUM] 2. Async streaming test coverage — added. New TestStreamResearchEventsTolerantAsync and TestResearchAndWaitStreamModeAsync mirror their sync counterparts using httpx.AsyncClient(MockTransport(...)). Also added TestModeValidation with sync + async mode="bogus" -> ValueError tests. The async streaming tests use a small _AsyncChunks(httpx.AsyncByteStream) helper since MockTransport returns sync streams by default which AsyncClient rejects for stream=True. All 4 new tests pass.

[LOW] 3. SDK title mislabeled — fixed. Hand-edited the README summary block ("You.com Finance Research API" -> "You.com API") and added a new overlay action (target: $["info"]["title"], update: "You.com API") so the next regen preserves the correct title. Overlay now has 6 actions (was 5).

Minor items:

• Unused deadline in poll mode: moved deadline = time.monotonic() + timeout_s to after the poll-mode early return in both sync and async variants, so it only computes in stream mode.
• is_custom belt-and-suspenders in YDCUserAgentOverrideHook: left as-is — the redundant condition is harmless and makes the intent explicit.

Test results: 18/18 MockTransport-based tests pass (4 new). Pylint 10.00/10. Server-dependent tests skipped (mockserver not running).

[claude]

Minor resource-cleanup nuance: returning out of this async for on response.done/complete does not deterministically run the finally: await stream.close() inside stream_research_events_raw_async. Unlike sync generators (finalized promptly by refcounting), an async generator's aclose() is only scheduled by the event loop's async-gen finalizer / GC, so the SSE HTTP connection can stay open past this return until then.

The comment in stream_research_events_raw_async says contextlib.aclosing was avoided because EventStreamAsync exposes close() (not aclose()) — but you can wrap the async generator itself (which does implement aclose()) here to force deterministic teardown:

import contextlib
async with contextlib.aclosing(
    stream_research_events_raw_async(client, task.task_id)
) as events:
    async for evt in events:
        ...

Not blocking (tests pass and the httpx read-timeout is an eventual backstop), but worth tightening given the module already goes out of its way for deterministic cleanup. Same applies to the sync path, though refcounting makes it far less of a concern there.

[claude]

Review summary

Reviewed with focus on the manually-maintained surface (tests/, CHANGELOG.md, MIGRATION.md, overlays/python_overlay.yaml, pyproject.toml, and the hand-written src/youdotcom/research_helpers.py / _hooks/registration.py / utils/security.py edits). This is a high-quality release PR — documentation is accurate and unusually honest about known limitations, and test coverage for the new hand-written code is strong.

Verified

• Docs accuracy: ResearchEffort (lite/standard/deep/exhaustive) and FinanceResearchEffort (deep/exhaustive) enum values match the code; boost_domains typing (GET search.unified = comma-separated str, POST search_post = List[str]) matches the generated models and USAGE.md; the env-var rename to YDC_API_KEY with YOU_API_KEY_AUTH fallback is implemented in get_security_from_env and locked in by tests/test_security_env.py.
• The extra=ignore data-loss caveat is real and correctly documented: Content and TaskDetail.Result are still generated as empty class ...(BaseModel): pass, and the base BaseModel uses pydantic default extra=ignore, so the output_schema/background result payloads genuinely drop at unmarshal time — exactly as CHANGELOG/MIGRATION/research_helpers.py describe.
• Overlay target is correct but not yet applied: content.oneOf[1] in out.openapi.yaml is the object branch (index 0 is string), so additionalProperties: true targets the right node. Confirmed the overlay is NOT applied in this regen (empty models, no additionalProperties in out.openapi.yaml) — consistent with the "staged for next regeneration cycle" comments.
• User-Agent hook: YDCUserAgentOverrideHook correctly passes through a custom sdk_configuration.user_agent and otherwise emits youdotcom-python-sdk/{version}. Both branches tested.
• Mockserver routing: the research_and_wait poll test drives two endpoints under one x-speakeasy-test-name; both pathPostV1Research and pathGetV1Research fall through to success default cases, so the submit-then-poll sequence works.
• Security: no hardcoded secrets (only test-api-key fixtures), no injection surface. Breaking changes (Union response, livecrawl_formats list requirement, error-class consolidation, env-var rename) are all covered in MIGRATION.md.

Note: I could not execute the test suite in this review environment (command sandboxing), so the above is static verification.

Worth confirming before merge

1. output_schema (beta) ships effectively non-functional for retrieving structured data. Because the overlay fix is not applied yet, a caller supplying output_schema gets content_type=object but an empty Content() — the structured payload is silently dropped, and the only way to get the data is the documented re-issue-synchronously workaround. It is thoroughly documented as a 2.4.0 caveat, so this is a product call rather than a bug, but please confirm the team is comfortable advertising the feature in this state (README/CHANGELOG present it as a usable beta). Consider applying the staged overlay before the 2.4.0 tag if a functional path is expected at release.
2. Inline comment on research_helpers.py (async stream early-return): the finally: await stream.close() does not run deterministically when the consumer returns early from async for — minor connection-lifetime nuance, not blocking.

Minor / non-blocking

• research_background TypeError guard (server returns ResearchResponse despite background=True) has no direct test.
• The pylint: disable-next=protected-access at research_helpers.py:405 sits above a call to the module-level _open_raw_stream (the protected access is actually inside that function, which has its own disable) — cosmetic.
• The regenerated README.md summary block has slightly jumbled ordering of the description lines (auto-generated from the spec description); not actionable here.

Nice work on the "Hand-maintained additions" list in the CHANGELOG — explicitly flagging what must be re-applied after the next Speakeasy regen is exactly right for a generated SDK.