How to recover from generation errors

[kaimibk]

BLUF: Is there a way to recover when a generative output is malformed?

I am currently experimenting with LangExtract using a custom Bedrock model provider with Claude 4.5 as the generative model (due to limitations of my environment, this is the only generative model/platform I can use). However, due to the very nature of generative models, sometimes (but often enough) we get malformed responses from the model. When such an error occurs, this causes the whole LangExtract pipeline to fail and lose all progress. For example Extraction text must be a string, integer, or float. Found: <class 'NoneType'>. In one case, I had extraction passes set to 3, and on the final pass the error arose and I lost all previous progress/extractions - this seems like such a waste!

In the case of errors:

1. Is there a way to trigger a retry on failed passes/batches?
2. Can we "skip" over failed passes/batches and keep the successful extractions? As to not waste time/money.

Mahalo! 🤙

[Yash-xoxo]

Great questions! Here's what you can do:

1. Retry on failures:
Wrap in a custom retry handler:

from tenacity import retry, stop_after_attempt, retry_if_exception_type

@retry(
    stop=stop_after_attempt(3),
    retry=retry_if_exception_type(ValueError)
)
def extract_with_retry(extractor, text):
    return extractor.extract(text)

2. Skip failed batches:
Process in chunks and catch errors per batch:

results = []
for batch in text_batches:
    try:
        result = extractor.extract(batch)
        results.append(result)
    except Exception as e:
        logger.warning(f"Batch failed: {e}")
        results.append(None)  # or skip
        continue

# Filter successful extractions
successful = [r for r in results if r is not None]

3. Save incremental progress:
Checkpoint after each pass:

for i, batch in enumerate(batches):
    result = extractor.extract(batch)
    with open(f"checkpoint_{i}.json", "w") as f:
        json.dump(result, f)

For Claude specifically: The NoneType error often means Claude refused to extract (guardrails). Try adding "If uncertain, return an empty string instead of null" to your prompt.

Would love to see native checkpoint/resume support in LangExtract though! 🤙

[AshrafMorningstar]

Dealing with malformed JSON or NoneType errors mid-pipeline is a common frustration when using high-reasoning models like Claude 4.5, especially when an unexpected "hallucination" in the output format breaks the schema validation in LangExtract.
Since you are locked into the Bedrock environment, you can't just switch models, so we have to harden the pipeline. Here is how you can handle retries and prevent data loss.
Triggering Retries on Failed Passes
LangExtract typically relies on the underlying orchestration layer (or your custom provider wrapper) to handle retries. If the error is occurring because the model returns a null value or malformed string that fails your schema, you can implement a Retry Strategy at the Provider level.
Implement a Wrapper: Wrap your Bedrock/Claude 4.5 call in a loop with a try-except block.
Backoff: If you hit a throttling limit or a transient error, use exponential backoff.
Validation Catch: If the model returns None or an invalid type, throw a custom exception within the provider to trigger the next loop iteration.

Example Logic for Custom Provider

def call_model_with_retry(prompt, max_retries=3):
for attempt in range(max_retries):
response = bedrock.invoke_model(prompt)
if response is not None and isValid(response):
return response
raise FinalPipelineError("Failed after 3 attempts")

"Skipping" and Partial Progress Recovery
To avoid losing progress during multi-pass extractions, you need to move away from a "single-shot" pipeline execution and adopt a Checkpointing or Atomic Batching approach.
Persistent State Storage
Instead of running all 3 passes in one volatile function call, break them into individual tasks.
Batching: Process your data in smaller batches (e.g., 5-10 items).
Intermediate Saving: Write the output of each successful pass to a local cache (like a JSONL file or a SQLite DB).
The "Skip" Logic: Before starting a pass, check your cache. If the data for "Pass 1" already exists, the script should skip it and move to "Pass 2."
Handling the NoneType Error
The error Extraction text must be a string... Found: <class 'NoneType'> usually happens when the model's output is empty or the parser fails. You can handle this by:
Default Values: In your extraction schema, set fields to be optional or provide a default string (e.g., "Not Found") so the pipeline doesn't crash on a null return.
Soft Failures: Modify your extraction logic to catch the error, log the failure for that specific record, and return an empty dictionary {} instead of letting the exception bubble up and kill the process.