Skip to content

[bug] RestrictToTopic validator occurs when disable_classifier is False and disable_llm is True #1192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
railsstudent opened this issue Dec 9, 2024 · 4 comments
Assignees
Labels
bug Something isn't working

Comments

@railsstudent
Copy link

Describe the bug
When disable_classifier is False and disable_llm is True, the _inference_local method returns error on

candidate_topics = model_input["valid_topics"] + model_input["invalid_topics"]

The error message is

TypeError: unsupported operand type(s) for +: 'set' and 'set'

It is because

        valid_topics = set(metadata.get('valid_topics', self._valid_topics))
        invalid_topics = set(metadata.get('invalid_topics', self._invalid_topics))

return two sets and + operator cannot form a new array

To Reproduce
Steps to reproduce the behavior:

restrictToTopic = RestrictToTopic(
    valid_topics=["environment"],
    invalid_topics=["conspiracy", "food", "entertainment", "politics", "religion", "violence", "hate"],
    disable_classifier=False,
    disable_llm=True,
    on_fail=OnFailAction.EXCEPTION
)

result = restrictToTopic.validate(value="Large corporations should act responsibly socially and environmentally.  They should reduce carbon emission. One way to monitor their behavior is by reading their ESG Report.")
print(result.outcome)

Expected behavior
The _inference_local should return a found_topics array.

Library version:
Version 0.5.2

Additional context
Add any other context about the problem here.

@railsstudent railsstudent added the bug Something isn't working label Dec 9, 2024
@JosephCatrambone
Copy link
Contributor

Good catch.

I think I might have fixed this incidentally as a side effect with this PR: tryolabs/restricttotopic#23

@JosephCatrambone JosephCatrambone self-assigned this Dec 10, 2024
@railsstudent
Copy link
Author

@JosephCatrambone Thank you. Do you know when guardrails will have a new release? I am evaluating guardrails and LLM guards. While guardrails ai has more broken validators and missing validators (e.g., prompt refusal), it has more validators for detecting hallucinations and better error messages. Both are very good for me.

@JosephCatrambone
Copy link
Contributor

JosephCatrambone commented Dec 13, 2024

Hi @railsstudent -- the PR got approved and merged yesterday, though I don't know if it's in the private pypi repo yet.

EDIT: Linking PR so it shows up: tryolabs/restricttotopic#23

EDIT: Also a way old PR: tryolabs/restricttotopic#21

EDIT: Just tried. It's up on Pipy!

@JosephCatrambone
Copy link
Contributor

I'm going to close this as resolved since the test in the first is passing, but if there's still a problem please feel free to reopen this ticket and I'll take another swing at it. Thanks again for reporting the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants