Responsible AI: Detecting and Reducing LLM Hallucination with Human in the Loop for Generative AI Applications
These notebooks represent some real-world use cases to detect and reduce LLM hallucinations with Amazon Bedrock Agents, Amazon Bedrock Knowledge Bases and RAGAS evaluation framework.
Project : Hallucination Detection and Mitigation
| Title | Studio lab |
|---|---|
| Chat with LLMs grounded by Amazon Guardrails and querying Amazon Knowledge Bases | lab1/lab1.ipynb |
| Detecting and mitigating hallucinations with Amazon Bedrock Agents and Amazon Knowledge Bases | lab2/lab2.ipynb |
If you have an event access code from a AWS hosted event or Self-paced event, you can run these labs using this Workshop Studio Link
Setup Instructions
The lab1 notebook will download the bedrock user guide from here: AWS docs for Amazon Bedrock User Guide
Please execute lab1.ipynb before proceeding to lab2.ipynb.
Note: Using the CloudFormation template and running the noteobooks end-to-end may create AWS service roles and AWS Managed KMS keys that will not incur cost in your account. If you do not run the optional cleanup infrastructure cell for every notebook run, there may be S3 buckets, SNS left behind in the account which would need manual cleanup.
Troubleshooting: CFN template
Please add the following independent role in SageMaker_Bedrock_Agents.yaml if you get the following error:
SageMaker is not authorized to perform: iam:CreateServiceLinkedRole on resource: arn:aws:iam::<account-id>:role/aws-service-role/observability.aoss.amazonaws.com/AWSServiceRoleForAmazonOpenSearchServerless because no identity-based policy allows the iam:CreateServiceLinkedRole action
AOSSLinkedRole:
Type: AWS::IAM::ServiceLinkedRole
Properties:
AWSServiceName: observability.aoss.amazonaws.com
The license for this repository depends on the section. Data set for the course is being provided to you by permission of Amazon and is subject to the terms of the Amazon License and Access. You are expressly prohibited from copying, modifying, selling, exporting or using this data set in any way other than for the purpose of completing this course. The lecture slides are released under the CC-BY-SA-4.0 License. This project is licensed under the Apache-2.0 License. See each section's LICENSE file for details.