Skip to content

Conversation

sanyo4ever
Copy link
Contributor

Summary

This PR enhances the Mailgun provider to handle DMARC reports and other emails without body content, and adds functionality to reprocess failed email events.

Problem

Mailgun provider was failing to parse DMARC reports and other emails without body content, resulting in errors:

DMARC reports (common automated emails from Google, Microsoft, etc.) have no body text - only subject and ZIP attachments - causing the provider to crash.

Solution

1. Enhanced Email Parsing

DMARC Detection:

  • Multi-indicator detection using sender, subject, content-type, and attachments
  • Properly classifies DMARC, SPF, and bounce emails

Flexible Message Handling:

  • Handles emails without body content
  • Extracts message from subject + attachment descriptions
  • Multiple fallback levels for robustness

Email Classification:

  • Tags emails with email_type metadata (dmarc_report, spf_report, bounce, alert)
  • Allows filtering by email type

Configuration Options:

  • skip_dmarc_reports (default: false) - Option to skip DMARC reports
  • skip_spf_reports (default: false) - Option to skip SPF reports
  • handle_emails_without_body (default: true) - Handle emails without body content

2. Error Alert Reprocessing

Backend:

  • New endpoint: POST /alerts/event/error/reprocess
  • Database helpers: get_error_alerts_to_reprocess(), dismiss_error_alert_by_id()
  • Reprocess single or all error alerts
  • Auto-dismiss successfully reprocessed alerts

Frontend:

  • Added "Reprocess" buttons to Error Alerts Modal
  • Loading states and toast notifications
  • Detailed success/failure feedback

Use Case:
After deploying code fixes, users can reprocess previously failed events without resending emails.

Changes

Backend

  • keep/providers/mailgun_provider/mailgun_provider.py - Enhanced email parsing logic
  • keep/api/routes/alerts.py - Reprocess endpoint and imports
  • keep/api/core/db.py - Database helper functions
  • docs/snippets/providers/mailgun-snippet-autogenerated.mdx - Updated documentation

Frontend

  • keep-ui/entities/alerts/model/useAlerts.ts - Reprocess hook function
  • keep-ui/features/alerts/alert-error-event-process/ui/alert-error-event-modal.tsx - UI buttons and handlers

Testing

Tested with:

  • ✅ DMARC reports from Google (no body, ZIP attachments)
  • ✅ Regular monitoring emails with body content
  • ✅ Reprocessing error alerts
  • ✅ Email classification and metadata

Results:

  • ✅ DMARC emails create alerts successfully
  • ✅ Message extracted from subject + attachments
  • ✅ No parsing errors
  • ✅ Reprocess functionality works for both new and old errors
  • ✅ Source facet counter updates correctly

Breaking Changes

None - fully backward compatible with existing functionality.

Screenshots

(Optional: Add screenshots of the Error Alerts Modal with Reprocess buttons)


Fixes: Mailgun DMARC email parsing failures
Related: #5366 (if applicable)

…d flexible handling

- Add comprehensive DMARC report detection using multiple indicators (sender, subject, content-type)
- Add email type classification (DMARC, SPF, bounce, alert)
- Add configurable skip options for DMARC/SPF reports via UI
- Handle emails without body content gracefully with fallback messages
- Improve error handling and logging for better debugging
- Add email_type metadata to all alerts for better tracking

Fixes parsing errors for DMARC reports that have no body content.
Backend:
- Add get_error_alerts_to_reprocess() helper function to db.py
- Add dismiss_error_alert_by_id() helper function to db.py
- Add POST /alerts/event/error/reprocess API endpoint
- Support reprocessing single alert or all error alerts
- Auto-dismiss successfully reprocessed alerts
- Return detailed results with success/failure counts

Frontend:
- Add reprocessErrorAlerts() function to useAlerts hook
- Add reprocess buttons to AlertErrorEventModal UI
- Add handleReprocessSelected() and handleReprocessAll() handlers
- Add loading states and toast notifications
- Disable buttons during operations to prevent race conditions

This allows users to reprocess failed alert events after code fixes
(e.g., DMARC detection improvements). Successfully reprocessed alerts
are automatically dismissed from the error alerts list.
Changed skip_dmarc_reports and skip_spf_reports defaults from True to False.
DMARC and SPF reports will now create alerts by default.

Users can still enable skipping via UI configuration if desired.

DMARC reports without body will get message: DMARC Report: {subject} + attachment info
- Add _extract_severity_from_email() method for keyword-based severity detection
- Detect critical, high, warning, low, and info severity from subject/body
- Assign severity based on email type (DMARC=low, SPF/bounce=warning)
- Priority keyword matching: critical > high > warning > low > info

Examples:
- DMARC reports: low severity (informational)
- [SUCCESS] emails: low severity
- [ERROR]/[CRITICAL] emails: high/critical severity
- [WARNING] emails: warning severity

This provides better alert prioritization in the UI with appropriate visual indicators.
- Add _extract_status_from_email() method for keyword-based status detection
- Detect resolved, acknowledged, and firing status from subject/body
- Support status transitions via email (e.g., resolved notifications)

Status mapping:
- resolved: resolved, cleared, recovered, fixed, closed, ok now, back to normal
- acknowledged: acknowledged, ack, investigating, working on
- firing: default for new alerts

This allows email alerts to properly reflect their lifecycle status.
…sender

Changed source field from email sender address to proper provider source format:
- Primary source: mailgun (for source facet filtering)
- Secondary source: email sender address (for detailed tracking)

This fixes the source counter in alerts feed and allows proper filtering by source=mailgun.

Before: source = [[email protected]]
After: source = [mailgun, [email protected]]
Reverted the source field change. After review, the original behavior is correct:
- source = [email_sender] allows filtering by specific email senders
- This is the intended behavior for email-based providers
- Users can filter by source to see alerts from specific monitoring systems

The source counter showing individual email addresses is intentional.
Users can use the email_type field to filter by provider (e.g., email_type=dmarc_report).
Add database script to refresh severity and status for Mailgun alerts
that were processed before the intelligent extraction logic was added.

Features:
- Updates severity using keyword-based detection
- Updates status using keyword-based detection
- Adds email_type classification if missing
- Dry-run mode by default (safe)
- Configurable time range (default: 30 days)
- Detailed reporting of changes
- Error handling for individual alerts

Usage:
  # Dry run (see what would change)
  python scripts/update_mailgun_alert_metadata.py --tenant-id keep

  # Actually update
  python scripts/update_mailgun_alert_metadata.py --tenant-id keep --apply

  # Check last 7 days only
  python scripts/update_mailgun_alert_metadata.py --tenant-id keep --days 7 --apply

This allows retroactive updates for alerts processed before severity/status
extraction improvements were added.
… values

Reverted back to original hardcoded severity and status values:
- severity = info (hardcoded)
- status = firing (hardcoded)

Removed:
- _extract_severity_from_email() method
- _extract_status_from_email() method
- update_mailgun_alert_metadata.py script

This matches the original Mailgun provider behavior where all email alerts
have the same severity/status regardless of content.
Update auto-generated documentation to include new configuration fields:
- skip_dmarc_reports: Skip DMARC reports
- skip_spf_reports: Skip SPF reports
- handle_emails_without_body: Handle emails without body content

Generated using: python scripts/docs_render_provider_snippets.py
@vercel
Copy link

vercel bot commented Oct 16, 2025

@sanyo4ever is attempting to deploy a commit to the KeepHQ Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions
Copy link
Contributor

No linked issues found. Please add the corresponding issues in the pull request description.
Use GitHub automation to close the issue when a PR is merged

@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. Documentation Improvements or additions to documentation Feature A new feature Provider Providers related issues UI User interface related issues labels Oct 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Documentation Improvements or additions to documentation Feature A new feature Provider Providers related issues size:XL This PR changes 500-999 lines, ignoring generated files. UI User interface related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants