Spring Boot web dashboard for Amazon CloudFront standard logs (JSON format). Fetches log files from S3, stores them in a local SQLite database, and displays interactive charts with drill-down detail pages.
This application is intended for local, single-user use only.
- There is no authentication or authorization. Anyone who can reach port 8080 can view the dashboard and trigger S3 fetches.
- CSRF protection is enabled (Spring Security defaults). The one state-changing operation,
Refresh from S3, is a POST whose token Thymeleaf injects viath:action. The application should still not be exposed to untrusted networks, since anyone who can reach it can load the form and submit it.
Do not expose this application on a public interface or behind a shared reverse proxy without adding authentication (e.g. Spring Security with HTTP Basic, or an authenticating proxy such as nginx/Authelia).
A security review found no exploitable vulnerabilities. The following controls are verified:
| Area | Control |
|---|---|
| SQL injection | All database queries use JdbcTemplate with ? bind parameters. User-supplied values are never interpolated into SQL strings. |
| XSS (server-side) | All Thymeleaf templates use th:text / th:content for user-controlled output, which applies automatic HTML entity escaping. th:utext is not used anywhere. |
| XSS (client-side) | API responses are rendered onto <canvas> via Chart.js. Canvas drawing APIs do not interpret HTML or JavaScript. encodeURIComponent() is applied to all user-derived values placed into URLs. |
| Path traversal | No file-serving endpoints with user-controlled paths exist. S3 object keys come from AWS API responses, not from user input. |
| CSRF | Spring Security's CSRF filter is enabled with session-backed tokens, and the token is resolved before the view renders (deferred loading is disabled). The only state-changing endpoint, POST /refresh, is therefore token-protected. |
| Data exposure | API endpoints return only CloudFront access-log data, which is the application's stated purpose. No credentials or internal state are exposed. |
Requires Java 25 and Maven.
mvn package
java -jar target/analyze-logs-1.0-SNAPSHOT.jar --spring.profiles.active=localOr without packaging:
mvn spring-boot:run -Dspring-boot.run.profiles=localOpen http://localhost:8080.
Key properties:
| Key | Default | Description |
|---|---|---|
app.aws.region |
us-east-1 |
AWS region of the S3 bucket |
app.aws.bucket |
`` | S3 bucket containing CloudFront logs |
app.aws.prefix |
`` | S3 key prefix (e.g. AWSLogs/123456789/CloudFront/) |
app.aws.profile |
`` | AWS credentials profile (~/.aws/credentials); empty = default chain |
app.db-path |
logs.db |
SQLite file path (relative to working directory) |
server.port |
8080 |
HTTP port |
uri-stem-filter.excluded-extensions |
.css, .js, .png, … |
File extensions excluded from all URL charts (static assets) |
referer-filter.self-referers |
[] |
Referer prefixes to exclude from Top Referers (your own domain). Matched with and without trailing slash, and as a bare domain without scheme. |
referer-normalizer.rules |
Google, Bing, Yahoo, … | Rules to group referer URLs into a single label. Each rule has label and one of domain (exact), domain-starts-with, or domain-ends-with. |
uri-stem-groups.groups |
PHP/WordPress, Env/credential file probing, Config/backup dump probing, Open-proxy / IP-echo probing, Java/Spring exploit probing, CGI-bin / router-IoT probing | Groups of uri_stem LIKE patterns collapsed into one label on the Top URLs chart. A group with security: true also feeds the Security traffic category and the Security page. |
Create src/main/resources/application-local.yml to override values without touching the committed file:
referer-filter:
self-referers:
- "https://your-site.example.com/"
- "http://your-site.example.com/"
app:
aws:
bucket: "my-cloudfront-logs"
prefix: "AWSLogs/123456789/CloudFront/"
profile: "my-aws-profile"
db-path: /absolute/path/to/logs.dbThis file is listed in .gitignore and will never be committed. Activate it with --spring.profiles.active=local.
The ua-classifier.rules list in application.yml maps UA substrings to human-readable labels.
Rules are evaluated top-to-bottom; first match wins. Add entries to classify custom bots or internal tools:
ua-classifier:
rules:
- pattern: "MyInternalBot"
label: "Internal crawler"Four detection methods, all scoped to the selected date range:
| Section | Detection signal |
|---|---|
| Top Probable Bots | IPs that fetched /robots.txt and then crawled another URL within 1 hour using the same IP and user agent. |
| No Static Assets Bots | IPs whose every request (per IP+UA pair) contains zero CSS, JS, WebP, ICO, or SVG fetches — real browsers always load static assets. Feed readers (/feed.xml, /rss.xml) are excluded. |
| Robots.txt Disobedient Bots | Known bots that requested a path their robots.txt group disallows (a bot with no named group falls back to User-agent: *; /robots.txt itself is always allowed). Requires a Refresh Robots to load the latest rules. |
| Robots.txt Obedient Bots | Known bots that fetched /robots.txt and never requested a path disallowed for them; the count is their other content requests (excluding /robots.txt); the bar covers all their requests. |
The Bot vs Human Traffic Over Time chart shows daily request volume split between known bots (AI Bots, Search Bots, Other Bots, Apps) and human browsers.
Date range presets and custom date picker are shared with the main dashboard.
Lists IPs that presented more than one known crawler identity (ua_name in a bot ua_group) within the
selected date range — real bots each operate from their own infrastructure, so one IP claiming several is
a strong UA-spoofing signal. Each entry shows first/last seen, the distinct user agents used (click one to
open its /ua-requests page), and the URLs requested with a Hit/Miss/Filtered/Error bar per URL.
Three stacked sections — Mastodon, WhatsApp, Facebook — each listing the most recent webpage requests
(URIs ending in /; static assets excluded) matching that network's known link-preview crawler user
agent (facebookexternalhit, WhatsApp) or a click-through Referer from its domain. Referer domains
are host-anchored (scheme + optional www. + exact domain) so facebook.com never matches a look-alike
host such as facebook.com.evil.example. Mastodon is matched on the user-agent classifier's ua_name
instead (fediverse UA strings vary too much for substring matching) and excludes the home page /,
which every instance polls often enough to bury the actual article previews. Only served pages count — Hit, RefreshHit
and Miss; Filtered and Error responses are dropped. Each row shows its timestamp, a link to the user
agent's /ua-detail page, the URI and the country.
Eight charts, all scoped to the selected date range:
| Chart | Description |
|---|---|
| Traffic Categories | Horizontal stacked bar — each (client_ip, user_agent) pair classified as Probable human, Declared bots (fetched /robots.txt), Feeds (/feed.xml, /rss.xml), Security (probed a uri-stem-groups pattern flagged security: true), or Other. Click a bar to open the category detail page. |
| Top User Agents | Horizontal stacked bar — classified UA names by request count, coloured by edge result type. Click a bar to open the UA detail page. |
| Top Countries | Horizontal stacked bar — request volume by country, coloured by edge result type. Click a bar to open the country detail page. |
| Top URLs | Horizontal stacked bar — most-requested paths, coloured by edge result type; static assets excluded, and uri-stem-groups patterns (PHP/WordPress, .env/.git probing, backup dumps, actuator/Struts, cgi-bin, etc.) collapsed into one label each. |
| Top Referers | Horizontal bar — most frequent Referer headers; self-referrals and null referers excluded. Known search engines and social sites (Google, Bing, Yahoo, DuckDuckGo, Qwant, Facebook, Babelio) are grouped under a single label. Configurable via referer-normalizer.rules. |
| User Agent Groups | Pie — request share by UA group (Browsers, AI Bots, Search Bots, Other Bots, Apps, Feed Readers). |
| Platforms | Pie — request share by OS/platform detected from the user agent (iOS, Android, Windows, Mac, Linux, Other). |
| Requests per Day | Stacked bar — daily breakdown by edge result type: Hit, Miss, Function, Redirect, Error. |
Date range presets: Today / 7 days / 30 days / 3 months or a custom date picker.
Hide bots & apps toggle removes traffic from the AI Bots, Search Bots, Other Bots, and Apps UA groups — as well as entries with no user agent — from all charts simultaneously. State is persisted in localStorage.
Refresh from S3 button triggers an incremental fetch (skips already-imported files).
Focuses on the traffic flagged by uri-stem-groups entries with security: true (scanner/exploit probing), scoped to the selected date range:
| Chart | Description |
|---|---|
| Traffic Categories | Horizontal bar — request count per security scan category (PHP/WordPress, Env/credential file probing, Config/backup dump probing, Open-proxy / IP-echo probing, Java/Spring exploit probing, CGI-bin / router-IoT probing). |
| Top Countries | Horizontal bar — countries the security-flagged requests originate from. |
| Requests per Day | Stacked bar — daily breakdown by scan category. |
Date range presets are shared with the main dashboard.
Opened by clicking a bar in Top User Agents. Shows charts scoped to a single classified user-agent:
| Chart | Description |
|---|---|
| Result Types | Pie — edge result type breakdown for this UA |
| Countries | Pie — geographic distribution of requests |
| Top URLs | Horizontal stacked bar — most-requested paths for this UA, coloured by edge result type (PHP / WordPress grouping applied) |
| Requests per Day | Line — daily request trend by edge result type |
Opened by clicking a bar in Top Blocked Countries. Shows charts scoped to a single country:
| Chart | Description |
|---|---|
| Result Types | Pie — edge result type breakdown for this country |
| Top URLs | Horizontal stacked bar — most-requested paths from this country, coloured by edge result type (PHP / WordPress grouping applied) |
| Requests per Day | Line — daily request trend by edge result type |
No credentials are stored by this application. Authentication is delegated to the AWS SDK default credentials chain:
- Environment variables —
AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY ~/.aws/credentials— populated byaws configure- EC2/ECS instance profile
{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetObject"],
"Resource": [
"arn:aws:s3:::my-cloudfront-logs",
"arn:aws:s3:::my-cloudfront-logs/*"
]
}Investigation queries are in etc/queries/. Run them with:
sqlite3 logs.db < etc/queries/suspicious_user_agents.sqlAgents hitting .php or /wp* URLs 10+ times per day, with no legitimate traffic on the site:
| User Agent | Date | Hits |
|---|---|---|
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/71.0.3542.0 Safari/537.36 |
2026-04-07 | 15 |
Requests with an empty
user_agentare excluded upfront — they are always suspicious by definition.
The Traffic Categories chart classifies each (client_ip, user_agent) pair as
Probable human if it ever requests a trailing-slash path (e.g. /) and ever
requests a static asset (image extension or /js/*) — regardless of the HTTP
result type. A scanner that pads its probe sequence with a fake / request and a
spoofed browser user agent can still qualify, even if every single request
(including /) comes back as Error. This lets burst scanners (dozens of
favicon/logo/admin-panel probes within a few seconds, all erroring) slip into the
human bucket. Known limitation, not currently fixed.
Logs are stored in a SQLite file (logs.db by default, relative to the working directory).
Query it directly with any SQLite-compatible tool:
sqlite3 logs.db "SELECT ua_name, COUNT(*) FROM cloudfront_logs GROUP BY ua_name ORDER BY 2 DESC"cloudfront_logs (
id INTEGER PRIMARY KEY,
timestamp TEXT NOT NULL, -- ISO-8601 UTC
edge_location TEXT,
sc_bytes INTEGER,
client_ip TEXT,
method TEXT,
uri_stem TEXT,
status INTEGER,
referer TEXT,
user_agent TEXT,
edge_result_type TEXT,
cs_bytes INTEGER,
time_taken REAL,
edge_response_result_type TEXT,
time_to_first_byte REAL,
edge_detailed_result_type TEXT,
content_type TEXT,
content_length INTEGER,
country TEXT, -- ISO 3166-1 alpha-2
ua_name TEXT, -- classified user-agent label
edge_location_iata TEXT -- IATA code from edge_location
)
fetched_files (
s3_key TEXT PRIMARY KEY,
fetched_at TEXT
)