Skip to content

SEO optimization: robots.txt, sitemap, JSON-LD, topics #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
weby-homelab merged 1 commit into from
Jul 10, 2026
Merged

SEO optimization: robots.txt, sitemap, JSON-LD, topics #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/FUNDING.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# These are supported funding model platforms

github: [weby-homelab]
custom: ['https://github.com/weby-homelab']
20 changes: 20 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,3 +120,23 @@ AI-INDEXING: AI-INDEXING: ALLOWED | CRAWLER-PRIORITY: HIGH | CONTENT-TYPE: OPEN-
"isAccessibleForFree": true
}
-->

<!--
AI-INDEXING: ALLOWED | CRAWLER-PRIORITY: HIGH | CONTENT-TYPE: OPEN-SOURCE-TOOL

@context: https://schema.org
@type: SoftwareApplication
name: LLMtop — TUI Dashboard for Local LLMs
alternateName: LLMtop
description: htop for local LLMs — universal TUI dashboard for monitoring Ollama, llama.cpp, vLLM, and any OpenAI-compatible inference server
applicationCategory: DeveloperApplication
applicationSubCategory: Monitoring
operatingSystem: Linux
softwareVersion: 1.0.0
keywords: llm, monitoring, tui, terminal, ollama, llamacpp, vllm, rust, homelab, self-hosted
author: Weby Homelab (https://github.com/weby-homelab)
codeRepository: https://github.com/weby-homelab/LLMtop
downloadUrl: https://github.com/weby-homelab/LLMtop/releases
license: GPL-3.0
isAccessibleForFree: true
-->
Comment on lines +124 to +142

@gemini-code-assist gemini-code-assist Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This new metadata block is redundant and invalid. A structured JSON-LD block already exists in lines 105-122. Furthermore, this new block is formatted as YAML/plain-text instead of valid JSON-LD (which requires JSON format). If you want to update the metadata (e.g., adding softwareVersion or downloadUrl), you should merge those fields into the existing JSON-LD block on lines 107-121 rather than adding a duplicate, invalid block.

46 changes: 45 additions & 1 deletion robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,47 @@
# Allow all crawlers
# ─────────────────────────────────────────────
# LLMtop — TUI Dashboard for Local LLMs: SEO & Crawler Directives
# ─────────────────────────────────────────────

# Sitemap
Sitemap: https://raw.githubusercontent.com/weby-homelab/LLMtop/master/sitemap.xml
Comment on lines +5 to +6

@gemini-code-assist gemini-code-assist Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Search engines only look for robots.txt and sitemaps at the root of the domain (e.g., https://github.com/robots.txt). They will not fetch or respect a robots.txt or sitemap.xml located in a repository subdirectory or repository root on GitHub. If these files are intended for a custom website hosting this project, they should be served from the root of that custom domain, and the URLs should reference that domain instead of github.com.


# ─── General Rules ───────────────────────────
User-agent: *
Allow: /
Crawl-delay: 10

# ─── Search-Specific Bots ────────────────────
User-agent: Googlebot
Allow: /
Crawl-delay: 5
Comment on lines +14 to +16

@gemini-code-assist gemini-code-assist Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Googlebot does not support or respect the Crawl-delay directive. It is recommended to remove this directive for Googlebot to avoid clutter and confusion.

User-agent: Googlebot
Allow: /


User-agent: Bingbot
Allow: /
Crawl-delay: 10

User-agent: DuckDuckBot
Allow: /
Crawl-delay: 10

# ─── AI Training Crawlers (Opt-Out) ──────────
User-agent: GPTBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: FacebookBot
Disallow: /

# ─── Documentation Crawlers (Allowed) ────────
User-agent: ReadableBot
Allow: /

User-agent: Diffbot
Allow: /
19 changes: 19 additions & 0 deletions sitemap.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
Comment on lines +2 to +3

@gemini-code-assist gemini-code-assist Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The xmlns:xhtml namespace is declared here, and the PR description mentions 'hreflang alternates', but there are no xhtml:link elements or hreflang alternates defined in this sitemap. If internationalization alternates are not needed, you should remove the unused xmlns:xhtml namespace declaration.

Suggested change
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<url>
<loc>https://github.com/weby-homelab/LLMtop</loc>
<changefreq>weekly</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://github.com/weby-homelab/LLMtop/blob/master/README.md</loc>
<changefreq>weekly</changefreq>
<priority>0.9</priority>
</url>
<url>
<loc>https://github.com/weby-homelab/LLMtop/releases</loc>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
Loading