Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/CATALOG_INGEST.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,26 @@ bundle 比上次少了大量 entry。检查上游是不是改了某个 source
| 代码量 | 540 行 fake-git/fake-registry 逻辑 | 一个 CatalogIngestService(~830 行带详尽注释) |
| 增量同步 | 全文件 walk + glob | 按 manifest entry 逐条对账,content_hash 命中即 skip |

## description 多语言(i18n)

上游 `catalog/index.json` 的每条 entry 同时携带三个描述字段:

- `description` — LLM 规范化的英文版本(ingest 默认消费)
- `description_zh` — 中文翻译(≈99.8% 覆盖率)
- `description_original` — 源仓库原文(暂不入库)

ingest 写入 `capability_items.descriptions jsonb` 列(同时给 `capability_versions` 写一份),shape 为 `{"en":"...","zh":"..."}`。`capability_items.description text` 列保留作为向后兼容的英文默认值(embedding 向量也由它生成)。

API 侧 `ItemResponse.description` 字段会按请求 locale resolve:

- 优先级:`?lang=<value>` query > `Accept-Language` header > 默认 `en`
- 归一化:`zh-CN` / `zh-TW` / `zh-Hant` / `zh-Hans` → `zh`;`en-*` → `en`;其它 locale 落回 `en`
- 同时在响应 JSON 中暴露原始 `descriptions: {"en":"...","zh":"..."}` 对象,前端切 locale 时无需重新请求

前端通过 `pickItemDescription(item, locale)` helper 消费(落回顺序与服务端一致)。

新增字段细节见 openspec change:`openspec/changes/add-item-description-i18n/`。

## 相关代码与文档

- 下游 service:`server/internal/services/catalog_ingest_service.go`
Expand Down
1 change: 1 addition & 0 deletions server/cmd/migrate/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"github.com/costrict/costrict-web/server/internal/services"
"github.com/costrict/costrict-web/server/internal/team"
migrations "github.com/costrict/costrict-web/server/migrations"
"github.com/google/uuid"
"github.com/pressly/goose/v3"
"gorm.io/gorm"
)
Expand Down
1 change: 1 addition & 0 deletions server/cmd/migrate/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ func newMigrateTestDB(t *testing.T) *gorm.DB {
item_type TEXT NOT NULL,
name TEXT NOT NULL,
description TEXT,
descriptions TEXT NOT NULL DEFAULT '{}',
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Keep test schema parity for capability_versions.descriptions.

Line 48 updates capability_items, but newMigrateTestDB still creates capability_versions without descriptions. That drift can hide/introduce migration-path failures when version rows are persisted through model paths.

🧪 Suggested schema parity patch
 		`CREATE TABLE IF NOT EXISTS capability_versions (
 			id TEXT PRIMARY KEY,
 			item_id TEXT NOT NULL,
 			revision INTEGER NOT NULL,
+			descriptions TEXT NOT NULL DEFAULT '{}',
 			content TEXT NOT NULL,
 			content_md5 TEXT DEFAULT '',
 			metadata TEXT DEFAULT '{}',
 			commit_msg TEXT,
 			created_by TEXT NOT NULL,
 			created_at DATETIME
 		)`,
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
descriptions TEXT NOT NULL DEFAULT '{}',
`CREATE TABLE IF NOT EXISTS capability_versions (
id TEXT PRIMARY KEY,
item_id TEXT NOT NULL,
revision INTEGER NOT NULL,
descriptions TEXT NOT NULL DEFAULT '{}',
content TEXT NOT NULL,
content_md5 TEXT DEFAULT '',
metadata TEXT DEFAULT '{}',
commit_msg TEXT,
created_by TEXT NOT NULL,
created_at DATETIME
)`,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/cmd/migrate/main_test.go` at line 48, The test DB schema created by
newMigrateTestDB is missing the descriptions column on capability_versions while
capability_items was updated; update newMigrateTestDB to add "descriptions TEXT
NOT NULL DEFAULT '{}'" to the CREATE TABLE for capability_versions so the test
schema matches the production change to capability_items.descriptions, ensuring
migration tests exercise the same column shape for version rows.

category TEXT,
version TEXT DEFAULT '1.0.0',
content TEXT,
Expand Down
25 changes: 15 additions & 10 deletions server/internal/handlers/capability_item.go
Original file line number Diff line number Diff line change
Expand Up @@ -482,7 +482,8 @@ type ItemResponse struct {
Slug string `json:"slug"`
ItemType string `json:"itemType"`
Name string `json:"name"`
Description string `json:"description"`
Description string `json:"description"` // Resolved per `?lang=` query or Accept-Language header; en is the default. Use `descriptions` for raw locale map.
Descriptions datatypes.JSON `json:"descriptions" swaggertype:"object"` // Raw locale → text map, e.g. {"en":"...","zh":"..."}.
Category string `json:"category"`
Version string `json:"version"`
Content string `json:"content"`
Expand Down Expand Up @@ -594,21 +595,23 @@ func reconcileItemCurrentRevision(db *gorm.DB, item *models.CapabilityItem) {
Update("current_revision", latestRevision).Error
}

func buildItemResponse(db *gorm.DB, item models.CapabilityItem, userID string) ItemResponse {
func buildItemResponse(c *gin.Context, db *gorm.DB, item models.CapabilityItem, userID string) ItemResponse {
reconcileItemCurrentRevision(db, &item)
if TagSvc != nil && item.ID != "" && len(item.Tags) == 0 {
if tagsMap, err := TagSvc.GetItemTags([]string{item.ID}); err == nil && tagsMap != nil {
item.Tags = tagsMap[item.ID]
}
}
locale := ResolveLocale(c)
resp := ItemResponse{
ID: item.ID,
RegistryID: item.RegistryID,
RepoID: item.RepoID,
Slug: item.Slug,
ItemType: item.ItemType,
Name: item.Name,
Description: item.Description,
Description: PickDescription(item.Descriptions, item.Description, locale),
Descriptions: item.Descriptions,
Category: item.Category,
Version: item.Version,
Content: item.Content,
Expand Down Expand Up @@ -837,6 +840,7 @@ func ListItems(c *gin.Context) {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to fetch items"})
return
}
ResolveItemListLocale(c, items)
c.JSON(http.StatusOK, gin.H{"items": items, "total": total, "page": page, "pageSize": pageSize, "hasMore": int64((page-1)*pageSize+pageSize) < total})
}

Expand Down Expand Up @@ -924,7 +928,7 @@ func CreateItem(c *gin.Context) {
CategorySvc.EnsureCategory(req.Category, req.CreatedBy)
}

c.JSON(http.StatusCreated, buildItemResponse(db, *item, c.GetString(middleware.UserIDKey)))
c.JSON(http.StatusCreated, buildItemResponse(c, db, *item, c.GetString(middleware.UserIDKey)))
}

// GetItem godoc
Expand All @@ -945,7 +949,7 @@ func GetItem(c *gin.Context) {
c.JSON(http.StatusNotFound, gin.H{"error": "Item not found"})
return
}
c.JSON(http.StatusOK, buildItemResponse(db, item, c.GetString(middleware.UserIDKey)))
c.JSON(http.StatusOK, buildItemResponse(c, db, item, c.GetString(middleware.UserIDKey)))
}

// ListItemAssets godoc
Expand Down Expand Up @@ -1206,7 +1210,7 @@ func (h *ItemHandler) updateItemFromJSON(c *gin.Context) {
}
}

c.JSON(http.StatusOK, buildItemResponse(db, item, c.GetString(middleware.UserIDKey)))
c.JSON(http.StatusOK, buildItemResponse(c, db, item, c.GetString(middleware.UserIDKey)))
}

// updateItemFromArchive handles multipart/form-data archive upload item update.
Expand Down Expand Up @@ -1336,7 +1340,7 @@ func (h *ItemHandler) updateItemFromArchive(c *gin.Context) {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update item"})
return
}
c.JSON(http.StatusOK, buildItemResponse(db, item, c.GetString(middleware.UserIDKey)))
c.JSON(http.StatusOK, buildItemResponse(c, db, item, c.GetString(middleware.UserIDKey)))
return
}

Expand Down Expand Up @@ -1516,7 +1520,7 @@ func (h *ItemHandler) updateItemFromArchive(c *gin.Context) {
}()
}

c.JSON(http.StatusOK, buildItemResponse(db, item, c.GetString(middleware.UserIDKey)))
c.JSON(http.StatusOK, buildItemResponse(c, db, item, c.GetString(middleware.UserIDKey)))
}

// DeleteItem godoc
Expand Down Expand Up @@ -1841,6 +1845,7 @@ func ListAllItems(c *gin.Context) {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to fetch items"})
return
}
ResolveItemListLocale(c, items)

// Collect unique repo IDs from preloaded registries and batch-fetch repo names
repoIDSet := make(map[string]bool)
Expand Down Expand Up @@ -2123,7 +2128,7 @@ func (h *ItemHandler) createItemFromJSON(c *gin.Context) {
}
}

c.JSON(http.StatusCreated, buildItemResponse(h.db, *item, c.GetString(middleware.UserIDKey)))
c.JSON(http.StatusCreated, buildItemResponse(c, h.db, *item, c.GetString(middleware.UserIDKey)))
}

// cleanupStorageKeys deletes uploaded objects after a later step fails.
Expand Down Expand Up @@ -2388,7 +2393,7 @@ func (h *ItemHandler) createItemFromArchive(c *gin.Context) {
}

enqueueScanAsync(item.ID, 1, "create")
c.JSON(http.StatusCreated, buildItemResponse(h.db, *item, c.GetString(middleware.UserIDKey)))
c.JSON(http.StatusCreated, buildItemResponse(c, h.db, *item, c.GetString(middleware.UserIDKey)))
}

// MoveItem godoc
Expand Down
2 changes: 2 additions & 0 deletions server/internal/handlers/capability_registry.go
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,8 @@ func ListMyItems(c *gin.Context) {
repoVisibilityMap["public"] = "public"
}

ResolveItemListLocale(c, items)

// Build response with repo info
result := make([]MyItem, len(items))
for i, item := range items {
Expand Down
116 changes: 116 additions & 0 deletions server/internal/handlers/locale.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
package handlers

import (
"encoding/json"
"strings"

"github.com/costrict/costrict-web/server/internal/models"
"github.com/gin-gonic/gin"
"gorm.io/datatypes"
)

// DefaultLocale is the fallback when neither `?lang=` nor `Accept-Language`
// yields a recognized locale, and the safe choice for "no header" callers
// (matches the pre-i18n API behavior so external consumers see no change).
const DefaultLocale = "en"

// ResolveLocale picks a locale tag from the request. Priority:
//
// 1. `?lang=` query parameter (highest)
// 2. `Accept-Language` header — first language tag
// 3. DefaultLocale ("en")
//
// Returned value is normalized to the primary subtag (zh-CN → zh, en-US → en);
// unrecognized locales fall back to DefaultLocale.
func ResolveLocale(c *gin.Context) string {
if c == nil {
return DefaultLocale
}
if q := strings.TrimSpace(c.Query("lang")); q != "" {
return normalizeLocale(q)
}
if h := strings.TrimSpace(c.GetHeader("Accept-Language")); h != "" {
// Accept-Language: zh-CN,en;q=0.9 → take "zh-CN" (first tag), ignore q values.
first := strings.SplitN(h, ",", 2)[0]
first = strings.SplitN(first, ";", 2)[0]
return normalizeLocale(first)
}
return DefaultLocale
}

// normalizeLocale collapses a BCP-47-ish tag down to the primary subtag we
// actually serve. Upstream currently ships only en + zh translations; any
// other language (ja/ko/de/…) falls back to en until upstream produces a
// matching `description_<locale>` field.
func normalizeLocale(raw string) string {
if raw == "" {
return DefaultLocale
}
primary := strings.ToLower(strings.SplitN(raw, "-", 2)[0])
primary = strings.SplitN(primary, "_", 2)[0]
switch primary {
case "zh":
return "zh"
case "en":
return "en"
default:
return DefaultLocale
}
}

// PickDescription resolves a single localized description string from the
// per-item descriptions JSONB map, falling back gracefully to keep
// pre-i18n rows readable.
//
// Order:
//
// 1. descriptions[locale] if non-empty
// 2. descriptions[DefaultLocale] if non-empty
// 3. fallbackText (the legacy capability_items.description column)
// 4. ""
func PickDescription(descriptions datatypes.JSON, fallbackText string, locale string) string {
if len(descriptions) > 0 {
var m map[string]string
if err := json.Unmarshal(descriptions, &m); err == nil {
if v := m[locale]; v != "" {
return v
}
if v := m[DefaultLocale]; v != "" {
return v
}
}
}
return fallbackText
}

// ResolveItemListLocale rewrites the `Description` field of each item in
// place to the locale-resolved value, so list endpoints that serialize
// raw `[]models.CapabilityItem` (or types embedding it) still honor
// Accept-Language without needing to wrap every row in ItemResponse.
// The original `descriptions` JSONB stays on the row so frontends can
// re-resolve on locale switch without re-fetching.
func ResolveItemListLocale(c *gin.Context, items []models.CapabilityItem) {
if len(items) == 0 {
return
}
locale := ResolveLocale(c)
for i := range items {
items[i].Description = PickDescription(items[i].Descriptions, items[i].Description, locale)
}
}

// ResolveCapabilityItemPointersLocale handles the case where the caller has
// pointers (e.g. inside a result struct field) and wants in-place rewrite
// without exposing the full slice surface to ResolveItemListLocale.
func ResolveCapabilityItemPointersLocale(c *gin.Context, items []*models.CapabilityItem) {
if len(items) == 0 {
return
}
locale := ResolveLocale(c)
for _, it := range items {
if it == nil {
continue
}
it.Description = PickDescription(it.Descriptions, it.Description, locale)
}
}
Loading
Loading