From 6ee872acae0228b6ba8b29b4681f653f1db66e83 Mon Sep 17 00:00:00 2001 From: Weijia Sun Date: Sat, 12 Sep 2026 00:09:06 +0800 Subject: [PATCH 1/4] docs(skills): define desktop offline recovery semantics --- .../decisions.md | 131 ++++++++++++++++++ .../spec.md | 21 ++- 2 files changed, 147 insertions(+), 5 deletions(-) diff --git a/docs/specs/2026-09-08-desktop-skill-capabilities/decisions.md b/docs/specs/2026-09-08-desktop-skill-capabilities/decisions.md index c1c2678b0..3d0ccac99 100644 --- a/docs/specs/2026-09-08-desktop-skill-capabilities/decisions.md +++ b/docs/specs/2026-09-08-desktop-skill-capabilities/decisions.md @@ -56,3 +56,134 @@ | 完整缓存增长 | 记录容量/下载错误；不未经授权清理在用或用户内容。 | 所有源码审计均为固定提交证据。文档定稿不表示实现、CI、评审、合并、部署或实机验证已经通过。 + +## 现场验收补充决策 + +### 术语边界 + +- **Desktop Device Offline**：当前 Desktop Runtime 没有活跃设备可接入，由 Bot `status=OFFLINE` 或可信的 BaaS 结构化 `NO_ACTIVE_DEVICES` 事实表达。它不是 Skill 资产下线，也不是 Runtime 代码错误。 +- **Transient Runtime Failure**：无法证明设备离线的 timeout、5xx、非法响应或连接故障；当前 Task 仍可按故障策略退避重试。 +- **Public Recoverability**：公开 `RuntimeProjectionIssue.retryable` 表达未来再次投影是否可能恢复，不表达当前 Queue Task 应继续执行。 +- **Recovery Continuation**：Desktop Recovery 深模块对当前 Task 的内部调度决策，包括等待事件后结束、故障退避、进度跟进和永久停止；不从公开 `retryable` 直接推导。 + +### R1. 明确离线与暂时通信故障分离 + +2026-09-11 预发发现：保留历史 Binding 的明确 OFFLINE Desktop 被恢复扫漏持续拉起，BaaS 返回 `NO_ACTIVE_DEVICES` 后又被泛化为可重试的 `SKILL_RUNTIME_UNAVAILABLE`。因此补充下列领域分类： + +- Bot 当前状态明确为 `OFFLINE`，或 BaaS 明确返回结构化 `NO_ACTIVE_DEVICES`，均表示 **Desktop Device Offline**：这是等待外部上线事件的正常状态，不是当前任务可通过高频重试修复的 Runtime 故障。 +- timeout、5xx、非法响应及无法确认设备状态的连接异常仍表示 **Transient Runtime Failure**，继续允许 TaskQueue 使用既有故障退避。 +- Device Offline 对外保持 Runtime 尚未收敛的 `PENDING` 事实，并保持 `retryable=true`，表示未来再次投影可以解决；该字段不决定当前 Queue Task 的调度。当前任务结束，后续由设备上线/重连事件重新确保同一 Bot 恢复，低频扫漏只负责漏事件兜底。 +- 明确离线不得记录为 Engine apply 失败，也不得产生 ERROR traceback；具体 Sweeper/Handler 调度与日志级别在后续 R2/R3 决策中确定。 + +该补充不改变 Installation Desired State，不回滚已提交的 SkillSet/Direct 变更，也不新增 Desktop 专属业务拒绝或第二套恢复任务。 + +### R2. 明确离线终止当前恢复任务 + +- Sweeper 只排除明确 `status=OFFLINE`，不收窄成仅扫描 `ACTIVE`，避免误伤仍可能具备设备连接的过渡状态。 +- Handler 在开始执行和正式 apply 前都重新读取 Bot；若此时已明确 OFFLINE，当前任务返回 `Complete`，不访问 Engine、不继续高频 Retry。 +- 执行期间 BaaS 明确返回 `NO_ACTIVE_DEVICES` 时同样结束当前任务。不能用十分钟 `Reschedule` 长期占用 live key，因为重复 `ensure` 不会提前既有任务的 `run_at`，会延迟真实重连后的 due-now 恢复。 +- 设备上线/重连事件重新 `ensure` 当前 Bot；若事件丢失，Bot 状态恢复为非 OFFLINE 后由下一轮约十分钟 Sweeper 兜底。 + +### R3. 明确离线的 Runtime 结果合同 + +明确离线使用稳定问题码 `DESKTOP_DEVICE_OFFLINE`：Runtime `status=PENDING`、该问题 `retryable=true`，原因表达“能力状态已保存，将在设备上线后自动同步”，建议动作是启动或重新连接 Desktop 客户端。`retryable=true` 遵守公开 DTO 的原义：未来一次投影可能解决；它不授权当前 Task 高频重试。 + +该结果继续与成功提交的 Desired State 一起返回，不改变公开 HTTP 结构、不回滚 Installation；不能再用通用 `SKILL_RUNTIME_UNAVAILABLE` 把明确离线误报成平台或 Engine 故障。 + +### R4. 离线恢复日志分级 + +- Sweeper 跳过明确 OFFLINE 时不逐 Bot 记录日志，只保留轮次汇总 INFO/指标。 +- Handler 在执行前识别明确 OFFLINE：INFO，无 traceback。 +- BaaS 返回结构化 `NO_ACTIVE_DEVICES`：若 DB 同为 OFFLINE 则 INFO；若 DB 仍显示在线则 WARN，表达短暂状态不一致；两者均不打印异常 traceback。 +- timeout、5xx、连接中断等状态未知的外部异常：WARN，并保留 TaskQueue 故障退避；避免在多层重复打印相同 traceback。 +- 非法响应、身份/绑定不变量破坏或已进入 Engine 后的 apply 异常：ERROR，并保留 traceback。 + +日志通用原则为：正常业务状态用 INFO，可恢复外部异常用 WARN，代码错误、合同破坏和不变量失败用 ERROR。明确离线必须在进入 `PerDomainRuntimeProjection` 的通用 ERROR 分支前转成结构化 Runtime 结果。 + +### R5. 自然重连使用 Runtime Reprojection 事件 + +Desktop 健康扫描确认 `OFFLINE → ACTIVE` 并提交状态更新后，发布既有 `RuntimeProjectionRequestedEvent`。不复用只表示首次激活的 `DeviceActivatedEvent`，避免重复触发首次初始化消费者；健康扫描也不直接依赖 Skill Recovery Service。`SkillSymlinkListener` 继续把该事件交给统一 `DesktopSkillRecoveryService.ensure()`，创建 due-now 工作。 + +事件发布失败不得回滚已经确认的在线状态，但必须可诊断；低频 Sweeper 在 Bot 已恢复为非 OFFLINE 后承担漏事件兜底。 + +### R6. Desktop Skill Recovery 状态资格 + +- `ACTIVE` 与 `PENDING` 可进入恢复。PENDING 请求若 BaaS 明确无设备，则结束当前任务并等待首次激活事件。 +- `OFFLINE`、`FAILED`、`RELEASING`、`RELEASED` 不创建高频恢复工作；已有 Handler 观察到这些状态时返回 `Complete`。 +- 未知状态不静默视为 ACTIVE：记录 WARN 并跳过/结束本轮。 + +资格判断收敛在 Recovery 深模块的单一内部规则，由 `ensure()`、Sweeper eligible 统计和 Handler 开始/正式 apply 前的二次复核共同使用；各业务入口不得重复状态判断。 + +### R7. NO_ACTIVE_DEVICES 的结构化边界转换 + +- BaaS Service 负责解析 HTTP/JSON，保留结构化 `status_code` 与 `error_code=NO_ACTIVE_DEVICES`，不把它压成只能匹配的字符串。 +- `BaasConnInfoBuilder` 把 provider-specific 错误转成 provider-neutral `DeviceOfflineError`；其它权限、绑定、协议和连接失败继续使用既有错误语义。 +- `PerDomainRuntimeProjection` 在通用异常分支前捕获 `DeviceOfflineError`，生成 R3 的 `DESKTOP_DEVICE_OFFLINE / PENDING / retryable=true`；当前任务是否继续由 R16 的内部 continuation 决定。 +- Recovery Handler 只消费结构化 Runtime issue，不感知 BaaS 响应文本或 provider 实现。 + +BaaS 的明确无设备记录 INFO；PerDomain 若 DB 仍显示 ACTIVE 则记录单行 WARN 表达状态竞态，若 DB 已 OFFLINE 则 INFO，均不打印 traceback。不得将整个 `ConnInfoBuildError` 降级，因为其中仍包含真正的合同和基础设施故障。 + +### R8. 状态提交后发布 best-effort 重投影事件 + +健康扫描确认 BaaS 在线后，先更新 Bot/Binding 为 ACTIVE，再发布 `RuntimeProjectionRequestedEvent`。Listener 按事件 `binding_id` 重新验证当前绑定，避免迟到事件投影到旧设备。事件继续使用非 required 的进程内发布；失败记录错误但不回滚在线状态、不增加 outbox，由后续健康扫描和 ACTIVE Bot Sweeper 兜底。 + +### R9. 多 Pod Sweeper 不新增分布式锁 + +允许多个 Backend Pod 重复执行低频扫描，继续依赖 Queue 的 `env + app + task_type + owner/bot` live-key 保证同一 Bot 至多一条 live recovery task。OFFLINE 过滤消除主要风暴后，本期接受少量重复 SELECT/ensure；只有实际指标证明扫描成为瓶颈时再单独优化，不为本修复引入锁生命周期。 + +### R10. 聚合观测优先于逐 Bot 正常日志 + +每轮 Sweeper 记录聚合 INFO/metric：`scanned`、`eligible`、`skipped_offline`、`skipped_terminal`、`skipped_unknown_status`、`task_created`、`task_joined_existing`、`ensure_failed`、`duration_ms`。普通 OFFLINE 不逐 Bot 记录；状态竞态、未知状态和持久化失败才带 owner/bot/binding 形成单项日志。 + +Task outcome/监控使用稳定原因分类，如 `DEVICE_OFFLINE_WAITING_FOR_RECONNECT`、`TRANSIENT_RUNTIME_FAILURE`、`POOL_TRANSITION_WAITING`、`CENTER_CONTENT_DOWNLOADING`，不得解析异常文本聚合。 + +### R11. 重连唤醒不升级为强一致事件事务 + +最终保证由“已提交在线状态 + best-effort Runtime Reprojection event + Queue 持久任务 + ACTIVE Bot 低频 Sweep”组合提供。事件 handler 不设 required，不把 Skill enqueue 失败反向耦合为健康状态更新失败；事件失败保留 ERROR，Sweep 提供最终兜底，但不宣传固定恢复 SLA。 + +### R12. BaaS 明确无设备使用专属结构化错误 + +新增 `BaasNoActiveDevicesError` 作为 `BaasServiceError` 的窄子类，保留 `status_code` 与稳定 `error_code=NO_ACTIVE_DEVICES`；不扩大所有既有 `BaasServiceError` 的构造协议，也不把 HTTP Response 透传到 Runtime。`BaasConnInfoBuilder` 将其转换为 `device_context.py` 中 provider-neutral 的 `DeviceOfflineError`，其余 BaaS/Binding/协议错误继续走现有失败类型。 + +### R13. Recovery 使用内部三态资格分类 + +不复用回答“Bot 是否可对外服务”的 `is_bot_ready()`。Desktop Recovery 内部定义纯分类 `DesktopRecoveryDisposition`：`RUN`、`WAIT_FOR_DEVICE`、`STOP`。ACTIVE/PENDING 映射 RUN，OFFLINE 映射 WAIT_FOR_DEVICE，FAILED/RELEASING/RELEASED/未知映射 STOP（未知同时 WARN）。`ensure()`、Sweeper 和 Handler 二次复核共用该规则；WAIT/STOP 都结束已有任务但分别计量。 + +### R14. 自然重连事件由 DesktopBotService 发布 + +`DesktopBotService._apply_decision()` 完成 `OFFLINE → ACTIVE` 状态更新后，通过窄 helper 重新读取当前 Binding 并发布已有 `RuntimeProjectionRequestedEvent`。不在 Health Scanner 拼 Event、不创建第二种 Event DTO、不直接调用 Skill Recovery，也不对 `ACTIVE → ACTIVE` 心跳重复发布。部分状态更新或迟到事件由 Listener current-binding 校验和 Recovery disposition 安全拒绝，下一轮扫描继续收敛。 + +### R15. 正确性修复不新增行为开关 + +不新增 `skip_offline`、日志级别或重连事件开关，避免产生只修一半的配置组合。现有 `desktop_skill_recovery.enabled` 仍只作为整套 Sweeper 的紧急控制；明确离线分类在所有环境一致生效。发布以预发任务/日志下降和真实重连自动恢复为门禁，回退使用代码版本回滚。 + +### R16. 可恢复性与当前任务调度分离 + +公开 `RuntimeProjectionIssue.retryable` 只回答“未来再次投影是否可能解决”，不能被 Backend 或前端解释成“当前 Queue Task 应立即重试”。Desktop Skill Recovery 内部使用独立 `RecoveryContinuation` 分类： + +- `DESKTOP_DEVICE_OFFLINE`：`COMPLETE_WAITING_FOR_EVENT`；公开仍为 PENDING/retryable=true。 +- timeout、5xx 等状态未知的暂时故障：`RETRY_WITH_BACKOFF`。 +- Center 内容正常准备或下载中：`RESCHEDULE_FOR_PROGRESS`，维持五秒进度跟进。 +- 永久包/合同问题：`COMPLETE_PERMANENT`，保留 DEGRADED/non-retryable。 + +该修订不增加公开字段或 code 枚举，不改变 Gateway schema。现有调用方可继续只处理 PENDING；前端可选识别 `DESKTOP_DEVICE_OFFLINE` 优化文案，但不得仅因 `retryable=true` 做秒级命令重放。 + +### R17. 只按可信结构化错误识别离线 + +仅在可信 BaaS 非2xx响应可解析为对象且 `detail.error` 精确等于 `NO_ACTIVE_DEVICES` 时进入专属离线分支；不匹配异常字符串，不要求固定为单一HTTP状态。401/403、其它code、坏JSON及缺字段继续走普通错误。现网404和历史约定503都需测试，避免状态码演进重新制造错误风暴。 + +### R18. 测试必须覆盖完整转换与竞态 + +修复测试需贯通 BaaS解析、Builder转换、PerDomain结果/日志、Recovery资格、Sweeper、Handler两次状态复核、既有live任务排空和自然重连事件。必须证明其它4xx/5xx/timeout仍按原故障语义、Center正常下载仍五秒跟进，并分别用OpenClaw/Hermes实机验证离线不访问Engine和上线后无需用户二次操作即可恢复。 + +### R19. 预发以任务风暴停止和真实重连为门禁 + +部署后允许既有任务执行一轮并Complete；稳定后明确OFFLINE样本不再新增高频恢复任务、不再由恢复链请求BaaS ws-info、不再产生PerDomain ERROR traceback。OpenClaw/Hermes各验证一次OFFLINE→ACTIVE事件、due-now任务、Engine apply、正确exact软链及可读SKILL.md时间线。5xx/timeout退避和Desired State不回滚同时回归；不承诺固定端到端秒数。 + +### R20. 文档落点与ADR边界 + +更新正式Spec D8/D9/Testing、本文决策账本、统一验收Issue #2105，并在实现PR同步Skill Center AGENTS/README的已实现行为。当前不新建ADR：这是既有Desktop Recovery合同的现场修订，Spec与决策账本已经是权威记录；避免产生第三份重复事实源。 + +### R21. 一个Avernet修复PR，OCB只做集成gitlink + +BaaS结构化错误、provider-neutral错误、Runtime结果/日志、Recovery资格/调度、自然重连事件、测试和实现文档必须作为一个Avernet原子切片交付，避免只修一半的部署组合。不新增OCB功能实现；Avernet合入后通过统一gitlink更新和Corp集成测试交付。公开DTO形状不变、code为自由字符串，因此不重新生成Gateway OpenAPI artifact。 diff --git a/docs/specs/2026-09-08-desktop-skill-capabilities/spec.md b/docs/specs/2026-09-08-desktop-skill-capabilities/spec.md index 1b728e235..838671eb8 100644 --- a/docs/specs/2026-09-08-desktop-skill-capabilities/spec.md +++ b/docs/specs/2026-09-08-desktop-skill-capabilities/spec.md @@ -1,6 +1,6 @@ # Desktop Skill 能力适配 Spec -版本：1.0，2026-09-09。状态：总体设计已定稿；本文为Q1–Q29最终决定的规范化整理，功能尚未实现、测试或部署。 +版本：1.1，2026-09-11。状态：G1–G4 已实现并合入 Avernet dev；统一集成和实机验收由 #2105 跟踪。本文补充预发发现的明确离线恢复语义，修复和部署状态仍须独立记录。 Owning Module：Skill Center。参与方：Avernet Backend/Engine/BaaS契约与OCB企业装配、Desktop客户端及Engine Adapter。目标分支：两仓`dev`。 @@ -171,7 +171,9 @@ Mounted保持原挂载及检查行为，不下载、不修mount、不写只读 **调度身份**：Queue既有`env/app/task_type`作用域加`owner_id + bot_id`；同Bot至多一条live工作项，不按入口、Skill、Reference或版本拆长期任务。payload只承载逻辑目标及必要控制信息，不存签名URL、冻结版本清单或待重放的激活动作。 -**入口**：正式Set/Direct变更收尾（含Reference最终add）、共同投影、启动/重连及Track Latest都请求同一恢复Interface。Bot未就绪或snapshot失败导致未进入Engine的提前PENDING，也要进入这一共同机制。inactive SKIPPED不是下载理由，业务权限/DB失败不是重试历史命令的授权。 +**入口**：正式Set/Direct变更收尾（含Reference最终add）、共同投影、启动/重连及Track Latest都请求同一恢复Interface。Bot未就绪或snapshot失败导致未进入Engine的提前PENDING，也要进入这一共同机制。inactive SKIPPED不是下载理由，业务权限/DB失败不是重试历史命令的授权。Desktop健康扫描确认`OFFLINE → ACTIVE`并提交状态后发布既有`RuntimeProjectionRequestedEvent`；不重放只表示首次激活的`DeviceActivatedEvent`，也不让健康扫描直接调用Skill服务。 + +**恢复资格**：Recovery内部使用独立三态分类，不复用回答产品可用性的`is_bot_ready()`。ACTIVE/PENDING为RUN，OFFLINE为WAIT_FOR_DEVICE，FAILED/RELEASING/RELEASED及未知状态为STOP（未知需告警）。`ensure()`、Sweeper和Handler执行前/正式apply前共用同一规则；WAIT/STOP不创建或继续高频任务。自然重连事件创建due-now任务，事件丢失时由后续RUN状态低频扫描兜底。 **一次执行**：核对目标仍适用及当前绑定；Reader读取当前有效身份并解析精确版本；后台逐项准备尚缺派生包，单项准备失败不能阻断其它可执行项；重工作结束后重新读取最新期望，再调用共同Projector的Skill scope及Engine apply；按逐项结果决定本行Queue outcome。已退出的目标不继续安排下载，迟到内容不能重新激活；任务自身不再enqueue另一条任务来代替Reschedule。 @@ -179,19 +181,24 @@ Mounted保持原挂载及检查行为，不下载、不修mount、不写只读 | --- | --- | | 新建due-now工作 | 使用现有wake_on_enqueue及worker领取，首轮不等待扫漏。 | | 内容正常准备/下载中 | `Reschedule(5s)`，不因正常等待指数退避。 | -| 网络/设备暂不可用 | 使用既有`Retry`退避；超时不能当已确认在下载。 | +| 明确OFFLINE或BaaS结构化`NO_ACTIVE_DEVICES` | 当前Task `Complete`并释放live key，等待首次激活/重连事件；不访问或继续轮询Engine。 | +| timeout、5xx、非法响应等状态未知故障 | 使用既有`Retry`退避；超时不能当已确认离线或正在下载。 | | 混合DEGRADED/PENDING | 看逐项可恢复工作，永久问题不能遮住其它等待项。 | | Skill域全部完成 | 结束快速跟进；不表示MCP/Passport全部完成。 | | 只剩不可自动解决的问题 | 停止该问题高频重试，保留具体DEGRADED，不能伪报成功。 | | 单轮达到30分钟deadline | 按现有Queue终态/释放live key机制处理；不回滚DB、不删除完整缓存。 | -| 漏唤醒、过期或较长离线 | 约10分钟分页扫漏，幂等ensure同一类任务；不在扫描线程下载/投影。 | +| 漏唤醒、过期或RUN状态恢复 | 约10分钟分页扫漏，幂等ensure同一类任务；明确OFFLINE/终态跳过，不在扫描线程下载/投影。 | -兜底不能只查现存任务或DB ACTIVE/状态跳变；必要时对目标范围内存活且有绑定的Desktop幂等再核对。允许已有完成Bot低频核对，不新增同步进度表。现有健康扫描只更新状态，不能偷换其dry-run/白名单保护或视为已经触发投影。Pool过渡期继续尊重既有迁移对映射的所有权，不让新任务抢写。 +兜底不能只查现存任务；按统一Recovery资格对RUN状态且有当前绑定的Desktop幂等再核对。明确OFFLINE、终态和未知状态不进入高频恢复，PENDING仍保留启动期资格。允许已有完成Bot低频核对，不新增同步进度表。健康扫描保持原dry-run/白名单保护，只在实际提交`OFFLINE → ACTIVE`后发布best-effort Runtime重投影事件。Pool过渡期继续尊重既有迁移对映射的所有权，不让新任务抢写。 5秒是下一次可领取时间；10分钟是扫描周期，均不是端到端SLA。deadline不是取消正在运行Handler的硬超时，租约续期不延长它。重复enqueue不更新payload、不提前run_at、不延deadline，亦不是所有前台命令的分布式锁。 保留现有入队非事务窗口，以及最后一次读取至Complete之间出现新期望的窗口，由已定扫漏兜底。不得因此新增outbox、generation表、任务进度表或修改通用Queue基础设施。入队失败必须保留真实错误/日志，不能声称持久恢复已保证。 +**明确离线错误合同**：BaaS层只根据可信非2xx JSON的精确`detail.error=NO_ACTIVE_DEVICES`产生窄结构化错误，不匹配异常字符串；HTTP状态可以是当前观察到的404或历史合同中的503。Builder将其转换为provider-neutral `DeviceOfflineError`，Runtime再返回`DESKTOP_DEVICE_OFFLINE / PENDING / retryable=true`。这里`retryable`仅表示未来投影可能恢复，不决定当前Task；Recovery内部将其分类为等待事件并`Complete`。401/403、其它code、坏JSON、timeout、5xx和普通`ConnInfoBuildError`不得误归为离线。 + +**日志与多Pod**：正常OFFLINE不逐Bot记录，Sweeper每轮输出扫描/资格/跳过/创建/加入既有任务/失败/耗时聚合；DB仍为ACTIVE但BaaS明确无设备时记录单行WARN，无traceback，DB已OFFLINE则INFO。普通网络故障WARN并退避，非法合同/不变量/Engine apply异常保持ERROR和traceback。多个Pod可重复低频扫描，继续依赖Queue live-key去重，不新增Sweeper分布式锁。 + ### D9. 版本切换、退出和MCP - V2未Ready时保留V1有效链接；Ready后由基于最新Reader的正式apply切换。Engine下载完成回调只发布缓存，不重放旧激活。 @@ -219,12 +226,16 @@ Desktop既有产品升级门禁继续使用，不新增Backend最低版本表。 公开Router/DTO如实际变更，必须从Backend正式实现生成Gateway产物并执行合同检查；不能只修改生成JSON或只更新前端说明。签名扩展为内部Engine合同，不直接成为前端字段。 +`DESKTOP_DEVICE_OFFLINE`只是现有`RuntimeProjectionIssue.code: string`的新稳定取值；状态、字段和HTTP结构不变，不要求Gateway schema或现有调用方升级。前端可选识别该code显示“设备离线、上线后自动同步”，但不能仅因`retryable=true`秒级重放业务命令。 + ## Testing Decisions 以用户可观察行为、正式Service/Plugin Interface及真实Router→DI为测试面。Mock应模拟外部依赖而非跳过需要验证的Module；必须证明真实Adapter被调用。共享Mounted/Downloaded运行同一契约套件，具体实现可补充测试。 测试分层：各组窄单测与协议/DI/架构门禁 → Standards/Spec双轴review并修高优问题 → 必要全量/CI收尾 → 匹配OCB gitlink与真实Desktop验证。已有CI要求不削弱；Fake/隔离测试不能代替真实设备。此文没有运行这些测试。 +明确离线修复至少覆盖：BaaS exact code与其它4xx/5xx/坏JSON分流；Builder provider-neutral转换；PerDomain结果和INFO/WARN/ERROR分级；Recovery三态资格；Sweeper排除OFFLINE/终态；Handler开始及apply前竞态；既有离线live task升级后完成；`OFFLINE → ACTIVE`仅发布一次Runtime重投影事件；事件失败由RUN状态Sweep补漏；timeout/5xx继续退避、Center下载继续五秒跟进。OpenClaw/Hermes均需真实验证。 + ### 验收索引 | ID | 必须验证 | 主责 | From c5b6f356a3b12319e558582b3f2cd7e73912eec2 Mon Sep 17 00:00:00 2001 From: Weijia Sun Date: Sat, 12 Sep 2026 00:14:39 +0800 Subject: [PATCH 2/4] fix(skills): stop retrying offline desktop recovery --- .../community/core/desktop_bot/README.md | 6 + .../services/desktop_bot_service.py | 55 ++++++ .../community/core/devices/services/README.md | 6 +- .../conn_info_builders/baas_builder.py | 21 ++- .../core/devices/services/device_context.py | 8 + .../community/core/service_bot/README.md | 2 + .../core/service_bot/services/baas_service.py | 66 ++++++- .../community/core/skill_center/AGENTS.md | 2 + .../services/desktop_skill_recovery.py | 173 ++++++++++++++++-- .../runtime_projections/per_domain.py | 37 ++++ .../services/skill_symlink_listener.py | 34 +++- .../services/test_desktop_bot_service.py | 111 +++++++++++ .../conn_info_builders/test_baas_builder.py | 34 +++- .../test_baas_service_ws_info_logging.py | 65 +++++-- .../services/test_skill_symlink_listener.py | 30 +++ .../test_desktop_skill_recovery.py | 127 ++++++++++++- .../test_skill_set_management_service.py | 103 ++++++++++- 17 files changed, 831 insertions(+), 49 deletions(-) diff --git a/src/backend/src/agentclaw/community/core/desktop_bot/README.md b/src/backend/src/agentclaw/community/core/desktop_bot/README.md index 3145fe153..76bd3ce3c 100644 --- a/src/backend/src/agentclaw/community/core/desktop_bot/README.md +++ b/src/backend/src/agentclaw/community/core/desktop_bot/README.md @@ -20,6 +20,7 @@ internal_dependencies: - agentclaw.community.core.bot_management - agentclaw.community.core.devices - agentclaw.community.core.errors + - agentclaw.community.core.events - agentclaw.community.core.mcp - agentclaw.community.core.service_bot - agentclaw.community.core.skill_center @@ -36,3 +37,8 @@ internal_dependencies: ### Change impact DesktopBot lifecycle is a parallel track to standard bot; refactors that unify the two need to touch this domain carefully. +When the health reconciliation commits an ``OFFLINE -> ACTIVE`` transition for +the current Bot and binding, it publishes the existing +``RuntimeProjectionRequestedEvent`` as a best-effort wake-up. Event delivery +failure never rolls back the confirmed ACTIVE state; Desktop Skill Recovery's +low-frequency sweep is the missed-event fallback. diff --git a/src/backend/src/agentclaw/community/core/desktop_bot/services/desktop_bot_service.py b/src/backend/src/agentclaw/community/core/desktop_bot/services/desktop_bot_service.py index 4678c59ec..57ebb4dc0 100644 --- a/src/backend/src/agentclaw/community/core/desktop_bot/services/desktop_bot_service.py +++ b/src/backend/src/agentclaw/community/core/desktop_bot/services/desktop_bot_service.py @@ -26,6 +26,8 @@ from agentclaw.community.core.desktop_bot.status_mapping import StatusDecision from agentclaw.community.core.repository.protocols.devices import DeviceBindingRepository from agentclaw.community.core.devices.services.device_service import DeviceService +from agentclaw.community.core.events.bus import get_event_bus +from agentclaw.community.core.events.types import RuntimeProjectionRequestedEvent from agentclaw.community.core.mcp.services.passport_scope import filter_passport_mcp_codes from agentclaw.community.core.service_bot.services.baas_service import ( BaasService, @@ -223,6 +225,12 @@ def _apply_decision( owner_id=owner_id, status=decision.target_status, ) + if current_status == "OFFLINE" and decision.target_status == "ACTIVE": + self._request_runtime_projection_after_reconnect( + bot_id=bot_id, + owner_id=owner_id, + binding_id=binding_id, + ) if decision.release_reason: self._merge_bot_ext(bot_id, owner_id, { "release_reason": decision.release_reason, @@ -238,6 +246,53 @@ def _apply_decision( "last_health_check": datetime.now().isoformat(), }) + def _request_runtime_projection_after_reconnect( + self, + *, + bot_id: str, + owner_id: str, + binding_id: int | str | None, + ) -> None: + """Wake current desired-state projection after a confirmed reconnect.""" + + if binding_id is None: + return + try: + bot = self._bot_repo.get_by_id_and_owner( + bot_id=bot_id, + owner_id=owner_id, + ) + if ( + bot is None + or str(bot.get("status") or "").upper() != "ACTIVE" + or str(bot.get("binding_id")) != str(binding_id) + ): + return + binding = self._binding_repo.get_by_id(binding_id=int(binding_id)) + if ( + binding is None + or str(binding.status or "").upper() != "ACTIVE" + ): + return + get_event_bus().publish( + RuntimeProjectionRequestedEvent( + device_id=binding.device_id, + binding_id=binding.id, + entity_id=binding.entity_id, + entity_type=binding.entity_type, + device_provider=binding.device_provider, + sandbox_id=(binding.device_props or {}).get("sandbox_id"), + ) + ) + except Exception: + logger.exception( + "[health-check] runtime projection wake-up failed " + "bot_id=%s owner_id=%s binding_id=%s", + bot_id, + owner_id, + binding_id, + ) + def _merge_bot_ext(self, bot_id: str, owner_id: str, patch: dict) -> None: """Read-modify-write bot ext field, merging only specified keys.""" try: diff --git a/src/backend/src/agentclaw/community/core/devices/services/README.md b/src/backend/src/agentclaw/community/core/devices/services/README.md index 845cdf3a8..70cf5713c 100644 --- a/src/backend/src/agentclaw/community/core/devices/services/README.md +++ b/src/backend/src/agentclaw/community/core/devices/services/README.md @@ -45,6 +45,8 @@ ctx = resolver.resolve_for_bot(bot_id: str, user_id: str) -> DeviceContext - `DeviceNotBoundError` — bot 无 active binding（未 apply / 已 release）。 - `UnknownProviderError` — `binding.device_provider` 是未知值（DB 异常）。 - `ConnInfoBuildError` — 底层 conn_info 计算失败。 +- `DeviceOfflineError` — provider 明确确认当前没有 active device；属于等待设备上线的正常状态。 +- `DeviceConnectionUnavailableError` — timeout、5xx 等无法确认设备状态的暂时连接故障。 ## `DeviceContext` 字段 @@ -67,7 +69,9 @@ frozen dataclass，不可变。 | `TeclawConnInfoBuilder`| 复用现 teclaw plugin 内 OSS 装配逻辑 | | `LocalConnInfoBuilder` | 兼容期 fallback（无 binding 场景；含 desktop pathlib path） | -builder 仅算 conn_info，不做权限 / 不做存活检查。 +builder 仅算 conn_info，不做权限判断。BaaS builder 会把边界层结构化 +`NO_ACTIVE_DEVICES` 转为 provider-neutral `DeviceOfflineError`，并把网络/5xx +转为 `DeviceConnectionUnavailableError`；不得靠异常字符串识别。 ## 2 个 Dispatcher diff --git a/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py b/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py index f9806423c..979f9f68f 100644 --- a/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py +++ b/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py @@ -6,7 +6,15 @@ from agentclaw.community.core.repository.protocols.bot import BotRepository from agentclaw.community.core.repository.protocols.devices import DeviceBindingRepository from agentclaw.community.core.devices.repository.record import DeviceBindingRecord -from agentclaw.community.core.devices.services.device_context import ConnInfoBuildError +from agentclaw.community.core.devices.services.device_context import ( + ConnInfoBuildError, + DeviceConnectionUnavailableError, + DeviceOfflineError, +) +from agentclaw.community.core.service_bot.services.baas_service import ( + BaasNoActiveDevicesError, + BaasTransientServiceError, +) from agentclaw.community.core.devices.services.baas_conn_info import build_baas_conn_info_for_http from agentclaw.community.core.devices.services.baas_template_resolver import ( SystemConfigBaasTemplateResolver, @@ -29,7 +37,8 @@ class BaasConnInfoBuilder: """provider=baas 的 conn_info 计算器。 底层复用 ``baas_service.get_ws_info`` + ``build_baas_conn_info_for_http``, - 不重写。失败统一包成 :class:`ConnInfoBuildError`。 + 不重写。明确离线和暂时连接失败保留为 ``ConnInfoBuildError`` 的窄子类， + 其它失败仍统一包成 :class:`ConnInfoBuildError`。 """ def __init__( @@ -57,6 +66,14 @@ def build( device_affinity=user_id, device_uuid=device_uuid, ) + except BaasNoActiveDevicesError as error: + raise DeviceOfflineError( + f"No active device for binding={binding.id}" + ) from error + except BaasTransientServiceError as error: + raise DeviceConnectionUnavailableError( + f"Device connection is temporarily unavailable for binding={binding.id}" + ) from error except Exception as e: raise ConnInfoBuildError( f"BaasConnInfoBuilder: get_ws_info failed for binding={binding.id}: {e}" diff --git a/src/backend/src/agentclaw/community/core/devices/services/device_context.py b/src/backend/src/agentclaw/community/core/devices/services/device_context.py index 1ee52b42e..f93c4c80e 100644 --- a/src/backend/src/agentclaw/community/core/devices/services/device_context.py +++ b/src/backend/src/agentclaw/community/core/devices/services/device_context.py @@ -58,3 +58,11 @@ class UnknownProviderError(RuntimeError): class ConnInfoBuildError(RuntimeError): """ConnInfoBuilder 调底层(baas /http-info、arca proxy 等)失败。""" + + +class DeviceOfflineError(ConnInfoBuildError): + """The provider confirms that the current device is offline.""" + + +class DeviceConnectionUnavailableError(ConnInfoBuildError): + """A transient provider connection failure with unknown device state.""" diff --git a/src/backend/src/agentclaw/community/core/service_bot/README.md b/src/backend/src/agentclaw/community/core/service_bot/README.md index 9a4722ba2..b67645862 100644 --- a/src/backend/src/agentclaw/community/core/service_bot/README.md +++ b/src/backend/src/agentclaw/community/core/service_bot/README.md @@ -14,6 +14,8 @@ provides: - "BotProcessRegistry" - "BotPublishService" - "BaasService" + - "BaasNoActiveDevicesError" + - "BaasTransientServiceError" - "ServiceSkillsManifestBuilder" - "ResolvedSharedCorpusDelivery" - "ArtifactBuildRequest" diff --git a/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py b/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py index 423fa5e1f..48a224072 100644 --- a/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py +++ b/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py @@ -14,8 +14,7 @@ The dataclasses (``BotWsConnectionInfoResponse``, ``HttpConnectionInfo``, ``Storage``, ``BotDeployConfig``, ``MountPointEntry``, ``BotConfig``) and the -``BaasServiceError`` exception are defined here and imported from ~15 call -sites. +``BaasServiceError`` hierarchy are defined here and imported by callers. """ from __future__ import annotations @@ -205,6 +204,38 @@ class BaasServiceError(Exception): pass +class BaasNoActiveDevicesError(BaasServiceError): + """BaaS confirms that the Bot currently has no active device.""" + + error_code = "NO_ACTIVE_DEVICES" + + def __init__(self, *, status_code: int) -> None: + self.status_code = status_code + super().__init__( + f"BaaS device is offline: status={status_code} " + f"error={self.error_code}" + ) + + +class BaasTransientServiceError(BaasServiceError): + """BaaS transport or 5xx failure that may succeed on a later attempt.""" + + +def _is_no_active_devices_response(response: httpx.Response) -> bool: + """Recognize only the trusted BaaS structured offline contract.""" + + if response.status_code not in {404, 503}: + return False + try: + payload = response.json() + except (ValueError, TypeError): + return False + if not isinstance(payload, dict): + return False + detail = payload.get("detail") + return isinstance(detail, dict) and detail.get("error") == "NO_ACTIVE_DEVICES" + + @dataclass class BotWsConnectionInfoResponse: """BAAS WebSocket connection info. @@ -2002,6 +2033,19 @@ def get_ws_info_by_bot_uuid( # intermittent redirects by bot/tenant/target device — the data # needed to tell a partial-instance/routing fault apart from a # blanket auth requirement. + if _is_no_active_devices_response(e.response): + logger.info( + "[BaasService.get_ws_info_by_bot_uuid] no active device: " + "status=%s bot_uuid=%s tenant=%s device_affinity=%s", + e.response.status_code, + bot_uuid, + effective_tenant, + device_affinity, + ) + raise BaasNoActiveDevicesError( + status_code=e.response.status_code + ) from e + location = e.response.headers.get("location") logger.warning( f"[BaasService.get_ws_info_by_bot_uuid] " @@ -2010,9 +2054,25 @@ def get_ws_info_by_bot_uuid( f"device_affinity={device_affinity}, location={location!r} - " f"{e.response.text}" ) + if e.response.status_code >= 500: + raise BaasTransientServiceError( + f"BaaS API transient error: {e.response.status_code}" + ) from e raise BaasServiceError( f"BaaS API error: {e.response.status_code} - {e.response.text}" ) + except httpx.RequestError as e: + logger.warning( + "[BaasService.get_ws_info_by_bot_uuid] transport failure: " + "bot_uuid=%s tenant=%s device_affinity=%s error_type=%s", + bot_uuid, + effective_tenant, + device_affinity, + type(e).__name__, + ) + raise BaasTransientServiceError( + f"BaaS transport failure: {type(e).__name__}" + ) from e except Exception as e: logger.error( f"[BaasService.get_ws_info_by_bot_uuid] " @@ -3797,6 +3857,8 @@ def get_bot_start_progress( __all__ = [ "BaasService", "BaasServiceError", + "BaasNoActiveDevicesError", + "BaasTransientServiceError", "BotWsConnectionInfoResponse", "HttpConnectionInfo", "Storage", diff --git a/src/backend/src/agentclaw/community/core/skill_center/AGENTS.md b/src/backend/src/agentclaw/community/core/skill_center/AGENTS.md index 188ea95b4..5e192adf8 100644 --- a/src/backend/src/agentclaw/community/core/skill_center/AGENTS.md +++ b/src/backend/src/agentclaw/community/core/skill_center/AGENTS.md @@ -148,6 +148,8 @@ SC Public 引用已是持久异步批量 Operation： `services/skill_symlink_listener.py` 处理设备激活和重投影事件；通过正式 Projector 恢复当前 Desired State。既有兼容 fallback 不是新增业务写入入口。 +Desktop Skill Recovery 使用 Bot 级持久任务，但明确设备离线不是当前任务可修复的故障：`ACTIVE/PENDING` 可运行，`OFFLINE` 等待重连事件，`FAILED/RELEASING/RELEASED` 与未知状态停止。BaaS 仅按可信非 2xx JSON 的精确 `detail.error=NO_ACTIVE_DEVICES` 产生结构化离线错误；Runtime 返回 `DESKTOP_DEVICE_OFFLINE / PENDING / retryable=true`，其中 `retryable` 表示未来再次投影可能恢复，不授权当前任务高频重试。当前任务结束并释放 live key，健康扫描实际提交 `OFFLINE -> ACTIVE` 后发布 `RuntimeProjectionRequestedEvent` 立即唤醒，约十分钟 Sweeper 只为可运行状态兜底。timeout/5xx 继续故障退避，Center 正常下载仍按五秒进度跟进。 + Service Bot 的边界在 `community/core/service_bot/services/`： - `publish_flow/build_stage.py` 与 `deploy/artifact_build_request.py`：文件型 Producer 使用本次 Runtime layout observation。 diff --git a/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py b/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py index 45de9ed4d..f18711ead 100644 --- a/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py +++ b/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py @@ -4,7 +4,9 @@ import asyncio import hashlib +import time from collections.abc import Callable +from enum import StrEnum from agentclaw.community.core.repository.protocols.bot import BotRepository from agentclaw.community.core.repository.protocols.skills_pool import ( @@ -67,10 +69,49 @@ } ) _POOL_TRANSITION_CODE = "SKILLS_POOL_TRANSITION_OWNS_MAPPING" +_DEVICE_OFFLINE_CODE = "DESKTOP_DEVICE_OFFLINE" _KEY_DIGEST_CHARS = 32 logger = get_logger() +class DesktopRecoveryDisposition(StrEnum): + """Whether a Bot is eligible for Desktop Skill recovery work.""" + + RUN = "RUN" + WAIT_FOR_DEVICE = "WAIT_FOR_DEVICE" + STOP = "STOP" + + +class RecoveryContinuation(StrEnum): + """What the current durable recovery task should do next.""" + + COMPLETE_WAITING_FOR_EVENT = "COMPLETE_WAITING_FOR_EVENT" + RETRY_WITH_BACKOFF = "RETRY_WITH_BACKOFF" + RESCHEDULE_FOR_PROGRESS = "RESCHEDULE_FOR_PROGRESS" + COMPLETE_PERMANENT = "COMPLETE_PERMANENT" + + +_RUN_STATUSES = frozenset({"ACTIVE", "PENDING"}) +_WAIT_STATUSES = frozenset({"OFFLINE"}) +_STOP_STATUSES = frozenset({"FAILED", "RELEASING", "RELEASED"}) + + +def _desktop_recovery_disposition(status: object) -> DesktopRecoveryDisposition: + """Classify persisted Bot status without guessing unknown values.""" + + normalized = str(status or "").upper() + if normalized in _RUN_STATUSES: + return DesktopRecoveryDisposition.RUN + if normalized in _WAIT_STATUSES: + return DesktopRecoveryDisposition.WAIT_FOR_DEVICE + return DesktopRecoveryDisposition.STOP + + +def _is_unknown_recovery_status(status: object) -> bool: + normalized = str(status or "").upper() + return normalized not in _RUN_STATUSES | _WAIT_STATUSES | _STOP_STATUSES + + def _required_text(value: object, field: str) -> str: if not isinstance(value, str) or not value or value != value.strip(): raise ValueError(f"{field} must be a non-empty unpadded string") @@ -105,6 +146,17 @@ def ensure(self, *, owner_id: str, bot_id: str) -> EnqueueResult | None: bot = self._bots.get_by_id_and_owner(bot_id, owner_id) if bot is None or bot.get("bot_type") != "desktop": return None + disposition = _desktop_recovery_disposition(bot.get("status")) + if disposition is not DesktopRecoveryDisposition.RUN: + if _is_unknown_recovery_status(bot.get("status")): + logger.warning( + "[DesktopSkillRecovery] skip unknown Bot status " + "owner_id=%s bot_id=%s status=%r", + owner_id, + bot_id, + bot.get("status"), + ) + return None return self._tasks.enqueue( DESKTOP_SKILL_RECOVERY_TASK, {"owner_id": owner_id, "bot_id": bot_id}, @@ -159,6 +211,8 @@ async def _recover(self, *, owner_id: str, bot_id: str) -> TaskOutcome: bot = self._current_desktop(owner_id=owner_id, bot_id=bot_id) if bot is None: return Complete() + if self._stop_for_status(bot, phase="start"): + return Complete() if bot.get("binding_id") is None: return Retry("Desktop Bot has no current binding") if self._pool_transition_owns_mappings(bot): @@ -211,6 +265,8 @@ async def _recover(self, *, owner_id: str, bot_id: str) -> TaskOutcome: latest_bot = self._current_desktop(owner_id=owner_id, bot_id=bot_id) if latest_bot is None: return Complete() + if self._stop_for_status(latest_bot, phase="before_apply"): + return Complete() if latest_bot.get("binding_id") is None: return Retry("Desktop Bot has no current binding after prepare") if self._pool_transition_owns_mappings(latest_bot): @@ -338,6 +394,30 @@ def _pool_transition_owns_mappings(self, bot: dict) -> bool: and state.phase is not SkillLayoutPhase.POOL_ACTIVE ) + @staticmethod + def _stop_for_status(bot: dict, *, phase: str) -> bool: + disposition = _desktop_recovery_disposition(bot.get("status")) + if disposition is DesktopRecoveryDisposition.RUN: + return False + if disposition is DesktopRecoveryDisposition.WAIT_FOR_DEVICE: + logger.info( + "[DesktopSkillRecovery] waiting for Desktop reconnect " + "owner_id=%s bot_id=%s phase=%s", + bot.get("owner_id"), + bot.get("bot_id"), + phase, + ) + elif _is_unknown_recovery_status(bot.get("status")): + logger.warning( + "[DesktopSkillRecovery] stop unknown Bot status " + "owner_id=%s bot_id=%s status=%r phase=%s", + bot.get("owner_id"), + bot.get("bot_id"), + bot.get("status"), + phase, + ) + return True + @staticmethod def _outcome( projection: RuntimeProjectionResult, @@ -346,21 +426,41 @@ def _outcome( prepare_retryable_error: bool, ) -> TaskOutcome: retryable = tuple(issue for issue in projection.issues if issue.retryable) - if any(issue.code == _POOL_TRANSITION_CODE for issue in retryable): + if any(issue.code == _DEVICE_OFFLINE_CODE for issue in retryable): + continuation = RecoveryContinuation.COMPLETE_WAITING_FOR_EVENT + elif any(issue.code == _POOL_TRANSITION_CODE for issue in retryable): + continuation = RecoveryContinuation.RESCHEDULE_FOR_PROGRESS + else: + abnormal = tuple( + issue + for issue in retryable + if not is_normal_center_content_wait(issue.code) + ) + if prepare_retryable_error or abnormal: + continuation = RecoveryContinuation.RETRY_WITH_BACKOFF + elif prepare_pending or any( + is_normal_center_content_wait(issue.code) for issue in retryable + ): + continuation = RecoveryContinuation.RESCHEDULE_FOR_PROGRESS + else: + continuation = RecoveryContinuation.COMPLETE_PERMANENT + + if continuation is RecoveryContinuation.COMPLETE_WAITING_FOR_EVENT: + return Complete() + if continuation is RecoveryContinuation.RESCHEDULE_FOR_PROGRESS: return Reschedule(DESKTOP_SKILL_RECOVERY_DELAY_SECONDS) - abnormal = tuple( - issue for issue in retryable if not is_normal_center_content_wait(issue.code) - ) - if prepare_retryable_error or abnormal: - codes = sorted({issue.code for issue in abnormal}) + if continuation is RecoveryContinuation.RETRY_WITH_BACKOFF: + abnormal_codes = sorted( + { + issue.code + for issue in retryable + if not is_normal_center_content_wait(issue.code) + } + ) return Retry( "Desktop Skill recovery transient failure" - + (f": {','.join(codes)}" if codes else "") + + (f": {','.join(abnormal_codes)}" if abnormal_codes else "") ) - if prepare_pending or any( - is_normal_center_content_wait(issue.code) for issue in retryable - ): - return Reschedule(DESKTOP_SKILL_RECOVERY_DELAY_SECONDS) if ( projection.status is RuntimeProjectionStatus.PENDING and not projection.issues @@ -421,8 +521,16 @@ async def _loop(self) -> None: def sweep_once(self) -> tuple[int, int]: """Return ``(eligible, ensured)`` for deterministic tests/monitoring.""" + started_at = time.monotonic() + scanned = 0 eligible = 0 ensured = 0 + skipped_offline = 0 + skipped_terminal = 0 + skipped_unknown_status = 0 + task_created = 0 + task_joined_existing = 0 + ensure_failed = 0 page = 1 while True: total, bots = self._bots.search_bots( @@ -431,6 +539,7 @@ def sweep_once(self) -> tuple[int, int]: page_size=self._config.sweep_page_size, ) for bot in bots: + scanned += 1 owner_id = bot.get("owner_id") bot_id = bot.get("bot_id") if ( @@ -439,14 +548,37 @@ def sweep_once(self) -> tuple[int, int]: or not isinstance(bot_id, str) ): continue + disposition = _desktop_recovery_disposition(bot.get("status")) + if disposition is DesktopRecoveryDisposition.WAIT_FOR_DEVICE: + skipped_offline += 1 + continue + if disposition is DesktopRecoveryDisposition.STOP: + if _is_unknown_recovery_status(bot.get("status")): + skipped_unknown_status += 1 + logger.warning( + "[DesktopSkillRecovery] sweep skipped unknown Bot " + "status owner_id=%s bot_id=%s status=%r", + owner_id, + bot_id, + bot.get("status"), + ) + else: + skipped_terminal += 1 + continue eligible += 1 try: - if self._recovery.ensure( + result = self._recovery.ensure( owner_id=owner_id, bot_id=bot_id, - ) is not None: + ) + if result is not None: ensured += 1 + if bool(getattr(result, "created", False)): + task_created += 1 + else: + task_joined_existing += 1 except Exception: + ensure_failed += 1 logger.exception( "[DesktopSkillRecovery] sweep ensure failed " "owner_id=%s bot_id=%s", @@ -456,6 +588,21 @@ def sweep_once(self) -> tuple[int, int]: if not bots or page * self._config.sweep_page_size >= total: break page += 1 + logger.info( + "[DesktopSkillRecovery] sweep completed scanned=%s eligible=%s " + "skipped_offline=%s skipped_terminal=%s " + "skipped_unknown_status=%s task_created=%s " + "task_joined_existing=%s ensure_failed=%s duration_ms=%s", + scanned, + eligible, + skipped_offline, + skipped_terminal, + skipped_unknown_status, + task_created, + task_joined_existing, + ensure_failed, + round((time.monotonic() - started_at) * 1000, 3), + ) return eligible, ensured diff --git a/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py b/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py index dd576ac09..a7fdd71e3 100644 --- a/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py +++ b/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py @@ -4,6 +4,10 @@ from collections.abc import Sequence +from agentclaw.community.core.devices.services.device_context import ( + DeviceConnectionUnavailableError, + DeviceOfflineError, +) from agentclaw.community.core.skill_center.runtime_projection_contract import ( EngineRuntimeProjection, ProjectionScope, @@ -90,6 +94,39 @@ async def apply( service_factory=service_factory, ) ) + except DeviceOfflineError: + log = ( + logger.info + if str(plan.bot.get("status") or "").upper() == "OFFLINE" + else logger.warning + ) + log( + "[PerDomainRuntimeProjection] Desktop device offline " + "bot_id=%s engine=%s db_status=%s", + plan.bot_id, + plan.engine, + plan.bot.get("status"), + ) + results.append( + RuntimeProjectionResult.pending( + code="DESKTOP_DEVICE_OFFLINE", + reason="Desktop 设备当前离线，能力状态已保存，将在设备上线后自动同步", + suggested_action="请启动或重新连接 Desktop 客户端。", + ) + ) + except DeviceConnectionUnavailableError: + logger.warning( + "[PerDomainRuntimeProjection] transient device connection " + "failure bot_id=%s engine=%s", + plan.bot_id, + plan.engine, + ) + results.append( + RuntimeProjectionResult.pending( + code="SKILL_RUNTIME_UNAVAILABLE", + reason="Skill 运行环境当前不可连接，能力状态已保存但尚未同步", + ) + ) except Exception: logger.exception( "[PerDomainRuntimeProjection] skill projection unavailable " diff --git a/src/backend/src/agentclaw/community/core/skill_center/services/skill_symlink_listener.py b/src/backend/src/agentclaw/community/core/skill_center/services/skill_symlink_listener.py index 4a2a1cf65..5d3077979 100644 --- a/src/backend/src/agentclaw/community/core/skill_center/services/skill_symlink_listener.py +++ b/src/backend/src/agentclaw/community/core/skill_center/services/skill_symlink_listener.py @@ -16,6 +16,10 @@ from typing import TYPE_CHECKING, Any from agentclaw.community.core.repository.protocols.bot import BotRepository +from agentclaw.community.core.devices.services.device_context import ( + DeviceConnectionUnavailableError, + DeviceOfflineError, +) from agentclaw.community.core.events.types import ( DeviceActivatedEvent, RuntimeProjectionRequestedEvent, @@ -150,6 +154,13 @@ def handle(self, event: _RuntimeReadyEvent) -> None: is_desktop=is_desktop, ) initial_authority = self._resolve_desktop_layout_authority(bot) + if is_desktop and self._desktop_skill_recovery_wakeup is not None: + # Persist the level-triggered recovery before any live-device + # lookup. A reconnect may race BaaS connection visibility; the + # durable handler will re-read the current Bot and binding. + self._enqueue_desktop_skill_recovery( + owner_id=str(owner_id), bot_id=str(bot_id) + ) ctx = self._resolver.resolve_for_bot(bot_id, owner_id) if ctx.binding_id != event.binding_id: logger.info( @@ -162,14 +173,6 @@ def handle(self, event: _RuntimeReadyEvent) -> None: ) return - if is_desktop and self._desktop_skill_recovery_wakeup is not None: - # This wake is independent of the Pool migration wake above. - # The durable recovery handler re-checks Pool transition - # ownership before writing Skill mappings. - self._enqueue_desktop_skill_recovery( - owner_id=str(owner_id), bot_id=str(bot_id) - ) - if initial_authority == _TRANSITION_AUTHORITY: logger.info( "[skill_symlink_listener] Desktop transitional mapping " @@ -237,6 +240,21 @@ def handle(self, event: _RuntimeReadyEvent) -> None: result.get("success"), result.get("message"), ) + except DeviceOfflineError: + logger.warning( + "[skill_symlink_listener] device still offline after runtime " + "wake; durable recovery will wait for the next reconnect: " + "device_id=%s binding_id=%s", + event.device_id, + event.binding_id, + ) + except DeviceConnectionUnavailableError: + logger.warning( + "[skill_symlink_listener] device connection is temporarily " + "unavailable after runtime wake: device_id=%s binding_id=%s", + event.device_id, + event.binding_id, + ) except Exception as exc: logger.exception( "[skill_symlink_listener] handler failed: device_id=%s exc=%s", diff --git a/src/backend/tests/community/core/desktop_bot/services/test_desktop_bot_service.py b/src/backend/tests/community/core/desktop_bot/services/test_desktop_bot_service.py index c4fcdbf22..288fde827 100644 --- a/src/backend/tests/community/core/desktop_bot/services/test_desktop_bot_service.py +++ b/src/backend/tests/community/core/desktop_bot/services/test_desktop_bot_service.py @@ -1555,6 +1555,117 @@ def _make_service_with_mocks(): return service, mocks +def test_offline_to_active_health_transition_requests_runtime_projection() -> None: + from agentclaw.community.core.desktop_bot.status_mapping import StatusDecision + from agentclaw.community.core.events.bus import get_event_bus, reset_event_bus + from agentclaw.community.core.events.types import RuntimeProjectionRequestedEvent + + service, mocks = _make_service_with_mocks() + binding = MagicMock( + id=17, + device_id="BOT-desktop-a", + entity_id="owner-a", + entity_type="staff", + device_provider="baas", + device_props={}, + status="ACTIVE", + ) + mocks["binding_repo"].get_by_id.return_value = binding + mocks["bot_repo"].get_by_id_and_owner.return_value = { + "bot_id": "desktop-a", + "owner_id": "owner-a", + "binding_id": 17, + "status": "ACTIVE", + } + projected: list[RuntimeProjectionRequestedEvent] = [] + reset_event_bus() + get_event_bus().subscribe(RuntimeProjectionRequestedEvent, projected.append) + try: + service._apply_decision( + "desktop-a", + "owner-a", + 17, + "OFFLINE", + StatusDecision(target_status="ACTIVE"), + ) + finally: + reset_event_bus() + + assert projected == [ + RuntimeProjectionRequestedEvent( + device_id="BOT-desktop-a", + binding_id=17, + entity_id="owner-a", + entity_type="staff", + device_provider="baas", + sandbox_id=None, + ) + ] + + +def test_active_health_heartbeat_does_not_repeat_runtime_projection() -> None: + from agentclaw.community.core.desktop_bot.status_mapping import StatusDecision + from agentclaw.community.core.events.bus import get_event_bus, reset_event_bus + from agentclaw.community.core.events.types import RuntimeProjectionRequestedEvent + + service, _mocks = _make_service_with_mocks() + projected: list[RuntimeProjectionRequestedEvent] = [] + reset_event_bus() + get_event_bus().subscribe(RuntimeProjectionRequestedEvent, projected.append) + try: + service._apply_decision( + "desktop-a", + "owner-a", + 17, + "ACTIVE", + StatusDecision(target_status="ACTIVE"), + ) + finally: + reset_event_bus() + + assert projected == [] + + +def test_reconnect_event_failure_does_not_rollback_active_status() -> None: + from agentclaw.community.core.desktop_bot.status_mapping import StatusDecision + + service, mocks = _make_service_with_mocks() + mocks["bot_repo"].get_by_id_and_owner.return_value = { + "bot_id": "desktop-a", + "owner_id": "owner-a", + "binding_id": 17, + "status": "ACTIVE", + } + mocks["binding_repo"].get_by_id.return_value = MagicMock( + id=17, + device_id="BOT-desktop-a", + entity_id="owner-a", + entity_type="staff", + device_provider="baas", + device_props={}, + status="ACTIVE", + ) + + with patch( + "agentclaw.community.core.desktop_bot.services.desktop_bot_service." + "get_event_bus" + ) as event_bus: + event_bus.return_value.publish.side_effect = RuntimeError("listener down") + service._apply_decision( + "desktop-a", + "owner-a", + 17, + "OFFLINE", + StatusDecision(target_status="ACTIVE"), + ) + + mocks["bot_repo"].update_by_owner.assert_any_call( + bot_id="desktop-a", + owner_id="owner-a", + update_data={"status": "ACTIVE"}, + ) + + def _setup_local_lookup(mocks, bot_id, device_id="m-001", active_engine="openclaw"): """Set up the mocks for _lookup_local success case.""" binding = MagicMock(id=1, entity_id="staff_u001", entity_type="staff") diff --git a/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py b/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py index 1797afbe8..7089a2f57 100644 --- a/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py +++ b/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py @@ -2,7 +2,15 @@ import pytest -from agentclaw.community.core.devices.services.device_context import ConnInfoBuildError +from agentclaw.community.core.devices.services.device_context import ( + ConnInfoBuildError, + DeviceConnectionUnavailableError, + DeviceOfflineError, +) +from agentclaw.community.core.service_bot.services.baas_service import ( + BaasNoActiveDevicesError, + BaasTransientServiceError, +) from agentclaw.community.core.devices.services.conn_info_builders.baas_builder import ( BaasConnInfoBuilder, ) @@ -104,6 +112,30 @@ def test_build_raises_conn_info_build_error_on_get_ws_info_failure( builder.build(fake_binding, user_id="user-1") +def test_build_preserves_explicit_device_offline_as_provider_neutral_error( + fake_binding, fake_baas_service, fake_bot_repo, fake_device_repo +): + fake_baas_service.get_ws_info.side_effect = BaasNoActiveDevicesError( + status_code=404 + ) + builder = _make_builder(fake_baas_service, fake_bot_repo, fake_device_repo) + + with pytest.raises(DeviceOfflineError): + builder.build(fake_binding, user_id="user-1") + + +def test_build_preserves_transient_connection_failure_without_calling_it_offline( + fake_binding, fake_baas_service, fake_bot_repo, fake_device_repo +): + fake_baas_service.get_ws_info.side_effect = BaasTransientServiceError( + "BaaS transport timeout" + ) + builder = _make_builder(fake_baas_service, fake_bot_repo, fake_device_repo) + + with pytest.raises(DeviceConnectionUnavailableError): + builder.build(fake_binding, user_id="user-1") + + # ── Step 1 命中:ac_bots.binding_id 直接关联(桌面 bot) ──────────── # # 现场:desktop bot owner 自己访问,ac_bots.binding_id 直接挂在 device binding 上 diff --git a/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py b/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py index cd824f90c..9d281d6f0 100644 --- a/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py +++ b/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py @@ -1,11 +1,9 @@ """Log-level coverage for BaasService.get_ws_info on HTTP errors. -get_ws_info re-raises every HTTP error as BaasServiceError, and all callers -(connection handler, device plugin, identity) catch and degrade. The common -cases — 404 BOT_NOT_FOUND (bot released/expired) and 503 NO_ACTIVE_DEVICES -(device still coming up) — are expected, self-healing business states, NOT -faults. They must be logged at WARNING, never ERROR, so they don't flood the -alarm/ticket pipeline. These tests pin that contract. +get_ws_info keeps every HTTP failure inside the BaasServiceError hierarchy. +Structured NO_ACTIVE_DEVICES is a normal external waiting state and logs at +INFO; transport/5xx failures are transient and log at WARNING; malformed or +unexpected failures keep the ERROR path. These tests pin that classification. The module logger is a SOFAPy logger (propagate=False, own handlers), so caplog can't see it — assert on the logger method called instead, which is what we @@ -22,8 +20,10 @@ ManagedDeployConfigComposer, ) from agentclaw.community.core.service_bot.services.baas_service import ( + BaasNoActiveDevicesError, BaasService, BaasServiceError, + BaasTransientServiceError, ) @@ -77,7 +77,7 @@ def _make_service_raising( def _ws_info_logged_calls(spy: MagicMock, level: str) -> list: - """get_ws_info log calls at the given level (warning/error).""" + """get_ws_info log calls at the requested level.""" return [ c for c in getattr(spy, level).call_args_list if c.args and "get_ws_info" in str(c.args[0]) @@ -101,27 +101,66 @@ def test_404_bot_not_found_logs_warning_not_error(self): assert _ws_info_logged_calls(spy, "warning"), "404 should log at WARNING" assert not _ws_info_logged_calls(spy, "error"), "404 must NOT log at ERROR" - def test_503_no_active_devices_logs_warning_not_error(self): + @pytest.mark.parametrize("status_code", [404, 503]) + def test_no_active_devices_is_structured_and_logs_info( + self, status_code: int + ) -> None: body = ( '{"detail":{"error":"NO_ACTIVE_DEVICES",' '"message":"No active devices available"}}' ) - service = _make_service_raising(503, body) + service = _make_service_raising(status_code, body) with patch( "agentclaw.community.core.service_bot.services.baas_service.logger" ) as spy: - with pytest.raises(BaasServiceError): + with pytest.raises(BaasNoActiveDevicesError) as raised: service.get_ws_info(bind_id=1) - assert _ws_info_logged_calls(spy, "warning"), "503 should log at WARNING" - assert not _ws_info_logged_calls(spy, "error"), "503 must NOT log at ERROR" + assert raised.value.status_code == status_code + assert raised.value.error_code == "NO_ACTIVE_DEVICES" + assert _ws_info_logged_calls(spy, "info") + assert not _ws_info_logged_calls(spy, "warning") + assert not _ws_info_logged_calls(spy, "error") + + @pytest.mark.parametrize( + ("status_code", "body"), + [ + (404, '{"detail":{"error":"BOT_NOT_FOUND"}}'), + (403, '{"detail":{"error":"NO_ACTIVE_DEVICES"}}'), + (500, '{"detail":{"error":"NO_ACTIVE_DEVICES"}}'), + (503, '{"detail":{"error":"SOME_OTHER_ERROR"}}'), + (503, "not-json"), + ], + ) + def test_other_http_errors_do_not_become_device_offline( + self, status_code: int, body: str + ) -> None: + service = _make_service_raising(status_code, body) + + with pytest.raises(BaasServiceError) as raised: + service.get_ws_info(bind_id=1) + + assert not isinstance(raised.value, BaasNoActiveDevicesError) def test_still_raises_baas_service_error(self): """Behavior unchanged: the HTTP error is still surfaced to callers.""" service = _make_service_raising(500, "boom") - with pytest.raises(BaasServiceError): + with pytest.raises(BaasTransientServiceError): service.get_ws_info(bind_id=1) + def test_transport_timeout_is_transient_and_logs_warning(self) -> None: + service = _make_service_raising(500, "unused") + service._http.get.side_effect = httpx.ReadTimeout("timeout") + + with patch( + "agentclaw.community.core.service_bot.services.baas_service.logger" + ) as spy: + with pytest.raises(BaasTransientServiceError): + service.get_ws_info(bind_id=1) + + assert _ws_info_logged_calls(spy, "warning") + assert not _ws_info_logged_calls(spy, "error") + def test_302_redirect_logs_location_and_correlation_fields(self): """A gateway 302 (Spanner) must surface the redirect ``Location`` plus bot_uuid/tenant/device_affinity in the WARNING log, so intermittent diff --git a/src/backend/tests/community/core/skill_center/services/test_skill_symlink_listener.py b/src/backend/tests/community/core/skill_center/services/test_skill_symlink_listener.py index 37d739cd1..c2dcfdffa 100644 --- a/src/backend/tests/community/core/skill_center/services/test_skill_symlink_listener.py +++ b/src/backend/tests/community/core/skill_center/services/test_skill_symlink_listener.py @@ -165,6 +165,36 @@ def test_restart_projection_event_wakes_the_same_skill_recovery_interface(self): recovery.assert_called_once_with("owner-1", "desktop-1") + def test_restart_wakes_durable_recovery_before_connection_resolution(self): + from agentclaw.community.core.devices.services.device_context import ( + DeviceOfflineError, + ) + + event = RuntimeProjectionRequestedEvent( + device_id="staff_u001_default", + binding_id=42, + entity_id="u001", + entity_type="staff", + device_provider="baas", + ) + bot_query = MagicMock() + bot_query.get_by_binding_id.return_value = { + "bot_id": "desktop-1", + "owner_id": "owner-1", + "bot_type": "desktop", + } + recovery = MagicMock() + listener, _, _, resolver = _make_listener( + bot_query=bot_query, + desktop_skill_recovery_wakeup=recovery, + runtime_reconcile=MagicMock(), + ) + resolver.resolve_for_bot.side_effect = DeviceOfflineError("offline") + + listener.handle(event) + + recovery.assert_called_once_with("owner-1", "desktop-1") + def test_restart_projection_trigger_is_layout_agnostic(self): bot_query = MagicMock() bot_query.get_by_binding_id.return_value = { diff --git a/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py b/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py index 40ae0d985..42518f404 100644 --- a/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py +++ b/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py @@ -6,7 +6,8 @@ from contextlib import contextmanager from dataclasses import replace from datetime import UTC, datetime -from unittest.mock import AsyncMock, MagicMock, call +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, call, patch import pytest from sqlalchemy import create_engine @@ -99,7 +100,7 @@ _UUID = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa" -def _bot(*, binding_id: int = 17) -> dict: +def _bot(*, binding_id: int = 17, status: str = "ACTIVE") -> dict: return { "bot_id": "bot-a", "owner_id": "owner-a", @@ -108,6 +109,7 @@ def _bot(*, binding_id: int = 17) -> dict: "bot_type": "desktop", "active_engine": "openclaw", "binding_id": binding_id, + "status": status, } @@ -844,6 +846,75 @@ def test_ensure_uses_minimal_payload_and_thirty_minute_deadline() -> None: assert "delay_seconds" not in tasks.enqueue.call_args.kwargs +@pytest.mark.parametrize( + "status", ["OFFLINE", "FAILED", "RELEASING", "RELEASED", "UNKNOWN"] +) +def test_ensure_does_not_create_recovery_for_non_runnable_status(status: str) -> None: + bots = MagicMock() + bots.get_by_id_and_owner.return_value = _bot(status=status) + tasks = MagicMock() + service = DesktopSkillRecoveryService(bots=bots, tasks=tasks) + + assert service.ensure(owner_id="owner-a", bot_id="bot-a") is None + tasks.enqueue.assert_not_called() + + +def test_handler_completes_explicit_offline_without_runtime_io() -> None: + bots = MagicMock() + bots.get_by_id_and_owner.return_value = _bot(status="OFFLINE") + projector = MagicMock() + handler = DesktopSkillRecoveryTaskHandler( + bots=bots, + projector=projector, + distribution=MagicMock(), + layouts=MagicMock(), + env_provider=lambda: "dev", + ) + + outcome = handler.handle({"owner_id": "owner-a", "bot_id": "bot-a"}) + + assert isinstance(outcome, Complete) + projector.resolve_plan.assert_not_called() + + +def test_handler_rechecks_offline_after_prepare_before_runtime_io() -> None: + bots = MagicMock() + bots.get_by_id_and_owner.side_effect = [ + _bot(status="ACTIVE"), + _bot(status="OFFLINE"), + ] + projector = _Projector() + layouts = MagicMock() + layouts.get.return_value = BotSkillLayoutState.legacy_default( + BotSkillLayoutScope(env="dev", entity_id="owner-a", bot_id="bot-a") + ) + handler = DesktopSkillRecoveryTaskHandler( + bots=bots, + projector=projector, + distribution=_Distribution(), + layouts=layouts, + env_provider=lambda: "dev", + ) + + outcome = handler.handle({"owner_id": "owner-a", "bot_id": "bot-a"}) + + assert isinstance(outcome, Complete) + assert projector.applied == [] + + +def test_explicit_device_offline_result_waits_for_reconnect_event() -> None: + outcome = DesktopSkillRecoveryTaskHandler._outcome( + RuntimeProjectionResult.pending( + code="DESKTOP_DEVICE_OFFLINE", + reason="offline", + ), + prepare_pending=False, + prepare_retryable_error=False, + ) + + assert isinstance(outcome, Complete) + + @pytest.mark.integration def test_terminal_recovery_releases_the_bot_key_for_a_later_sweep( recovery_queue, @@ -878,22 +949,62 @@ def test_sweeper_pages_all_live_bound_desktop_bots_and_only_ensures() -> None: ( 3, [ - {"owner_id": "o1", "bot_id": "b1", "binding_id": 1}, - {"owner_id": "o2", "bot_id": "b2", "binding_id": None}, + { + "owner_id": "o1", + "bot_id": "b1", + "binding_id": 1, + "status": "ACTIVE", + }, + { + "owner_id": "o2", + "bot_id": "b2", + "binding_id": None, + "status": "PENDING", + }, + ], + ), + ( + 4, + [ + { + "owner_id": "o3", + "bot_id": "b3", + "binding_id": 3, + "status": "OFFLINE", + }, + { + "owner_id": "o4", + "bot_id": "b4", + "binding_id": 4, + "status": "PENDING", + }, ], ), - (3, [{"owner_id": "o3", "bot_id": "b3", "binding_id": 3}]), ] recovery = MagicMock() - recovery.ensure.return_value = object() + recovery.ensure.side_effect = [ + SimpleNamespace(created=True), + SimpleNamespace(created=False), + ] sweeper = DesktopSkillRecoverySweeper( bots=bots, recovery=recovery, config=DesktopSkillRecoveryConfig(sweep_page_size=2), ) - assert sweeper.sweep_once() == (2, 2) + with patch( + "agentclaw.community.core.skill_center.services." + "desktop_skill_recovery.logger" + ) as log: + assert sweeper.sweep_once() == (2, 2) + assert recovery.ensure.call_args_list == [ call(owner_id="o1", bot_id="b1"), - call(owner_id="o3", bot_id="b3"), + call(owner_id="o4", bot_id="b4"), ] + summary = next( + call + for call in log.info.call_args_list + if "sweep completed" in str(call.args[0]) + ) + assert summary.args[1:9] == (4, 2, 1, 0, 0, 1, 1, 0) diff --git a/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py b/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py index 5d65f139b..a5997c4b5 100644 --- a/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py +++ b/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py @@ -9,7 +9,7 @@ import re from dataclasses import replace from types import SimpleNamespace -from unittest.mock import MagicMock +from unittest.mock import AsyncMock, MagicMock, patch import pytest @@ -3719,6 +3719,107 @@ async def test_failed_skill_delivery_does_not_skip_the_mcp_half(): assert len(factory.service.mcp_projections) == 1 +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("bot_status", "log_method"), + [("OFFLINE", "info"), ("ACTIVE", "warning")], +) +async def test_explicit_desktop_offline_is_pending_without_error_traceback( + bot_status: str, log_method: str +) -> None: + from agentclaw.community.core.devices.services.device_context import ( + DeviceOfflineError, + ) + from agentclaw.community.core.skill_center.runtime_projection_contract import ( + ResolvedSkillPlan, + RuntimeProjectionStatus, + ) + from agentclaw.community.core.skill_center.runtime_resolver import ( + RuntimeSkillProjection, + ) + from agentclaw.community.core.skill_center.services.runtime_projections.per_domain import ( + PerDomainRuntimeProjection, + ) + + delivery = MagicMock() + delivery.deliver = AsyncMock(side_effect=DeviceOfflineError("offline")) + projection = PerDomainRuntimeProjection(skill_delivery=delivery) + plan = ResolvedSkillPlan( + bot_id="desktop-a", + owner_id="owner-a", + bot={"status": bot_status}, + engine="hermes", + projection=RuntimeSkillProjection(skill_mappings=(), skill_assets=()), + ) + + with patch( + "agentclaw.community.core.skill_center.services.runtime_projections." + "per_domain.logger" + ) as log: + result = await projection.apply( + plan=plan, + scope=ProjectionScope(skills=True), + service_factory=MagicMock(), + ) + + assert result.status is RuntimeProjectionStatus.PENDING + assert result.issues[0].code == "DESKTOP_DEVICE_OFFLINE" + assert result.issues[0].retryable is True + assert "上线后自动同步" in result.issues[0].reason + calls = getattr(log, log_method).call_args_list + assert any("Desktop device offline" in str(call.args[0]) for call in calls) + log.exception.assert_not_called() + + +@pytest.mark.asyncio +async def test_transient_desktop_connection_failure_is_retryable_warning() -> None: + from agentclaw.community.core.devices.services.device_context import ( + DeviceConnectionUnavailableError, + ) + from agentclaw.community.core.skill_center.runtime_projection_contract import ( + ResolvedSkillPlan, + RuntimeProjectionStatus, + ) + from agentclaw.community.core.skill_center.runtime_resolver import ( + RuntimeSkillProjection, + ) + from agentclaw.community.core.skill_center.services.runtime_projections.per_domain import ( + PerDomainRuntimeProjection, + ) + + delivery = MagicMock() + delivery.deliver = AsyncMock( + side_effect=DeviceConnectionUnavailableError("timeout") + ) + projection = PerDomainRuntimeProjection(skill_delivery=delivery) + plan = ResolvedSkillPlan( + bot_id="desktop-a", + owner_id="owner-a", + bot={"status": "ACTIVE"}, + engine="openclaw", + projection=RuntimeSkillProjection(skill_mappings=(), skill_assets=()), + ) + + with patch( + "agentclaw.community.core.skill_center.services.runtime_projections." + "per_domain.logger" + ) as log: + result = await projection.apply( + plan=plan, + scope=ProjectionScope(skills=True), + service_factory=MagicMock(), + ) + + assert result.status is RuntimeProjectionStatus.PENDING + assert result.issues[0].code == "SKILL_RUNTIME_UNAVAILABLE" + assert result.issues[0].retryable is True + assert any( + "transient" in str(call.args[0]).lower() + for call in log.warning.call_args_list + ) + log.exception.assert_not_called() + + # ── Skill mutations carry the Skill's MCP dependencies ─────────────── # # A Skill's ``mcp_dependencies`` join the Bot's MCP set along with the Skill, From 0debf50267556e8d675faea75aeac91a9c7114b2 Mon Sep 17 00:00:00 2001 From: Weijia Sun Date: Sat, 12 Sep 2026 00:25:00 +0800 Subject: [PATCH 3/4] fix(skills): tighten desktop recovery contracts --- .../src/agentclaw/community/api/README.md | 1 + .../agentclaw/community/api/baas_service.py | 16 ++++- .../conn_info_builders/baas_builder.py | 2 +- .../community/core/service_bot/README.md | 1 + .../core/service_bot/baas_service_errors.py | 31 ++++++++ .../core/service_bot/services/baas_service.py | 27 ++----- .../services/desktop_skill_recovery.py | 37 ++++++++-- .../runtime_projections/per_domain.py | 51 ++++++++----- .../conn_info_builders/test_baas_builder.py | 8 +-- .../test_baas_service_ws_info_logging.py | 11 +-- .../test_desktop_skill_recovery.py | 72 +++++++++++++++++-- .../test_skill_set_management_service.py | 37 +++++++++- 12 files changed, 232 insertions(+), 62 deletions(-) create mode 100644 src/backend/src/agentclaw/community/core/service_bot/baas_service_errors.py diff --git a/src/backend/src/agentclaw/community/api/README.md b/src/backend/src/agentclaw/community/api/README.md index cb9fad15f..093ead419 100644 --- a/src/backend/src/agentclaw/community/api/README.md +++ b/src/backend/src/agentclaw/community/api/README.md @@ -153,6 +153,7 @@ internal_dependencies: - agentclaw.community.core.task.task_runner.client.ports # OpenApiBotPort — typed in task_grant_service.py - agentclaw.community.core.work_orders.callbacks # Work-order callback credential typed in work_order_service.py - agentclaw.community.core.service_bot.services.baas_service # BotWsConnectionInfoResponse / HttpConnectionInfo — typed in baas_service.py (BaasService is a plain core service) + - agentclaw.community.core.service_bot.baas_service_errors # Stable BaaS Service error hierarchy re-exported by the Protocol module - agentclaw.community.core.service_bot.types # PublishStage enum — typed in baas_service.py - agentclaw.community.core.skills_pool # Skills Pool rollout/query/recovery domain DTOs used by operator Service API Protocols - agentclaw.community.core.skill_center # Local Skill desired-state query service DTOs diff --git a/src/backend/src/agentclaw/community/api/baas_service.py b/src/backend/src/agentclaw/community/api/baas_service.py index 87ecd28f6..5924a59fc 100644 --- a/src/backend/src/agentclaw/community/api/baas_service.py +++ b/src/backend/src/agentclaw/community/api/baas_service.py @@ -16,6 +16,11 @@ from typing import TYPE_CHECKING, Any, Dict, Optional, Protocol, runtime_checkable +from agentclaw.community.core.service_bot.baas_service_errors import ( + BaasNoActiveDevicesError, + BaasServiceError, + BaasTransientServiceError, +) from agentclaw.community.core.service_bot.services.baas_service import ( BotWsConnectionInfoResponse, HttpConnectionInfo, @@ -149,6 +154,10 @@ def get_ws_info( ``device_uuid`` (optional) locks a specific instance in a multi-instance service bot; omitted → BaaS auto-selects an active instance. + + Raises ``BaasNoActiveDevicesError`` for trusted structured device-offline + responses, ``BaasTransientServiceError`` for transport/5xx failures, and + ``BaasServiceError`` for other service failures. """ ... @@ -279,4 +288,9 @@ def get_bot_start_progress( ... -__all__ = ["BaasServiceProtocol"] +__all__ = [ + "BaasNoActiveDevicesError", + "BaasServiceError", + "BaasServiceProtocol", + "BaasTransientServiceError", +] diff --git a/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py b/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py index 979f9f68f..6f0740547 100644 --- a/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py +++ b/src/backend/src/agentclaw/community/core/devices/services/conn_info_builders/baas_builder.py @@ -11,7 +11,7 @@ DeviceConnectionUnavailableError, DeviceOfflineError, ) -from agentclaw.community.core.service_bot.services.baas_service import ( +from agentclaw.community.core.service_bot.baas_service_errors import ( BaasNoActiveDevicesError, BaasTransientServiceError, ) diff --git a/src/backend/src/agentclaw/community/core/service_bot/README.md b/src/backend/src/agentclaw/community/core/service_bot/README.md index b67645862..01e49fb24 100644 --- a/src/backend/src/agentclaw/community/core/service_bot/README.md +++ b/src/backend/src/agentclaw/community/core/service_bot/README.md @@ -14,6 +14,7 @@ provides: - "BotProcessRegistry" - "BotPublishService" - "BaasService" + - "BaasServiceError" - "BaasNoActiveDevicesError" - "BaasTransientServiceError" - "ServiceSkillsManifestBuilder" diff --git a/src/backend/src/agentclaw/community/core/service_bot/baas_service_errors.py b/src/backend/src/agentclaw/community/core/service_bot/baas_service_errors.py new file mode 100644 index 000000000..bb1dfdee3 --- /dev/null +++ b/src/backend/src/agentclaw/community/core/service_bot/baas_service_errors.py @@ -0,0 +1,31 @@ +"""Stable BaaS Service error contract shared by implementation and consumers.""" + +from __future__ import annotations + + +class BaasServiceError(Exception): + """Base error for BaaS Service failures.""" + + +class BaasNoActiveDevicesError(BaasServiceError): + """BaaS confirms that the Bot currently has no active device.""" + + error_code = "NO_ACTIVE_DEVICES" + + def __init__(self, *, status_code: int) -> None: + self.status_code = status_code + super().__init__( + f"BaaS device is offline: status={status_code} " + f"error={self.error_code}" + ) + + +class BaasTransientServiceError(BaasServiceError): + """BaaS transport or 5xx failure that may succeed on a later attempt.""" + + +__all__ = [ + "BaasNoActiveDevicesError", + "BaasServiceError", + "BaasTransientServiceError", +] diff --git a/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py b/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py index 48a224072..fb07d391d 100644 --- a/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py +++ b/src/backend/src/agentclaw/community/core/service_bot/services/baas_service.py @@ -25,6 +25,11 @@ import time import httpx +from agentclaw.community.core.service_bot.baas_service_errors import ( + BaasNoActiveDevicesError, + BaasServiceError, + BaasTransientServiceError, +) from agentclaw.community.core.caller_identity.credential import ( CALLER_CREDENTIAL_REQUEST_INVALID, CALLER_OUTBOUND_INVALID, @@ -199,28 +204,6 @@ def __init__(self, reason: str) -> None: super().__init__(reason) -class BaasServiceError(Exception): - """BaaS service error.""" - pass - - -class BaasNoActiveDevicesError(BaasServiceError): - """BaaS confirms that the Bot currently has no active device.""" - - error_code = "NO_ACTIVE_DEVICES" - - def __init__(self, *, status_code: int) -> None: - self.status_code = status_code - super().__init__( - f"BaaS device is offline: status={status_code} " - f"error={self.error_code}" - ) - - -class BaasTransientServiceError(BaasServiceError): - """BaaS transport or 5xx failure that may succeed on a later attempt.""" - - def _is_no_active_devices_response(response: httpx.Response) -> bool: """Recognize only the trusted BaaS structured offline contract.""" diff --git a/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py b/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py index f18711ead..ae9ad1d14 100644 --- a/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py +++ b/src/backend/src/agentclaw/community/core/skill_center/services/desktop_skill_recovery.py @@ -70,6 +70,22 @@ ) _POOL_TRANSITION_CODE = "SKILLS_POOL_TRANSITION_OWNS_MAPPING" _DEVICE_OFFLINE_CODE = "DESKTOP_DEVICE_OFFLINE" +_TRANSIENT_FAILURE_CODES = frozenset( + { + "CENTER_CONTENT_DESCRIPTOR_WRITE_FAILED", + "CENTER_CONTENT_DNS_UNAVAILABLE", + "CENTER_CONTENT_DOWNLOAD_FAILED", + "CENTER_CONTENT_DOWNLOAD_START_FAILED", + "CENTER_CONTENT_LOOKUP_FAILED", + "CENTER_CONTENT_PACKAGE_WRITE_FAILED", + "CENTER_CONTENT_SIGNING_FAILED", + "CENTER_RUNTIME_RESTART_REQUIRED", + "MANAGED_SOURCE_MISSING", + "MAPPING_PUBLISH_IO_ERROR", + "SKILL_MAPPING_RUNTIME_UNAVAILABLE", + "SKILL_RUNTIME_UNAVAILABLE", + } +) _KEY_DIGEST_CHARS = 32 logger = get_logger() @@ -431,12 +447,12 @@ def _outcome( elif any(issue.code == _POOL_TRANSITION_CODE for issue in retryable): continuation = RecoveryContinuation.RESCHEDULE_FOR_PROGRESS else: - abnormal = tuple( + transient = tuple( issue for issue in retryable - if not is_normal_center_content_wait(issue.code) + if issue.code in _TRANSIENT_FAILURE_CODES ) - if prepare_retryable_error or abnormal: + if prepare_retryable_error or transient: continuation = RecoveryContinuation.RETRY_WITH_BACKOFF elif prepare_pending or any( is_normal_center_content_wait(issue.code) for issue in retryable @@ -444,6 +460,19 @@ def _outcome( continuation = RecoveryContinuation.RESCHEDULE_FOR_PROGRESS else: continuation = RecoveryContinuation.COMPLETE_PERMANENT + unknown_retryable = sorted( + { + issue.code + for issue in retryable + if not is_normal_center_content_wait(issue.code) + } + ) + if unknown_retryable: + logger.warning( + "[DesktopSkillRecovery] unsupported retryable issues " + "will not authorize task retry codes=%s", + unknown_retryable, + ) if continuation is RecoveryContinuation.COMPLETE_WAITING_FOR_EVENT: return Complete() @@ -454,7 +483,7 @@ def _outcome( { issue.code for issue in retryable - if not is_normal_center_content_wait(issue.code) + if issue.code in _TRANSIENT_FAILURE_CODES } ) return Retry( diff --git a/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py b/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py index a7fdd71e3..7d776b9bd 100644 --- a/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py +++ b/src/backend/src/agentclaw/community/core/skill_center/services/runtime_projections/per_domain.py @@ -95,25 +95,40 @@ async def apply( ) ) except DeviceOfflineError: - log = ( - logger.info - if str(plan.bot.get("status") or "").upper() == "OFFLINE" - else logger.warning - ) - log( - "[PerDomainRuntimeProjection] Desktop device offline " - "bot_id=%s engine=%s db_status=%s", - plan.bot_id, - plan.engine, - plan.bot.get("status"), - ) - results.append( - RuntimeProjectionResult.pending( - code="DESKTOP_DEVICE_OFFLINE", - reason="Desktop 设备当前离线，能力状态已保存，将在设备上线后自动同步", - suggested_action="请启动或重新连接 Desktop 客户端。", + if str(plan.bot.get("bot_type") or "").lower() != "desktop": + logger.warning( + "[PerDomainRuntimeProjection] runtime device has no " + "active instance bot_id=%s engine=%s bot_type=%s", + plan.bot_id, + plan.engine, + plan.bot.get("bot_type"), + ) + results.append( + RuntimeProjectionResult.pending( + code="SKILL_RUNTIME_UNAVAILABLE", + reason="Skill 运行环境当前不可连接，能力状态已保存但尚未同步", + ) + ) + else: + log = ( + logger.info + if str(plan.bot.get("status") or "").upper() == "OFFLINE" + else logger.warning + ) + log( + "[PerDomainRuntimeProjection] Desktop device offline " + "bot_id=%s engine=%s db_status=%s", + plan.bot_id, + plan.engine, + plan.bot.get("status"), + ) + results.append( + RuntimeProjectionResult.pending( + code="DESKTOP_DEVICE_OFFLINE", + reason="Desktop 设备当前离线，能力状态已保存，将在设备上线后自动同步", + suggested_action="请启动或重新连接 Desktop 客户端。", + ) ) - ) except DeviceConnectionUnavailableError: logger.warning( "[PerDomainRuntimeProjection] transient device connection " diff --git a/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py b/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py index 7089a2f57..121a617c7 100644 --- a/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py +++ b/src/backend/tests/community/core/devices/services/conn_info_builders/test_baas_builder.py @@ -2,15 +2,15 @@ import pytest +from agentclaw.community.core.service_bot.baas_service_errors import ( + BaasNoActiveDevicesError, + BaasTransientServiceError, +) from agentclaw.community.core.devices.services.device_context import ( ConnInfoBuildError, DeviceConnectionUnavailableError, DeviceOfflineError, ) -from agentclaw.community.core.service_bot.services.baas_service import ( - BaasNoActiveDevicesError, - BaasTransientServiceError, -) from agentclaw.community.core.devices.services.conn_info_builders.baas_builder import ( BaasConnInfoBuilder, ) diff --git a/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py b/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py index 9d281d6f0..dada5c9ee 100644 --- a/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py +++ b/src/backend/tests/community/core/service_bot/services/test_baas_service_ws_info_logging.py @@ -16,15 +16,15 @@ import httpx import pytest -from agentclaw.community.core.service_bot.services.deploy.managed_composer import ( - ManagedDeployConfigComposer, -) -from agentclaw.community.core.service_bot.services.baas_service import ( +from agentclaw.community.core.service_bot.baas_service_errors import ( BaasNoActiveDevicesError, - BaasService, BaasServiceError, BaasTransientServiceError, ) +from agentclaw.community.core.service_bot.services.deploy.managed_composer import ( + ManagedDeployConfigComposer, +) +from agentclaw.community.core.service_bot.services.baas_service import BaasService def _make_service_raising( @@ -126,6 +126,7 @@ def test_no_active_devices_is_structured_and_logs_info( ("status_code", "body"), [ (404, '{"detail":{"error":"BOT_NOT_FOUND"}}'), + (401, '{"detail":{"error":"NO_ACTIVE_DEVICES"}}'), (403, '{"detail":{"error":"NO_ACTIVE_DEVICES"}}'), (500, '{"detail":{"error":"NO_ACTIVE_DEVICES"}}'), (503, '{"detail":{"error":"SOME_OTHER_ERROR"}}'), diff --git a/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py b/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py index 42518f404..b2652adec 100644 --- a/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py +++ b/src/backend/tests/community/core/skill_center/test_desktop_skill_recovery.py @@ -830,9 +830,12 @@ def insert(self, _record): assert projector.applied == [combined] -def test_ensure_uses_minimal_payload_and_thirty_minute_deadline() -> None: +@pytest.mark.parametrize("status", ["ACTIVE", "PENDING"]) +def test_ensure_uses_minimal_payload_and_thirty_minute_deadline( + status: str, +) -> None: bots = MagicMock() - bots.get_by_id_and_owner.return_value = _bot() + bots.get_by_id_and_owner.return_value = _bot(status=status) tasks = MagicMock() service = DesktopSkillRecoveryService(bots=bots, tasks=tasks) @@ -859,9 +862,14 @@ def test_ensure_does_not_create_recovery_for_non_runnable_status(status: str) -> tasks.enqueue.assert_not_called() -def test_handler_completes_explicit_offline_without_runtime_io() -> None: +@pytest.mark.parametrize( + "status", ["OFFLINE", "FAILED", "RELEASING", "RELEASED", "UNKNOWN"] +) +def test_handler_completes_non_runnable_status_without_runtime_io( + status: str, +) -> None: bots = MagicMock() - bots.get_by_id_and_owner.return_value = _bot(status="OFFLINE") + bots.get_by_id_and_owner.return_value = _bot(status=status) projector = MagicMock() handler = DesktopSkillRecoveryTaskHandler( bots=bots, @@ -877,11 +885,16 @@ def test_handler_completes_explicit_offline_without_runtime_io() -> None: projector.resolve_plan.assert_not_called() -def test_handler_rechecks_offline_after_prepare_before_runtime_io() -> None: +@pytest.mark.parametrize( + "status", ["OFFLINE", "FAILED", "RELEASING", "RELEASED", "UNKNOWN"] +) +def test_handler_rechecks_non_runnable_status_after_prepare_before_runtime_io( + status: str, +) -> None: bots = MagicMock() bots.get_by_id_and_owner.side_effect = [ _bot(status="ACTIVE"), - _bot(status="OFFLINE"), + _bot(status=status), ] projector = _Projector() layouts = MagicMock() @@ -915,6 +928,19 @@ def test_explicit_device_offline_result_waits_for_reconnect_event() -> None: assert isinstance(outcome, Complete) +def test_unknown_retryable_issue_does_not_authorize_task_retry() -> None: + outcome = DesktopSkillRecoveryTaskHandler._outcome( + RuntimeProjectionResult.pending( + code="UNKNOWN_RUNTIME_WAIT", + reason="unknown", + ), + prepare_pending=False, + prepare_retryable_error=False, + ) + + assert isinstance(outcome, Complete) + + @pytest.mark.integration def test_terminal_recovery_releases_the_bot_key_for_a_later_sweep( recovery_queue, @@ -943,6 +969,40 @@ def test_terminal_recovery_releases_the_bot_key_for_a_later_sweep( assert second.record.id != first.record.id +@pytest.mark.integration +def test_existing_live_task_drains_when_bot_is_now_offline(recovery_queue) -> None: + bots = MagicMock() + bots.get_by_id_and_owner.side_effect = [ + _bot(status="ACTIVE"), + _bot(status="OFFLINE"), + ] + recovery = DesktopSkillRecoveryService(bots=bots, tasks=recovery_queue) + enqueued = recovery.ensure(owner_id="owner-a", bot_id="bot-a") + assert enqueued is not None + claimed = recovery_queue._repo.claim_batch( + worker_id="worker-1", + env="dev", + app=DEFAULT_APP, + limit=1, + lease_seconds=60, + ) + handler = DesktopSkillRecoveryTaskHandler( + bots=bots, + projector=MagicMock(), + distribution=MagicMock(), + layouts=MagicMock(), + env_provider=lambda: "dev", + ) + + outcome = handler.handle(claimed[0].payload) + + assert isinstance(outcome, Complete) + assert recovery_queue._repo.complete( + task_id=claimed[0].id, + worker_id="worker-1", + ) + + def test_sweeper_pages_all_live_bound_desktop_bots_and_only_ensures() -> None: bots = MagicMock() bots.search_bots.side_effect = [ diff --git a/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py b/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py index a5997c4b5..2d4c3b99d 100644 --- a/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py +++ b/src/backend/tests/community/core/skill_center/test_skill_set_management_service.py @@ -3747,7 +3747,7 @@ async def test_explicit_desktop_offline_is_pending_without_error_traceback( plan = ResolvedSkillPlan( bot_id="desktop-a", owner_id="owner-a", - bot={"status": bot_status}, + bot={"status": bot_status, "bot_type": "desktop"}, engine="hermes", projection=RuntimeSkillProjection(skill_mappings=(), skill_assets=()), ) @@ -3771,6 +3771,41 @@ async def test_explicit_desktop_offline_is_pending_without_error_traceback( log.exception.assert_not_called() +@pytest.mark.asyncio +async def test_non_desktop_no_active_device_keeps_generic_runtime_contract() -> None: + from agentclaw.community.core.devices.services.device_context import ( + DeviceOfflineError, + ) + from agentclaw.community.core.skill_center.runtime_projection_contract import ( + ResolvedSkillPlan, + ) + from agentclaw.community.core.skill_center.runtime_resolver import ( + RuntimeSkillProjection, + ) + from agentclaw.community.core.skill_center.services.runtime_projections.per_domain import ( + PerDomainRuntimeProjection, + ) + + delivery = MagicMock() + delivery.deliver = AsyncMock(side_effect=DeviceOfflineError("offline")) + projection = PerDomainRuntimeProjection(skill_delivery=delivery) + plan = ResolvedSkillPlan( + bot_id="service-a", + owner_id="owner-a", + bot={"status": "ACTIVE", "bot_type": "service"}, + engine="openclaw", + projection=RuntimeSkillProjection(skill_mappings=(), skill_assets=()), + ) + + result = await projection.apply( + plan=plan, + scope=ProjectionScope(skills=True), + service_factory=MagicMock(), + ) + + assert result.issues[0].code == "SKILL_RUNTIME_UNAVAILABLE" + + @pytest.mark.asyncio async def test_transient_desktop_connection_failure_is_retryable_warning() -> None: from agentclaw.community.core.devices.services.device_context import ( From 425a548800cee7cd78bc489e0e66d6e7af324afc Mon Sep 17 00:00:00 2001 From: Weijia Sun Date: Sat, 12 Sep 2026 00:35:40 +0800 Subject: [PATCH 4/4] test(skills): align recovery fixtures with bot status contract --- .../test_baas_service_get_ws_info_by_bot_uuid.py | 9 +++++---- .../core/skill_center/test_local_skill_upload_service.py | 1 + .../skill_center/test_permanent_package_recovery.py | 1 + 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/src/backend/tests/community/core/service_bot/services/test_baas_service_get_ws_info_by_bot_uuid.py b/src/backend/tests/community/core/service_bot/services/test_baas_service_get_ws_info_by_bot_uuid.py index d784a78ce..d875dc81f 100644 --- a/src/backend/tests/community/core/service_bot/services/test_baas_service_get_ws_info_by_bot_uuid.py +++ b/src/backend/tests/community/core/service_bot/services/test_baas_service_get_ws_info_by_bot_uuid.py @@ -339,8 +339,8 @@ def test_404_logs_warning_not_error(self): assert _ws_info_by_bot_uuid_logged_calls(spy, "warning"), "404 should log at WARNING" assert not _ws_info_by_bot_uuid_logged_calls(spy, "error"), "404 must NOT log at ERROR" - def test_503_logs_warning_not_error(self): - """503 HTTP error logs at WARNING, not ERROR.""" + def test_503_no_active_devices_logs_info_not_warning_or_error(self): + """Structured NO_ACTIVE_DEVICES is a normal external waiting state.""" body = ( '{"detail":{"error":"NO_ACTIVE_DEVICES",' '"message":"No active devices available"}}' @@ -352,7 +352,8 @@ def test_503_logs_warning_not_error(self): with pytest.raises(BaasServiceError): service.get_ws_info_by_bot_uuid(bot_uuid="BOT-xyz") - assert _ws_info_by_bot_uuid_logged_calls(spy, "warning"), "503 should log at WARNING" + assert _ws_info_by_bot_uuid_logged_calls(spy, "info"), "503 should log at INFO" + assert not _ws_info_by_bot_uuid_logged_calls(spy, "warning") assert not _ws_info_by_bot_uuid_logged_calls(spy, "error"), "503 must NOT log at ERROR" def test_http_error_still_raises_baas_service_error(self): @@ -559,4 +560,4 @@ def test_get_ws_info_forwards_parameters(self): assert params["device_uuid"] == "device-instance-456" assert result.tenant == "override-tenant" - assert result.engine_port == 18900 \ No newline at end of file + assert result.engine_port == 18900 diff --git a/src/backend/tests/community/core/skill_center/test_local_skill_upload_service.py b/src/backend/tests/community/core/skill_center/test_local_skill_upload_service.py index 9ff185e45..3d5c0ed9c 100644 --- a/src/backend/tests/community/core/skill_center/test_local_skill_upload_service.py +++ b/src/backend/tests/community/core/skill_center/test_local_skill_upload_service.py @@ -1437,6 +1437,7 @@ def get_by_id_and_owner(self, bot_id, owner_id): "bot_id": bot_id, "owner_id": owner_id, "bot_type": "desktop", + "status": "ACTIVE", } class _FailingTasks: diff --git a/src/backend/tests/community/integration/skill_center/test_permanent_package_recovery.py b/src/backend/tests/community/integration/skill_center/test_permanent_package_recovery.py index 00383370d..3fb213ce2 100644 --- a/src/backend/tests/community/integration/skill_center/test_permanent_package_recovery.py +++ b/src/backend/tests/community/integration/skill_center/test_permanent_package_recovery.py @@ -76,6 +76,7 @@ def __init__(self, distribution) -> None: "bot_type": "desktop", "active_engine": "openclaw", "binding_id": 17, + "status": "ACTIVE", }, engine="openclaw", projection=RuntimeSkillProjection(