diff --git a/plugins/aem/cloud-service/skills/aem-workflow/workflow-debugging/SKILL.md b/plugins/aem/cloud-service/skills/aem-workflow/workflow-debugging/SKILL.md index 9642d8c29..2ba169b96 100644 --- a/plugins/aem/cloud-service/skills/aem-workflow/workflow-debugging/SKILL.md +++ b/plugins/aem/cloud-service/skills/aem-workflow/workflow-debugging/SKILL.md @@ -48,7 +48,7 @@ This skill is largely self-contained but routes back into the dev skills when th | Workflow not starting (launcher) | workflow_not_starting_launcher | Launcher enabled; path/event match payload. | | Workflow fails or shows error | workflow_fails_or_shows_error | Instance history; error.log for instance ID; payload and process. | | Step failed, retries exhausted | step_failed_retries_exhausted | Logs → `process.label` → Inbox Retry, or bulk via custom servlet (see Step 6). | -| Stale (no current work item) | stale_workflow_no_work_item | Deploy a custom `StaleWorkflowServlet` to your `core` bundle; invoke with `?dryRun=true`. | +| Stale (no current work item) | stale_workflow_no_work_item | Deploy a custom `StaleWorkflowServlet` to your `core` bundle (a privileged endpoint — secure it per the Step 6 checklist first); invoke with `?dryRun=true`. | | Repository bloat / too many instances | repository_bloat_too_many_instances | Purge Scheduler OSGi config in Git (PID: `com.adobe.granite.workflow.purge.Scheduler`). | | User cannot see or complete item | user_cannot_see_or_complete_item | Assignee / initiator / superuser group; `enforce*Permissions` flags. | | Cannot delete model | cannot_delete_model | Count RUNNING instances via Workflow Console → terminate → delete model. | @@ -60,7 +60,7 @@ This skill is largely self-contained but routes back into the dev skills when th ## Step 2: Decision tree (workflow stuck) -1. **No current work item?** → Stale. Deploy a custom `StaleWorkflowServlet` to your `core` bundle; call `GET /bin/support/workflow/stale?dryRun=true` to enumerate, then `POST ...?dryRun=false` to restart. +1. **No current work item?** → Stale. Deploy a custom `StaleWorkflowServlet` to your `core` bundle — this is a privileged endpoint; secure it per the checklist in Step 6 before deploying. Call `GET /bin/support/workflow/stale?dryRun=true` to enumerate, then `POST ...?dryRun=false` to restart. 2. **Participant step** → Assignee exists? Inbox visible? Payload accessible? Dynamic participant resolver returning correct user? 3. **Process step** → Search error.log for instance ID. Check: `process.label` registered, payload path exists, bundle active, no exception in `execute()`. 4. **OR/AND Split** → Condition evaluates correctly? Routes exist? No dead-end branches? Model synced? @@ -198,13 +198,20 @@ If the numbers don't change, the PID is Adobe-managed on your environment — ** |--------|------------------------| | Retry failed work item (single) | `/aem/inbox` → select failure → **Retry**. History and audit trail preserved. | | Retry failed work items (bulk) | **Preferred:** iterate `/aem/inbox` UI — single-item Retry preserves the original instance, its history, and its audit trail. **Not recommended:** a "bulk" servlet using `terminateWorkflow(wf)` + `startWorkflow(model, data)` — this creates a **new** instance and **loses** the original history, step durations, and comments. Only use the replay approach with explicit customer approval and *never* for audit-regulated workflows (pharma, finance, legal). | -| Restart stale workflows | Deploy a custom `StaleWorkflowServlet` to your `core` bundle. Always invoke `GET /bin/support/workflow/stale?dryRun=true` first; confirm scope; then `POST ...?dryRun=false`. Scope with `&model=` if you only want one model. | +| Restart stale workflows | Deploy a custom `StaleWorkflowServlet` to your `core` bundle — **secure it per the checklist below before shipping.** Always invoke `GET /bin/support/workflow/stale?dryRun=true` first; confirm scope; then `POST ...?dryRun=false`. Scope with `&model=` if you only want one model. | | Purge completed | Deploy `com.adobe.granite.workflow.purge.Scheduler-.cfg.json` with `scheduledpurge.workflowStatus=["COMPLETED"]` (array-typed) and `scheduledpurge.daysold=`. Triggered by the **Granite Maintenance Task window** — this PID has **no** `scheduledpurge.cron`; any cron property is silently ignored. **Do not** reference `/libs/granite/operations/config/maintenance` — on AEMaaCS `/libs` is the read-only code layer. One purge config file per schedule. Deploy via pipeline. | | Increase parallelism | `queue.maxparallel` on `org.apache.sling.event.jobs.QueueConfiguration-.cfg.json` (topics: `com/adobe/granite/workflow/job/*`). Adobe's *Workflows Best Practices* recommends staying between half and three-quarters of available CPU cores. The commonly cited `cq.workflow.job.max.procs` is an **orphaned metatype label** with no Java code path that reads it (verified against AEM source on `master`) — do not waste a deployment on it. **Verify after deploy:** Developer Console → `/system/console/slingevent` → find the **Granite Workflow Queue** row and confirm `queue.maxparallel` shows your value. If it still shows the OOTB value (`0.5` on the AEMaaCS SDK), your override lost the `service.ranking` tiebreak — raise `service.ranking` in your override (e.g. from `100` to `1000`) and redeploy. If your ranking *matches* Adobe's OOB ranking exactly, Sling can register both queues against the same topic and occasionally execute a workflow step twice — always set a higher, non-equal ranking. Watch for `refreshing the session since we had to wait for a lock` after raising; if it appears, lower parallelism or stagger launchers. | | Fix thread pool exhaustion | Short-term: **open an Adobe Support ticket** requesting a pod restart for the affected environment — AEMaaCS does **not** expose a customer-facing restart action in Cloud Manager. Long-term, all via Git + pipeline: (1) fix the stuck scheduler (add HTTP timeouts; `@Component scheduler.concurrent=false`); (2) set `blockPolicy=RUN` in `org.apache.sling.commons.threads.impl.DefaultThreadPool-default.cfg.json`; (3) raise `maxPoolSize` to 50. Verify the thread-pool config actually applied — see the AEMaaCS caveat in Step 5. | | Fix process not found | Redeploy the `core` bundle; the `@Component process.label` must exactly match the model's Process step. Re-sync the workflow model from `/libs/cq/workflow/admin` after deploy. | | Fix auto-advancement | Verify `sling-default-*` pool not saturated in thread dump; `com/adobe/granite/workflow/timeout/job` topic active on the Sling Jobs page (`/system/console/slingevent`) — it is a Sling Job topic, not a Sling Scheduler entry; `blockPolicy=RUN` on the `default` pool. | +> **Securing a custom support servlet (e.g. `StaleWorkflowServlet`, bulk-retry servlet):** These endpoints restart, replay, or purge workflows and must be treated as privileged. A servlet that does any of this with no access control is a remote-code-style hazard — do not ship one without all of the following: +> - **Author-only, non-public path.** Bind it under an author-only path such as `/bin/support/...` and confirm your Dispatcher/CDN config does **not** route `/bin/support` on publish. Where practical, bind by resource type behind an access-controlled resource rather than an open path. +> - **Authorize every request in code.** Resolve the request user and verify membership in an operations/admin group (created via `repoinit`) before doing any work; return `403` otherwise. Never rely on the path alone for protection. +> - **Service user for the elevated work — never an admin session.** Map a dedicated sub-service via `org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.amended-*` and grant it only the workflow ACLs it needs (see `workflow-development`). Never call `loginAdministrative()`. +> - **Default to dry run.** Treat a missing or invalid `dryRun` as `true`; require an explicit `dryRun=false` on `POST` for the destructive path. Log the caller, scope (`model`, instance count), and outcome for audit. +> - **Scope and cap.** Always support `&model=` so an operator can limit blast radius, and cap the number of instances acted on per call. + > **Pod-restart reality on AEMaaCS:** Cloud Manager does **not** expose a customer-facing pod-restart or env-restart action. The only way a customer can trigger a restart is an Adobe Support ticket. A restart bounces the running author/publish node — in-flight authoring sessions are lost, active jobs are requeued, there is no hot-swap. Treat it as last-resort mitigation, not a fix, and always file the long-term code/config fix in the same support conversation. --- @@ -247,7 +254,7 @@ If the numbers don't change, the PID is Adobe-managed on your environment — ** **Root cause:** `Cannot archive workitem` during transition; JCR session crash during step completion. -**Diagnosis:** grep Cloud Manager logs for `Cannot archive workitem`. For live count, deploy a custom `StaleWorkflowServlet` and invoke `GET /bin/support/workflow/stale?dryRun=true` — it returns a JSON report without side effects. +**Diagnosis:** grep Cloud Manager logs for `Cannot archive workitem`. For live count, deploy a custom `StaleWorkflowServlet` (secure it per the Step 6 checklist) and invoke `GET /bin/support/workflow/stale?dryRun=true` — it returns a JSON report without side effects. --- diff --git a/plugins/aem/cloud-service/skills/aem-workflow/workflow-triaging/SKILL.md b/plugins/aem/cloud-service/skills/aem-workflow/workflow-triaging/SKILL.md index fcaeaf9f8..bf5926dda 100644 --- a/plugins/aem/cloud-service/skills/aem-workflow/workflow-triaging/SKILL.md +++ b/plugins/aem/cloud-service/skills/aem-workflow/workflow-triaging/SKILL.md @@ -6,21 +6,32 @@ license: Apache-2.0 # AEM Workflow Triaging — Cloud Service -Classify workflow issues, determine what logs and data to gather, and map to the correct runbook or log search. Optimized for **production support** on **AEM as a Cloud Service**. +Classify workflow issues, determine what logs and data to gather, and map to the correct runbook or log search. Optimized for **production support** on **AEM as a Cloud Service (AEMaaCS)**. + +## Audience + +AEMaaCS developers and operators (and the IDE LLM acting on their behalf) classifying workflow incidents across one or more environments — using environment ID + time-range + Cloud Manager Logs (or a log aggregator such as Splunk, if you forward AEMaaCS logs there), before drilling into one instance. Use this skill for cross-environment log mining and symptom classification; switch to `workflow-debugging` once the instance and root cause are identified. ## Variant Scope -- This skill is **cloud-service-only**. -- Log access via Cloud Manager download or log streaming. -- No JMX — workflow counts and queue metrics come from logs, APIs, or Developer Console. +- AEM as a Cloud Service only. +- **Not for AEM 6.5 LTS / AMS.** If the target is 6.5 LTS, stop and use the 6.5-lts variant of this skill — Splunk index/sourcetype paths, the JMX surface, and several log signatures there do not apply as written on AEMaaCS. +- Log access via **Cloud Manager** → Environments → Logs (download or streaming), or Splunk if logs are indexed there. +- **No JMX on AEMaaCS production.** Workflow counts and queue metrics come from logs, Developer Console status producers, and the Sling Job Console. JMX MBeans exist on the local AEMaaCS SDK (`localhost:4502`) but must not be assumed available on cloud environments. +- **All remediation lands via Git + Cloud Manager pipeline.** There is no Felix Console write access or Package Manager on production AEMaaCS environments. + +## Dependencies + +- `workflow-debugging` — once a symptom is classified and an environment/instance is identified, route here for the step-by-step runbook and remediation. +- `workflow-debugging/reference.md` — canonical diagnostic tool pointers, log patterns, and external doc links for AEMaaCS. --- ## When to use this skill -- User asks: "Workflow errors on <host> for the past X hours", "Workflow activity on <host>", "Why did workflow X fail?", "What should I collect to debug this workflow ticket?" +- User asks: "Workflow errors on `` for the past X hours", "Workflow activity on ``", "Why did workflow X fail?", "What should I collect to debug this workflow ticket?" - User needs: Symptom classification, log patterns to search, Splunk queries, or required inputs for a runbook. -- Context: AEM Cloud Service (e.g. cm-p12345-e67890). +- Context: AEM Cloud Service (environment ID format: `cm-p-e`). --- @@ -40,8 +51,11 @@ Map the user's description to a **symptom_id** and runbook. | User cannot see work item or complete/delegate/return | user_cannot_see_or_complete_item | runbook-inbox-and-permissions.md | | Cannot delete workflow model (running instances) | cannot_delete_model | runbook-model-delete-and-update.md | | Jobs queued a long time; slow completion; queue depth high | slow_throughput_queue_backlog | runbook-job-throughput-and-concurrency.md | +| Auto-advance / timeout jobs not firing; participant step stuck past its configured timeout | workflow_auto_advance_failure | runbook-job-throughput-and-concurrency.md | | New or changed workflow not starting or step not executing | workflow_setup_validation | runbook-validate-workflow-setup.md | +> **WorkItem vs. TaskManager task — do not confuse these.** Most workflow Inbox items are workflow work items (`WorkItem`), created by Participant steps and managed by the workflow engine; they are stored under `/var/workflow/instances`, not in TaskManager. TaskManager (`/var/taskmanagement/tasks`) only holds tasks created explicitly via the Task API — used by Projects, Assets tasks, and custom integrations. For `task_not_in_inbox` and `user_cannot_see_or_complete_item` symptoms on a workflow: investigate the Participant step assignee configuration, Inbox filters, and workflow permissions — not TaskManager storage. Diagnosing the wrong backend wastes significant time. + --- ## Step 2: Required inputs for triage @@ -50,34 +64,37 @@ Before suggesting a runbook or Splunk search, try to obtain: | Input | Purpose | |-------|---------| -| **Host / instance** | e.g. cm-p163724-e1759416 (Cloud Service program-environment format). | +| **Environment ID** | AEMaaCS format: `cm-p-e` (e.g. `cm-p163724-e1759416`). | | **Time range** | e.g. "past 4 hours", "past 10 hours" – for log/Splunk scope. | | **Workflow model or step name** | e.g. "Dynamic Media Reupload", "DAM Update Asset", "testmodel". | -| **Instance ID** (if known) | From Workflow console URL or payload; ties logs to one instance. | -| **Payload path** (if known) | e.g. /content/dam/...; for path-related errors. | +| **Instance ID** (if known) | From Workflow Console URL or payload; ties logs to one instance. | +| **Payload path** (if known) | e.g. `/content/dam/...`; for path-related errors. | | **Log source** | Cloud Manager log download, log streaming, or Splunk index/sourcetype. | -If the user only provides host + time, respond with the **generic** workflow error searches and note that narrowing by model/instance ID will improve accuracy. +If the user only provides environment ID + time, respond with the **generic** workflow error searches and note that narrowing by model or instance ID will improve accuracy. --- ## Step 3: Log patterns and Splunk (what to search) -Logs on Cloud Service are accessed via **Cloud Manager** → Environments → Logs (download or streaming). When logs are in **Splunk** (or any log aggregator), use these patterns. +Logs on AEMaaCS are accessed via **Cloud Manager** → Environments → Logs (download or streaming). The primary file is `error.log`. When logs are indexed in **Splunk** (or any log aggregator), use these patterns. -| Scenario | Primary log pattern(s) | Splunk hint | -|----------|------------------------|-------------| +| Scenario | Primary log pattern(s) | Note | +|----------|------------------------|------| | Step failed | `Error executing workflow step` | Add instance ID or model name to narrow. | -| Process not found | `getProcess for '*' failed` | Extract process name for OSGi check. | +| Process not found | `getProcess for '*' failed` | Extract process name; check OSGi Components for `process.label` mismatch. | | Stuck at Process step | Same as step failed + `getProcess` | Combine with payload path. | -| Stale workflow | `Cannot archive workitem` | Correlate time with instance. | -| Lock / throughput | `wait for a lock` or `refreshing the session since we had to wait` | Timechart by host. | -| Permission | `Terminate failed` / `Resume failed` / `Suspend failed` + verifyAccess | Or `AccessControlException`. | -| Payload path | `PathNotFoundException` + workflow/payload | Launcher: "launcher config". | +| Stale workflow | `Cannot archive workitem` | Correlate time with instance ID. | +| Lock / throughput | `refreshing the session since we had to wait for a lock` | Reduce effective concurrency — on AEMaaCS, job queue settings are not directly tunable at runtime; address via code changes: split workflows, offload heavy steps asynchronously, or externalize processing. Raising concurrency makes lock contention worse. | +| Permission | `Terminate failed` / `Resume failed` / `Suspend failed` + verifyAccess | Or `AccessControlException`. Check `enforceWorkflowInitiatorPermissions`. | +| Payload path | `PathNotFoundException` (workflow/payload) | Payload deleted, or launcher config path missing. | | Launcher not starting | `Error adding launcher config` / `Error retrieving launcher config entries` | Path: `/conf/global/settings/workflow/launcher/config`. | | Purge failure | `Workflow purge '*' :` | Filter by repository exception / invalid state. | +| Transient workflow retries exhausted | `retrys exceeded - remove isTransient` | Process step kept throwing after `cq.workflow.job.retry` retries. Fix step code; instance persisted for admin handling. | +| Thread pool full | `RejectedExecutionException` | `default` pool saturated with `blockPolicy=ABORT`; timeout/auto-advance jobs dropped. | +| Operation on finished instance | `Workflow is already finished` | Check logic that calls terminate/resume on a completed or aborted instance. | -**Example Splunk searches (replace index/sourcetype/field names as needed):** +**Example Splunk searches (replace index/sourcetype/field names for your environment):** - All workflow step errors (last 24h): `index=aem sourcetype=aem:error "Error executing workflow step" | table _time host message | sort - _time` @@ -86,50 +103,85 @@ Logs on Cloud Service are accessed via **Cloud Manager** → Environments → Lo - By workflow model or instance: `index=aem ("Error executing workflow step" OR WorkflowException) (message=** OR message=**) | sort - _time` - Lock contention: - `index=aem "wait for a lock" OR "refreshing the session since we had to wait" | table _time host message` + `index=aem "refreshing the session since we had to wait for a lock" | table _time host message` +- Thread pool exhaustion (auto-advance impact): + `index=aem "RejectedExecutionException" | table _time host message` + +> **Note:** Indexes and sourcetypes vary by organization; adapt queries accordingly. + +--- + +## Step 4: Developer Console and Sling Job diagnostics + +On AEMaaCS production, use the **Developer Console** status producers and the **Sling Jobs page** for metrics not available from logs alone. JMX is not available on production AEMaaCS; these are the equivalents. + +| What to check | Tool / URL | Purpose | +|---------------|-----------|---------| +| Workflow queue depth and failed jobs | Sling Jobs page: `/system/console/slingevent` | `Queued Jobs > 0` with `Active Jobs = 0` → jobs not being picked up. `Failed Jobs` count per topic. | +| Workflow job topic statistics | Sling Jobs page: topic `com/adobe/granite/workflow/job/var/workflow/models/` | High `Failed Jobs` / low `Finished Jobs` → process step throwing exceptions. | +| Sling `default` thread pool saturation | Thread Pools page: `/system/console/status-slingthreadpools` | `active count = max pool size` AND `blockPolicy = ABORT` → new scheduled tasks (including workflow timeout detection) are silently rejected. | +| Thread stack trace | Thread Dump: `/system/console/status-jstack-threaddump` | All `sling-default-*` threads stuck on same stack → blocking culprit for auto-advance failure. | +| Sling Scheduler status | Scheduler page: `/system/console/status-slingscheduler` | Confirm `ApacheSlingdefault` uses `ThreadPool: default`. Note: `com/adobe/granite/workflow/timeout/job` is a Sling Job topic, not visible here — check the Sling Jobs page instead. | +| OSGi bundle / process registration | OSGi Components: `/system/console/components` | Confirm WorkflowProcess component with matching `process.label` is Active. | +| Instance state | Workflow Console: `/libs/cq/workflow/admin/console/content/instances.html` | Instance status, current work item, history. | + +**Developer Console access:** AEM Cloud Service → Developer Console. Status producers (thread dumps, Sling Jobs, thread pools) are read-only on all tiers. On the local AEMaaCS SDK (`localhost:4502/system/console/jmx`) JMX MBeans are also available — use them for local development only; do not document JMX steps for production. + +**Safety:** Never recommend remediation operations that bypass Git + Cloud Manager pipeline (e.g. Felix Console config changes) on cloud environments. All config changes go in `ui.config` and deploy via pipeline. --- -## Step 4: Example triage prompts and responses +## Step 5: Example triage prompts and responses | User prompt | Triage response | -|-------------|------------------| -| "Workflow errors on <host> for the past X hours" | Classify as workflow_fails_or_shows_error / step_failed_retries_exhausted. Search Cloud Manager logs or Splunk for "Error executing workflow step", "Error processing workflow job", "getProcess for … failed" on that host. Route to runbook-workflow-fails-or-shows-error. | -| "Workflow activity on <host> for the past X hours" | Clarify: "activity" = counts (started/completed/failed) or list of errors? For errors, use same searches. For counts on Cloud Service, use log aggregation or custom reporting API — no JMX. | -| "Why did <workflow-or-step> fail? Show failure details." | Need: host, time range, and if possible instance ID. Search Cloud Manager logs for "Error executing workflow step" + model/step name or instance ID; return exception type, message, and stack. Route to runbook-workflow-fails-or-shows-error. | -| "Task not in Inbox" | symptom_id: task_not_in_inbox. Route to runbook-task-not-in-inbox. Gather: instance ID, assignee, whether user is initiator/assignee; check Inbox filters and enforceWorkitemAssigneePermissions. | -| "Workflow not starting" | symptom_id: workflow_not_starting_launcher. Route to runbook-launcher-not-starting. Gather: model name, payload path, launcher config path; search logs for launcher errors. | -| "Workflow stuck / not progressing" | symptom_id: workflow_stuck_not_progressing. Route to runbook-workflow-stuck. First: Does instance have a current work item? If no → stale. If yes, follow decision tree by step type. | +|-------------|-----------------| +| "Workflow errors on `` for the past X hours" | Classify as `workflow_fails_or_shows_error` / `step_failed_retries_exhausted`. Download or stream `error.log` from Cloud Manager; search for `Error executing workflow step`, `Error processing workflow job`, `getProcess for … failed`. Check Sling Jobs page for failed job count per topic. Route to `runbook-workflow-fails-or-shows-error`. | +| "Workflow activity on `` for the past X hours" | Clarify: counts (started/completed/failed) or list of errors? For errors, use log searches above. For counts on AEMaaCS, use Cloud Manager log aggregation or the Sling Jobs page — no JMX. | +| "Why did `` fail? Show failure details." | Need: environment ID, time range, instance ID if known. Search `error.log` for `Error executing workflow step` + model/step name or instance ID; return exception type, message, and stack. Route to `runbook-workflow-fails-or-shows-error`. | +| "Task not in Inbox" | `symptom_id: task_not_in_inbox`. Route to `runbook-task-not-in-inbox`. Gather: instance ID, assignee, whether user is initiator/assignee. Check Inbox filters and `enforceWorkitemAssigneePermissions` via Developer Console OSGi config view. | +| "Workflow not starting" | `symptom_id: workflow_not_starting_launcher`. Route to `runbook-launcher-not-starting`. Gather: model name, payload path, launcher config path; search logs for launcher errors. | +| "Workflow stuck / not progressing" | `symptom_id: workflow_stuck_not_progressing`. Route to `runbook-workflow-stuck`. First: does the instance have a current work item? If no → stale. If yes, follow decision tree by step type. | +| "Auto-advance / timeout jobs not firing" | `symptom_id: workflow_auto_advance_failure`. Route to `runbook-job-throughput-and-concurrency`. Check Developer Console thread dump for `sling-default-*` thread saturation; check Sling Jobs page for `com/adobe/granite/workflow/timeout/job` topic; search `error.log` for `RejectedExecutionException`. | --- -## Step 5: What logs can and cannot answer +## Step 6: What logs and Developer Console can and cannot answer + +**Can answer (with AEM workflow logs from Cloud Manager + Developer Console on AEMaaCS):** -**Can answer (with AEM workflow logs in Cloud Manager / Splunk):** +- Step failures: exception type, message, stack (by environment, time, model, step). +- Process not registered: which `process.label` is missing (logs + Developer Console OSGi Components). +- Stuck: step errors, `getProcess` failures, lock wait, payload/path errors. +- Stale: `Cannot archive workitem` and transition errors in logs. +- Queue metrics: Sling Jobs page (`/system/console/slingevent`) → queued, active, failed per topic. +- Thread pool saturation: Thread Pools page (`/system/console/status-slingthreadpools`). +- Throughput: lock wait, session refresh, JobHandler volume in logs. +- Permission: Terminate/Resume/Suspend failed (`verifyAccess`), `AccessControlException` in logs. +- Payload/launcher: `PathNotFoundException`, launcher config errors in logs. +- Purge: `Workflow purge …` repository exception or invalid state in logs. -- Step failures: exception type, message, stack (by host, time, model, step). -- Process not registered: which `process.label` is missing. -- Stuck: step errors, getProcess failures, lock wait, payload/path errors. -- Stale: "Cannot archive workitem" and transition errors. -- Throughput: lock wait, session refresh, JobHandler volume. -- Permission: Terminate/Resume/Suspend failed (verifyAccess), AccessControlException. -- Payload/launcher: PathNotFoundException, launcher config errors. -- Purge: "Workflow purge …" repository exception or invalid state. +**Cannot answer directly (AEMaaCS limitations vs 6.5 LTS):** -**Cannot answer directly (Cloud Service limitations):** +| What is needed | AEMaaCS alternative | +|----------------|---------------------| +| JMX `countStaleWorkflows` | Deploy a custom `StaleWorkflowServlet` (see `workflow-debugging` Step 6); call with `?dryRun=true`. | +| JMX `countRunningWorkflows` | Workflow Console UI, or a custom count servlet. | +| JMX `retryFailedWorkItems` | Inbox UI → Retry (single); or a custom bulk-retry servlet (see `workflow-debugging` Step 6). | +| JMX `purgeCompleted` | `com.adobe.granite.workflow.purge.Scheduler-.cfg.json` deployed via pipeline. | +| JMX `restartStaleWorkflows` | Custom `StaleWorkflowServlet` with `POST ...?dryRun=false`. | +| Config status ZIP | Developer Console status producers; or request from Adobe Support. | +| Console state (current work item) | Workflow Console UI (`/libs/cq/workflow/admin/...`) or custom API. | +| Runtime process step code behavior | Requires code review + log correlation. | +| Pod restart | Adobe Support ticket — Cloud Manager does not expose a customer-facing restart action. | -- Console state (e.g. "is there a current work item?"). Use Workflow Console UI or custom API. -- JMX counts (e.g. countStaleWorkflows, queue depth). No JMX on Cloud Service — use log aggregation, custom HTTP APIs, or Developer Console. -- Thread pool metrics. Request thread dump via Developer Console or support. -- Configuration status ZIP. Request from support. +> **Custom servlets are privileged — secure them before deploying.** The `StaleWorkflowServlet` and any bulk-retry/count servlet above restart, replay, or enumerate workflows. Treat them as admin-only: authorize the caller against an operations group, use a service user (never an admin session), default to `dryRun=true`, scope by model, and keep the endpoint off publish. Follow the secure-write/deploy checklist in `workflow-debugging` Step 6 before shipping one. -Always pair log-based triage with the appropriate runbook for actions (retry via Inbox, Purge Scheduler config, pipeline deploy). +Always pair log-based triage with Developer Console diagnostics and the appropriate runbook for actions (Inbox Retry, Purge Scheduler config, Cloud Manager pipeline deploy). --- ## References (in repo) -- **Machine-readable index:** `aem-agent-marketplace-workflow-knowledge-base/docs/debugging-index.md` -- **Decision guide:** `runbooks/runbook-decision-guide.md` -- **Splunk scenarios and queries:** `Workflow-docs/splunk-workflow-triaging.md` -- **Error patterns:** `docs/error-patterns.md` +- **Diagnostic tool pointers and log patterns:** [`../workflow-debugging/reference.md`](../workflow-debugging/reference.md) +- **Step-by-step runbook (per symptom):** [`../workflow-debugging/SKILL.md`](../workflow-debugging/SKILL.md) +- **Cloud Service guardrails (paths, service users, OSGi annotations):** [`../workflow-development/references/workflow-foundation/cloud-service-guardrails.md`](../workflow-development/references/workflow-foundation/cloud-service-guardrails.md)