From 4e0352d548e87dcd39c0f124e318bffcda34a12d Mon Sep 17 00:00:00 2001 From: Damian Zaremba Date: Thu, 6 Nov 2025 17:07:29 +0100 Subject: [PATCH] Move existing cluebot3 docs undex Sphinx --- cluebot3/architecture.md | 17 -- cluebot3/deployment.md | 80 -------- pyproject.toml | 3 + source/cluebot3/architecture.rst | 23 +++ source/cluebot3/deployment.rst | 175 ++++++++++++++++++ .../README.md => source/cluebot3/index.rst | 9 +- source/index.rst | 2 + 7 files changed, 211 insertions(+), 98 deletions(-) delete mode 100644 cluebot3/architecture.md delete mode 100644 cluebot3/deployment.md create mode 100644 source/cluebot3/architecture.rst create mode 100644 source/cluebot3/deployment.rst rename cluebot3/README.md => source/cluebot3/index.rst (58%) diff --git a/cluebot3/architecture.md b/cluebot3/architecture.md deleted file mode 100644 index d9af93c..0000000 --- a/cluebot3/architecture.md +++ /dev/null @@ -1,17 +0,0 @@ -# ClueBot 3 Architecture - -## Runtime dependencies - -- `CLUEBOT3_BOT_PASSWORD` environment variable containing the Wiki account password - -## Build dependencies - -- https://github.com/cluebotng/wikipedia.git (managed via `composer.json`) - -## Health checking - -The job framework uses `health_check.php` which causes the pod (job) to be restarted if the bot hasn't edited within the last 24 hours. - -A cron running on one of [https://en.wikipedia.org/wiki/User:DamianZaremba](Damian)'s servers checks the last contribution time, -updating [https://en.wikipedia.org/wiki/User:ClueBot_III/running](https://en.wikipedia.org/wiki/User:ClueBot_III/running) and emailing -[https://en.wikipedia.org/wiki/User:DamianZaremba](Damian) & [https://en.wikipedia.org/wiki/User:Rich_Smith](Rich) if the bot is not running. diff --git a/cluebot3/deployment.md b/cluebot3/deployment.md deleted file mode 100644 index 90de097..0000000 --- a/cluebot3/deployment.md +++ /dev/null @@ -1,80 +0,0 @@ -# ClueBot 3 Deployment - -The bot runs as a [https://wikitech.wikimedia.org/wiki/Portal:Toolforge](Toolforge) job on Wikimedia Cloud Services. - -There is currently only 1 (production) instance of the bot. - -## Production - -Toolforge user: `cluebot3` -Toolforge job: `cluebot3` - -## Setup - -We handle secrets using `envvars`, which need to be created by hand prior to a deployment. - -This should only be required if setting up a new account, or a secret needs to be rotated. - -From within the tool account (see below): -``` -toolforge envvars create CLUEBOT3_BOT_PASSWORD -Enter the value of your envvar (Hit Ctrl+C to cancel): -``` - -## Deployment - -The code is packaged as a container using pack (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images), -which is then deployed as a `continuous` job. - -Any tagged release will be deployed via GitHub actions using (`https://github.com/cluebotng/cluebot3/blob/main/fabfile.py`)(fabric). - -## Manual deployment - -It can be useful to manually execute a deployment, for example when adjusting the `jobs.yaml` or `fabric.py` logic. - -Assuming `fabric` is installed locally, within the root of the `cluebot3` repo you can directly execute: -* `fab deploy` - everything -* `fab deploy-jobs` - update only the `jobs` config - -Note: This requires SSH access to the tool user account. - -## Troubleshooting - -First login to the tool account: -``` -$ ssh login.toolforge.org -$ become cluebot3 -tools.cluebot3@tools-bastion-13:~$ -``` - -## Check the job is running -``` -tools.cluebot3@tools-bastion-12:~$ toolforge jobs list -+-----------+------------+---------+ -| Job name: | Job type: | Status: | -+-----------+------------+---------+ -| cluebot3 | continuous | Running | -+-----------+------------+---------+ -``` - -If the job is not running, check the logs, or investigate the pod status (`kubectl get pod `, `kubectl logs `, `kubectl events`). - -## Check the logs -``` -toolforge jobs logs [--follow] cluebot3 -``` - -For example: -``` -tools.cluebot3@tools-bastion-12:~$ toolforge jobs logs cluebot3 | tail -n 10 -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] Stack trace: -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #0 /workspace/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php(104): Monolog\Handler\StreamHandler->createDir() -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #1 /workspace/vendor/monolog/monolog/src/Monolog/Handler/RotatingFileHandler.php(120): Monolog\Handler\StreamHandler->write() -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #2 /workspace/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(39): Monolog\Handler\RotatingFileHandler->write() -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #3 /workspace/vendor/monolog/monolog/src/Monolog/Logger.php(344): Monolog\Handler\AbstractProcessingHandler->handle() -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #4 /workspace/vendor/monolog/monolog/src/Monolog/Logger.php(422): Monolog\Logger->addRecord() -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #5 /workspace/lib/bot.php(610): Monolog\Logger->addInfo() -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #6 /workspace/cluebot3.php(69): ClueBot3\parsetemplate() -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] #7 {main} -2025-08-07T14:36:32+00:00 [cluebot3-7c774d569b-ndhsz] [job] thrown in /workspace/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php on line 189 -``` diff --git a/pyproject.toml b/pyproject.toml index 389f6d5..9848a69 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,6 +17,9 @@ ruff = "*" bandit = "*" doc8 = "*" +[tool.doc8] +max-line-length = 120 + [build-system] requires = ["poetry-core>=2.0.0,<3.0.0"] build-backend = "poetry.core.masonry.api" diff --git a/source/cluebot3/architecture.rst b/source/cluebot3/architecture.rst new file mode 100644 index 0000000..687c9a4 --- /dev/null +++ b/source/cluebot3/architecture.rst @@ -0,0 +1,23 @@ +Architecture +============ + +Runtime dependencies +-------------------- + +- `CLUEBOT3_BOT_PASSWORD` environment variable containing the Wiki account password + +Build dependencies +------------------ + +- https://github.com/cluebotng/wikipedia.git (managed via `composer.json`) + +Health checking +--------------- + +The runtime executes `health_check.php` which causes a restart if the bot hasn't edited within the last 24 hours. + +The last edit time is exported via the `monitoring-probes `_, +with alerting via the `monitoring `_ setup. + +.. _monitoring_repo: https://github.com/cluebotng/monitoring +.. _probes_repo: https://github.com/cluebotng/monitoring-probes diff --git a/source/cluebot3/deployment.rst b/source/cluebot3/deployment.rst new file mode 100644 index 0000000..9a71170 --- /dev/null +++ b/source/cluebot3/deployment.rst @@ -0,0 +1,175 @@ +Deployment +========== + +The bot runs on `Toolforge `_ (part of Wikimedia Cloud Services). + +.. _toolforge: https://wikitech.wikimedia.org/wiki/Portal:Toolforge + +There is currently only 1 (production) instance of the bot. + +Production +---------- + +- Toolforge user: `cluebot3` +- Toolforge component/job: `cluebot3` + +Setup +----- + +We handle secrets using `envvars `_, which need to be created by hand prior to a deployment. + +This should only be required if setting up a new account, or a secret needs to be rotated. + +From within the tool account (see below) + +.. code-block:: bash + + toolforge envvars create CLUEBOT3_BOT_PASSWORD + Enter the value of your envvar (Hit Ctrl+C to cancel): + +.. _envvars_url: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars + +Deployment +---------- + +Deployments are handled via `components `_, +which coordinates building the image via `pack `_ and running the component via `jobs `_. + +.. _deploy_url: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Deploy_your_tool +.. _jobs_url: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_jobs +.. _pack_url: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images + +Any tagged releases will be deployed via GitHub actions using `component-configs `_. + +.. _component_configs_url: https://github.com/cluebotng/component-configs/ + +Troubleshooting +--------------- + +First login to the tool account: + +.. code-block:: bash + + $ ssh login.toolforge.org + $ become cluebot3 + tools.cluebot3@tools-bastion-13:~$ + +Check the job is running +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + tools.cluebot3@tools-bastion-12:~$ toolforge jobs list + +-----------+------------+---------+ + | Job name: | Job type: | Status: | + +-----------+------------+---------+ + | cluebot3 | continuous | Running | + +-----------+------------+---------+ + +If the job is missing, check the recent deployment status + +.. code-block:: bash + + tools.cluebot3@tools-bastion-15:~$ toolforge components deployment show + Deployment ID: 20251104-171541-c2415tqylj + Created: 20251104-171541 + Status: successful + Long status: + Finished at 2025-11-04 17:15:43.053902 + + Builds: + cluebot3(skipped): id:cluebot3-buildpacks-pipelinerun-45jbr Reusing existing build + + Runs: + cluebot3(successful): job cluebot3 is already up to date, [info](Job cluebot3 is already up to date) + + Tool config: + components: + cluebot3: + build: + ref: refs/tags/v1.2.1 + repository: https://github.com/cluebotng/cluebot3.git + use_latest_versions: true + run: + command: run-bot + cpu: '3' + health_check_script: health-check + memory: 1Gi + +Check the logs +______________ + +.. code-block:: bash + + toolforge jobs logs [--follow] cluebot3 + +For example: + +.. code-block:: bash + + tools.cluebot3@tools-bastion-15:~$ toolforge jobs logs -f cluebot3 + 2025-11-06T14:50:20Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:20] cluebot3.INFO: doarchive(Talk:CYP4F8,Talk:CYP4F8/Archive, %%i,17520,0,0,{{Talkarchive}},{{User:ClueBot III/ArchiveNow}},2,0,0,0,,0,1,) [] [] + 2025-11-06T14:50:20Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:20] cluebot3.INFO: [Talk:CYP4F8] calculated sections: 0 old, 0 current, 0 keep, 0 archive [] [] + 2025-11-06T14:50:20Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:20] cluebot3.INFO: [Talk:CYP4F8] generating index page [] [] + 2025-11-06T14:50:22Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:22] cluebot3.INFO: doarchive(Talk:Eleanor of Castile (1307–1359),Talk:Eleanor of Castile (1307–1359)/Archive, %%i,2160,0,0,{{Talkarchive}},{{User:ClueBot III/ArchiveNow}},2,0,1,0,,150000,1,) [] [] + 2025-11-06T14:50:22Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:22] cluebot3.INFO: [Talk:Eleanor of Castile (1307–1359)] calculated sections: 0 old, 0 current, 0 keep, 0 archive [] [] + 2025-11-06T14:50:22Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:22] cluebot3.INFO: [Talk:Eleanor of Castile (1307–1359)] generating index page [] [] + 2025-11-06T14:50:25Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:25] cluebot3.INFO: doarchive(Talk:United Way,Talk:United Way/Archive,,8760,0,0,{{Talkarchive}},{{User:ClueBot III/ArchiveNow}},2,0,0,0,,0,1,) [] [] + 2025-11-06T14:50:25Z [cluebot3-ffdc8f79c-l9k7m] [job] [2025-11-06 14:50:25] cluebot3.INFO: [Talk:United Way] calculated sections: 0 old, 0 current, 0 keep, 0 archive [] [] + 2025-11-06T14:50:25Z [cluebot3-ffdc8f79c-l9k7m] [job] PHP Warning: Undefined array key "format" in /workspace/lib/bot.php on line 619 + 2025-11-06T14:50:25Z [cluebot3-ffdc8f79c-l9k7m] [job] PHP Warning: Undefined array key "format" in /workspace/lib/bot.php on line 640 + +Check the job status +____________________ + +.. code-block:: bash + + tools.cluebot3@tools-bastion-15:~$ toolforge jobs show cluebot3 + +---------------+------------------------------------------------------------------------+ + | Job name: | cluebot3 | + +---------------+------------------------------------------------------------------------+ + | Command: | run-bot | + +---------------+------------------------------------------------------------------------+ + | Job type: | continuous | + +---------------+------------------------------------------------------------------------+ + | Image: | tool-cluebot3/cluebot3:latest | + +---------------+------------------------------------------------------------------------+ + | Port: | none | + +---------------+------------------------------------------------------------------------+ + | File log: | no | + +---------------+------------------------------------------------------------------------+ + | Output log: | | + +---------------+------------------------------------------------------------------------+ + | Error log: | | + +---------------+------------------------------------------------------------------------+ + | Emails: | none | + +---------------+------------------------------------------------------------------------+ + | Resources: | mem: 1.0Gi, cpu: 3.0 | + +---------------+------------------------------------------------------------------------+ + | Replicas: | 1 | + +---------------+------------------------------------------------------------------------+ + | Mounts: | none | + +---------------+------------------------------------------------------------------------+ + | Retry: | no | + +---------------+------------------------------------------------------------------------+ + | Timeout: | no | + +---------------+------------------------------------------------------------------------+ + | Health check: | script: health-check | + +---------------+------------------------------------------------------------------------+ + | Status: | Running | + +---------------+------------------------------------------------------------------------+ + | Hints: | Last run at 2025-10-29T10:32:03Z. Pod in 'Running' phase. Pod has been | + | | restarted 64 times. State 'running'. Started at | + | | '2025-11-06T14:43:35Z'. | + +---------------+------------------------------------------------------------------------+ + +Check the runtime (kubernetes) +______________________________ + +Sometimes jobs-api is not useful for debugging a failure within kubernetes (the runtime). + +Kubernetes can be interrogated via the usual commands from the tool account e.g.: + +- `kubectl get pod -l app.kubernetes.io/name=cluebot3` +- `kubectl describe pod -l app.kubernetes.io/name=cluebot3` +- `kubectl events` diff --git a/cluebot3/README.md b/source/cluebot3/index.rst similarity index 58% rename from cluebot3/README.md rename to source/cluebot3/index.rst index 52f500c..54ab995 100644 --- a/cluebot3/README.md +++ b/source/cluebot3/index.rst @@ -1,5 +1,12 @@ -# ClueBot III +ClueBot III +=========== This bot provides page archiving on English Wikipedia. Wiki account: https://en.wikipedia.org/wiki/User:ClueBot_III + +.. toctree:: + :maxdepth: 2 + + architecture + deployment diff --git a/source/index.rst b/source/index.rst index 1c9c942..1cb597a 100644 --- a/source/index.rst +++ b/source/index.rst @@ -8,3 +8,5 @@ It is intended as a long term replacement for external sources such as https://d .. toctree:: :maxdepth: 2 :caption: Contents: + + cluebot3/index