Prompt management and redid the webhooks page #445
Conversation
WalkthroughUpdated organization settings docs: split SSO and LLM entries, added Webhooks, Course Action Data, and Prompt Management pages; reorganized Organization Owners feature list and updated wording across navigation and feature bullets. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Pre-merge checks✅ Passed checks (3 passed)
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro 📒 Files selected for processing (3)
🧰 Additional context used📓 Path-based instructions (1)**/*.rst⚙️ CodeRabbit configuration file
Files:
🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
source/instructors/admin/organization/webhook.rst (1)
2-2: Fix typos in heading and description.Lines 2 and 9 contain typos that diminish document quality:
- Line 2: "webhoooks" → "webhooks"
- Line 9: "Webbhooks" → "Webhooks"
Suggested fixes:
- :description: Allow webhoooks to support passing of student data to extenal servers + :description: Allow webhooks to support passing of student data to external servers-Webbhooks provide a way to receive Codio events at an endpoint you specify. +Webhooks provide a way to receive Codio events at an endpoint you specify.Also note: Line 2 has another typo—"extenal" should be "external".
Also applies to: 9-9
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (2)
source/img/manage_organization/addprompt.pngis excluded by!**/*.png,!**/*.pngsource/img/manage_organization/promptmanagement.pngis excluded by!**/*.png,!**/*.png
📒 Files selected for processing (3)
source/instructors/admin/organization/organisation.rst(3 hunks)source/instructors/admin/organization/prompt-management.rst(1 hunks)source/instructors/admin/organization/webhook.rst(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rst
⚙️ CodeRabbit configuration file
**/*.rst: Review files for:
- Consistent formatting (e.g., headings, lists, links).
- Anywhere there are tables, they should use
list-table.- Clear and concise language.
- Correct grammar and spelling.
- Proper use of rst syntax (e.g., avoid broken links or invalid code blocks).
- Adherence to style guides (e.g., consistent tone, terminology).
Suggest improvements to enhance readability and accessibility.- Make sure to find all improvements on the first pass.
Go through the material multiple times before submitting comments.
Files:
source/instructors/admin/organization/webhook.rstsource/instructors/admin/organization/prompt-management.rstsource/instructors/admin/organization/organisation.rst
🔇 Additional comments (1)
source/instructors/admin/organization/organisation.rst (1)
57-61: All cross-reference labels are correctly defined.The five reference labels in lines 57–61 of organisation.rst all have matching anchor definitions in their target files:
sso-integration→ sso.rst (line 4)org_llm→ llms.rst (line 4)webhooks→ webhook.rst (line 4)course-action-data→ course-action-data.rst (line 4)prompt-management→ prompt-management.rst (line 4)No broken cross-references detected. Build warnings should not occur from these references.
| - **Overview** - Update organization profile, manage the ability to create public objects, manage Codio support access, manage education settings, and delete the organization. | ||
| - **Members** - View, add, and remove users in your organization and invite teachers to the organization. | ||
| - **Billing** - View your Codio subscription information. | ||
| - **Rubrics** - Create and manage your grading templates. | ||
| - **Dashboard** - Specify Dashboard configurations for students, teachers and read-only teachers. | ||
| - **IP Consent** - Enable or disable IP Consent and manage the versions. | ||
| - **Custom Scripts** - Manage custom scripts used to integrate third-party systems to help and track students. | ||
| - **Extensions** - Add and enable custom extensions for the organization. | ||
| - **LTI Integrations** - View and manage LTI integration settings. | ||
| - **Integrations** - Specify API integrations, Webhooks and SSO Integration. | ||
| - **Assessment Libraries** - Create and manage assessment libraries for your organization. | ||
| - **LLMs** - Specify API keys for LLM models or add a custom LLM provider. | ||
| - **Course Action Data** - Download a report of all course management activities. | ||
| - **Prompt Management** - Manage prompts for the entire organization. |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Review feature description consistency and completeness.
The reorganized feature descriptions (lines 77–90) are now better structured. However, ensure the new descriptions for Integrations (line 86) and Prompt Management (line 90) align with the content and complexity level of other feature descriptions:
- Line 86: "Integrations - Specify API integrations, Webhooks and SSO Integration." — Clear and concise.
- Line 90: "Prompt Management - Manage prompts for the entire organization." — Consider whether this needs more detail about the business value (e.g., "...for use in custom assistants or course materials").
Additionally, verify that the description for Course Action Data (line 89) matches the purpose of that feature.
🤖 Prompt for AI Agents
In source/instructors/admin/organization/organisation.rst around lines 77–90,
update the short feature descriptions to ensure consistency and completeness:
keep "Integrations - Specify API integrations, Webhooks and SSO Integration."
as-is, expand "Prompt Management" to something like "Prompt Management - Manage
and version prompts for the entire organization for use in custom assistants,
course materials, and automated grading workflows" to convey business value, and
verify/adjust the "Course Action Data" line so it clearly reflects the feature
purpose (for example "Course Action Data - Download detailed reports of course
management activities, including timestamps, user actions and affected
resources") so all entries match the level of detail and clarity of the others.
| This feature allows instructors to store prompts at an organization level. These prompts have unique **Prompt IDs** that can be used in custom assistants. | ||
|
|
||
| 1. Click your username in the top-right corner, then select **Organization** from the menu. | ||
|
|
||
| 2. In the **Organizations** tab, click the name of your organization. | ||
|
|
||
| 3. Click the **Prompt Management** tab and then click **Add Prompt**. | ||
|
|
||
| .. image:: /img/manage_organization/addprompt.png | ||
| :alt: Add a prompt | ||
|
|
||
| Creating the Prompt | ||
| ~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| Enter the information you want to provide with your prompt along with the text of the prompt. | ||
|
|
||
| - Each Prompt must have a unique **Prompt ID**. | ||
| - Use the buttons to create templates to specify the data you want to send. For example, **Add Instructor View** will add the tag: ``{{"type": "INSTRUCTOR_VIEW", "filepath": ""}}`` which will pull the contents of the specified file. | ||
| - You can use this feature to access solutions in the ``.guides\secure`` folder. See the example below. |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
wc -l source/instructors/admin/organization/prompt-management.rstRepository: codio/knowledge
Length of output: 122
🏁 Script executed:
cat -n source/instructors/admin/organization/prompt-management.rstRepository: codio/knowledge
Length of output: 4073
Clarify when file contents are captured and what happens if files are later modified or deleted.
The documentation doesn't specify whether file contents are captured at prompt creation time, retrieved dynamically when the prompt is used, or cached. It also doesn't explain what happens if a referenced file is modified or deleted after the prompt is created. Add clarity on:
- Whether file contents are read at prompt creation or retrieved dynamically during usage
- Whether subsequent file changes are reflected in prompt execution
- Error handling if a referenced file is moved or deleted
- Any data retention or versioning approach for captured file contents
🤖 Prompt for AI Agents
In source/instructors/admin/organization/prompt-management.rst around lines 10
to 28, the docs lack clarity about when referenced file contents are read and
what happens if those files change or are deleted; update the Prompt Management
section to explicitly state (1) whether file contents are captured at prompt
creation time or fetched dynamically at runtime, (2) whether later file edits
are reflected in prompt execution (e.g., immediate/dynamic, cached until manual
refresh, or immutable snapshot), (3) the error behavior when a referenced file
is missing or moved (what user sees and any fallback), and (4) any
retention/versioning policy for stored/captured file contents (how long
snapshots are kept, whether version history is preserved, and how to refresh or
re-capture); add a short example sentence showing how to reference files and a
brief note on how admins can manage or re-scan captured files if applicable.
| .. code:: javascript | ||
| (async function(codioIDE, window) { | ||
| const systemPrompt = "System Prompt for the LLM goes here" | ||
| codioIDE.coachBot.register("iNeedHelpButton", "I have a question", onButtonPress) | ||
| async function onButtonPress() { | ||
| const context = await codioIDE.coachBot.getContext() | ||
| const userPrompt = "{% prompt 'TEST_PROMPT_1' %}" | ||
| const result = await codioIDE.coachBot.ask({ | ||
| systemPrompt: systemPrompt, | ||
| userPrompt: userPrompt, | ||
| vars: { | ||
| "GUIDE_CONTENT": context.guidesPage.content, | ||
| } | ||
| }) | ||
| } | ||
| })(window.codioIDE, window) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -type f -name "*.rst" 2>/dev/null | wc -lRepository: codio/knowledge
Length of output: 62
🌐 Web query:
Codio IDE coachBot API context guidesPage documentation
💡 Result:
Found the Codio coachBot API docs and the Guides (guidesPage) authoring docs:
-
coachBot API reference — shows methods (getContext(), getHistory(), ask(), write(), showTooltip(), etc.) and the CoachBotContext type (which includes guidesPage). See coachBot.getContext() returns CoachBotContext with a guidesPage property. [1]
-
Guides (authoring) documentation — explains Guides pages and authoring features (what a guidesPage can control in the IDE). [2]
If you want, I can extract the exact coachBot.getContext() schema fields for guidesPage and show example code calling coachBot.getContext() and using guidesPage.
🌐 Web query:
Codio coachBot prompt template syntax {% prompt %}
💡 Result:
I can't find official Codio docs showing a "{% prompt %}" template tag. Could you paste where you saw that syntax (a guide file, assignment metadata, or a screenshot)? I can then look it up and explain the exact semantics and examples. [Codio coachBot docs / Virtual Coach overview I checked]. [1][2]
Sources:
- Codio coachBot API docs. [1]
- Codio Virtual Coach / assignment settings. [2]
🌐 Web query:
codioIDE.coachBot.ask context object properties documentation
💡 Result:
The Coach Bot context object (returned by codioIDE.coachBot.getContext / available to ask) has these top-level properties:
- guidesPage — opened guides page info.
- assignmentData — assignment info.
- files — array of opened files info.
- error — current error info.
Source: codioIDE.coachBot API docs. [1]
Reference:
[1] codioIDE.coachBot (API docs) — CoachBotContext properties.
🏁 Script executed:
find . -type f -name "*prompt*" 2>/dev/nullRepository: codio/knowledge
Length of output: 120
🏁 Script executed:
rg "context\.guidesPage" -A 5 -B 5Repository: codio/knowledge
Length of output: 1018
🏁 Script executed:
rg "TEST_PROMPT_1" -A 10 -B 10Repository: codio/knowledge
Length of output: 1906
🏁 Script executed:
cat -n source/instructors/admin/organization/prompt-management.rstRepository: codio/knowledge
Length of output: 4073
🏁 Script executed:
rg "userPrompt.*prompt" -A 2 -B 2Repository: codio/knowledge
Length of output: 591
🏁 Script executed:
rg "template.*syntax\|prompt.*syntax" -iRepository: codio/knowledge
Length of output: 41
🏁 Script executed:
grep -r "guidesPage" source/ | head -10Repository: codio/knowledge
Length of output: 175
Provide documentation for template syntax, Prompt ID handling, and context object properties.
The code example uses three undocumented features:
-
Template syntax
{% prompt 'TEST_PROMPT_1' %}— No explanation of how this is processed (server-side rendering, client-side replacement, or custom syntax). How should developers reference Prompt IDs? -
Error handling — What happens if the Prompt ID doesn't exist or is invalid? Should the code include try-catch or validation?
-
Context object —
context.guidesPage.contentis accessed without documenting what properties are available. Are there other accessible properties oncontext?
Add explanatory text before or after the code block describing these aspects, or link to API documentation for the coachBot API.
| Enter the URL of your server and click **Create**, the system will send a test request to check endpoint validity. | ||
|
|
||
|
|
||
| All requests contain a JWT signature you can verify using Codio keys https://apollo.codio.com/lti/oidc/certs (or https://apollo.codio.co.uk/lti/oidc/certs if you are working on codio.co.uk). |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Expand JWT verification guidance for clarity.
Line 27 references JWT signature verification using Codio keys, which is important security information but may be unclear to non-technical readers. Consider adding:
- A brief explanation of what JWT is and why verification matters
- Whether verification is mandatory or optional
- Link to more detailed security documentation if available
🤖 Prompt for AI Agents
In source/instructors/admin/organization/webhook.rst around line 27, the current
sentence about verifying the JWT with Codio keys is too terse; update it to add
a one-sentence plain-language definition of JWT (signed JSON Web Token used to
assert identity and claims), state explicitly that verification of the signature
is mandatory for security, show the exact Certs URL(s) to use (apollo.codio.com
and apollo.codio.co.uk) and briefly note the verification steps (fetch keys,
validate signature and audience/issuer/expiry), and add a link to the platform
security or developer security docs for full guidance.
Mohit - can you please take a pass through this and let me know what else needs to be added? Don't worry too much about formatting or grammar because coderabbit and other reviewers will focus on those. More about should there be more information.. I didn't really explain things about when the file data is taken and I'm not sure if customers need to know that. But maybe more needs to be said about the files that are used as data.
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.