Skip to content

feat(messaging): add a messaging category with SQS and Pub/Sub - #157

Merged
fredpena merged 5 commits into
floci-io:mainfrom
TheSaifZaman:feat/messaging-category
Aug 30, 2026
Merged

feat(messaging): add a messaging category with SQS and Pub/Sub#157
fredpena merged 5 commits into
floci-io:mainfrom
TheSaifZaman:feat/messaging-category

Conversation

@TheSaifZaman

Copy link
Copy Markdown
Contributor

Split out of #155 so each service lands on its own. Independent of #155 and #156 — this branch is off main.

The sidebar carried a route-less "Queue" placeholder that could never light up, while Floci core has had three seeded SQS queues the whole time:

$ curl -s .../aws/services/messaging/resources | jq -r '.[]|"\(.name) msgs=\(.metadata.approximateMessages)"'
orders-queue msgs=0
dead-letter-queue msgs=0
notifications-queue msgs=0

One category for both SQS and Pub/Sub rather than separate queue/topic ones: the delivery semantics differ, but a console renders both the same way — a named endpoint with a depth and a few settings — so splitting them would double the nav for nothing.

Contract details worth a look

  • SQS ids. SQS addresses queues by URL, but a URL embeds the endpoint and changes if the runtime is re-pointed, which makes it a poor stable id. The adapter uses the queue name and resolves it via GetQueueUrl per call, as the AWS console does. Epoch-second timestamps are converted to the ISO strings the contract expects.
  • FIFO. Attributes.FifoQueue=true is set for a .fifo name. Worth calling out because neither layer of verification caught its earlier absence: the unit stub accepted any input, and Floci core infers FIFO from the suffix so it worked locally too. It would only have failed against AWS. Caught in review on feat(gcp): add Cloud SQL and GKE adapters #155 and fixed here.
  • Pub/Sub creates with PUT on the resource path, not POST to the collection.
  • A queue whose GetQueueAttributes fails still lists, with detail omitted — a describe failure should degrade one row, not blank the table.

One SPI change

CloudResource.type was a closed union of eleven literals; queue, fifo-queue and topic are the first additions since. It is now open with a KnownResourceType alias documenting the convention — nothing dispatches exhaustively on it, so a closed union meant editing two packages per adapter for no safety. CloudServiceType stays closed, because that one is route-addressable and a typo there should 404.

The identical widening also appears on #156, so whichever lands second is a no-op for that hunk.

Adds @aws-sdk/client-sqs (both bun.lock and pnpm-lock.yaml updated — the API installs with bun --frozen-lockfile, which I learned the hard way).

Verification

pnpm lint, type-check, test, build pass. Live round-trip on both providers:

$ curl -X POST .../aws/services/messaging/resources -d '{"queueName":"probe","visibilityTimeout":"45"}'
  create: 201 → inspect: vis=45 arn=arn:aws:sqs:us-east-1:000000000000:probe → delete: 200 → 404

$ curl -X POST .../gcp/services/messaging/resources -d '{"topicName":"probe"}'
  create: 201 → list: [('probe','topic')] → delete: 200 → []

README service matrix regenerated from the registry.

Type of change

  • Bug fix (fix:)
  • New feature / service UI (feat:)
  • Breaking change (feat!: or fix!:)
  • Docs / chore

Area

  • Frontend (packages/frontend)
  • API / Cloud Proxy (packages/api)
  • Cloud Explorer adapter / schema
  • Build / CI / Docker

Checklist

  • pnpm lint, pnpm type-check, pnpm test, and pnpm build pass locally
  • New or updated tests added where it makes sense (bun test in packages/api)
  • No fake/mock data added — unwired states stay empty or show an explicit placeholder
  • Commit messages / PR title follow Conventional Commits

@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds a schema-driven messaging category with AWS SQS and GCP Pub/Sub adapters registered on the cloud proxy.

  • New AwsSqsAdapter (list/create/get/delete/health) using queue name as id, FIFO FifoQueue attribute, and SQS range validation
  • New GcpPubSubAdapter creating topics via PUT on the resource path
  • Opens CloudResource.type with KnownResourceType on API and frontend; wires client-sqs and registry entries; updates catalog display names and README matrix

Confidence Score: 5/5

The PR appears safe to merge; no blocking failures remain from prior threads or this follow-up pass.

No blocking failure remains.

Important Files Changed

Filename Overview
packages/api/src/adapter-aws/AwsSqsAdapter.ts SQS messaging adapter with name-based ids, attribute normalization, FIFO create, and range validation.
packages/api/src/adapter-gcp/GcpPubSubAdapter.ts Pub/Sub topics adapter using PUT create and short topic ids.
packages/api/src/cloudProxy.ts Registers AwsSqsAdapter and GcpPubSubAdapter on the shared registry.
packages/api/src/cloud-spi/types.ts Widens CloudResource.type to KnownResourceType plus open string brand.
packages/api/src/cloud-spi/messagingSchema.ts AWS/GCP messaging ServiceSchema fields, columns, and CRUD capabilities.

Reviews (4): Last reviewed commit: "chore(messaging): merge main into messag..." | Re-trigger Greptile

@hectorvent hectorvent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Saif, this is a strong one. Using the queue name as the stable id and resolving it per call with GetQueueUrl matches what the AWS console does, and catching the FifoQueue attribute gap (which neither the stub nor Floci core would have surfaced) shows real care for the actual AWS contract. Both lockfiles are regenerated, tests are colocated and thorough on both adapters, and the catalog row follows the new AGENTS.md pattern exactly.

One coordination note before anything else: SQS and the messaging category are the most contested spot in the queue right now. Three earlier open PRs add SQS under a queue category (#137, #142, #148), and #144 and #146 also introduce a messaging category. I am coordinating the taxonomy (queue vs messaging) and how these land, so please treat that as an open question rather than settled by whichever PR merges first. Nothing for you to do yet, but a rebase or renaming may be requested once that call is made.

Two small things on the code itself, neither blocking:

  1. The new doc comment on CloudResource.type gives sqs-queue as an example, but the adapter emits queue and fifo-queue. Could you align the example so the convention reads unambiguously?
  2. visibilityTimeout and messageRetentionPeriod are free-text and passed straight through, so a non-numeric value only fails at the SQS call. A quick numeric check in create() with a ValidationError would give the form a friendlier error. Fine as a follow-up if you prefer.

The Pub/Sub PUT-on-resource-path detail and the degrade-one-row behavior on a failed describe are both exactly right. Nice work.

The sidebar carried a route-less "Queue" placeholder that could never light up,
while Floci core had three seeded SQS queues the whole time. This replaces it
with a real category serving both AWS SQS and GCP Pub/Sub.

One category rather than separate queue and topic ones: the delivery semantics
differ, but a console renders both the same way — a named endpoint with a depth
and a few settings — so splitting them would double the nav for nothing.

Two contract details worth naming:

- SQS identifies a queue by URL, but a URL embeds the endpoint and changes if
  the runtime is re-pointed, which makes it a poor resource id. The adapter uses
  the queue name and resolves it via GetQueueUrl per call, as the AWS console
  does. Timestamps are converted from SQS epoch seconds to the ISO strings the
  contract expects.
- Pub/Sub creates with PUT on the resource path, not POST to the collection.

A queue whose GetQueueAttributes fails still lists, with the detail omitted — a
describe failure should degrade one row, not blank the table.

This is also the first service to need `CloudResource.type` beyond its original
eleven literals. The union is now open with a documented convention and a
`KnownResourceType` alias: nothing dispatches exhaustively on it, so a closed
union meant editing two packages per adapter for no safety. `CloudServiceType`
stays closed, because that one is route-addressable.

Verified against the live stack: the three seeded queues list with their real
ARNs, depths and visibility timeouts, and create/inspect/delete round-trips on
both providers.
The adapter accepted a .fifo name and reported the resulting queue as
`fifo-queue`, but never sent `Attributes.FifoQueue=true` on create. Real SQS
rejects that request outright.

It went unnoticed because neither layer of verification could see it: the unit
test used a stub that accepts any input, and Floci core infers FIFO from the
name suffix, so creating one locally succeeded. The gap would only have appeared
against AWS.

Adds tests asserting the attribute is sent for a .fifo name and omitted
otherwise.
@TheSaifZaman
TheSaifZaman force-pushed the feat/messaging-category branch from baa8e11 to 95a9e0d Compare July 29, 2026 02:50
@TheSaifZaman

Copy link
Copy Markdown
Contributor Author

Rebased onto fd3bd2f after #147, #152 and #155 merged. Mechanical: the CloudResource.type union collided with the 'secret' member #147 added, and cloudProxy.ts collided with the Cloud SQL and GKE registrations #155 added. Both resolved by keeping every member and every registration — never picking a side.

This branch's open KnownResourceType | (string & {}) union supersedes main's closed one, so the collision disappears for good once this lands.

README regenerated from service-matrix.ts rather than hand-merged, so it now shows both the Integration row this PR adds and the Key Vault column main gained.

Gate green after the rebase: lint, type-check, 463 tests, build.

visibilityTimeout and messageRetentionPeriod were free text passed straight through, so a bad value only failed at CreateQueue. Also corrects the CloudResource.type doc example, which said sqs-queue where the adapter emits queue and fifo-queue.
@TheSaifZaman
TheSaifZaman force-pushed the feat/messaging-category branch from caff3da to 3084c30 Compare July 29, 2026 03:32
@TheSaifZaman

Copy link
Copy Markdown
Contributor Author

@hectorvent Both taken, thank you.

1. Doc example corrected. The CloudResource.type comment gave sqs-queue, which this adapter never emits — it is now fifo-queue, so the example is one of the values actually produced.

2. Numeric validation added, with SQS's own bounds rather than a generic "must be a number": visibilityTimeout 0–43200 and messageRetentionPeriod 60–1209600 seconds. Blank still means "omit the attribute", so the optional fields stay optional. Two tests: one for non-numeric and both out-of-range cases, one asserting a valid value is forwarded while a blank one is not.

Encoding the provider's range rather than just the type follows the same principle as the FifoQueue fix — the runtime would have accepted values real SQS rejects.

On the taxonomy coordination: understood, and no objection to whatever the maintainer decides. Flagging one input in case it is useful: this branch now carries the open KnownResourceType | (string & {}) union, so if messaging were renamed to queue or vice versa, the resource type values would not need to change — only the catalog slug and the schema. That should make the rename cheap whichever way it lands.

Gate green: lint, type-check, 465 tests, build. (I pushed a version with a type error in the test file first and amended it within the minute — the checks you see on the final SHA are the real ones.)

@fredpena
fredpena self-requested a review as a code owner August 30, 2026 22:50
@fredpena
fredpena merged commit dfa6d1c into floci-io:main Aug 30, 2026
6 checks passed
@hectorvent

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.4.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

fredpena pushed a commit to TheSaifZaman/floci-ui that referenced this pull request Sep 1, 2026
# [0.4.0](floci-io/floci-ui@0.3.0...0.4.0) (2026-09-01)

### Bug Fixes

* **ec2:** include catalog AMIs in launch selector ([floci-io#191](floci-io#191)) ([b72135d](floci-io@b72135d))

### Features

* **aws:** add CloudFormation adapter to Cloud Explorer ([floci-io#184](floci-io#184)) ([f3d6105](floci-io@f3d6105)), closes [floci-io#81](floci-io#81) [floci-io#75](floci-io#75) [floci-io#81](floci-io#81) [floci-io#75](floci-io#75) [floci-io#81](floci-io#81)
* **aws:** add EventBridge explorer ([floci-io#146](floci-io#146)) ([42944e9](floci-io@42944e9)), closes [floci-io#85](floci-io#85)
* **aws:** add IAM to Cloud Explorer ([floci-io#145](floci-io#145)) ([fc50d19](floci-io@fc50d19)), closes [floci-io#79](floci-io#79)
* **aws:** add Secrets Manager resource adapter ([floci-io#193](floci-io#193)) ([4811866](floci-io@4811866))
* **azure:** add databases and split Cosmos NoSQL ([floci-io#149](floci-io#149)) ([2e704f4](floci-io@2e704f4)), closes [floci-io#92](floci-io#92) [floci-io#67](floci-io#67) [floci-io/floci-az#138](floci-io/floci-az#138) [floci-io#143](floci-io#143)
* **azure:** add Service Bus explorer ([floci-io#144](floci-io#144)) ([a9f0d06](floci-io@a9f0d06)), closes [floci-io#89](floci-io#89)
* **eks:** Manage nodegroups and Fargate profiles via Cloud Proxy ([floci-io#194](floci-io#194)) ([3091cc9](floci-io@3091cc9)), closes [floci-io#106](floci-io#106)
* **loadbalancing:** add an AWS Elastic Load Balancing adapter ([floci-io#168](floci-io#168)) ([10d4298](floci-io@10d4298)), closes [floci-io#162](floci-io#162) [floci-io#162](floci-io#162) [floci-io#156](floci-io#156)
* **messaging:** add a messaging category with SQS and Pub/Sub ([floci-io#157](floci-io#157)) ([dfa6d1c](floci-io@dfa6d1c)), closes [floci-io#155](floci-io#155) [floci-io#155](floci-io#155) [floci-io#156](floci-io#156) [floci-io#156](floci-io#156)
* **secretsmanager:** add JSON key-value editor for secret values ([floci-io#195](floci-io#195)) ([8e88961](floci-io@8e88961)), closes [floci-io#151](floci-io#151)
* **ses:** Add AWS SES mailbox to Cloud Explorer ([floci-io#196](floci-io#196)) ([6389a56](floci-io@6389a56)), closes [floci-io#130](floci-io#130)
fredpena added a commit that referenced this pull request Sep 3, 2026
… Manager (#156)

Narrowed: this is now **secrets only**, off `main`. Cloud SQL/GKE are
#155 and messaging is #157 — all three independent.

Both runtimes serve secret metadata, but the console had only an
AWS-specific page and nothing for GCP.

| Cloud | Was | Now |
|---|---|---|
| AWS Secrets Manager | dedicated page outside Cloud Explorer | SPI
adapter; page retained for value reveal |
| GCP Secret Manager | not present | list, create, inspect, delete |

## No secret value reaches a resource object

Neither adapter reads a value and no schema returns one. A value on
`CloudResource.metadata` would flow into the inspector, the client-side
query cache and the request telemetry — reveal has to be an explicit,
uncached action.

Both test files assert this rather than trusting the implementation:

```ts
// AWS: GetSecretValue is never sent, and no value-shaped key survives
expect(sent.every((c) => c.constructor.name !== 'GetSecretValueCommand')).toBe(true)
for (const key of Object.keys(resource.metadata)) expect(key.toLowerCase()).not.toContain('value')

// GCP: reading a payload needs versions/latest:access — never requested
expect(calls.every((c) => !c.url.includes(':access'))).toBe(true)
```

## On the create-payload review comment

Half right, and the useful half is now fixed.

The premise isn't: AWS marks both `SecretString` and `SecretBinary`
**optional** on `CreateSecret`, and a valueless create succeeds against
the runtime too —

```
$ curl -X POST .../aws/services/secrets/resources -d '{"secretName":"floci-noval"}'
  create: 201  → exists: floci-noval
```

— so create wasn't failing and the matrix wasn't wrong. But a secret you
can't put a value in is close to useless, and the schema offered no way
to supply one. Create now accepts an optional `secretValue` and forwards
it once. That direction is safe: input never lands on a resource, and
there's a test asserting the value does not appear in the returned
object.

## `routeByCloud`

New catalog field, and the part I'd most like a second opinion on.

AWS keeps its dedicated page because that's still the only way to read a
value; GCP goes through the generic explorer. A single `route` per
catalog entry can't express that, and pointing GCP at an AWS-only page
would be worse. So the entry carries `routeByCloud: {aws:
'/secretsmanager'}`, and the override disappears when the page migrates.

I considered migrating the AWS page here instead, but that drops value
reveal — a real regression — until the row-action mechanism exists. This
felt like the honest intermediate rather than a silent capability loss.

## Small correctness note

AWS delete uses `ForceDeleteWithoutRecovery`. Without it the secret
enters a 30-day recovery window and keeps appearing in the list after
the user deleted it, which reads as a broken delete.

## Overlap to be aware of

`CloudResource.type` is widened identically here and on #157, so
whichever lands second is a no-op for that hunk.

## Verification

`pnpm lint`, `type-check`, `test`, `build` pass. Live round-trip on both
providers, with routing resolving per cloud:

```
aws    Secrets Manager   available   route=/secretsmanager
gcp    Secret Manager    available   route=secrets
azure  Secrets Manager   coming_soon route=secrets
```

## Type of change

- [ ] Bug fix (`fix:`)
- [x] New feature / service UI (`feat:`)
- [ ] Breaking change (`feat!:` or `fix!:`)
- [ ] Docs / chore

## Area

- [ ] Frontend (`packages/frontend`)
- [x] API / Cloud Proxy (`packages/api`)
- [x] Cloud Explorer adapter / schema
- [ ] Build / CI / Docker

## Checklist

- [x] `pnpm lint`, `pnpm type-check`, `pnpm test`, and `pnpm build` pass
locally
- [x] New or updated tests added where it makes sense (`bun test` in
`packages/api`)
- [x] No fake/mock data added
- [x] Commit messages / PR title follow [Conventional
Commits](https://www.conventionalcommits.org/)

---------

Co-authored-by: fredpena <f.ant.pena@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants