docs(timestream-influxdb-mcp-server): Add schema design, query, and ingestion guidance #9
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
fredjoonpark marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| --- | ||
| name: "amazon-timestream-for-influxdb" | ||
| displayName: "Build a Time-Series Database with Amazon Timestream for InfluxDB" | ||
| description: "Query and analyze time-series data with Amazon Timestream for InfluxDB - built for IoT monitoring, DevOps metrics, real-time analytics, and high-cardinality data at scale. Deploy v2 and v3 instances, manage schemas, handle migrations and more." | ||
| keywords: ["influxdb","timestream","time-series","iot","observability","metrics","monitoring","devops","analytics","sql","telemetry"] | ||
| author: "AWS" | ||
| --- | ||
|
|
||
| # Amazon Timestream for InfluxDB Power | ||
|
|
||
| ## Overview | ||
| The Amazon Timestream for InfluxDB Power provides access to Amazon Timestream for InfluxDB, a fully managed time-series database service compatible with InfluxDB 2 and InfluxDB 3. InfluxDB is purpose-built for high-ingest, high-cardinality workloads such as IoT telemetry, observability, metrics, and real-time analytics. | ||
|
|
||
| This Power enables you to deploy and operate Timestream for InfluxDB instances and clusters, write and query time-series data using InfluxDB APIs and query languages, and manage schemas, retention, and migrations while leveraging AWS-managed scaling, durability, and security. | ||
|
|
||
| - **Dual Version Support**: Deploy and manage both InfluxDB 2 and InfluxDB 3 instances and clusters | ||
|
|
||
| - **Multi-Protocol Query Support**: Run Flux, InfluxQL, or SQL queries directly against your Timestream for InfluxDB instance | ||
|
|
||
| - **Ingest Line Protocol**: Write high-throughput time-series data using InfluxDB's native line protocol | ||
|
|
||
| - **Migration Support**: Handle migrations across InfluxDB versions | ||
|
|
||
| - **Schema Management**: Create and manage buckets/databases, measurements/tables, and retention policies | ||
|
|
||
|
|
||
| ## Available Steering Files | ||
|
|
||
| - `getting-started.md` - InfluxDB guidelines and operational Rules | ||
| - ALWAYS load before implementing schema changes or database operations | ||
| - MAY load when planning database application design | ||
|
|
||
| - `influxdb-2-vs-3.md` - Highlights key differences between InfluxDB 2 and 3 | ||
| - ALWAYS load when there is ambiguity in InfluxDB versions | ||
| - ALWAYS load on migrations between InfluxDB 2 and 3 | ||
|
|
||
| - `glossary.md` - Glossary of terms related to InfluxDB | ||
| - SHOULD load when there is vagueness in terminology or overlapping concepts | ||
|
|
||
| - `troubleshooting.md` - Common errors while working with InfluxDB and how to solve them | ||
| - SHOULD load on errors or when debugging | ||
|
|
||
| - `line-protocol.md` - Details the text-based format used when writing to InfluxDB v2 and v3 | ||
| - SHOULD load when writing or using line protocol | ||
|
|
||
| - `influxdb3/` | ||
| - `development-guide.md` - Contains guide for managing instances/clusters, queries/writes, example workflows | ||
| - `query-guide.md` - Query examples (SQL, InfluxQL) | ||
| - `troubleshooting.md` - Common errors in InfluxDB 3 and how to solve them | ||
| - `migrations.md` | ||
| - ALWAYS load when migrating TO Timestream for InfluxDB 3 from another source | ||
| - Timestream for LiveAnalytics -> Timestream for InfluxDB 3 | ||
| - Timestream for InfluxDB (2) -> Timestream for InfluxDB 3 | ||
| - InfluxDB v1 -> Timestream for InfluxDB 3 | ||
| - InfluxDB OSS 2.x -> Timestream for InfluxDB 3 | ||
| - `dashboard-guide.md` - Generate grafana dashboards and visualizations | ||
| - SHOULD load when building Grafana dashboards or visualizations | ||
| - `schema-design.md` - Schema modeling: tables, tags vs. fields, cardinality | ||
| - ALWAYS load when designing or modifying schema (table/tag/field choices, cardinality planning) | ||
| - `ingestion.md` - Data ingestion & writing: line protocol, batching, write endpoints | ||
| - ALWAYS load when writing data or ingesting via line protocol | ||
| - `parameters.md` - InfluxDBv3 Core/Enterprise parameter reference and duration formats | ||
|
fredjoonpark marked this conversation as resolved.
|
||
| - SHOULD load when configuring or tuning cluster/database parameters | ||
| - `gotchas.md` - Common gotchas & pain points: batching, cardinality, replica lag, limits | ||
| - SHOULD load when ingesting at scale or diagnosing performance/limit issues | ||
| - `onboarding.md` - Interactive get-started: provision cluster, retrieve token, first write/query | ||
| - SHOULD load when getting started or provisioning a new cluster | ||
|
|
||
| - `influxdb2/` | ||
| - `development-guide.md` - Contains guide for managing instances/clusters, queries/writes, example workflows | ||
| - `query-guide.md` - Query examples (Flux, InfluxQL) | ||
| - `troubleshooting.md` - Common errors in InfluxDB 2 and how to solve them | ||
| - `migrations.md` | ||
| - ALWAYS load when migrating TO Timestream for InfluxDB 2 from another source | ||
| - InfluxDB OSS 2.x -> Timestream for InfluxDB (2) | ||
| - InfluxDB v1 -> Timestream for InfluxDB (2) | ||
| - Timestream for LiveAnalytics -> Timestream for InfluxDB (2) | ||
| - `dashboard-guide.md` - Generate grafana dashboards and visualizations | ||
| - SHOULD load when building Grafana dashboards or visualizations | ||
| - `schema-design.md` - Schema modeling: buckets/measurements, tags vs. fields, cardinality | ||
| - ALWAYS load when designing or modifying schema (measurement/tag/field choices, cardinality planning) | ||
| - `ingestion.md` - Data ingestion & writing: line protocol, batching, write endpoints | ||
| - ALWAYS load when writing data or ingesting via line protocol | ||
| - `parameters.md` - InfluxDBv2 parameter reference | ||
| - SHOULD load when configuring or tuning instance parameters | ||
| - `gotchas.md` - Common gotchas & pain points: cardinality, batching, limits | ||
| - SHOULD load when ingesting at scale or diagnosing performance/limit issues | ||
| - `onboarding.md` - Interactive get-started: provisioning through first write and query | ||
| - SHOULD load when getting started or provisioning a new instance | ||
|
|
||
|
|
||
| ## Available MCP Tools | ||
|
|
||
| 1. [Timestream for InfluxDB MCP Server](https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server#available-tools) | ||
|
|
||
| The Timestream for InfluxDB MCP server provides tools for managing Amazon Timestream for InfluxDB clusters, instances, and parameter groups, along with InfluxDB-specific operations for writing data, querying with Flux, and managing buckets and organizations. | ||
|
|
||
| 2. [InfluxDB 3 MCP Server](https://github.com/influxdata/influxdb3_mcp_server?tab=readme-ov-file#available-tools) | ||
|
|
||
| The InfluxDB 3 MCP Server provides tools for data operations (read/write), database lifecycle management (create/update/delete), schema inspection, and authentication token management across all InfluxDB versions (Core, Enterprise, Cloud). | ||
|
|
||
|
|
||
| ## Configuration | ||
|
|
||
| #### Timestream for InfluxDB MCP | ||
|
|
||
| Prerequisites: | ||
| - A configured AWS profile | ||
| - (Optional) Provide instance (or cluster) details: | ||
| - `INFLUXDB_URL`: "https://your-influxdb-2-endpoint:8086" | ||
| - `INFLUXDB_TOKEN`: "your-influxdb-2-token" | ||
| - `INFLUXDB_ORG`: "your-influxdb-org" | ||
|
|
||
| Note that this MCP server can be used to deploy both instances (V2) and clusters (V2, V3). | ||
|
|
||
| #### InfluxDB 3 MCP | ||
|
|
||
| Prerequisites: | ||
| - Set up the InfluxDB 3 MCP Server. [See here](https://github.com/influxdata/influxdb3_mcp_server?tab=readme-ov-file#2-integration-with-mcp-clients) for instructions. | ||
| - An existing InfluxDB 3 instance (or cluster). The following details are required: | ||
| - `INFLUX_DB_INSTANCE_URL`: "https://your-influxdb-3-endpoint:8181" | ||
| - `INFLUX_DB_TOKEN`: "your-influxdb-3-token" | ||
| - `INFLUX_DB_PRODUCT_TYPE`: "core" | ||
|
|
||
| > See [mcp.json](./mcp.json) for an example configuration. | ||
|
|
||
| ## Additional Resources | ||
|
|
||
| - [Timestream for InfluxDB 3](https://docs.aws.amazon.com/timestream/latest/developerguide/influxdb3.html) | ||
| - [Timestream for InfluxDB (InfluxDB 2)](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influxdb.html) | ||
| - [Code Samples Repository](https://github.com/awslabs/amazon-timestream-tools) | ||
|
|
||
| #### MCP Servers | ||
| - [Amazon Timestream for InfluxDB MCP Server](https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server) (Apache-2.0 license) | ||
| - [InfluxDB 3 MCP Server](https://github.com/influxdata/influxdb3_mcp_server/tree/main) (Apache-2.0 license) | ||
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
fredjoonpark marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| # Glossary | ||
|
|
||
| Terms used across Amazon Timestream for InfluxDB (v2 and v3). Where a term differs | ||
| between versions, the version is noted. For a fuller treatment of version | ||
| differences, see [influxdb-2-vs-3.md](./influxdb-2-vs-3.md). | ||
|
|
||
| ## Version & terminology map | ||
|
|
||
| InfluxDB 2 (Timestream for InfluxDB) and InfluxDB 3 (Timestream for InfluxDB 3) | ||
| name several concepts differently: | ||
|
|
||
| | InfluxDB 2 | InfluxDB 3 | Notes | | ||
| |---|---|---| | ||
| | Organization (`org`) | — (no orgs) | V3 has no organization concept | | ||
| | Bucket | Database (`db`) | Top-level write/query namespace | | ||
| | Measurement | Table | A measurement becomes a table in V3 (auto-created on first write) | | ||
| | Flux / InfluxQL | SQL / InfluxQL | V3 has no Flux; SQL is primary | | ||
| | Token (`Token` prefix) | Token (`Bearer` prefix) | Data-plane auth header differs | | ||
| | Tasks (Flux) | Processing engine (Python) | Scheduled automation | | ||
| | TSM + TSI | Parquet on S3 + DataFusion | Storage + query engine | | ||
| | Port 8086 | Port 8181 | Default endpoint port | | ||
|
|
||
| ## Data model | ||
|
|
||
| - **Line protocol** — InfluxDB's text format for writing data: | ||
| `measurement,tag=val field=1.0 timestamp`. Unchanged across versions. See | ||
| [line-protocol.md](./line-protocol.md). | ||
| - **Point** — a single data record: a measurement + tag set + field set + timestamp. | ||
| - **Measurement** (V2) / **Table** (V3) — logical container for points of the same | ||
| kind (e.g. `cpu`). | ||
| - **Tag** — indexed key/value metadata (e.g. `host=server01`), used for filtering and | ||
| grouping. High tag cardinality is the main driver of V2 performance limits. | ||
| - **Tag set** — the unique combination of all tag key/values on a point. | ||
| - **Field** — the measured value(s) (e.g. `usage=42.0`). Not indexed. Field type is | ||
| inferred on first write and then locked. | ||
| - **Field set** — all field key/values on a point. | ||
| - **Timestamp** — the time of a point; default precision is nanoseconds. Always specify | ||
| precision explicitly to avoid silent shifts. | ||
| - **Series** — a unique combination of measurement + tag set + field key. | ||
| - **Series cardinality** — the number of unique series. V2 (TSM/TSI) degrades above | ||
| ~10M series; V3 is effectively unbounded. | ||
|
|
||
| ## Namespaces & auth | ||
|
|
||
| - **Organization (org)** — V2-only top-level tenant that groups buckets. | ||
| - **Bucket** — V2 named container for data with a retention period; the write/query | ||
| target (`bucket`). | ||
| - **Database (db)** — V3 top-level namespace (replaces org/bucket). | ||
| - **Retention period / policy** — how long data is kept before it expires. | ||
| - **Token** — data-plane credential. V2 uses `Authorization: Token <token>`; V3 uses | ||
| `Authorization: Bearer <token>`. Operator/all-access tokens differ in scope. | ||
| - **Control plane vs data plane** — the control plane (create/manage instances and | ||
| clusters) uses AWS SigV4/IAM; the data plane (read/write) is authorized only by the | ||
| engine token (IAM does not gate it). | ||
|
|
||
| ## Engines & storage | ||
|
|
||
| - **TSM (Time-Structured Merge Tree)** — V2's on-disk storage engine. | ||
| - **TSI (Time-Series Index)** — V2's index over tags; large indexes drive memory use | ||
| and the cardinality limit. | ||
| - **Apache Parquet** — V3's columnar on-disk (S3) file format. | ||
| - **Apache DataFusion** — V3's SQL query engine/optimizer. | ||
| - **Apache Arrow / Arrow Flight SQL** — Arrow is V3's in-memory columnar format; Flight | ||
| SQL is its high-performance gRPC query protocol. | ||
| - **WAL (write-ahead log)** — durability buffer for recent writes before they are | ||
| persisted to Parquet. | ||
| - **Compaction** — background merging of files for query efficiency; V3 Enterprise can | ||
| use a dedicated compactor node. | ||
| - **Last Value Cache (LVC)** — V3 cache of the most recent value per field (fast | ||
| "last point" lookups). | ||
| - **Distinct Value Cache (DVC)** — V3 cache of unique tag/field values (fast metadata | ||
| lookups, e.g. template-variable queries). | ||
|
|
||
| ## Query & automation | ||
|
|
||
| - **Flux** — V2's functional data-scripting language (primary in V2; absent in V3). | ||
| - **InfluxQL** — SQL-like legacy language; supported in both V2 and V3 (via a | ||
| compatibility endpoint). | ||
| - **SQL** — V3's primary query language (DataFusion). | ||
| - **Tasks** — V2 scheduled Flux scripts (downsampling, alerting). | ||
| - **Processing engine** — V3's embedded Python VM that runs plugins on schedule or on | ||
|
forestmvey marked this conversation as resolved.
Outdated
|
||
| WAL-flush events (replaces V2 Tasks). | ||
| - **Telegraf** — InfluxData's plugin-based metrics collection agent; writes line | ||
| protocol to either version. | ||
|
|
||
| ## Deployment | ||
|
|
||
| - **Instance** — V2 single-node deployment (`create-db-instance`); optionally with a | ||
| multi-AZ standby (failover only, does not serve reads). | ||
| - **Read-replica cluster** — V2 multi-node deployment that adds read-serving nodes | ||
| (adds a license fee). | ||
| - **Cluster** — V3's only deployment shape (`create-db-cluster`). | ||
| - **Core** — single-node V3 cluster; no dedicated compactor; best for | ||
| recent data. Uses an open-source binary with more limitations than Enterprise. | ||
| - **Enterprise** — multi-node V3 cluster (up to 15 nodes) with | ||
| role separation; adds a per-node AWS Marketplace license. | ||
| - **Parameter group** — immutable named set of engine parameters; change by creating a | ||
| new group and reassigning it (which triggers a reboot). | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are these typos or did this actually change? same as below on 108
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is accurate.
In Kiro,
/toolsshows this (excerpt):There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you take a look at the inconsistency with the MCP and figure out what's outdated?
https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server#database-instance-management
and have you tried invoking those tools to see if it works?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Their README is incorrect. From what I see, the inconsistencies are just those two,
LsInstancesByStatusandLsInstancesOfCluster.I've tested these tools, by asking Kiro to use them, and they work: