diff --git a/src/timestream-for-influxdb-mcp-server/README.md b/src/timestream-for-influxdb-mcp-server/README.md index e20178fe38..9873e29d14 100644 --- a/src/timestream-for-influxdb-mcp-server/README.md +++ b/src/timestream-for-influxdb-mcp-server/README.md @@ -19,6 +19,7 @@ An AWS Labs Model Context Protocol (MCP) server for Timestream for InfluxDB. Thi - You need an AWS account with appropriate permissions - Configure AWS credentials with `aws configure` or environment variables - Consider starting with Read-only permission if you don't want the LLM to modify any resources +4. *(Only for InfluxDB v3 data operations)* Install **Node.js ≥ 20.11** — required to build and run the separate InfluxDB 3 MCP server (see [InfluxDB v3 data operations](#influxdb-v3-data-operations)). ## Installation @@ -48,6 +49,8 @@ You can modify the settings of your MCP client to run your local server (e.g. fo } } ``` +> **Note:** The `INFLUXDB_URL` / `INFLUXDB_TOKEN` / `INFLUXDB_ORG` variables are **optional** and apply to **InfluxDB v2** data operations only (default port **8086**). Control-plane tools (clusters, instances, parameter groups, tags) work with AWS credentials alone. For **InfluxDB v3** data operations (port 8181), see [InfluxDB v3 data operations](#influxdb-v3-data-operations). + ### Windows Installation For Windows users, the MCP server configuration format is slightly different: @@ -93,8 +96,7 @@ The Timestream for InfluxDB MCP server provides the following tools: - `DeleteDbCluster`: Delete a Timestream for InfluxDB database cluster - `ListDbClusters`: List all Timestream for InfluxDB database clusters - `UpdateDbCluster`: Update a Timestream for InfluxDB database cluster -- `ListDbClusters`: List all Timestream for InfluxDB database clusters -- `ListDbInstancesForCluster`: List DB instances belonging to a specific cluster +- `LsInstancesOfCluster`: List DB instances belonging to a specific cluster - `ListClustersByStatus`: List DB clusters filtered by status ##### Database Instance Management @@ -103,7 +105,7 @@ The Timestream for InfluxDB MCP server provides the following tools: - `DeleteDbInstance`: Delete a Timestream for InfluxDB database instance - `ListDbInstances`: List all Timestream for InfluxDB database instances - `UpdateDbInstance`: Update a Timestream for InfluxDB database instance -- `ListDbInstancesByStatus`: List DB instances filtered by status +- `LsInstancesByStatus`: List DB instances filtered by status ##### Parameter Group Management - `CreateDbParamGroup`: Create a new DB parameter group @@ -131,3 +133,66 @@ The Timestream for InfluxDB MCP server provides the following tools: ##### Organization Management - `InfluxDBListOrgs`: List all organizations in InfluxDB - `InfluxDBCreateOrg`: Create a new organization in InfluxDB + +## InfluxDB v3 data operations + +This MCP server covers the **AWS control plane** (clusters, instances, parameter groups, tags) and **InfluxDB v2 data** operations (Flux query, writes, buckets, organizations). **InfluxDB v3 data operations** (SQL queries, v3 schema, v3 token management) are provided by a separate, InfluxData-maintained MCP server: [`influxdata/influxdb3_mcp_server`](https://github.com/influxdata/influxdb3_mcp_server). The [Kiro Power](#kiro-power)'s `kiro_power/mcp.json` references both servers. + +### Set up the InfluxDB 3 MCP server + +Requires **Node.js ≥ 20.11**. It is not published to npm yet, so clone and build it (or use the Docker option in its README): + +```bash +git clone https://github.com/influxdata/influxdb3_mcp_server.git +cd influxdb3_mcp_server +npm install +npm run build # produces build/index.js +``` + +Then add it to your MCP client config alongside this server: + +```json +{ + "mcpServers": { + "influxdb3": { + "command": "node", + "args": ["/absolute/path/to/influxdb3_mcp_server/build/index.js"], + "env": { + "INFLUX_DB_INSTANCE_URL": "https://your-influxdb-v3-endpoint:8181/", + "INFLUX_DB_TOKEN": "your-influxdb-v3-token", + "INFLUX_DB_PRODUCT_TYPE": "core" + } + } + } +} +``` + +- `INFLUX_DB_PRODUCT_TYPE`: use `core` for a single-node V3 cluster, `enterprise` for multi-node. +- If the `node` on your PATH is older than 20.11, point `command` at an absolute path to a Node ≥ 20.11 binary. + +See the [InfluxDB 3 MCP server README](https://github.com/influxdata/influxdb3_mcp_server) for Cloud Dedicated/Clustered/Serverless variants and the full tool list. + +## AI Rules + +This repository also contains AI Rules (Steering). These markdown files serve as simple +context and guidance for best practices and patterns that AI assistants automatically apply +when generating code to improve the quality of agentic development. + +Recommended path: +* [Kiro Power](#kiro-power) - button-click installation + +Alternative: +The [Timestream for InfluxDB power](https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server/kiro_power/) can also be cloned into your tool's respective `rules` directory +for use with other coding assistants. + +### Kiro Power + +To setup the Kiro power: +1. Install directly from the [Kiro Powers Registry](https://kiro.dev/launch/powers/amazon-timestream-for-influxdb/) +2. Once redirected to the Power in the IDE either: + 1. Select the **`Try Power`** button. Suggested for people who want: + - The AI to guide MCP server setup + - An interactive onboarding experience with Timestream for InfluxDB to create a new instance or cluster + 2. Open a new Kiro chat and ask anything related to Timestream for InfluxDB + - The Kiro agent will automatically activate the power if it identifies the power as valuable for completing + the user's task. diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/POWER.md b/src/timestream-for-influxdb-mcp-server/kiro_power/POWER.md new file mode 100644 index 0000000000..e30ef9023d --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/POWER.md @@ -0,0 +1,135 @@ +--- +name: "amazon-timestream-for-influxdb" +displayName: "Build a Time-Series Database with Amazon Timestream for InfluxDB" +description: "Query and analyze time-series data with Amazon Timestream for InfluxDB - built for IoT monitoring, DevOps metrics, real-time analytics, and high-cardinality data at scale. Deploy v2 and v3 instances, manage schemas, handle migrations and more." +keywords: ["influxdb","timestream","time-series","iot","observability","metrics","monitoring","devops","analytics","sql","telemetry"] +author: "AWS" +--- + +# Amazon Timestream for InfluxDB Power + +## Overview +The Amazon Timestream for InfluxDB Power provides access to Amazon Timestream for InfluxDB, a fully managed time-series database service compatible with InfluxDB 2 and InfluxDB 3. InfluxDB is purpose-built for high-ingest, high-cardinality workloads such as IoT telemetry, observability, metrics, and real-time analytics. + +This Power enables you to deploy and operate Timestream for InfluxDB instances and clusters, write and query time-series data using InfluxDB APIs and query languages, and manage schemas, retention, and migrations while leveraging AWS-managed scaling, durability, and security. + +- **Dual Version Support**: Deploy and manage both InfluxDB 2 and InfluxDB 3 instances and clusters + +- **Multi-Protocol Query Support**: Run Flux, InfluxQL, or SQL queries directly against your Timestream for InfluxDB instance + +- **Ingest Line Protocol**: Write high-throughput time-series data using InfluxDB's native line protocol + +- **Migration Support**: Handle migrations across InfluxDB versions + +- **Schema Management**: Create and manage buckets/databases, measurements/tables, and retention policies + + +## Available Steering Files + +- `getting-started.md` - InfluxDB guidelines and operational Rules + - ALWAYS load before implementing schema changes or database operations + - MAY load when planning database application design + +- `influxdb-2-vs-3.md` - Highlights key differences between InfluxDB 2 and 3 + - ALWAYS load when there is ambiguity in InfluxDB versions + - ALWAYS load on migrations between InfluxDB 2 and 3 + +- `glossary.md` - Glossary of terms related to InfluxDB + - SHOULD load when there is vagueness in terminology or overlapping concepts + +- `troubleshooting.md` - Common errors while working with InfluxDB and how to solve them + - SHOULD load on errors or when debugging + +- `line-protocol.md` - Details the text-based format used when writing to InfluxDB v2 and v3 + - SHOULD load when writing or using line protocol + +- `influxdb3/` + - `development-guide.md` - Contains guide for managing instances/clusters, queries/writes, example workflows + - `query-guide.md` - Query examples (SQL, InfluxQL) + - `troubleshooting.md` - Common errors in InfluxDB 3 and how to solve them + - `migrations.md` + - ALWAYS load when migrating TO Timestream for InfluxDB 3 from another source + - Timestream for LiveAnalytics -> Timestream for InfluxDB 3 + - Timestream for InfluxDB (2) -> Timestream for InfluxDB 3 + - InfluxDB v1 -> Timestream for InfluxDB 3 + - InfluxDB OSS 2.x -> Timestream for InfluxDB 3 + - `dashboard-guide.md` - Generate grafana dashboards and visualizations + - SHOULD load when building Grafana dashboards or visualizations + - `schema-design.md` - Schema modeling: tables, tags vs. fields, cardinality + - ALWAYS load when designing or modifying schema (table/tag/field choices, cardinality planning) + - `ingestion.md` - Data ingestion & writing: line protocol, batching, write endpoints + - ALWAYS load when writing data or ingesting via line protocol + - `parameters.md` - InfluxDBv3 Core/Enterprise parameter reference and duration formats + - SHOULD load when configuring or tuning cluster/database parameters + - `gotchas.md` - Common gotchas & pain points: batching, cardinality, replica lag, limits + - SHOULD load when ingesting at scale or diagnosing performance/limit issues + - `onboarding.md` - Interactive get-started: provision cluster, retrieve token, first write/query + - SHOULD load when getting started or provisioning a new cluster + +- `influxdb2/` + - `development-guide.md` - Contains guide for managing instances/clusters, queries/writes, example workflows + - `query-guide.md` - Query examples (Flux, InfluxQL) + - `troubleshooting.md` - Common errors in InfluxDB 2 and how to solve them + - `migrations.md` + - ALWAYS load when migrating TO Timestream for InfluxDB 2 from another source + - InfluxDB OSS 2.x -> Timestream for InfluxDB (2) + - InfluxDB v1 -> Timestream for InfluxDB (2) + - Timestream for LiveAnalytics -> Timestream for InfluxDB (2) + - `dashboard-guide.md` - Generate grafana dashboards and visualizations + - SHOULD load when building Grafana dashboards or visualizations + - `schema-design.md` - Schema modeling: buckets/measurements, tags vs. fields, cardinality + - ALWAYS load when designing or modifying schema (measurement/tag/field choices, cardinality planning) + - `ingestion.md` - Data ingestion & writing: line protocol, batching, write endpoints + - ALWAYS load when writing data or ingesting via line protocol + - `parameters.md` - InfluxDBv2 parameter reference + - SHOULD load when configuring or tuning instance parameters + - `gotchas.md` - Common gotchas & pain points: cardinality, batching, limits + - SHOULD load when ingesting at scale or diagnosing performance/limit issues + - `onboarding.md` - Interactive get-started: provisioning through first write and query + - SHOULD load when getting started or provisioning a new instance + + +## Available MCP Tools + +1. [Timestream for InfluxDB MCP Server](https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server#available-tools) + +The Timestream for InfluxDB MCP server provides tools for managing Amazon Timestream for InfluxDB clusters, instances, and parameter groups, along with InfluxDB-specific operations for writing data, querying with Flux, and managing buckets and organizations. + +2. [InfluxDB 3 MCP Server](https://github.com/influxdata/influxdb3_mcp_server?tab=readme-ov-file#available-tools) + +The InfluxDB 3 MCP Server provides tools for data operations (read/write), database lifecycle management (create/update/delete), schema inspection, and authentication token management across all InfluxDB versions (Core, Enterprise, Cloud). + + +## Configuration + +#### Timestream for InfluxDB MCP + +Prerequisites: +- A configured AWS profile +- (Optional) Provide instance (or cluster) details: + - `INFLUXDB_URL`: "https://your-influxdb-2-endpoint:8086" + - `INFLUXDB_TOKEN`: "your-influxdb-2-token" + - `INFLUXDB_ORG`: "your-influxdb-org" + +Note that this MCP server can be used to deploy both instances (V2) and clusters (V2, V3). + +#### InfluxDB 3 MCP + +Prerequisites: +- Set up the InfluxDB 3 MCP Server. [See here](https://github.com/influxdata/influxdb3_mcp_server?tab=readme-ov-file#2-integration-with-mcp-clients) for instructions. +- An existing InfluxDB 3 instance (or cluster). The following details are required: + - `INFLUX_DB_INSTANCE_URL`: "https://your-influxdb-3-endpoint:8181" + - `INFLUX_DB_TOKEN`: "your-influxdb-3-token" + - `INFLUX_DB_PRODUCT_TYPE`: "core" + +> See [mcp.json](./mcp.json) for an example configuration. + +## Additional Resources + +- [Timestream for InfluxDB 3](https://docs.aws.amazon.com/timestream/latest/developerguide/influxdb3.html) +- [Timestream for InfluxDB (InfluxDB 2)](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influxdb.html) +- [Code Samples Repository](https://github.com/awslabs/amazon-timestream-tools) + +#### MCP Servers +- [Amazon Timestream for InfluxDB MCP Server](https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server) (Apache-2.0 license) +- [InfluxDB 3 MCP Server](https://github.com/influxdata/influxdb3_mcp_server/tree/main) (Apache-2.0 license) diff --git a/src/timestream-for-influxdb-mcp-server/power/mcp.json b/src/timestream-for-influxdb-mcp-server/kiro_power/mcp.json similarity index 100% rename from src/timestream-for-influxdb-mcp-server/power/mcp.json rename to src/timestream-for-influxdb-mcp-server/kiro_power/mcp.json diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/getting-started.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/getting-started.md similarity index 56% rename from src/timestream-for-influxdb-mcp-server/power/steering/getting-started.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/getting-started.md index f01d93e332..bb6775d4d7 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/getting-started.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/getting-started.md @@ -2,15 +2,15 @@ Timestream for InfluxDB is the PREFERRED time-series database. This decision was made for handling high-ingest, high-cardinality workloads among other advantages. -For all documentation and guides related to InfluxDB 3 (Timestream for InfluxDB 3), refer to the documentation in the `influxdb3` directory. +For all documentation and guides related to InfluxDB 3 (Timestream for InfluxDB 3), refer to the documentation in the [`influxdb3`](./influxdb3) directory. -For all documentation and guides related to InfluxDB 2 (Timestream for InfluxDB), refer to the documentation in the `influxdb2` directory. +For all documentation and guides related to InfluxDB 2 (Timestream for InfluxDB), refer to the documentation in the [`influxdb2`](./influxdb2) directory. ## Best Practices - SHOULD ask user about which version of InfluxDB they are using if not specified. -- SHOULD refer to `influxdb-2-vs-3` when working across both versions. -- SHOULD refer to `line-protocol.md` when dealing with Line Protocol. +- SHOULD refer to [`influxdb-2-vs-3.md`](./influxdb-2-vs-3.md) when working across both versions. +- SHOULD refer to [`line-protocol.md`](./line-protocol.md) when dealing with Line Protocol. ## Troubleshooting diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/glossary.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/glossary.md new file mode 100644 index 0000000000..7b888953ea --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/glossary.md @@ -0,0 +1,97 @@ +# Glossary + +Terms used across Amazon Timestream for InfluxDB (v2 and v3). Where a term differs +between versions, the version is noted. For a fuller treatment of version +differences, see [influxdb-2-vs-3.md](./influxdb-2-vs-3.md). + +## Version & terminology map + +InfluxDB 2 (Timestream for InfluxDB) and InfluxDB 3 (Timestream for InfluxDB 3) +name several concepts differently: + +| InfluxDB 2 | InfluxDB 3 | Notes | +|---|---|---| +| Organization (`org`) | — (no orgs) | V3 has no organization concept | +| Bucket | Database (`db`) | Top-level write/query namespace | +| Measurement | Table | A measurement becomes a table in V3 (auto-created on first write) | +| Flux / InfluxQL | SQL / InfluxQL | V3 has no Flux; SQL is primary | +| Tasks (Flux) | Processing engine (Python) | Scheduled automation | +| TSM + TSI | Parquet on S3 + DataFusion | Storage + query engine | +| Port 8086 | Port 8181 | Default endpoint port | + +## Data model + +- **Line protocol** — InfluxDB's text format for writing data: + `measurement,tag=val field=1.0 timestamp`. Unchanged across versions. See + [line-protocol.md](./line-protocol.md). +- **Point** — a single data record: a measurement + tag set + field set + timestamp. +- **Measurement** (V2) / **Table** (V3) — logical container for points of the same + kind (e.g. `cpu`). +- **Tag** — indexed key/value metadata (e.g. `host=server01`), used for filtering and + grouping. High tag cardinality is the main driver of V2 performance limits. +- **Tag set** — the unique combination of all tag key/values on a point. +- **Field** — the measured value(s) (e.g. `usage=42.0`). Not indexed. Field type is + inferred on first write and then locked. +- **Field set** — all field key/values on a point. +- **Timestamp** — the time of a point; default precision is nanoseconds. Always specify + precision explicitly to avoid silent shifts. +- **Series** — a unique combination of measurement + tag set + field key. +- **Series cardinality** — the number of unique series. V2 (TSM/TSI) degrades above + ~10M series; V3 is effectively unbounded. + +## Namespaces & auth + +- **Organization (org)** — V2-only top-level tenant that groups buckets. +- **Bucket** — V2 named container for data with a retention period; the write/query + target (`bucket`). +- **Database (db)** — V3 top-level namespace (replaces org/bucket). +- **Retention period / policy** — how long data is kept before it expires. +- **Token** — data-plane credential. V2 and V3 use `Authorization: Bearer `. + Operator/all-access tokens differ in scope. +- **Control plane vs data plane** — the control plane (create/manage instances and + clusters) uses AWS SigV4/IAM; the data plane (read/write) is authorized only by the + engine token (IAM does not gate it). + +## Engines & storage + +- **TSM (Time-Structured Merge Tree)** — V2's on-disk storage engine. +- **TSI (Time-Series Index)** — V2's index over tags; large indexes drive memory use + and the cardinality limit. +- **Apache Parquet** — V3's columnar on-disk (S3) file format. +- **Apache DataFusion** — V3's SQL query engine/optimizer. +- **Apache Arrow / Arrow Flight SQL** — Arrow is V3's in-memory columnar format; Flight + SQL is its high-performance gRPC query protocol. +- **WAL (write-ahead log)** — durability buffer for recent writes before they are + persisted to Parquet. +- **Compaction** — background merging of files for query efficiency; V3 Enterprise can + use a dedicated compactor node. +- **Last Value Cache (LVC)** — V3 cache of the most recent value per field (fast + "last point" lookups). +- **Distinct Value Cache (DVC)** — V3 cache of unique tag/field values (fast metadata + lookups, e.g. template-variable queries). + +## Query & automation + +- **Flux** — V2's functional data-scripting language (primary in V2; absent in V3). +- **InfluxQL** — SQL-like legacy language; supported in both V2 and V3 (via a + compatibility endpoint). +- **SQL** — V3's primary query language (DataFusion). +- **Tasks** — V2 scheduled Flux scripts (downsampling, alerting). +- **Processing engine** — V3's embedded Python VM that runs plugins on schedule, HTTP invocation, or on + WAL-flush events (replaces V2 Tasks). +- **Telegraf** — InfluxData's plugin-based metrics collection agent; writes line + protocol to either version. + +## Deployment + +- **Instance** — V2 single-node deployment (`create-db-instance`); optionally with a + multi-AZ standby (failover only, does not serve reads). +- **Read-replica cluster** — V2 multi-node deployment that adds read-serving nodes + (adds a license fee). +- **Cluster** — V3's only deployment shape (`create-db-cluster`). +- **Core** — single-node V3 cluster; no dedicated compactor; best for + recent data. Uses an open-source binary with more limitations than Enterprise. +- **Enterprise** — multi-node V3 cluster (up to 15 nodes) with + role separation; adds a per-node AWS Marketplace license. +- **Parameter group** — immutable named set of engine parameters; change by creating a + new group and reassigning it (which triggers a reboot). diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb-2-vs-3.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb-2-vs-3.md new file mode 100644 index 0000000000..61d5256c74 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb-2-vs-3.md @@ -0,0 +1,131 @@ +# InfluxDB 2 vs 3 + +Covers the differences between **Timestream for InfluxDB** (the InfluxDB 2.x engine) and **Timestream for InfluxDB 3** (the InfluxDB 3 engine). Load this when there is ambiguity about which version a user is on, or when planning a migration between the two. + +## At-a-Glance Comparison + +| Aspect | V2 (Timestream for InfluxDB) | V3 (Timestream for InfluxDB 3) | +|--------|----|----| +| Underlying engine | InfluxDB OSS 2.x branch (Go) | InfluxDB 3 (Rust) | +| Storage engine | TSM (Time-Structured Merge Tree) + TSI index | Apache Parquet on object storage | +| Query engine | InfluxDB built-in | Apache DataFusion | +| Storage layer | Influx IOPS Included volumes (local, block) | Amazon S3 (object storage) | +| Namespace | Organization → Bucket | Database → Table (no orgs) | +| Query language | Flux (primary), InfluxQL | SQL (primary), InfluxQL — **no Flux** | +| Query/network protocol | HTTP | Apache Arrow Flight SQL (gRPC) + HTTP | +| Default port | 8086 | 8181 | +| Write namespacing | `org` + `bucket` | `db` (database) | +| Cardinality | ~10M series practical limit | Virtually unlimited | +| **Deployment topology** | **Single node** (optionally + standby); **read-replica cluster** for multiple nodes | **Cluster only** (Core = single-node, Enterprise = multi-node) | +| Control-plane create op | `create-db-instance` or `create-db-cluster` | `create-db-cluster` only | +| Max storage | Up to 16 TiB provisioned | Elastic via S3 | +| Processing/automation | Tasks (Flux) | Processing engine (embedded Python VM) | +| Extra licensing | Read replicas add a license fee | Enterprise adds a per-node Marketplace license; Core has none | +| Table limit | No limit | For Core, 2,000 across all databases. For Enterprise, 10,000 across all databases | +| Bucket/database limit | 20, recommended for performance, not a hard limit | For core, 5. For Enterprise, 100 | + +## Deployment Topology + +This is the most commonly misunderstood difference. The two versions have fundamentally different deployment models. + +### V2 — single node by default +A Timestream for InfluxDB **instance** (`create-db-instance`) is a **single node**. Availability options change this only slightly: + +- **`SINGLE_AZ`** (default) — one node, one Availability Zone. +- **`WITH_MULTIAZ_STANDBY`** — a primary node plus a secondary **standby** in another AZ. The standby is for **failover only**; it does not serve reads or add throughput. + +To get more than one serving node, you must create a **read-replica cluster** (`create-db-cluster` with `--deployment-type MULTI_NODE_READ_REPLICAS`). Only then does V2 run multiple nodes. **Without a read-replica cluster, V2 is a single node.** + +### V3 — cluster only +Timestream for InfluxDB 3 is **always deployed as a cluster** (`create-db-cluster`). There is no single-instance API for V3. The cluster's topology is determined by the parameter group, not a `--deployment-type` flag (do **not** pass `--deployment-type` for V3): + +- **Core** — a **single-node** cluster. Ingestion, querying, compaction, and the processing engine all run on one node and share compute. Core has **no dedicated compactor**, so it is best for recent data (typically the last 3–5 days). This can be adjusted by changing the `queryFileLimit` and `gen1Duration` configuration options in a cluster's parameter group. The default values of these options, `432` and 10 minutes, mean queries can access up to a maximum of 72 hours of data. +- **Enterprise** — a **multi-node** cluster, up to 15 nodes: 1–4 writer (ingest) nodes, 0–13 read-only (query) nodes, and 1 compactor node. Nodes can be assigned distinct roles to isolate ingest/query/compaction/processing. All multi-node deployments are spread across multiple AZs for availability. + +**Summary:** V2 is single-node unless you build a read-replica cluster; V3 is only ever a cluster (single-node for Core, multi-node for Enterprise). + +## Architecture & Storage + +**V2** uses the InfluxDB 2.x **TSM** storage engine with a **TSI** (time-series index). Tags are indexed, which is what makes high cardinality expensive — every unique tag-value combination becomes a series key, and large indexes drive memory pressure and slow writes/queries. Practical guidance caps a V2 instance at **~10M series**. Storage is local **Influx IOPS Included** block volumes (up to 16 TiB). + +**V3** is a ground-up redesign built on open formats: +- **Rust** core for performance. +- **Apache Arrow** for in-memory columnar processing (vectorized execution, predicate/projection pushdown). +- **Apache Parquet** for persisted columnar files. +- **Apache DataFusion** as the query optimizer/engine. +- **Apache Arrow Flight SQL** (gRPC) as the high-performance query protocol. +- **Amazon S3** as the core storage layer — 11 nines (99.999999999%) durability, multi-AZ, and effectively elastic capacity. + +V3 also adds two in-memory caches that accelerate common patterns: +- **Last Value Cache (LVC)** — most recent value per field; millisecond "last point" lookups for dashboards. +- **Distinct Value Cache (DVC)** — unique tag/field values per table; fast metadata queries (e.g., listing host or sensor IDs). + +## Query Languages + +- **V2:** **Flux** is the primary language; **InfluxQL** is also supported, **SQL** is not. +- **V3:** **SQL** is the primary language (via DataFusion); a v1-compatible **InfluxQL** endpoint is provided for legacy apps. **Flux is not supported.** + +There is **no automatic Flux-to-SQL conversion**. Migrating from V2 to V3 requires rewriting all Flux queries, tasks, and dashboards into SQL or InfluxQL. + +## Data Model & API + +- **V2** organizes data as **Organization → Bucket**. Writes specify `org` and `bucket`; default port **8086**. +- **V3** organizes data as **Database → Table** (no organizations). Tables are created automatically on first write (a measurement becomes a table). Writes specify `db`; data-plane auth uses the `Bearer` prefix; default port **8181**. +- **`422` HTTP status code meaning** - For v2, a status code of `422` means that some or all of the data was rejected. For v3, `422` means that writing the line protocol points would lead to the maximum number of databases, tables, columns, tags, or fields being exceeded. +- **`400` HTTP status code meaning** - For v2, a status code of `400` means that the `org` or `orgID` parameter doesn't match an existing organization. For v3, a status code of `400` means that some or all of the data was rejected. +**Backward compatibility:** V3 exposes v1- and v2-compatible write endpoints (including `/api/v2/write`), so existing line-protocol writers and Telegraf configs can keep writing without changes during a migration. Line protocol itself is unchanged across versions. + +## Automation / Processing + +- **V2** uses **Tasks** — scheduled Flux scripts for downsampling, alerting, and transforms. +- **V3** ships a **processing engine**: an embedded Python virtual machine that runs plugins on scheduled events or WAL-flush events (downsampling, alerting, anomaly detection) with zero-copy access to data. Timestream for InfluxDB 3 provides a curated, security-hardened set of plugins. + - **Gotcha:** on multi-node Enterprise, scheduled triggers run on **every** node and can OOM small instances — target nodes with `node_spec` or scale up. + +## Cost & Licensing + +- **V2:** billed on instance hours + provisioned storage (per GiB) + data transfer. **Read replicas add a license fee**. +- **V3 Core:** compute + S3 storage only — **no license fee**. +- **V3 Enterprise:** compute + S3 storage **plus** an Enterprise license billed per node-hour via **AWS Marketplace** (InfluxData). The license is separate from AWS compute spend, and **EDP discounts do not apply** to Marketplace charges. + +## Choosing / Migration Guidance + +- **Choose V3** for high or unpredictable cardinality, SQL/BI integration, large historical retention (cheap S3), or when you need a multi-node solution for high ingest. V3 is also the forward path for workloads leaving Timestream for LiveAnalytics (in maintenance mode). +- **Stay on / choose V2** when you depend on **Flux**, existing v2 tooling/dashboards, or org/bucket semantics, and your cardinality stays well under ~10M series. +- **Migration checklist (V2 → V3):** rewrite Flux → SQL/InfluxQL; map orgs/buckets → databases/tables; switch port `8086` → `8181`; re-point writers (or use V3's v2-compatible write endpoint); re-provision as a cluster (Core or Enterprise). +- **Data migration tooling:** the engines share no storage format, so there is no snapshot/restore between them. Use the [**Amazon Timestream for InfluxDB v2 to v3 Migration Script**](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influxdb_v2_to_v3_migration), which automates `influx backup` → line-protocol translation (`influxd inspect export-lp`) → write to the V3 HTTP API (buckets → databases, measurements → tables). Full runbook: `influxdb3/migrations.md`. + +## Retrieving tokens + +To interact with your deployed Timestream for InfluxDB instance or cluster, creating buckets, executing queries, ingesting data, etc., using the InfluxDB v2 or v3 HTTP APIs, you must retrieve or create an InfluxDB token. + +### V2 + +Once your v2 instance or cluster has been deployed, create a new operator token, using the [Influx v2 CLI](https://docs.influxdata.com/influxdb/v2/tools/influx-cli/?section=influxdb%252Fv2%252Ftools): + +```shell +influx config create --config-name CONFIG_NAME1 --host-url "https://yourinstanceid.eu-central-1.timestream-influxdb.amazonaws.com:8086" --org [YOURORG] --username-password [YOURUSERNAME] --active + +influx auth create --org [YOURORG] --operator +``` + +### V3 + +After you have deployed a Timestream for InfluxDB v3 cluster, a secret in AWS Secrets Manager will be associated with your cluster. The ID of this secret will be returned as part of the `aws timestream-influxdb get-db-cluster` AWS CLI call and is visible in the AWS console. + +Given the cluster ID, you can retrieve your token with the following command, replacing `` with your cluster ID: +```shell +aws secretsmanager \ + get-secret-value \ + --secret-id READONLY-InfluxDB-auth-parameters- \ + --query SecretString \ + --output text | jq -r '.token' +``` + +## Sources + +- [What is Timestream for InfluxDB?](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influxdb.html) +- [Amazon Timestream for InfluxDB 3](https://docs.aws.amazon.com/timestream/latest/developerguide/influxdb3.html) +- [Features and workflows with Amazon Timestream for InfluxDB 3](https://aws.amazon.com/blogs/database/features-and-workflows-with-amazon-timestream-for-influxdb-3/) +- [Timestream for InfluxDB 3 workload analysis and best practices](https://aws.amazon.com/blogs/database/timestream-for-influxdb-3-workload-analysis-and-best-practices/) +- [InfluxDB v2 HTTP API](https://docs.influxdata.com/influxdb/v2/api/?section=influxdb%252Fv2%252Fapi) +- [InfluxDB v3 HTTP API](https://docs.influxdata.com/influxdb3/enterprise/api/) diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/dashboard-guide.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/dashboard-guide.md new file mode 100644 index 0000000000..5ddf8532f1 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/dashboard-guide.md @@ -0,0 +1,213 @@ +# Grafana Dashboards for InfluxDB 2 + +How to visualize **Timestream for InfluxDB v2** data in Grafana. + +## Data source +Grafana ships a built-in [**InfluxDB** data source plugin](https://grafana.com/docs/grafana/latest/datasources/influxdb/). Add it via **Connections → Data sources → InfluxDB**. The plugin supports three query languages; for v2 use **Flux** (primary) or **InfluxQL**. + +### Connection settings +- **URL:** `https://:8086` (v2 default port). +- **Network reachability:** the Grafana server must reach the instance. For private instances, run Grafana inside the VPC or front the endpoint with a bastion/port-forward (see `influxdb2/onboarding.md`). + +### Flux (recommended for v2) +Set **Query language: Flux**, then provide: +- **Organization:** your InfluxDB org name. +- **Token:** a data-plane token with read access (retrieve from InfluxDB UI — see `influxdb2/onboarding.md`). +- **Default Bucket:** optional. + +Example panel query: +```flux +from(bucket: "my-bucket") + |> range(start: v.timeRangeStart, stop: v.timeRangeStop) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage") +``` +`v.timeRangeStart`/`v.timeRangeStop` bind the panel to Grafana's time picker; `v.windowPeriod` is the auto-interval for `aggregateWindow`. + +### InfluxQL (alternative) +Set **Query language: InfluxQL**. InfluxQL needs a **DBRP mapping** (database/retention-policy → bucket) on the instance; without it queries return no databases. Configure **Database**, and use **Bearer** auth (header `Authorization: Bearer `). Use Flux instead, unless you need InfluxQL compatibility. + +Example panel query: +```sql +SELECT mean("usage") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) fill(null) +``` +In Grafana, the `$timeFilter` macro expands to the panel's time range (the InfluxQL equivalent of Flux's `v.timeRangeStart`/`v.timeRangeStop`), and `$__interval` is the auto-interval for the `GROUP BY time(...)` bucket. For raw rows without aggregation, use `SELECT "usage" FROM "cpu" WHERE $timeFilter`. + +## Building dashboards +- Use the visual query builder or raw editor in the panel. **Time series** and **Stat** panels cover most metrics; **Table** for raw rows. +- Use **template variables** (e.g. a query variable over tag values) for reusable, filterable dashboards. +- Grafana **alerting** and **annotations** both work against this data source. + +### Example: template variable over tag values +Create a **Query** variable named `host` that lists the values of the `host` tag, so panels can be filtered by host. + +InfluxQL variable query: +```sql +SHOW TAG VALUES WITH KEY = "host" +``` +Flux variable query: +```flux +import "influxdata/influxdb/schema" +schema.tagValues(bucket: "my-bucket", tag: "host") +``` + +### Example: reference the variable in a panel query +Filter the panel by the selected `host` value. In InfluxQL, wrap the variable in a regex match (`=~ /^$host$/`) — Grafana interpolates multi-value selections as a regex like `(host1|host2)`: +```sql +SELECT mean("usage") FROM "cpu" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($__interval) fill(null) +``` +In Flux, compare against the variable directly: +```flux +from(bucket: "my-bucket") + |> range(start: v.timeRangeStart, stop: v.timeRangeStop) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage" and r.host == "${host}") + |> aggregateWindow(every: v.windowPeriod, fn: mean) +``` + +### Example: Stat panel for a current value +A **Stat** panel showing the latest single value works well with a `last()` selector: +```sql +SELECT last("usage") FROM "cpu" WHERE $timeFilter +``` + +### Example: one series per tag value +`GROUP BY` a tag to render a separate line per host in a **Time series** panel: +```sql +SELECT mean("usage") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval), "host" fill(null) +``` +Flux equivalent — `group()` by the `host` column before aggregating: +```flux +from(bucket: "my-bucket") + |> range(start: v.timeRangeStart, stop: v.timeRangeStop) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage") + |> group(columns: ["host"]) + |> aggregateWindow(every: v.windowPeriod, fn: mean) +``` + +### Example: Table panel for raw rows +For a **Table** panel showing raw, unaggregated rows, select the fields directly and skip the `GROUP BY time(...)`: +```sql +SELECT "usage", "host" FROM "cpu" WHERE $timeFilter ORDER BY time DESC LIMIT 100 +``` + +### Example: annotation query +Grafana **annotations** overlay events on a graph. Configure a dashboard annotation against this data source with a query that returns a time column plus text: +```sql +SELECT "text" FROM "deploy_events" WHERE $timeFilter +``` + +### Editing panels as JSON +Grafana exposes the underlying JSON in two places: +- **Panel JSON** — open a panel's menu and choose **Inspect → Panel JSON** (the **JSON** tab in the panel inspector). This lets you view and copy the panel JSON, panel data JSON, and data frame structure JSON. It's primarily an inspect/export view (handy when provisioning or administering Grafana); inline editing of panel JSON is limited and varies by Grafana version. +- **Dashboard JSON Model** — open **Dashboard settings → JSON Model** to see the entire dashboard's JSON, including every panel definition (queries, visualization type, field config, grid layout). This is the raw representation you edit or commit when managing dashboards as code. + +## Monitoring the database itself +To dashboard the health of the Timestream for InfluxDB instance (CPU, memory, write throughput, query latency), use the AWS sample in **awslabs/amazon-timestream-tools** → [`integrations/influxdb_metrics_dashboard`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/integrations/influxdb_metrics_dashboard). It is a CDK app that deploys a Telegraf collector to scrape the instance `/metrics` endpoint into CloudWatch and ships a pre-built Grafana dashboard. CloudWatch metrics (`CPUUtilization`, `MemoryUtilization`, etc.) can also be charted directly via Grafana's CloudWatch data source. + +## Managing dashboards via the Grafana HTTP API + +Grafana exposes a REST API for creating, reading, updating, and deleting dashboards. This is useful for provisioning panels as code or automating dashboard rollout across environments. See the [Grafana Dashboard HTTP API](https://grafana.com/docs/grafana/latest/developer-resources/api-reference/http-api/dashboard/) for more information. + +These calls target your **Grafana** server (not the InfluxDB endpoint), authenticate with a Grafana **service account token** (`Authorization: Bearer `), and use the dashboard API available in Grafana 12+. `:namespace` is your org namespace (`default` for the default org), the dashboard JSON (panels, InfluxQL/Flux targets, variables) goes in `spec`, and `metadata.name` is the dashboard UID. + +**Create a dashboard** — `POST .../dashboards`, returns `201` (`409` if the UID already exists): +```bash +curl -X POST "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "metadata": { "generateName": "cpu-" }, + "spec": { "title": "CPU Dashboard", "schemaVersion": 41, "panels": [], "tags": ["influxdb"] } + }' +``` +Set `metadata.generateName` to let Grafana assign a random UID, or `metadata.name` to choose your own. Add `metadata.annotations."grafana.app/folder"` to place it in a folder. + +**Get a dashboard** — `GET .../dashboards/:uid`, returns `200` (or `404` if not found): +```bash +curl "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards/" \ + -H "Authorization: Bearer " +``` + +**List dashboards** — `GET .../dashboards`; page with `limit` and the `metadata.continue` token returned by the previous response: +```bash +curl "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards?limit=20" \ + -H "Authorization: Bearer " +``` + +**Delete a dashboard** — `DELETE .../dashboards/:uid`, returns `200`: +```bash +curl -X DELETE "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards/" \ + -H "Authorization: Bearer " +``` + +### Create an InfluxDB data source + +A dashboard panel references a data source by its `uid`, so create the InfluxDB data source first with `POST /api/datasources` (see the [Data source HTTP API](https://grafana.com/docs/grafana/latest/developer-resources/api-reference/http-api/api-legacy/data_source/)). Use the `jsonData`/`secureJsonData` fields documented for [InfluxDB provisioning](https://grafana.com/docs/grafana/latest/datasources/influxdb/configure/#provisioning-examples); the response returns the new data source's `uid`. + +**Flux** (`version: Flux`, with organization, default bucket, and token): +```bash +curl -X POST "https:///api/datasources" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Timestream-InfluxDB-Flux", + "type": "influxdb", + "access": "proxy", + "url": "https://:8086", + "jsonData": { "version": "Flux", "organization": "", "defaultBucket": "" }, + "secureJsonData": { "token": "" } + }' +``` + +**InfluxQL** (token passed as an `Authorization` header; the bucket needs a DBRP mapping): +```bash +curl -X POST "https:///api/datasources" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Timestream-InfluxDB-InfluxQL", + "type": "influxdb", + "access": "proxy", + "url": "https://:8086", + "jsonData": { "dbName": "", "httpHeaderName1": "Authorization" }, + "secureJsonData": { "httpHeaderValue1": "Bearer " } + }' +``` + +### Create a dashboard with panels + +Panels live in the dashboard `spec.panels` array. Each panel sets a `type`, a `gridPos` (layout), and `targets` (the queries), where each target's `datasource.uid` is the InfluxDB data source created above. This example creates a time series panel driven by a Flux query: +```bash +curl -X POST "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "metadata": { "generateName": "cpu-" }, + "spec": { + "title": "CPU Usage", + "schemaVersion": 41, + "tags": ["influxdb"], + "time": { "from": "now-6h", "to": "now" }, + "panels": [ + { + "type": "timeseries", + "title": "Mean usage by host", + "datasource": { "type": "influxdb", "uid": "" }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "influxdb", "uid": "" }, + "query": "from(bucket: \"\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"cpu\" and r._field == \"usage\")\n |> group(columns: [\"host\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean)" + } + ] + } + ] + } + }' +``` +Add more objects to the `panels` array — each with its own `gridPos` — to build a multi-panel dashboard. Set `"type": "stat"` for a single-value Stat panel or `"type": "table"` for a raw-rows Table panel. + +## See also +- Ingestion via Telegraf: [`integrations/telegraf`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/integrations/telegraf) +- [Grafana InfluxDB data source docs](https://grafana.com/docs/grafana/latest/datasources/influxdb/) +- Multiagent observability with Grafana: [`sample_apps/python/multiagent_observability`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/sample_apps/python/multiagent_observability#grafana-dashboard-overview). diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/development-guide.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/development-guide.md similarity index 59% rename from src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/development-guide.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/development-guide.md index 8f0679c547..4bccbff454 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/development-guide.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/development-guide.md @@ -11,10 +11,15 @@ ### Queries +For query guidance, see [query-guide.md](./query-guide.md). + ### Writes +For ingestion and writing guidance, see [ingestion.md](./ingestion.md). + ### Schema Operations +For schema operations, see [query-guide.md](./query-guide.md) and [schema-design.md](./schema-design.md). ## Workflow Examples diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/gotchas.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/gotchas.md similarity index 92% rename from src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/gotchas.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/gotchas.md index 4b98649944..8d5cf486a1 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/gotchas.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/gotchas.md @@ -22,12 +22,6 @@ **No Direct Host Access**: Cannot SSH into instances. All management is via APIs, Console, or InfluxDB UI. -## Operational — Will Cause Scaling Issues - -**10K Tables Limit**: Default max tables per database. Siemens hit this. Increasing beyond recommendation requires understanding compaction and query performance implications. - -**No Storage Scaling for V2**: Cannot increase EBS storage after creation. Plan storage capacity upfront. - ## Cost — Will Cause Bill Shock **Read Replica License**: V2 read replicas add 50% license fee on top of base instance cost. diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/ingestion.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/ingestion.md new file mode 100644 index 0000000000..6a0344b0df --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/ingestion.md @@ -0,0 +1,169 @@ +# InfluxDB v2 Data Ingestion & Writing + +## Write Endpoints + +``` +POST /api/v2/write?org=&bucket=&precision= +Authorization: Bearer +Content-Type: text/plain +``` + +- Use `org` ID (not name) to avoid breakage if the org is renamed. +- Default precision is `ns` — always specify explicitly to avoid silent timestamp errors. +- Returns `204` on success, `400` on malformed line protocol, `422` on field type conflict, `401` on bad token. + +## Batching + +**Always batch writes.** Single-point writes with many concurrent writers cause severe replica lag. + +- Target **5,000+ points per request** minimum +- Optimal batch size: 5,000–10,000 points +- Max recommended: ~50,000 points per request (beyond this, HTTP timeouts become a risk) +- For high-throughput workloads, tune write timeout in the parameter group (see Tuning section below) + +## Python Client + +```bash +pip install influxdb-client +``` + +```python +from influxdb_client import InfluxDBClient, Point +from influxdb_client.client.write_api import SYNCHRONOUS + +client = InfluxDBClient( + url="https://:8086", + token="", + org="" # use org ID, not name +) +write_api = client.write_api(write_options=SYNCHRONOUS) + +# Write a single point +point = Point("cpu") \ + .tag("host", "server01") \ + .tag("region", "us-east") \ + .field("usage_idle", 98.2) \ + .field("usage_system", 1.8) +write_api.write(bucket="", record=point) + +# Write a batch +points = [ + Point("cpu").tag("host", f"server{i}").field("usage_idle", 90.0 + i) + for i in range(5000) +] +write_api.write(bucket="", record=points) + +write_api.close() +client.close() +``` + +For async/high-throughput use `write_options=WriteOptions(batch_size=5000, flush_interval=1000)` instead of `SYNCHRONOUS`: + +```python +from influxdb_client import InfluxDBClient, Point, WriteOptions + +client = InfluxDBClient(url="https://:8086", token="", org="") +write_api = client.write_api(write_options=WriteOptions(batch_size=5000, flush_interval=1000)) +# Points are buffered and flushed automatically +write_api.write(bucket="", record=Point("cpu").tag("host", "server01").field("usage_idle", 98.2)) +write_api.close() # flushes remaining buffer +client.close() +``` + +## Go Client + +```bash +go get github.com/influxdata/influxdb-client-go/v2 +``` + +```go +import ( + "time" + influxdb2 "github.com/influxdata/influxdb-client-go/v2" +) + +client := influxdb2.NewClient("https://:8086", "") +// Non-blocking API batches writes automatically +writeAPI := client.WriteAPI("", "") // use org ID, not name + +p := influxdb2.NewPoint("cpu", + map[string]string{"host": "server01", "region": "us-east"}, + map[string]interface{}{"usage_idle": 98.2, "usage_system": 1.8}, + time.Now(), +) +writeAPI.WritePoint(p) +writeAPI.Flush() // ensure buffered points are sent +client.Close() +``` + +## Telegraf + +Telegraf is the recommended agent for infrastructure metrics. + +### Output + +```toml +[[outputs.influxdb_v2]] + urls = ["https://:8086"] + token = "" + organization = "" # use org ID, not name + bucket = "" +``` + +### Common Telegraf settings + +```toml +# Example: collect CPU metrics +[[inputs.cpu]] + percpu = false + totalcpu = true + collect_cpu_time = false +``` + +```bash +# Test config before running +telegraf --config /etc/telegraf/telegraf.conf --test + +# Run +telegraf --config /etc/telegraf/telegraf.conf +``` + +Telegraf automatically batches writes — default `metric_batch_size = 1000`. Increase to 5000+ for high-throughput: + +```toml +[agent] + metric_batch_size = 5000 + metric_buffer_limit = 100000 +``` + +## Write Error Handling + +| HTTP Status | Cause | Action | +|-------------|-------|--------| +| 204 | Success | — | +| 400 | Malformed line protocol | Check syntax | +| 401 | Invalid or expired token | Verify token has write permission | +| 404 | Bucket not found | Verify names or IDs | +| 413 | Payload too large | Reduce batch size | +| 422 | Field type conflict | Conflicting points are **dropped** (partial write); valid points in the batch succeed. Check field types haven't changed. | +| 429 | Too many requests | Back off and retry with exponential backoff | +| 500 | Server error | Retry; check instance health via `/health` | + +On 400 (malformed line protocol), the entire batch is rejected. On 422 (type conflict), only the conflicting points are dropped — the rest of the batch succeeds. + +## Tuning Write Performance + +### Parameter group settings + +| Parameter | Default | Recommendation | +|-----------|---------|----------------| +| `httpWriteTimeout` | 0 | Increase to 60–120s to protect against large amounts of open connections | +| `storageWalMaxWriteDelay` | 10ms | Increase if seeing write stalls under load | +| `storageCacheMaxMemorySize` | 1073741824 | Increase on large instances to reduce WAL flushes | + +To change: create a new parameter group with updated values and update the instance to reference it (`UpdateDbInstance`). `UpdateDbParameterGroup` is not available via SDK. + + +## Line Protocol + +For information on the line protocol data format, see [line-protocol.md](../line-protocol.md). diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/migrations.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/migrations.md new file mode 100644 index 0000000000..63985e193b --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/migrations.md @@ -0,0 +1,162 @@ +# Migrating to Timestream for InfluxDB v2 + +How to migrate **into** Timestream for InfluxDB v2 from common +sources. Pick the section that matches your source system. All of the tooling referenced +here lives in the [awslabs/amazon-timestream-tools](https://github.com/awslabs/amazon-timestream-tools) +repository. + +> **Migrating off V2 to V3 instead?** See [`influxdb3/migrations.md`](../influxdb3/migrations.md). +> For the differences between the engines, see [`influxdb-2-vs-3.md`](../influxdb-2-vs-3.md). + +## Source matrix + +| Source | Tooling | Mechanism | +|--------|---------|-----------| +| InfluxDB **OSS 2.x** (self-managed) | [`tools/python/influx-migration`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influx-migration) | Backup/restore + v2 API (also moves dashboards, tasks, users) | +| InfluxDB **v1** | [`guides/influxdb_v1_to_v2_migration`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/guides/influxdb_v1_to_v2_migration) | `influxd upgrade` → `export-lp` → ingest | +| **Timestream for LiveAnalytics** | [`tools/python/liveanalytics_migration_scripts`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts) | Unload → transform to line protocol → ingest → validate | + +--- + +## From InfluxDB OSS 2.x + +This is the most direct path: both source and target run the same 2.x engine, so buckets, +dashboards, tasks, users, and other key-value data can all move across. Use the AWS +**InfluxDB migration script** ([`tools/python/influx-migration`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influx-migration), +`influx_migration.py`), which drives the Influx CLI `backup`/`restore` flow and the v2 API. + +**Prerequisites** +- A machine (Linux/macOS/Windows) with Python 3.7+, the [Influx CLI](https://docs.influxdata.com/influxdb/v2/tools/influx-cli/) on `PATH`, and the `influxdb_client` + `boto3` Python libraries. +- Source operator token in `INFLUX_SRC_TOKEN`; destination (Timestream for InfluxDB) operator token in `INFLUX_DEST_TOKEN`. +- Network access to both source and destination. +- Enough local disk for the export, **or** an S3 bucket mounted via [Mountpoint for S3](https://aws.amazon.com/s3/features/mountpoint/)/rclone (`--s3-bucket`) to avoid local storage pressure. + +**Migrate a single bucket** +```shell +python3 influx_migration.py \ + --src-host https://:8086 \ + --src-bucket \ + --dest-host https://:8086 \ + --dest-bucket +``` + +**Migrate everything** (buckets, dashboards, tasks, users, tokens, key-value data): +```shell +python3 influx_migration.py \ + --src-host https://:8086 \ + --dest-host https://.timestream-influxdb.amazonaws.com:8086 \ + --full +``` + +Other useful flags: `--src-org`/`--dest-org`, `--csv` (CSV instead of native backup), +`--s3-bucket` (offload the backup to S3), `--retry-restore-dir` (resume). Run +`python3 influx_migration.py -h` for the full list. + +**Validate** by listing buckets (`influx bucket list`) and counting records on the +destination (`SELECT COUNT(*) ...` via `influx v1 shell`, or a Flux `... |> count()`). + +**Near-zero-downtime (live) migration:** the script itself incurs downtime while data +copies and while clients are re-pointed. For a live cutover, front both databases with an +**API Gateway → Kinesis → Lambda** dual-write pipeline (with endpoints held in Secrets +Manager), seed the target with a `--full` run, then flip the read endpoint once the target +catches up. Full architecture and Lambda samples: +[Use the AWS InfluxDB migration script to migrate your InfluxDB OSS 2.x data to Amazon Timestream for InfluxDB](https://aws.amazon.com/blogs/database/use-the-aws-influxdb-migration-script-to-migrate-your-influxdb-oss-2-x-data-to-amazon-timestream-for-influxdb/). + +## From InfluxDB v1 + +There is no v1 write/restore path into the managed service, so v1 data has to be **upgraded +to the 2.x on-disk format first, exported to line protocol, then ingested**. The +[`influxdb_v1_to_v2_migration` guide](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/guides/influxdb_v1_to_v2_migration) +documents this for **read-replica clusters**, which is the case that *requires* line protocol +ingestion — read replicas do not support `restore`. + +> If your target is a **single V2 instance** (not a read-replica cluster), the simpler path +> is to `influxd upgrade` locally and then use the **OSS 2.x migration script above** (native +> backup/restore is far more efficient than ingesting line protocol). Use the line-protocol +> route below when the target is a read-replica cluster. + +1. **Upgrade v1 → 2.x on-disk format** with the InfluxDB 2 daemon (this copies data to a new + directory and leaves the v1 instance intact): + ```shell + influxd upgrade # set --engine-path if your v1 engine isn't in the default location + ``` +2. **Export each bucket to line protocol** with `influxd inspect export-lp` (parallelize per + bucket; compress with `--compress`): + ```shell + influxd inspect export-lp \ + --bucket-id \ + --engine-path ~/.influxdbv2/engine \ + --output-path buckets//.gz \ + --compress + ``` + Bucket-name→ID mapping comes from `GET /api/v2/buckets?org=` while the local daemon + is running. +3. **Ingest into the read-replica cluster** with the Timestream ingestion script + ([`...targets/timestream_for_influxdb/ingestion/influxdb_ingestion.py`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts/targets/timestream_for_influxdb/ingestion)). + Set `INFLUXDB_V2_URL` / `INFLUXDB_V2_ORG` / `INFLUXDB_V2_TOKEN` for the cluster, create the + target bucket, then point the script at the directory of `.gz` files: + ```shell + python3 influxdb_ingestion.py --skip-bucket-check + ``` + +## From Timestream for LiveAnalytics + +Timestream for LiveAnalytics is in maintenance mode; Timestream for InfluxDB is a supported +migration target for **moderate-cardinality** workloads. Use the +[**LiveAnalytics migration tooling**](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts), +a four-stage pipeline orchestrated by `main.py --config `: + +1. **Cardinality assessment** — map your LiveAnalytics schema to an InfluxDB schema and check + the series cardinality. **InfluxDB is the wrong target if cardinality exceeds ~10M series** + (consider RDS for PostgreSQL instead). The + [`cardinality.py`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts/cardinality) + script computes it and recommends an instance type; high-cardinality dimensions (e.g. a + unique `request_id`) can be demoted from **tags to fields** to bring cardinality down. +2. **Unload** — export the LiveAnalytics table(s) to S3 in **Parquet with no compression** + (the format the ingestion stage expects). +3. **Transform** — Athena converts the Parquet to line protocol, applying the tag/field schema + and adding an `la_unload=1` marker field. +4. **Ingest** — the same `influxdb_ingestion.py` loads the line protocol into the V2 instance. +5. **Validate** — compare logical row counts between source and target. + +**Configure the V2 target** with environment variables: +```shell +export INFLUXDB_V2_URL="https://:8086" +export INFLUXDB_V2_ORG="" +export INFLUXDB_V2_TOKEN="" +``` + +The pipeline runs in two modes (set in the config): +- **`batch`** — migrate everything in a fixed `start_time`…`end_time` window. Use when there + are no new writes to LiveAnalytics. +- **`live_replication`** — run continuously, backfilling and replicating new writes on an + interval (`batch_sleep_min`) with optional `backfill_min_overlap` for late-arriving data and + an optional `cutoff_time`. This is the basis for a near-zero-downtime cutover. + +For the end-to-end live cutover (data replication → gradual application cutover → cleanup), +follow the +[Live Migration Guide](https://github.com/awslabs/amazon-timestream-tools/blob/mainline/tools/python/liveanalytics_migration_scripts/targets/timestream_for_influxdb/live_migration_guide.md). + +--- + +## Things to get right (any source) + +- **Schema / namespace mapping:** target data lands under an `org` + `bucket`; each source + measurement becomes an InfluxDB measurement. Decide the bucket layout before you start. +- **Tags vs. fields:** the tag set defines series cardinality. Only index (tag) what you + filter/group on; demote high-cardinality attributes to fields. +- **Cardinality ceiling:** keep well under **~10M series** per instance — it drives memory use + and write/query performance. Size the instance from the cardinality assessment. +- **Timestamp precision:** carry the original precision through ingestion (`--precision`, + default `ns`); a mismatch silently shifts every point. +- **Batching:** ingest **thousands of points per request**; many tiny writes cause replica lag + (see [`gotchas.md`](./gotchas.md)). +- **Backfill order & instance sizing:** large historical loads benefit from a high-IOPS + instance and a network-optimized EC2 box in the same Region as the target (see the ingestion + script's performance notes). + +## AWS migration tools and documentation +- **InfluxDB OSS 2.x → Timestream for InfluxDB** — [AWS InfluxDB migration script](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influx-migration) and walkthrough: [Use the AWS InfluxDB migration script to migrate your InfluxDB OSS 2.x data to Amazon Timestream for InfluxDB](https://aws.amazon.com/blogs/database/use-the-aws-influxdb-migration-script-to-migrate-your-influxdb-oss-2-x-data-to-amazon-timestream-for-influxdb/). +- **InfluxDB v1 → Timestream for InfluxDB** — [InfluxDB v1 to v2 migration guide](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/guides/influxdb_v1_to_v2_migration). +- **Timestream for LiveAnalytics → Timestream for InfluxDB** — [LiveAnalytics migration tooling](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts) and its [Timestream for InfluxDB target guide](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts/targets/timestream_for_influxdb). +- **Service documentation** — [Amazon Timestream for InfluxDB developer guide](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influxdb.html) and [Applying the AWS Well-Architected Framework for Amazon Timestream for InfluxDB](https://docs.aws.amazon.com/prescriptive-guidance/latest/timestream-for-influxdb-well-architected-framework/introduction.html). diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/onboarding.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/onboarding.md similarity index 96% rename from src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/onboarding.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/onboarding.md index 1a9328bc56..c7ac38a82c 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/onboarding.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/onboarding.md @@ -129,8 +129,6 @@ Use `--cidr /32` rather than `0.0.0.0/0` to avoid exposing the instance ## Step 1 — Provision the Instance -**Storage sizing:** EBS storage cannot be increased after creation — plan capacity upfront. Minimum is 400GB; for production workloads estimate based on retention period × write rate × bytes per point. - **Standalone (single node):** ```bash # 1. Create parameter group (name must be alphanumeric only — no hyphens) @@ -270,7 +268,7 @@ Use `$ALL_ACCESS_TOKEN` as the Token for all subsequent data plane operations. T curl https://:8086/ping # Returns 204 — no auth required, confirms network connectivity -curl -H "Authorization: Token $ALL_ACCESS_TOKEN" \ +curl -H "Authorization: Bearer $ALL_ACCESS_TOKEN" \ https://:8086/health # Returns: {"name":"influxdb","status":"pass","version":"..."} ``` @@ -284,12 +282,12 @@ Best practice: use tokens scoped to the minimum permissions required for each op ```bash # Get bucket ID BUCKET_ID=$(curl -s "https://:8086/api/v2/buckets?org=my-org" \ - -H "Authorization: Token $ALL_ACCESS_TOKEN" \ + -H "Authorization: Bearer $ALL_ACCESS_TOKEN" \ | python3 -c "import sys,json; print([b for b in json.load(sys.stdin)['buckets'] if not b['name'].startswith('_')][0]['id'])") # Create a read/write token scoped to the bucket curl -X POST https://:8086/api/v2/authorizations \ - -H "Authorization: Token $ALL_ACCESS_TOKEN" \ + -H "Authorization: Bearer $ALL_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d "{ \"orgID\": \"$ORG_ID\", @@ -312,7 +310,7 @@ TOKEN="${APP_TOKEN:-$ALL_ACCESS_TOKEN}" # Use current Unix timestamp so the data appears in range(start: -1h) queries NOW=$(date +%s) curl -X POST "https://:8086/api/v2/write?org=my-org&bucket=my-bucket&precision=s" \ - -H "Authorization: Token $TOKEN" \ + -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: text/plain" \ -d "cpu,host=server01,region=us-east usage_idle=98.2,usage_system=1.8 $NOW cpu,host=server02,region=us-east usage_idle=92.1,usage_system=3.4 $NOW" @@ -327,7 +325,7 @@ Returns `204` on success. Common errors: **Flux:** ```bash curl -X POST "https://:8086/api/v2/query?org=my-org" \ - -H "Authorization: Token $TOKEN" \ + -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "Accept: application/csv" \ -d '{ @@ -339,7 +337,7 @@ curl -X POST "https://:8086/api/v2/query?org=my-org" \ **InfluxQL (V1 compatibility endpoint):** ```bash curl "https://:8086/query?db=my-bucket&q=SELECT+*+FROM+cpu+WHERE+time+>+now()+-+1h" \ - -H "Authorization: Token $TOKEN" + -H "Authorization: Bearer $TOKEN" ``` ## What's Created by `create-db-instance` diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/parameters.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/parameters.md new file mode 100644 index 0000000000..d3cf5959e3 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/parameters.md @@ -0,0 +1,172 @@ +# InfluxDB v2 Parameter Reference + +> **Single top-level key.** The `--parameters` JSON is a **tagged union** — for InfluxDB v2 +> you set exactly one top-level key, `InfluxDBv2`. You cannot mix it with `InfluxDBv3Core` +> or `InfluxDBv3Enterprise` in the same parameter group. Keys are case-sensitive. + +> **Read replicas are not configured here.** Unlike v3 Enterprise — where cluster topology +> (`ingestQueryInstances`, `queryOnlyInstances`, `dedicatedCompactor`) lives in the parameter +> group — v2 read replicas are a **cluster-deployment** setting, not a parameter. Configure +> them on `create-db-cluster` via `--deployment-type MULTI_NODE_READ_REPLICAS` and +> `--failover-mode` (`AUTOMATIC` / `NO_FAILOVER`), and choose the node size with +> `--db-instance-type`. The same `InfluxDBv2` parameter group applies uniformly to the primary +> and all read replicas in the cluster. See `onboarding.md` for the full cluster setup workflow. + +## Table of Contents +- [InfluxDBv2 Parameters](#influxdbv2-parameters) + - [Query and Logging](#query-and-logging) + - [HTTP Timeouts](#http-timeouts) + - [InfluxQL Query Limits](#influxql-query-limits) + - [Query Memory](#query-memory) + - [Session](#session) + - [Storage — Cache](#storage--cache) + - [Storage — Compaction](#storage--compaction) + - [Storage — Index and Series](#storage--index-and-series) + - [Storage — WAL](#storage--wal) +- [Duration Type Format](#duration-type-format) +- [CLI Examples](#cli-examples) + +## InfluxDBv2 Parameters + +All parameters below belong to the single `InfluxDBv2` top-level key. + +### Query and Logging + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `fluxLogEnabled` | boolean | — | Enable logging of Flux query execution | +| `logLevel` | string | `debug`, `info`, `error` | Log output level | +| `noTasks` | boolean | — | Disable the task scheduler on startup | +| `queryConcurrency` | integer | 0-256 | Number of queries allowed to execute concurrently | +| `queryQueueSize` | integer | 0-256 | Max number of queries allowed in execution queue | +| `tracingType` | string | `log`, `jaeger`, `disabled` | Tracing backend type | +| `metricsDisabled` | boolean | — | Disable the Prometheus `/metrics` endpoint | +| `pprofDisabled` | boolean | — | Disable the `/debug/pprof` profiling endpoint | +| `uiDisabled` | boolean | — | Disable the InfluxDB web UI | + +### HTTP Timeouts + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `httpIdleTimeout` | Duration | — | Max idle duration before timing out an HTTP connection | +| `httpReadHeaderTimeout` | Duration | — | Max duration to read HTTP request headers | +| `httpReadTimeout` | Duration | — | Max duration to read an entire HTTP request | +| `httpWriteTimeout` | Duration | — | Max duration before timing out an HTTP response write | + +### InfluxQL Query Limits + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `influxqlMaxSelectBuckets` | long | 0-1000000000000 | Max number of `GROUP BY time()` buckets a SELECT can process | +| `influxqlMaxSelectPoint` | long | 0-1000000000000 | Max number of points a SELECT can process | +| `influxqlMaxSelectSeries` | long | 0-1000000000000 | Max number of series a SELECT can process | + +### Query Memory + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `queryInitialMemoryBytes` | long | 0-1000000000000 | Initial bytes of memory allocated per query | +| `queryMaxMemoryBytes` | long | 0-1000000000000 | Max bytes of memory allowed across all running queries | +| `queryMemoryBytes` | long | 0-1000000000000 | Max bytes of memory allowed per individual query | + +### Session + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `sessionLength` | integer | 1-2880 | Session duration in minutes (max 48 hours) | +| `sessionRenewDisabled` | boolean | — | Disable automatic session renewal on activity | + +### Storage — Cache + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `storageCacheMaxMemorySize` | long | 0-1000000000000 | Max size of the in-memory write cache in bytes | +| `storageCacheSnapshotMemorySize` | long | 0-1000000000000 | Cache size in bytes that triggers a snapshot to a TSM file | +| `storageCacheSnapshotWriteColdDuration` | Duration | — | Idle duration before a cold shard's cache is snapshotted | + +### Storage — Compaction + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `storageCompactFullWriteColdDuration` | Duration | — | Idle duration before a cold shard is fully compacted | +| `storageCompactThroughputBurst` | long | 0-1000000000000 | Max bytes/sec burst rate for compaction throughput | +| `storageMaxConcurrentCompactions` | integer | 0-64 | Max number of concurrent full/level compactions | + +### Storage — Index and Series + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `storageMaxIndexLogFileSize` | long | 0-1000000000000 | Size threshold (bytes) at which the index log file is compacted | +| `storageNoValidateFieldSize` | boolean | — | Skip validation of maximum field/value size | +| `storageRetentionCheckInterval` | Duration | — | Interval between retention policy enforcement checks | +| `storageSeriesFileMaxConcurrentSnapshotCompactions` | integer | 0-64 | Max concurrent snapshot compactions for the series file | +| `storageSeriesIdSetCacheSize` | long | 0-1000000000000 | Size of the internal series ID set cache | + +### Storage — WAL (Write-Ahead Log) + +| Parameter | Type | Range | Description | +|-----------|------|-------|-------------| +| `storageWalMaxConcurrentWrites` | integer | 0-256 | Max number of concurrent WAL writes | +| `storageWalMaxWriteDelay` | Duration | — | Max duration a WAL write waits when the limit is reached before timing out | + +## Duration Type Format + +Duration parameters use a structure with `durationType` and `value`: + +```json +{ + "durationType": "seconds", + "value": 300 +} +``` + +Valid `durationType` values: `hours`, `minutes`, `seconds`, `milliseconds`, `days` + +Both `durationType` and `value` are **required** when the duration parameter is set. `value` must be `>= 0`. + +CLI shorthand: `durationType=seconds,value=300` + +## CLI Examples + +### Creating a v2 parameter group with custom settings + +```bash +aws timestream-influxdb create-db-parameter-group \ + --name my-v2-tuned \ + --description "Tuned v2 params for high-concurrency querying" \ + --parameters '{ + "InfluxDBv2": { + "queryConcurrency": 128, + "queryQueueSize": 256, + "logLevel": "info", + "fluxLogEnabled": true, + "influxqlMaxSelectSeries": 1000000, + "queryMaxMemoryBytes": 4294967296, + "httpReadTimeout": {"durationType": "seconds", "value": 60}, + "storageRetentionCheckInterval": {"durationType": "minutes", "value": 30} + } + }' \ + --region us-east-1 +``` + +### Creating a v2 parameter group with shorthand syntax + +```bash +aws timestream-influxdb create-db-parameter-group \ + --name my-v2-shorthand \ + --description "v2 params using shorthand syntax" \ + --parameters 'InfluxDBv2={queryConcurrency=64,logLevel=debug,httpIdleTimeout={durationType=minutes,value=5}}' \ + --region us-east-1 +``` + +### Associating the parameter group with an instance + +Parameter groups are immutable. To apply one (or change parameters), associate the group with +a DB instance and reboot: + +```bash +aws timestream-influxdb update-db-instance \ + --identifier my-v2-instance \ + --db-parameter-group-identifier my-v2-tuned \ + --region us-east-1 +``` diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/query-guide.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/query-guide.md new file mode 100644 index 0000000000..cc64b1b419 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/query-guide.md @@ -0,0 +1,285 @@ +# InfluxDB 2 Query Guide + +V2 supports two query languages: +- **Flux** — primary language, functional pipeline style, full-featured +- **InfluxQL** — SQL-like, available via V1 compatibility endpoint, limited vs Flux + +## Flux Query Endpoint + +``` +POST /api/v2/query?orgID= +Authorization: Bearer +Content-Type: application/vnd.flux +Accept: application/csv +``` + +Raw body: '' + +- `Accept: application/csv` is required — Flux always returns CSV. Omitting it causes a 400. +- `Content-Type: application/vnd.flux` is required. +- Use org ID (not name) in the `orgID` query param. +- Include the header `Accept-Encoding: gzip` for responses over 1.4 KB. Using compression saves network bandwidth but increases server-side load. + +For example: +```shell +curl \ + --request POST \ + http://localhost:8086/api/v2/query?orgID=ORG_ID \ + --header 'Authorization: Bearer API_TOKEN' \ + --header 'Accept: application/csv' \ + --header 'Content-Type: application/vnd.flux' \ + --data 'from(bucket:"BUCKET_NAME") + |> range(start: -12h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> aggregateWindow(every: 1h, fn: mean)' +``` + +or + +``` +POST /api/v2/query +Authorization: Bearer +Content-Type: application/json +Accept: application/csv +``` + +JSON body: +- `dialect`: Options for tabular data output. Default output is annotated CSV with headers. [See W3 metadata vocabulary for tabular data](https://www.w3.org/TR/2015/REC-tabular-metadata-20151217/#dialect-descriptions). +- `extern`: Represents a source from a file with a list of Flux statements. +- `now`: Specifies the time that should be treated as the current time. Default is the server's current time. +- `query`: Required. The query to execute. +- `type`: The type of query. Must be `flux`. + +For example: +```shell +curl --request POST \ + "http://localhost:8086/api/v2/query" \ + --header "Authorization: Bearer INFLUX_TOKEN" \ + --header "Content-Type: application/json" \ + --data-raw '{ + "dialect": {}, + "extern": {}, + "now": "NOW", + "query": "QUERY", + "type": "flux" +}' + +Change InfluxDB URL +``` + +Passing `"dialect": {}` and `"extern": {}` (as above) just means "use defaults / inject nothing". Both are defined in InfluxData's OpenAPI source, not the prose docs — see [`Query.yml`](https://github.com/influxdata/openapi/blob/master/src/common/schemas/Query.yml), [`Dialect.yml`](https://github.com/influxdata/openapi/blob/master/src/common/schemas/Dialect.yml), and [`File.yml`](https://github.com/influxdata/openapi/blob/master/src/common/schemas/File.yml). + +### `dialect` — output (annotated CSV) formatting + +| Field | Meaning | Default | +|-------|---------|---------| +| `annotations` | Annotation rows to include: any of `group`, `datatype`, `default` | `[]` (none) | +| `header` | Include the column-name header row | `true` | +| `delimiter` | Column separator (single char) | `,` | +| `commentPrefix` | Prefix marking annotation rows | `#` | +| `dateTimeFormat` | `RFC3339` or `RFC3339Nano` (nanosecond precision) | `RFC3339` | + +The Flux CSV parsers expect the full annotation set, so for round-trippable output set `"annotations": ["group","datatype","default"]`. + +### `extern` — a Flux AST prepended to the query + +`extern` is a Flux **AST `File`** node whose `body` is a list of statements injected before your `query`. Use it to supply variables without string-concatenating values into the query text. Example — inject `mybucket = "telegraf"`: + +```json +{ + "type": "flux", + "query": "from(bucket: mybucket) |> range(start: -1h)", + "extern": { + "type": "File", + "body": [ + { + "type": "VariableAssignment", + "id": { "type": "Identifier", "name": "mybucket" }, + "init": { "type": "StringLiteral", "value": "telegraf" } + } + ] + } +} +``` + +InfluxDB runs `mybucket = "telegraf"` then the query. `init` accepts any Flux expression node (`StringLiteral`, `IntegerLiteral`, `ObjectExpression`, …). This is how Grafana injects its `v` time-range record. + +- **Simpler alternative:** for plain value injection use `params` instead — `"params": {"mybucket":"telegraf"}` with `query: "from(bucket: params.mybucket) ..."`. You **cannot** use `params` and `extern` together. +- **Generating the AST:** don't hand-write it — POST Flux text to `/api/v2/query/ast` (or run `flux ast`) to get the JSON for `extern`. + +## Flux Patterns + +### Basic filter + +```flux +from(bucket: "my-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_idle") +``` + +`range()` is always required — Flux will not scan without a time bound. + +`range()` can be relative to the current time, such as `range(start: -1h, stop: -10m)` or absolute, such as `range(start: 2026-01-01T00:00:00Z, stop: 2026-01-01T12:00:00Z)`. + +### Aggregation over time windows + +```flux +from(bucket: "my-bucket") + |> range(start: -24h) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_idle") + |> aggregateWindow(every: 5m, fn: mean, createEmpty: false) +``` + +`createEmpty: false` prevents null rows for windows with no data. + +### Multiple fields with pivot + +Flux returns one row per field by default. Use `pivot` to get all fields as columns: + +```flux +from(bucket: "my-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu") + |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") +``` + +### Filter by tag value + +```flux +from(bucket: "my-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu" and r.host == "server01") +``` + +Tag filters go in the same `filter()` call as measurement/field filters. + +### Top N values + +```flux +from(bucket: "my-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_idle") + |> top(n: 5, columns: ["_value"]) +``` + +### Last value per tag group + +```flux +from(bucket: "my-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_idle") + |> last() +``` + +### Downsampling with aggregateWindow + write to another bucket + +```flux +from(bucket: "raw") + |> range(start: -task.every) + |> filter(fn: (r) => r._measurement == "cpu") + |> aggregateWindow(every: 5m, fn: mean, createEmpty: false) + |> to(bucket: "downsampled", org: "my-org") +``` + +Used in scheduled tasks (see Tasks section below). + +### Math between fields + +```flux +from(bucket: "my-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "cpu") + |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with usage_total: r.usage_user + r.usage_system })) +``` + +## InfluxQL Query Endpoint + +``` +GET /query?db=&q=&epoch= +Authorization: Bearer +``` + +Or via POST with form encoding: +``` +POST /query +Authorization: Bearer +Content-Type: application/x-www-form-urlencoded + +db=&q= +``` + +- `db` maps to the bucket name (not ID) +- `epoch` sets timestamp format in results (optional) + +### InfluxQL Patterns + +**Basic select:** +```sql +SELECT * FROM cpu WHERE time > now() - 1h +``` + +**Aggregation with time grouping:** +```sql +SELECT mean(usage_idle), max(usage_system) +FROM cpu +WHERE time > now() - 24h +GROUP BY time(5m), host +``` + +**Filter by tag:** +```sql +SELECT mean(usage_idle) FROM cpu +WHERE host = 'server01' AND time > now() - 1h +GROUP BY time(5m) +``` + +**Show measurements and tag keys:** +```sql +SHOW MEASUREMENTS +SHOW TAG KEYS FROM cpu +SHOW TAG VALUES FROM cpu WITH KEY = "host" +SHOW FIELD KEYS FROM cpu +``` + +**Limitations vs Flux:** InfluxQL cannot join across measurements, has no `pivot`, no custom functions, and limited math operations. Use Flux for anything beyond basic aggregations. + +## Scheduled Tasks (Flux) + +Tasks run Flux scripts on a schedule — used for downsampling, alerting, and data transformation. + +```bash +# Create a task +curl -X POST "https://:8086/api/v2/tasks" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "orgID": "", + "name": "downsample-cpu", + "every": "1h", + "flux": "option task = {name: \"downsample-cpu\", every: 1h}\nfrom(bucket: \"raw\") |> range(start: -task.every) |> filter(fn: (r) => r._measurement == \"cpu\") |> aggregateWindow(every: 5m, fn: mean, createEmpty: false) |> to(bucket: \"downsampled\")" + }' + +# List tasks +curl "https://:8086/api/v2/tasks?orgID=" \ + -H "Authorization: Bearer " + +# Manually trigger a task run +curl -X POST "https://:8086/api/v2/tasks//runs" \ + -H "Authorization: Bearer " + +# Check run history +curl "https://:8086/api/v2/tasks//runs" \ + -H "Authorization: Bearer " +``` + +Task `status`: `active` (runs on schedule) | `inactive` (paused). Use `PATCH /api/v2/tasks/` with `{"status": "inactive"}` to pause. + +## Query Performance + +- Always use `range()` with the narrowest time window needed — full scans are expensive +- Filter on tags (indexed) before fields (not indexed) in `filter()` calls +- Use `aggregateWindow` instead of `window` + `reduce` for standard aggregations — it's optimized +- For "current state" queries (latest value per series), use `last()` after a short range rather than scanning all time +- Tune `queryConcurrency` and `queryMaxMemoryBytes` in the parameter group for high-concurrency workloads +- If queries timeout, increase `httpReadTimeout` in the parameter group diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/schema-design.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/schema-design.md new file mode 100644 index 0000000000..661f743c33 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/schema-design.md @@ -0,0 +1,223 @@ +# InfluxDB v2 Schema Design & Data Modeling + +## V2 vs V3 Data Model + +| Concept | V2 | V3 | +|---------|----|----| +| Top-level namespace | Organization | *(none)* | +| Data container | Bucket (with retention) | Database (with retention) | +| Logical grouping | Measurement | Table (auto-created from measurement) | +| Indexed metadata | Tags (string only) | Tags → columns (indexed) | +| Value storage | Fields | Fields → columns (non-indexed) | +| Retention config | `retentionRules[].everySeconds` | `retentionPeriod` (e.g. `"30d"`) | +| Series cardinality | ~10M typical threshold (varies by instance) | Virtually unlimited | +| Query language | Flux (primary), InfluxQL | SQL (primary), InfluxQL | + +For key differences between InfluxDB v2 and v3, see [influxdb-2-vs-3.md](../influxdb-2-vs-3.md). + +## Measurement Naming + +Measurements are auto-created on first write — there is no explicit create step. In V3, each measurement becomes a table. + +Rules: +- Use simple, descriptive names that describe the data: `cpu`, `memory`, `http_requests`, `sensor_reading` +- Do **not** encode data in the measurement name. `blueberries.plot-1.north` is wrong — use tags for `crop`, `plot`, `region` instead. +- Do not use dots, hyphens, or concatenated attributes in measurement names — they force regex queries and prevent filtering. +- Avoid Flux keywords (`from`, `to`, `filter`, `range`, `yield`) — they require quoting. +- Case-sensitive: `CPU` and `cpu` are different measurements. + +When to use one measurement vs multiple: +- Use **one measurement** when data shares the same tags and fields (e.g., all CPU metrics in `cpu` with fields `usage_idle`, `usage_system`, `usage_user`). +- Use **separate measurements** when data has different tag/field schemas (e.g., `cpu` and `disk` have different fields and tags). +- Do **not** create a measurement per entity (e.g., `cpu_server01`, `cpu_server02`) — use a `host` tag instead. + +## Tag vs Field Decision + +| Criterion | Tag | Field | +|-----------|-----|-------| +| Indexed? | Yes — fast filtering | No — full scan | +| Data type | Strings only | Float, integer, string, boolean | +| Use for filtering? | Yes | Avoid if possible | +| Use for grouping? | Yes | No | +| Unique values | Low-to-moderate (< 100K distinct on V2) | Unlimited | +| Numeric data | No — store as field | Yes | + +Rules: +- **Tags** store metadata shared across many points: `host`, `region`, `sensor_id`, `environment`. +- **Fields** store numeric measurements and unique/variable data: `temperature`, `usage_idle`, `request_count`. +- Never store continuously changing values (timestamps, UUIDs, log messages) as tags — this causes cardinality explosion on V2 and wastes index space on V3. +- Avoid duplicate names for a tag key and field key within the same measurement — query results become unpredictable. +- Sort tags alphabetically in line protocol for best write compression (both engines). + +Cardinality considerations: Tags with high unique values directly increase series cardinality. Keep individual tags under ~100K distinct values. Total series cardinality above ~10M (varies by instance size) causes TSM degradation. + +## Bucket Design + +A bucket is a named container with a retention period. All data in a bucket shares the same retention policy. + +Design principle: **one bucket per retention period**. If you need 7-day and 90-day retention, create two buckets. + +Common patterns: + +| Pattern | Buckets | Use case | +|---------|---------|----------| +| By retention | `raw-7d`, `downsampled-90d`, `archive-365d` | Different data lifetimes | +| By environment | `metrics-prod`, `metrics-staging` | Isolation + different retention | +| By team/tenant | `team-a-metrics`, `team-b-metrics` | Access control via scoped tokens | + +Guidance: +- The initial bucket is created automatically by `create-db-instance` — use it for your primary workload. +- Create additional buckets via `POST /api/v2/buckets` for different retention needs. +- Scope tokens to specific buckets for access control — one token per bucket per application. +- Do **not** create a bucket per measurement or per host — this creates management overhead with no benefit. +- On Timestream for InfluxDB V2, **do not delete and recreate buckets** when using read replicas — the bucket ID mismatch breaks replication. Update the bucket instead. + +### Bucket creation example + +```bash +curl -X POST "https://:8086/api/v2/buckets" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "downsampled-90d", + "orgID": "", + "retentionRules": [{ "type": "expire", "everySeconds": 7776000 }], + "description": "90-day downsampled metrics" + }' +``` + +## Retention Policy + +Retention is configured per bucket via `retentionRules[].everySeconds`. The retention enforcement service runs every 30 minutes by default and deletes entire shard groups (not individual points) when the shard group's time range is fully beyond the retention period. + +Common retention values: + +| Duration | `everySeconds` | Use case | +|----------|---------------|----------| +| 1 hour | `3600` | Debugging / ephemeral data | +| 1 day | `86400` | Short-term operational metrics | +| 7 days | `604800` | Standard monitoring | +| 30 days | `2592000` | Default for most workloads | +| 90 days | `7776000` | Compliance / trend analysis | +| 1 year | `31536000` | Long-term capacity planning | +| Infinite | `0` | Never expire (use with caution — disk fills) | + +When data is actually deleted: +- **Minimum**: after `retention-period` has elapsed +- **Maximum**: after `retention-period + shard-group-duration` has elapsed +- Shard group duration is auto-calculated from retention period. For a 7-day retention, shard group duration is typically 1 day, so data persists 7–8 days. +- Data remains queryable until the shard group is deleted. + +Retention can be updated on an existing bucket via `PATCH /api/v2/buckets/{bucketID}`: + +```bash +curl -X PATCH "https://:8086/api/v2/buckets/" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ "retentionRules": [{ "type": "expire", "everySeconds": 604800 }] }' +``` + +### Downsampling pattern + +Use a Flux task to aggregate raw data into a longer-retention bucket: + +1. Create `raw-7d` bucket (7-day retention) for high-resolution data +2. Create `downsampled-90d` bucket (90-day retention) for aggregated data +3. Schedule a task to downsample hourly: + +```flux +option task = {name: "downsample-cpu", every: 1h} + +from(bucket: "raw-7d") + |> range(start: -task.every) + |> filter(fn: (r) => r._measurement == "cpu") + |> aggregateWindow(every: 5m, fn: mean, createEmpty: false) + |> to(bucket: "downsampled-90d") +``` + +## Field Type Conflicts + +Field types are **locked on first write** per measurement per field key on both V2 and V3. Writing a different type to the same field causes an error and the conflicting points are dropped. + +| First write | Subsequent write | Result | +|-------------|-----------------|--------| +| `temp=72.3` (float) | `temp=72i` (integer) | Error — `temp` is locked as float | +| `count=10i` (integer) | `count=10.0` (float) | Error — `count` is locked as integer | +| `status="ok"` (string) | `status=true` (boolean) | Error — `status` is locked as string | + +InfluxDB v2 returns **422**. Conflicting points are dropped; valid points in the same batch succeed (partial write). + +Prevention: +- Document your schema before writing. Agree on types per field across all writers. +- Use explicit type suffixes in line protocol: `i` for integer, `"quotes"` for string, bare number for float. +- Be careful with numeric fields — `10` is a float, `10i` is an integer. Mixing these is the most common conflict. +- Telegraf plugins have fixed output types — check plugin docs before adding new inputs. + +Detection: use `SHOW FIELD KEYS FROM ` (InfluxQL) to check current field types. + +Resolution: +- You **cannot change** a field's type after first write. Options: + 1. Write to a new field name (e.g., `temp_f` instead of `temp`) and update queries. + 2. Delete all data in the measurement and rewrite with the correct type. + 3. Create a new measurement with the correct schema and migrate data via a task. + +## Series Cardinality + +A **series** is a unique combination of measurement name + tag set. Series cardinality = total number of unique series across all measurements. + +Example: `cpu,host=A,region=us` and `cpu,host=B,region=us` are 2 series. + +V2 TSM engine: performance typically degrades above **~10M series**, depending on instance size and workload. Symptoms: slow queries, high memory, compaction stalls. + +Measure cardinality with Flux: + +```flux +import "influxdata/influxdb/schema" + +schema.tagValues(bucket: "my-bucket", tag: "host") + |> count() +``` + +Reduce cardinality: +- Move high-cardinality values from tags to fields +- Remove unnecessary tags +- Use bounded values (e.g., `region` not `ip_address`) +- Delete old high-cardinality data + +## Schema Examples + +### IoT sensor monitoring + +``` +sensor_reading,device_id=D001,location=warehouse-a,type=temperature value=22.5 1709251200000000000 +sensor_reading,device_id=D001,location=warehouse-a,type=humidity value=45.2 1709251200000000000 +sensor_reading,device_id=D002,location=warehouse-b,type=temperature value=19.8 1709251200000000000 +``` + +- Measurement: `sensor_reading` (one measurement for all sensor types) +- Tags: `device_id` (bounded set of devices), `location`, `type` +- Field: `value` (numeric reading) +- Cardinality: devices × locations × types + +### Infrastructure monitoring + +``` +cpu,host=web01,region=us-east-1 usage_idle=92.3,usage_system=3.1,usage_user=4.6 1709251200000000000 +memory,host=web01,region=us-east-1 used_percent=67.2,available=8589934592i 1709251200000000000 +disk,host=web01,region=us-east-1,device=sda1 used_percent=45.0,free=107374182400i 1709251200000000000 +``` + +- Separate measurements for `cpu`, `memory`, `disk` (different field schemas) +- Tags: `host`, `region`, `device` (bounded) +- Fields: numeric metrics + +### Application metrics + +``` +http_requests,method=GET,endpoint=/api/users,status=200 count=1523i,latency_ms=45.2 1709251200000000000 +http_requests,method=POST,endpoint=/api/users,status=201 count=89i,latency_ms=120.5 1709251200000000000 +``` + +- Tags: `method`, `endpoint`, `status` (all bounded) +- Fields: `count` (integer), `latency_ms` (float) +- Do **not** tag `request_id` or `user_id` — this causes high cardinality. diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/troubleshooting.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/troubleshooting.md new file mode 100644 index 0000000000..f1040839ef --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb2/troubleshooting.md @@ -0,0 +1,98 @@ +# Troubleshooting in InfluxDB 2 + +Version-specific errors for **Timestream for InfluxDB** (the InfluxDB 2.x engine) and how +to resolve them. For cross-cutting connection/authorization problems (port, token prefix, +networking, namespace), start with the parent [troubleshooting.md](../troubleshooting.md). +For terminology, see [glossary.md](../glossary.md). + +> V2 quick facts: default port **8086**, auth header `Authorization: Bearer `, +> write/query namespace is `org` + `bucket`, Flux is primary (InfluxQL also supported). +> See [influxdb-2-vs-3.md](../influxdb-2-vs-3.md). + +## Provisioning (instances & clusters) + +- **Cluster/instance stuck in `CREATING` or `FAILED`** — run `get-db-cluster` / + `get-db-instance` to read `status`. If `FAILED`: verify subnets and security groups are + valid and in the same VPC, the security group allows inbound on **8086**, then delete + and recreate with corrected parameters. Check CloudTrail for the create-event error. +- **`ValidationException` on `--password`** — the password accepts only `[a-zA-Z0-9]+`; + special characters are rejected. Omit it to have one auto-generated and stored in + Secrets Manager. +- **Parameter group name rejected** — `--name` for `create-db-parameter-group` accepts + only `[a-zA-Z0-9]+`. Reference the group by its **ID** in + `--db-parameter-group-identifier`. +- **`ResourceNotFoundException` on `--db-cluster-id`** — use the service-generated + `dbClusterId` (from the create response or `list-db-clusters`), not the name you chose. + +## Writing / ingestion + +Status codes for the v2 write API (see the +[InfluxDB v2 write troubleshooting docs](https://docs.influxdata.com/influxdb/v2/write-data/troubleshoot/)): + +- **`400` Bad request** — malformed line protocol; **all request data is rejected** and + the response body contains the first malformed line. Validate syntax (see + [line-protocol.md](../line-protocol.md)). +- **`401` Unauthorized** — missing/malformed `Authorization: Bearer` header or insufficient + token permissions. +- **`404` Not found** — a resource such as the `org` or `bucket` wasn't found. +- **`413` Request entity too large** — reduce batch size. +- **`422` Unprocessable entity** — well-formed request, but some/all points were rejected + for **semantic** reasons (schema/field-type conflict or retention-policy violation). + Non-rejected points are still ingested (partial write); field types are **locked on + first write**, so keep them consistent. +- **`503` Service unavailable** — server temporarily can't accept writes; retry per the + `Retry-After` header. +- **Replica lag / slow writes** — batch **5,000+ points per request** instead of many + tiny writes (see [`gotchas.md`](./gotchas.md)). + +## Querying (Flux / InfluxQL) + +Patterns and endpoints: [`query-guide.md`](./query-guide.md). + +- **InfluxQL returns "no databases" / empty results** — InfluxQL needs a **DBRP mapping** + (database/retention-policy → bucket). Without it, queries find nothing. Configure the + mapping, or use Flux. +- **Queries are slow** — narrow the `range()`, filter on **tags before fields** (tags are + indexed), use `aggregateWindow` for standard aggregations, and use `last()` after a + short range for "current value" lookups (see the Query Performance section of + [`query-guide.md`](./query-guide.md)). +- **Queries time out or OOM under load** — tune `queryConcurrency`, + `queryMaxMemoryBytes`, and `httpReadTimeout` via the parameter group. + +## Schema & cardinality + +See [`schema-design.md`](./schema-design.md) and [`gotchas.md`](./gotchas.md). + +- **Slow queries, high memory, compaction stalls** — classic **high series cardinality**. + The TSM/TSI engine degrades above **~10M series**. Audit tags for high-uniqueness values + (e.g. request IDs) and move them to **fields** instead of tags. +- **Expecting very high cardinality (1M+ unique tag combos)** — consider **V3** instead, + which has no practical cardinality limit (see [influxdb-2-vs-3.md](../influxdb-2-vs-3.md)). + +## Parameters & storage + +See [`gotchas.md`](./gotchas.md). + +- **Parameter change didn't take effect** — parameter groups are **immutable** and changes + require a **reboot**. Create a new group and reassign with + `update-db-cluster --db-parameter-group-identifier NEW_ID` (reboots automatically). + There is no `UpdateDbParameterGroup` operation. + +## Cost + +- **Unexpectedly high bill** — V2 **read replicas add a license fee** on top of base + instance cost; regional pricing multipliers apply (see [`gotchas.md`](./gotchas.md)). + +## Migrating to V3 + +Planning a move off V2 to V3? See [influxdb3/migrations.md](../influxdb3/migrations.md) and +[influxdb-2-vs-3.md](../influxdb-2-vs-3.md) — note there is no turnkey query migration and +Flux must be rewritten to SQL/InfluxQL. + +## Health check + +`GET /health` is unauthenticated — use it to confirm the engine is reachable before +chasing auth or query issues: +```shell +curl https://:8086/health +``` diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/dashboard-guide.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/dashboard-guide.md new file mode 100644 index 0000000000..5a470aa920 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/dashboard-guide.md @@ -0,0 +1,205 @@ +# Grafana Dashboards for InfluxDB 3 + +How to visualize **Timestream for InfluxDB v3** data in Grafana. + +## Data source +Use Grafana's built-in [**InfluxDB** data source plugin](https://grafana.com/docs/grafana/latest/datasources/influxdb/) (Grafana 12+ recommended for full v3 support). Add it via **Connections → Data sources → InfluxDB**. For v3 the query language is **SQL** (primary) or **InfluxQL**. **Flux is not supported in v3** — do not select it. + +### Connection settings +- **URL:** `https://:8181` (v3 default port). +- **Query language: SQL** (or InfluxQL). +- **Database:** your v3 database name (replaces v2's org/bucket). +- **Token:** a data-plane token with read access; v3 uses the `Bearer` prefix (retrieve from Secrets Manager — see `influxdb3/onboarding.md`). +- **Network reachability:** Grafana must reach the cluster. Private clusters need in-VPC Grafana or a bastion port-forward; the reader endpoint is used when the cluster has query-only nodes (Enterprise). + +### SQL example +```sql +SELECT date_bin($__interval, time) AS time, mean(usage) AS usage +FROM cpu +WHERE $__timeFilter(time) +GROUP BY 1 ORDER BY 1 +``` +`$__timeFilter(time)` and `$__interval` are Grafana macros that bind to the panel time range and auto-interval. Leave `$__interval` unquoted — Grafana substitutes a SQL interval literal, so quoting it breaks the `date_bin` call. + +### InfluxQL (alternative) +Set **Query language: InfluxQL** to reuse legacy v1-style queries against v3's compatibility endpoint. Use SQL for new dashboards — it is the native, best-supported path. + +InfluxQL ships several time-series functions that have no direct SQL equivalent. They run against v3's InfluxQL compatibility endpoint and are useful in panels (`$timeFilter` and `$__interval` are the Grafana macros for the panel time range and auto-interval): + +- **Rate of change** of a counter, per second. Use the non-negative variant to ignore counter resets: + ```sql + SELECT non_negative_derivative(mean("usage"), 1s) FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) + ``` +- **Smoothing** with a moving average over N points: + ```sql + SELECT moving_average(mean("usage"), 5) FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) + ``` +- **Percentile** (e.g. p95) for latency-style panels: + ```sql + SELECT percentile("usage", 95) FROM "cpu" WHERE $timeFilter + ``` +- **Top-N** points: + ```sql + SELECT top("usage", 5) FROM "cpu" WHERE $timeFilter + ``` +- **Cumulative sum** and **point-to-point difference**: + ```sql + SELECT cumulative_sum(mean("usage")) FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) + SELECT difference(mean("usage")) FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) + ``` +- **Spread / median / stddev** aggregations in a single panel: + ```sql + SELECT spread("usage"), median("usage"), stddev("usage") FROM "cpu" WHERE $timeFilter + ``` +- **Gap filling** for continuous lines — `fill(previous)` carries the last value forward, `fill(linear)` interpolates: + ```sql + SELECT mean("usage") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) fill(previous) + ``` + +> Not every InfluxQL function is implemented in v3 — for example `sample()` and `holt_winters()` return an error. Prefer SQL for anything not covered here. + +### Arrow Flight SQL (optional) +v3's native high-performance protocol is Arrow Flight SQL. The community **FlightSQL** Grafana data source plugin can connect over gRPC for lower-latency querying, but the built-in InfluxDB data source (SQL) is the simplest and recommended starting point. + +## Building dashboards +- **Time series** and **Stat** panels for metrics; **Table** for raw rows. Use the SQL builder or raw editor. +- Use **template variables** (e.g. `SELECT DISTINCT host FROM cpu`) for filterable dashboards — the v3 Distinct Value Cache makes these fast. +- Grafana **alerting** and **annotations** work against the data source. + +## Monitoring the cluster itself +To dashboard cluster health (CPU, memory, write throughput, query latency), use the AWS sample in **awslabs/amazon-timestream-tools** → [`integrations/influxdb_metrics_dashboard`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/integrations/influxdb_metrics_dashboard): a CDK app that deploys a Telegraf collector scraping the cluster `/metrics` endpoint into CloudWatch with a pre-built Grafana dashboard. v3's processing engine also includes a System Metrics plugin, and CloudWatch metrics can be charted directly via Grafana's CloudWatch data source. + +## Managing dashboards via the Grafana HTTP API + +Grafana exposes a REST API for creating, reading, updating, and deleting dashboards. This is useful for provisioning panels as code or automating dashboard rollout across environments. See the [Grafana Dashboard HTTP API](https://grafana.com/docs/grafana/latest/developer-resources/api-reference/http-api/dashboard/) for more information. + +These calls target your **Grafana** server (not the InfluxDB cluster endpoint), authenticate with a Grafana **service account token** (`Authorization: Bearer `), and use the dashboard API available in Grafana 12+. `:namespace` is your org namespace (`default` for the default org), the dashboard JSON (panels, SQL/InfluxQL targets, variables) goes in `spec`, and `metadata.name` is the dashboard UID. + +**Create a dashboard** — `POST .../dashboards`, returns `201` (`409` if the UID already exists): +```bash +curl -X POST "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "metadata": { "generateName": "cpu-" }, + "spec": { "title": "CPU Dashboard", "schemaVersion": 41, "panels": [], "tags": ["influxdb"] } + }' +``` +Set `metadata.generateName` to let Grafana assign a random UID, or `metadata.name` to choose your own. Add `metadata.annotations."grafana.app/folder"` to place it in a folder. + +**Get a dashboard** — `GET .../dashboards/:uid`, returns `200` (or `404` if not found): +```bash +curl "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards/" \ + -H "Authorization: Bearer " +``` + +**List dashboards** — `GET .../dashboards`; page with `limit` and the `metadata.continue` token returned by the previous response: +```bash +curl "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards?limit=20" \ + -H "Authorization: Bearer " +``` + +**Delete a dashboard** — `DELETE .../dashboards/:uid`, returns `200`: +```bash +curl -X DELETE "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards/" \ + -H "Authorization: Bearer " +``` + +### Create an InfluxDB data source + +A dashboard panel references a data source by its `uid`, so create the InfluxDB data source first with `POST /api/datasources` (see the [Data source HTTP API](https://grafana.com/docs/grafana/latest/developer-resources/api-reference/http-api/api-legacy/data_source/)). Use the `jsonData`/`secureJsonData` fields documented for [InfluxDB provisioning](https://grafana.com/docs/grafana/latest/datasources/influxdb/configure/#provisioning-examples); the response returns the new data source's `uid`. + +**SQL** (`version: SQL`, with database name and token; SQL uses the FlightSQL/gRPC protocol): +```bash +curl -X POST "https:///api/datasources" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Timestream-InfluxDB-SQL", + "type": "influxdb", + "access": "proxy", + "url": "https://:8181", + "jsonData": { "version": "SQL", "dbName": "", "httpMode": "POST", "insecureGrpc": false }, + "secureJsonData": { "token": "" } + }' +``` +For InfluxQL instead, drop `version`/`insecureGrpc` and keep `"dbName": ""` with the token in an `Authorization: Bearer` header (`httpHeaderName1`/`httpHeaderValue1`). + +### Create a dashboard with panels + +Panels live in the dashboard `spec.panels` array. Each panel sets a `type`, a `gridPos` (layout), and `targets` (the queries), where each target's `datasource.uid` is the InfluxDB data source created above. For the SQL query language, the target carries the statement in the `rawSql` field (with `rawQuery: true`). It's easiest to put the body in a file and post it with `-d @dashboard.json`: +```json +{ + "metadata": { "generateName": "cpu-" }, + "spec": { + "title": "CPU Usage", + "schemaVersion": 41, + "tags": ["influxdb"], + "time": { "from": "now-6h", "to": "now" }, + "panels": [ + { + "type": "timeseries", + "title": "Mean usage over time", + "datasource": { "type": "influxdb", "uid": "" }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "influxdb", "uid": "" }, + "rawQuery": true, + "rawSql": "SELECT date_bin($__interval, time) AS time, mean(usage) AS usage FROM cpu WHERE $__timeFilter(time) GROUP BY 1 ORDER BY 1" + } + ] + } + ] + } +} +``` +```bash +curl -X POST "https:///apis/dashboard.grafana.app/v1/namespaces/default/dashboards" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d @dashboard.json +``` +Add more objects to the `panels` array — each with its own `gridPos` — to build a multi-panel dashboard. Set `"type": "stat"` for a single-value Stat panel or `"type": "table"` for a raw-rows Table panel. + +### Example: reference the variable in a panel query +Filter the panel by the selected `host` value. In InfluxQL, wrap the variable in a regex match (`=~ /^$host$/`) — Grafana interpolates multi-value selections as a regex like `(host1|host2)`: +```sql +SELECT mean("usage") FROM "cpu" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($__interval) fill(null) +``` +In SQL, compare against the variable directly — Grafana substitutes the selected value into the string, so quote it like any string literal: +```sql +SELECT date_bin($__interval, time) AS time, mean(usage) AS usage +FROM cpu +WHERE $__timeFilter(time) AND host = '$host' +GROUP BY 1 ORDER BY 1 +``` +For a **multi-value** variable, use `IN` with the `:singlequote` format so each selected value is quoted: `... AND host IN (${host:singlequote})`. +### Example: one series per tag value +`GROUP BY` a tag to render a separate line per host in a **Time series** panel: +```sql +SELECT mean("usage") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval), "host" fill(null) +``` +SQL equivalent — select the `host` tag as a column and add it to the `GROUP BY`; Grafana renders each distinct `host` as its own series: +```sql +SELECT date_bin($__interval, time) AS time, host, mean(usage) AS usage +FROM cpu +WHERE $__timeFilter(time) +GROUP BY 1, host ORDER BY 1 +``` + +### Example: Table panel for raw rows +For a **Table** panel showing raw, unaggregated rows, select the fields directly and skip the `date_bin(...)`/`GROUP BY`: +```sql +SELECT time, host, usage +FROM cpu +WHERE $__timeFilter(time) +ORDER BY time DESC +LIMIT 100 +``` + +## See also +- Ingestion via Telegraf: [`integrations/telegraf`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/integrations/telegraf) +- [Grafana InfluxDB data source docs](https://grafana.com/docs/grafana/latest/datasources/influxdb/) +- Multiagent observability with Grafana: [`sample_apps/python/multiagent_observability`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/sample_apps/python/multiagent_observability#grafana-dashboard-overview). diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/development-guide.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/development-guide.md new file mode 100644 index 0000000000..93c6d743bb --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/development-guide.md @@ -0,0 +1,31 @@ +# InfluxDB 3 Development Guide + +## Overview + +## Best Practices Guide + +- SHOULD never attempt FLUX queries + +## Tool Examples + +### Queries + +For query guidance, see [query-guide.md](./query-guide.md). + +### Writes + +For ingestion and writing guidance, see [ingestion.md](./ingestion.md). + +### Schema Operations + +For schema operations, see [query-guide.md](./query-guide.md) and [schema-design.md](./schema-design.md). + +## Workflow Examples + +## Limitations + +Core has significant restrictions compared to Enterprise. See [influxdb3/troubleshooting.md](./troubleshooting.md#core-vs-enterprise) for more information. + +## Troubleshooting + +See [troubleshooting.md](./troubleshooting.md). diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/gotchas.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/gotchas.md similarity index 75% rename from src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/gotchas.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/gotchas.md index b55e974874..8f389fb2d7 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/gotchas.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/gotchas.md @@ -22,13 +22,11 @@ ## Operational — Will Cause Scaling Issues -**10K Tables Limit**: Default max tables per database. Siemens hit this. Increasing beyond recommendation requires understanding compaction and query performance implications. - -**S3 Endpoint for Private V3**: Private V3 instances require an S3 VPC endpoint in the same account VPC. Without it, writes fail silently or timeout. +**10K Tables Limit**: Default is **10,000 tables across all databases** (not per database), set by the `--num-table-limit` server config and **not exposed in the Timestream parameter group**. Each unique measurement is a table. Also note the **500-columns-per-table** limit (1 timestamp + up to 499 tag/field columns). More tables means more compaction work and more object-store PUTs — review compaction/query implications before relying on a high table count. See [Database, table, and column limits](https://docs.influxdata.com/influxdb3/enterprise/admin/databases/#database-table-and-column-limits). **Compactor Node OOM**: In V3 Enterprise, uneven load distribution can cause compactor nodes to OOM. Monitor `system.parquet_files` for growing file counts. Consider dedicated compactor nodes. -**IO Threads Default**: V3 default is 2 IO threads (`--num-io-threads`). Often insufficient for production workloads. Tune via parameter group. +**DataFusion Threads**: V3 has **no IO-threads parameter** (`num-io-threads` does not exist in the parameter group). To tune query parallelism, set `dataFusionNumThreads` (and related `dataFusionRuntime*` options) via the parameter group. ## Cost — Will Cause Bill Shock diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/ingestion.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/ingestion.md new file mode 100644 index 0000000000..afc6bdc761 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/ingestion.md @@ -0,0 +1,201 @@ +# InfluxDB v3 Data Ingestion & Writing + +## Write Endpoints + +``` +POST /api/v3/write_lp?db=&precision= +Authorization: Bearer +Content-Type: text/plain +``` + +- No `org` parameter — V3 has no organizations. +- `db` is the database name (required). +- `accept_partial` (boolean, default true) — when true, valid points succeed even if some fail. +- Default port is **8181** (not 8086 like V2). +- Auth uses `Bearer` prefix (not `Token`). + +### Backward-compatible endpoint + +V3 also supports the V2 write endpoint for migration: + +``` +POST /api/v2/write?bucket=&precision= +Authorization: Bearer +Content-Type: text/plain +``` + +- `bucket` maps to the V3 database name. +- `org` parameter is accepted but **ignored**. +- Allows existing V2 client code (including `influxdb-client` Python library) to write to V3 without changes — just update the URL, port, and token prefix. + +## Batching + +**Always batch writes.** Single-point writes with many concurrent writers cause severe replica lag. + +- Target **5,000+ points per request** minimum +- Optimal batch size: 5,000–10,000 points +- Max recommended: ~50,000 points per request (beyond this, HTTP timeouts become a risk) +- For high-throughput workloads, tune write timeout in the parameter group (see Tuning section below) + +## Python Client + +```bash +pip install influxdb3-python +``` + +```python +from influxdb_client_3 import InfluxDBClient3, Point + +client = InfluxDBClient3( + host="", # Hostname, including scheme and port. For example: http://localhost:8181 + token="", + database="" +) + +# Write a single point +point = Point("cpu") \ + .tag("host", "server01") \ + .tag("region", "us-east") \ + .field("usage_idle", 98.2) \ + .field("usage_system", 1.8) +client.write(point) + +# Write with line protocol string +client.write("cpu,host=server02,region=us-west usage_idle=92.1 1709251200000000000") + +# Write a batch of points +points = [ + Point("cpu").tag("host", f"server{i}").field("usage_idle", 90.0 + i) + for i in range(5000) +] +client.write(points) + +client.close() +``` + +For batching with callbacks (high-throughput): + +```python +from influxdb_client_3 import InfluxDBClient3, Point, WriteOptions, InfluxDBError, write_client_options + +def on_success(conf, data: str): + print(f"Written batch: {conf}") + +def on_error(conf, data: str, exception: InfluxDBError): + print(f"Write failed: {exception}") + +def on_retry(conf, data: str, exception: InfluxDBError): + print(f"Retrying: {exception}") + +write_opts = WriteOptions(batch_size=5000, flush_interval=1_000) +wco = write_client_options( + success_callback=on_success, + error_callback=on_error, + retry_callback=on_retry, + write_options=write_opts, +) + +with InfluxDBClient3( + host="", + token="", + database="", + write_client_options=wco, +) as client: + for i in range(10000): + client.write(Point("cpu").tag("host", f"server{i % 100}").field("usage_idle", 90.0 + (i % 10))) +# Buffer is flushed on context manager exit +``` + +### Migrating V2 Python code to V3 + +If you have existing V2 `influxdb-client` code, you can write to V3 without changing libraries by using the V2 compatibility endpoint: + +```python +from influxdb_client import InfluxDBClient, Point +from influxdb_client.client.write_api import SYNCHRONOUS + +# Point to V3 instance — use port 8181, org is ignored +client = InfluxDBClient( + url="https://:8181", + token="", + org="-" # required by client but ignored by V3 +) +write_api = client.write_api(write_options=SYNCHRONOUS) + +# bucket maps to V3 database name +write_api.write(bucket="", record=Point("cpu").tag("host", "server01").field("usage_idle", 98.2)) + +write_api.close() +client.close() +``` + +Note: The V2 client sends `Authorization: Token ` by default. If V3 requires `Bearer` prefix, set `auth_scheme="Bearer"` in the `InfluxDBClient` constructor. + +## Telegraf + +Telegraf is the recommended agent for infrastructure metrics. + +### Output + +Telegraf can write to V3 using the same `influxdb_v2` output plugin via the V2 compatibility endpoint: + +```toml +[[outputs.influxdb_v2]] + urls = ["https://:8181"] + token = "" + organization = "-" # required by plugin but ignored by V3 + bucket = "" # maps to V3 database name +``` + +### Common Telegraf settings + +```toml +# Example: collect CPU metrics +[[inputs.cpu]] + percpu = false + totalcpu = true + collect_cpu_time = false +``` + +```bash +# Test config before running +telegraf --config /etc/telegraf/telegraf.conf --test + +# Run +telegraf --config /etc/telegraf/telegraf.conf +``` + +Telegraf automatically batches writes — default `metric_batch_size = 1000`. Increase to 5000+ for high-throughput: + +```toml +[agent] + metric_batch_size = 5000 + metric_buffer_limit = 100000 +``` + +## Write Error Handling + +| HTTP Status | Cause | Action | +|-------------|-------|--------| +| 204 | Success | — | +| 400 | Malformed line protocol | Check syntax | +| 401 | Invalid or expired token | Verify token has write permission | +| 404 | Database not found | Verify names or IDs | +| 413 | Payload too large | Reduce batch size | +| 422 | Writing the line protocol points would exceed the maximum number of tables, databases, columns, tags, or fields | Check line protocol and existing data | +| 429 | Too many requests | Back off and retry with exponential backoff | +| 500 | Server error | Retry; check instance health via `/health` | + +## Tuning Write Performance + +### Parameter group settings + +| Parameter | Default | Recommendation | +|-----------|---------|----------------| +| `walMaxWriteBufferSize` | 100000 | Increase for bursty write workloads | +| `ingestQueryInstances` | 2 | Only for Enterprise. Increase for overall throughput | +| `queryOnlyInstances` | 0 | Only for Enterprise. Increase for higher query throughput | + +## Line Protocol + +For information on the line protocol data format, see [line-protocol.md](../line-protocol.md). diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/migrations.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/migrations.md new file mode 100644 index 0000000000..be98c1fdad --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/migrations.md @@ -0,0 +1,173 @@ +# Migrating to Timestream for InfluxDB 3 + +How to migrate **into** Timestream for InfluxDB v3 from common sources. Pick the section that +matches your source system. For the differences +between the engines, see [`influxdb-2-vs-3.md`](../influxdb-2-vs-3.md). + +> **Migrating to V2 instead?** See [`influxdb2/migrations.md`](../influxdb2/migrations.md). + +## What carries over + +V3 is a different engine from V2/OSS/v1: storage is **Apache Parquet on S3**, the data model +is **Database → Table** (no orgs/buckets), and the query language is **SQL/InfluxQL — there is +no Flux**. Consequences for every migration path: + +- **Time-series data** moves via **line protocol**. V3 exposes **v1- and v2-compatible write + endpoints** (`/api/v2/write` on port **8181**), so any tool that writes line protocol to V2 + can target V3 by re-pointing the URL. +- **Metadata does *not* port.** Dashboards, **Flux** tasks, and v2 key-value data have no V3 + equivalent. Rewrite Flux queries/tasks into **SQL or InfluxQL**, and replace Flux tasks with + the V3 **processing engine** (embedded Python plugins) for downsampling/alerting. +- **Namespace mapping:** V2 `org`/`bucket` (and v1 database) → V3 **database**; each source + measurement → a V3 **table** (created automatically on first write). + +| Source | Tooling | Mechanism | +|--------|---------|-----------| +| **Timestream for LiveAnalytics** | [`tools/python/liveanalytics_influxdb3_migration_plugin`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_influxdb3_migration_plugin) (`influxdb_version: v3`) | Unload to S3 → ingest using processing engine → validate | +| **Timestream for InfluxDB v2** | [`tools/python/influxdb_v2_to_v3_migration`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influxdb_v2_to_v3_migration) | `influx backup` → translate to line protocol → write to V3 HTTP API | +| **InfluxDB v1** | No direct tool | Two-hop: `influxd upgrade` → `export-lp` → write line protocol to V3 | +| **InfluxDB OSS 2.x** | Export line protocol | Write line protocol to V3's v2-compatible endpoint | + +--- + +## From Timestream for LiveAnalytics + +Use the same [**LiveAnalytics migration tooling**](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts) +as the V2 path — it targets **both V2 and V3**. The four stages are identical (unload to S3 in +Parquet → Athena transform to line protocol → ingest → validate), orchestrated by +`main.py --config ` in either `batch` or `live_replication` mode. + +To target **V3**, make two changes versus the V2 setup: + +1. Set `influxdb_version: v3` in the config (remember *buckets* become *databases*). +2. Configure the V3 endpoint via environment variables — port **8181**, and **omit + `INFLUXDB_V2_ORG`** (V3 has no organizations); ingestion uses V3's V2-compatible write API: + ```shell + export INFLUXDB_V2_URL="https://:8181" + export INFLUXDB_V2_TOKEN="" + ``` + +Cardinality is far less of a concern on V3 than on V2 (Parquet/columnar storage, no TSI), so +the strict ~10M-series ceiling does not apply — though a sensible tag/field design still helps +query performance. + +> **Large datasets (≥1 TB):** line-protocol ingestion can be slow at terabyte scale. A +> Parquet-based V3 ingestion path is in development; if you are moving terabytes, check the +> repo for its availability before committing to the line-protocol route. + +For the end-to-end live cutover (continuous replication → gradual application cutover → +cleanup), follow the +[Live Migration Guide](https://github.com/awslabs/amazon-timestream-tools/blob/mainline/tools/python/liveanalytics_migration_scripts/targets/timestream_for_influxdb/live_migration_guide.md). + +--- + +## From Timestream for InfluxDB v2 + +Use the [**Amazon Timestream for InfluxDB v2 to v3 Migration Script**](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influxdb_v2_to_v3_migration) +(`influxdb_v2_to_v3_migration.py`). It is the purpose-built, turnkey path between the two +managed engines and runs the whole migration end to end: + +1. **Backs up** the selected V2 buckets to local storage with `influx backup` (the InfluxDB v2 CLI). +2. **Translates** the backup to line protocol with the InfluxDB v2 daemon (`influxd inspect export-lp`). +3. **Writes** the line protocol to V3 via the [InfluxDB v3 HTTP API](https://docs.influxdata.com/influxdb3/enterprise/api/v3/). + +Buckets map to **databases** and measurements map to **tables** during the migration. + +The script is available **standalone**, or as an **automated solution** that provisions an EC2 +instance with the script and all prerequisites pre-installed — see the +[`automated_deployment/` README](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influxdb_v2_to_v3_migration/automated_deployment). + +### Prerequisites +- **Python 3.13+** (3.12 will not work). +- The **InfluxDB v2 CLI** installed, and the **InfluxDB v2 daemon (`influxd`) on your `PATH`**. + The daemon does **not** need to be running — it is used in isolation to extract line protocol + (extraction cannot be done over a network, which is why the daemon is required locally). +- Network connectivity to both instances — V2 on **8086** (`influx ping --host`), V3 on **8181** + (`curl /health -H "Authorization: Bearer "`). +- Enough local disk to hold all migrated data **uncompressed** (backed up under `~/engine`). +- A **V2 operator token** and a **V3 token** (the V3 token lives in the AWS Secrets Manager + secret associated with the instance). Store both in a single Secrets Manager secret: + ```shell + aws secretsmanager create-secret \ + --region us-west-2 \ + --name influxdb_v2_to_v3_migration \ + --secret-string '{"INFLUXDB_V2_TOKEN": "", "INFLUXDB_V3_TOKEN": ""}' + ``` + +### Run the migration +Provide the source/destination URLs, the buckets (or whole orgs) to migrate, and the secret name: +```shell +python3.13 influxdb_v2_to_v3_migration.py \ + --source-url "https://:8086" \ + --destination-url "https://:8181" \ + --source-buckets-and-orgs "bucket-one:org-one,bucket-two:org-two" \ + --tokens-secret-name "influxdb_v2_to_v3_migration" +``` +Use `--source-orgs` instead of `--source-buckets-and-orgs` to migrate **all** buckets from the +named organizations. When finished, remove the local backup (`rm -rf ~/engine`). + +> **Already have a backup?** A companion `influxdb_v3_ingestion.py` script ingests pre-backed-up +> V2 data (organized under `engine/data//.lp`) straight into V3 — useful if +> you have already run `influx backup` + `export-lp` yourself. + +> **Cutoff behavior:** only points written **before the migration begins** are migrated; points +> written during or after are not (a consequence of how the V2 CLI backup works). For a live +> cutover, dual-write or replay the gap after the bulk migration completes. + +### Verify +Buckets become databases and measurements become tables, so compare counts per table. On V2, +count a measurement with Flux (`from(bucket:"...") |> range(start: 0) |> filter(...) |> group() |> count()`) +or `wc -l ~/engine/data//.lp`; on V3, run `SELECT COUNT(*) FROM ` for +each table in the database. + +> The same script can also migrate **V2 → V2** (e.g. into a read-replica cluster, which does not +> support `backup`/`restore`) by passing `--destination-org `. + +--- + +## From InfluxDB v1 + +There is no direct v1 → V3 tool. Take a **two-hop** approach: + +1. **Upgrade and export v1 to line protocol** exactly as in the + [v1 → v2 path](../influxdb2/migrations.md#from-influxdb-v1): `influxd upgrade` to the 2.x + on-disk format, then `influxd inspect export-lp` per bucket (compressed). +2. **Write the line protocol to V3** through its **v2-compatible write endpoint** (port 8181, + `Authorization: Bearer `, no org). You can re-point the Timestream + [`influxdb_ingestion.py`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts/targets/timestream_for_influxdb/ingestion) + script at the V3 endpoint (set `INFLUXDB_V2_URL` to the `:8181` URL, omit + `INFLUXDB_V2_ORG`), since V3 accepts the V2 write API. + +Alternatively, migrate **v1 → managed V2 first** (see [`influxdb2/migrations.md`](../influxdb2/migrations.md)) +and then run the **[V2 → V3 migration script](#from-timestream-for-influxdb-v2)** above — useful +if you also want a V2 instance, or want to validate on V2 before going to V3. + +--- + +## From InfluxDB OSS 2.x + +The AWS [`influx-migration`](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influx-migration) +script uses **backup/restore**, which is **V2-only** — it cannot target V3. For V3: + +1. **Export** OSS 2.x data to line protocol locally (`influxd inspect export-lp`, since you + control the host), or query it out as a DataFrame with the v2 client. +2. **Write** to V3's v2-compatible endpoint (`/api/v2/write` on 8181), or use the InfluxDB 3 + Python client's `write_file()` / `write_dataframe()`. + +Note again that **dashboards and Flux tasks do not carry over** — rebuild them as SQL/InfluxQL +and processing-engine plugins on V3. + +--- + +## Validate (any source) +After loading, compare per-table row counts and cardinality between source and target, and +spot-check a few time ranges (`SELECT count(*)`, min/max timestamps, sample rows) before +decommissioning the source. Note that `SELECT count(*)` is an expensive query and if the data range +hasn't reached compaction then a timeout may occur. + +## AWS migration tools and documentation +- **Timestream for InfluxDB v2 → v3** — [Amazon Timestream for InfluxDB v2 to v3 Migration Script](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influxdb_v2_to_v3_migration) (standalone or [automated EC2 deployment](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/influxdb_v2_to_v3_migration/automated_deployment)). +- **Timestream for LiveAnalytics → Timestream for InfluxDB 3** — [LiveAnalytics migration tooling](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts) (set `influxdb_version: v3`) and its [Timestream for InfluxDB target guide](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/tools/python/liveanalytics_migration_scripts/targets/timestream_for_influxdb). +- **InfluxDB v1 → v2 (first hop)** — [InfluxDB v1 to v2 migration guide](https://github.com/awslabs/amazon-timestream-tools/tree/mainline/guides/influxdb_v1_to_v2_migration). +- **Other migration tools and guides** — see the [awslabs/amazon-timestream-tools](https://github.com/awslabs/amazon-timestream-tools) repository. +- **Service documentation** — [Amazon Timestream for InfluxDB 3 developer guide](https://docs.aws.amazon.com/timestream/latest/developerguide/influxdb3.html), [Features and workflows with Amazon Timestream for InfluxDB 3](https://aws.amazon.com/blogs/database/features-and-workflows-with-amazon-timestream-for-influxdb-3/), and the [InfluxDB v3 compatibility (v1/v2) write APIs](https://docs.influxdata.com/influxdb3/enterprise/write-data/compatibility-apis/). diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/onboarding.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/onboarding.md similarity index 91% rename from src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/onboarding.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/onboarding.md index 4aae393ae9..b2c2a138bb 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/onboarding.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/onboarding.md @@ -34,7 +34,7 @@ Examples: ## Provisioning Checklist - [ ] Determine engine: Core (single-node) or Enterprise (multi-node) -- [ ] Create parameter group (or use literal `InfluxDBV3Core` / `InfluxDBV3Enterprise`) +- [ ] Choose a parameter group: pass a **service-owned** group to `--db-parameter-group-identifier` — `InfluxDBV3Core` (single-node) or `InfluxDBV3Enterprise` (multi-node) — or create a custom group and pass its returned **id**. **Mind the casing:** service-owned group **identifiers** use an uppercase `V` (`InfluxDBV3Core`), while the `--parameters` **option keys** use a lowercase `v` (`InfluxDBv3Core`). - [ ] Create cluster with `create-db-cluster` — **you MUST NOT pass** `--username`, `--password`, `--organization`, `--bucket`, or `--deployment-type` — these switch the cluster into a non-v3 initialization mode - [ ] Poll until `status` is `AVAILABLE` - [ ] Retrieve token from Secrets Manager: `READONLY-InfluxDB-auth-parameters-` @@ -43,7 +43,9 @@ Examples: → Next step: retrieve the token and make a data-plane call (see Authentication below). -**V3 Core (default parameter group):** +**V3 Core (service-owned default parameter group):** + +Use the service-owned `InfluxDBV3Core` parameter group (uppercase `V`). `--db-parameter-group-identifier` is optional — omitting it yields the same Core defaults. ```bash aws timestream-influxdb create-db-cluster \ --name my-v3-cluster \ @@ -181,7 +183,7 @@ When a user asks how to write data — read `development-guide.md`. Key facts: When a user asks about databases, tables, retention, or cardinality — read `schema-design.md`. Key facts: - Design databases by retention period - Measurements become tables automatically on first write -- Virtually unlimited cardinality; watch `maxTables` (default 500) and `maxColumnsPerTable` (default 200) +- Virtually unlimited cardinality; watch the **10,000-table** limit (across all databases) and the **500-column-per-table** limit ([docs](https://docs.influxdata.com/influxdb3/enterprise/admin/databases/#database-table-and-column-limits)) ### Querying diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/parameters.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/parameters.md similarity index 95% rename from src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/parameters.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/parameters.md index 4c95e0939e..40f922805b 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/parameters.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/parameters.md @@ -1,5 +1,10 @@ # InfluxDB v3 Parameter Reference +> **Casing matters.** The `--parameters` JSON **option keys** use a lowercase `v` — +> `InfluxDBv3Core` / `InfluxDBv3Enterprise`. This is distinct from the two **service-owned +> parameter groups**, `InfluxDBV3Core` / `InfluxDBV3Enterprise` (uppercase `V`), which are +> passed to `--db-parameter-group-identifier`. Both are case-sensitive; don't interchange them. + ## Table of Contents - [InfluxDBv3Core Parameters](#influxdbv3core-parameters) - [InfluxDBv3Enterprise Additional Parameters](#influxdbv3enterprise-additional-parameters) diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/query-guide.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/query-guide.md new file mode 100644 index 0000000000..a27aa6bc28 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/query-guide.md @@ -0,0 +1,354 @@ +# InfluxDB 3 Query Guide + +V3 supports two query languages: +- **SQL** — primary language +- **InfluxQL** — SQL-like, limited vs SQL + +## SQL Query Endpoint + +``` +GET /api/v3/query_sql?db=DATABASE_NAME&q=SELECT+*+FROM+TABLE_NAME+LIMIT+10 +Authorization: Bearer +Accept: application/json +``` + +**Example** +```shell +curl --request GET \ + "https://localhost:8181/api/v3/query_sql?db=DATABASE_NAME&q=Q" \ + --header "Authorization: Bearer INFLUX_TOKEN" +``` + +or + +``` +POST /api/v3/query_sql +Authorization: Bearer +Content-Type: application/json +Accept: application/json +``` + +JSON body, for POST request: +```json +{ + "db": "", + "q": "", + "format": "json", + "params": "" +} +``` + +**Example** +```shell +curl --request POST \ + "https://localhost:8181/api/v3/query_sql" \ + --header "Authorization: Bearer INFLUX_TOKEN" \ + --header "Content-Type: application/json" \ + --data-raw '{"db":"DATABASE_NAME","format":"json","params":{},"q":"SELECT * FROM mytable"}' +``` + +**NOTE**: Setting `"format"` to `"jsonl"` is preferred because it streams data back to the client. Keep in mind that this means query response bodies must be read, otherwise queries will be considered cancelled by InfluxDB. + +### SQL Patterns + +**Syntax** +```sql +[ WITH with_query [, …] ] +SELECT [ ALL | DISTINCT ] select_expr [, …] +[ FROM from_item [, …] ] +[ JOIN join_item [, …] ] +[ WHERE condition ] +[ GROUP BY grouping_element [, …] ] +[ HAVING condition] +[ UNION [ ALL ] ] +[ ORDER BY expression [ ASC | DESC ][, …] ] +[ LIMIT count ] +``` + +**Keywords** +```sql +AND +ALL +ANALYZE +AS +ASC +AT TIME ZONE +BETWEEN +BOTTOM +CASE +DESC +DISTINCT +EXISTS +EXPLAIN +FROM +GROUP BY +HAVING +IN +INNER JOIN +JOIN +LEFT JOIN +LIKE +LIMIT +NOT +OR +ORDER BY +FULL OUTER JOIN +RIGHT JOIN +SELECT +TOP +TYPE +UNION +WHERE +WITH +``` + +**Numeric Literals** +```sql +-- Integers +10 ++10 +-10 + +-- Unsigned integers +10::BIGINT UNSIGNED ++10::BIGINT UNSIGNED + +-- Floats +10.78654 +-100.56 +``` + +**Date and Time Literals** +```sql +'2022-01-31T06:30:30.123Z' -- (RFC3339) +'2022-01-31T06:30:30.123' -- (RFC3339-like) +'2022-01-31 06:30:30.123' -- (RFC3339-like) +'2022-01-31 06:30:30' -- (RFC3339-like, no fractional seconds) +to_timestamp_nanos(1643610630123000000) -- (Unix epoch nanoseconds to a timestamp) +to_timestamp(1643610630) -- (Unix epoch seconds to a timestamp) +``` + +**General examples** +```sql +-- Double-quote identifiers that contain whitespace +SELECT "water temperature", "buoy location" FROM buoy + +-- Double-quote measurement names with special characters +SELECT * FROM "h2o-temperature" + +-- Double-quote identifiers that should be treated as case-sensitive +SELECT "pH" FROM "Water" + +-- Selecting with a WHERE clause +SELECT + * +FROM + "h2o_feet" +WHERE + "location" = 'santa_monica' + AND "level description" = 'below 3 feet' +``` + +**Show schema information** +```sql +SHOW tables +SHOW columns FROM +``` + +**Joins** +InfluxDB v3 SQL supports joins across tables (measurements) within a database. + +The following joins are supported: +```sql +-- Inner join +SELECT + * +FROM + home +INNER JOIN home_actions ON + home.room = home_actions.room + AND home.time = home_actions.time; + +-- Left outer join +SELECT + * +FROM + home +LEFT OUTER JOIN home_actions ON + home.room = home_actions.room + AND home.time = home_actions.time; + +-- Right outer join +SELECT + * +FROM + home +RIGHT OUTER JOIN home_actions ON + home.room = home_actions.room + AND home.time = home_actions.time; + +-- Full outer join +SELECT + * +FROM + home +FULL OUTER JOIN home_actions ON + home.room = home_actions.room + AND home.time = home_actions.time; +``` + +**Selector functions** +Selector functions are unique to InfluxDB. Selector functions behave like aggregate functions except they return a time value in addition to the computed value. + +Selector functions: +- `SELECTOR_FIRST()`: Returns the first value of a selected column and timestamp. +- `SELECTOR_LAST()`: Returns the last value of a selected column and timestamp. +- `SELECTOR_MIN()`: Returns the smallest value of a selected column and timestamp. +- `SELECTOR_MAX()`: Returns the largest value of a selected column and timestamp. + +Examples: +```sql +SELECT +SELECTOR_MAX("pH", time)['value'], +SELECTOR_MAX("pH", time)['time'] +FROM "h2o_pH" + +SELECT +SELECTOR_LAST("water_level", time)['value'], +SELECTOR_LAST("water_level", time)['time'] +FROM "h2o_feet" +WHERE time >= timestamp '2019-09-10T00:00:00Z' AND time <= timestamp '2019-09-19T00:00:00Z' +``` + +## InfluxQL Query Endpoint + +InfluxQL can be executed using either the `/api/v3/query_influxql` or the `/query` V1 backwards compatible endpoint. + +``` +GET /api/v3/query_influxql?db=DATABASE_NAME&q=SELECT+*+FROM+TABLE_NAME+LIMIT+10 +Authorization: Bearer +Accept: application/json +``` + +The `query_influxql` endpoint supports the following query parameters: +- `db`: The name of the database. If you provide a query that specifies the database, you can omit the ‘db’ parameter from your request. +- `q`: Required. The query to execute. +- `format`: The format of the response. Valid options are: `json`, `jsonl`, `csv`, `pretty`, or `parquet`. `jsonl` is preferred because it streams results back to the client. `pretty` is for human-readable output. The default is `json`. +- `params`: JSON-encoded query parameters for parameterized queries. + +**Example** +```shell +curl --request GET \ + "https://localhost:8181/api/v3/query_influxql?q=Q" \ + --header "Authorization: Bearer INFLUX_TOKEN" +``` + +or + +``` +POST /api/v3/query_influxql +Authorization: Bearer +Content-Type: application/json +Accept: application/json +``` + +**Example** +```shell +curl --request POST \ + "https://localhost:8181/api/v3/query_influxql" \ + --header "Authorization: Bearer INFLUX_TOKEN" \ + --header "Content-Type: application/json" \ + --data-raw '{"db":"DATABASE_NAME","format":"json","params":{},"q":"SELECT * FROM mytable"}' +``` + +or + +``` +GET /query?db=DATABASE_NAME&q=QUERY&chunk_size=10000&chunked=true&u=USERNAME&p=PASSWORD +Authorization: Bearer +Accept: text/csv +``` + +The `query` endpoint supports the following query parameters: +- `chunked`: Returns points in streamed batches. When set to `true`, InfluxDB chunks responses by series or by every 10,000 points, whichever occurs first. +- `chunk_size`: Specifies the number of points to include in a chunk, if `chunked` is `true`. +- `db`: Required. Database name. +- `epoch`: Timestamp precision. Valid values are: `h`, `m`, `s`, `ms`, `us`, and `ns`. +- `u`: For query string authentication, the user's username. +- `p`: For query string authentication, the user's password. InfluxDB v3 enterprise expects this to be a token with read access to the database and will ignore `u`. +- `q`: The InfluxQL query to execute. + +**Example** +```shell +curl --request GET \ + "https://localhost:8181/query?q=Q" \ + --header "Authorization: Bearer INFLUX_TOKEN" +``` + +or + +``` +POST /query +Authorization: Bearer +Content-Type: application/json +Accept: text/csv +``` + +JSON body, for POST request: +```json +{ + "db": "", + "q": "", + "chunk_size": 10000, + "chunked": true, + "epoch": "ns" +} +``` + +**Example** +```shell +curl --request POST \ + "https://localhost:8181/query" \ + --header "Authorization: Bearer INFLUX_TOKEN" \ + --header "Content-Type: application/json" \ + --data-raw '{ + "chunk_size": 10000, + "chunked": false, + "db": "DATABASE_NAME", + "epoch": "ns", + "pretty": false, + "q": "Q" +}' +``` + +### InfluxQL Patterns + +**Basic select:** +```sql +SELECT * FROM cpu WHERE time > now() - 1h +``` + +**Aggregation with time grouping:** +```sql +SELECT mean(usage_idle), max(usage_system) +FROM cpu +WHERE time > now() - 24h +GROUP BY time(5m), host +``` + +**Filter by tag:** +```sql +SELECT mean(usage_idle) FROM cpu +WHERE host = 'server01' AND time > now() - 1h +GROUP BY time(5m) +``` + +**Show measurements and tag keys:** +```sql +SHOW MEASUREMENTS +SHOW TAG KEYS FROM cpu +SHOW TAG VALUES FROM cpu WITH KEY = "host" +SHOW FIELD KEYS FROM cpu +``` + +**InfluxQL Limitations vs SQL:** InfluxQL cannot join across measurements, has no `pivot`, no custom functions, and limited math operations. Use SQL for anything beyond basic aggregations. diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/schema-design.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/schema-design.md similarity index 86% rename from src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/schema-design.md rename to src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/schema-design.md index 8a7d17465a..27b22562f1 100644 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/schema-design.md +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/schema-design.md @@ -1,4 +1,4 @@ -# InfluxDB Schema Design & Data Modeling +# InfluxDB v3 Schema Design & Data Modeling ## V2 vs V3 Data Model @@ -17,7 +17,8 @@ Key differences: - V3 has **no organizations** — databases are the top-level container. - V3 measurements become **tables** automatically on first write. Tables have explicit column schemas. - V3 has **no practical cardinality limit** — the Parquet/S3 storage engine handles high-cardinality workloads that would degrade V2's TSM engine. -- V3 databases have `maxTables` (default 500) and `maxColumnsPerTable` (default 200) limits that can be tuned. +- V3 Enterprise limits the total number of tables to **10,000 across all databases**. +- V3 Core limits the total number of tables to **2,000 across all databases**. ## Measurement Naming @@ -34,7 +35,7 @@ When to use one measurement vs multiple: - Use **one measurement** when data shares the same tags and fields (e.g., all CPU metrics in `cpu` with fields `usage_idle`, `usage_system`, `usage_user`). - Use **separate measurements** when data has different tag/field schemas (e.g., `cpu` and `disk` have different fields and tags). - Do **not** create a measurement per entity (e.g., `cpu_server01`, `cpu_server02`) — use a `host` tag instead. -- Be aware of the `maxTables` limit per database (default 500). Each unique measurement name creates a table. +- Be aware of table limits (each unique measurement name creates a table). ## Tag vs Field Decision @@ -54,7 +55,7 @@ Rules: - Avoid duplicate names for a tag key and field key within the same measurement — query results become unpredictable. - Sort tags alphabetically in line protocol for best write compression (both engines). -InfluxDB v3 supports virtually unlimited cardinality. High-cardinality tags are safe, but each unique tag key adds a column — respect the `maxColumnsPerTable` limit (default 200). +InfluxDB v3 supports virtually unlimited cardinality. High-cardinality tags are safe, but each unique tag key adds a column — respect the **500-columns-per-table limit** (1 timestamp + up to 499 tag/field columns). ## Database Design @@ -65,8 +66,8 @@ Design principle: same as V2 — **one database per retention period**. Guidance: - Create databases via `POST /api/v3/configure/database`. - Retention is set via `retentionPeriod` (e.g. `"7d"`, `"90d"`, `"1y"`). Null or omitted = infinite. -- Each database has a `maxTables` limit (default 500) — plan measurement names accordingly. -- Each table has a `maxColumnsPerTable` limit (default 200) — this covers both tag keys and field keys. +- Tables are limited to **10,000 across all databases** (default) — plan measurement names accordingly. +- Each table is limited to **500 columns** (1 timestamp + up to 499 tag/field keys). ### Database creation example @@ -75,10 +76,8 @@ curl -X POST "https://:8181/api/v3/configure/database" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ - "name": "downsampled-90d", - "retentionPeriod": "90d", - "maxTables": 500, - "maxColumnsPerTable": 200 + "db": "downsampled-90d", + "retention_period": "90d" }' ``` @@ -98,7 +97,7 @@ Field types are **locked on first write** per measurement per field key on both | `count=10i` (integer) | `count=10.0` (float) | Error — `count` is locked as integer | | `status="ok"` (string) | `status=true` (boolean) | Error — `status` is locked as string | -InfluxDB v3 returns **400** or **422** depending on `accept_partial` param. Same partial-write behavior. +InfluxDB v3 returns **400** if points have been rejected. Depending of the value of the `accept_partial` query parameter, this may mean some points have been written. A **422** status code will be returned if writing the line protocol points would exceed the number of allowed databases, tables, columns, tags, or fields. Prevention: - Document your schema before writing. Agree on types per field across all writers. @@ -121,8 +120,9 @@ A **series** is a unique combination of measurement name + tag set. Series cardi Example: `cpu,host=A,region=us` and `cpu,host=B,region=us` are 2 series. InfluxDB V3 Parquet/S3 storage engine has **virtually unlimited cardinality**. High-cardinality tags that would cripple V2 are handled efficiently. The main limits to watch are: -- `maxTables` per database (default 500, tunable) -- `maxColumnsPerTable` (default 200, tunable) — each tag key and field key is a column +- For Enterprise: 10,000 tables across all databases +- For Core: 2,000 tables across all databases +- **500 columns per table** (1 timestamp + up to 499 tag/field columns) — each tag key and field key is a column If a V2 user is hitting cardinality limits, migrating to V3 is the recommended long-term solution. @@ -153,7 +153,7 @@ disk,host=web01,region=us-east-1,device=sda1 used_percent=45.0,free=107374182400 - Separate measurements for `cpu`, `memory`, `disk` (different field schemas) - Tags: `host`, `region`, `device` (bounded) - Fields: numeric metrics -- V3: creates 3 tables — count toward `maxTables` limit +- V3: creates 3 tables — count toward table limit (across all databases) ### Application metrics diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/troubleshooting.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/troubleshooting.md new file mode 100644 index 0000000000..f5b9d8f6ab --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/influxdb3/troubleshooting.md @@ -0,0 +1,114 @@ +# Troubleshooting in InfluxDB 3 + +Version-specific errors for **Timestream for InfluxDB 3** and how to resolve them. For +cross-cutting connection/authorization problems (port, token prefix, networking, +namespace), start with the parent [troubleshooting.md](../troubleshooting.md). For +terminology, see [glossary.md](../glossary.md). + +> V3 quick facts: default port **8181**, auth header `Authorization: Bearer `, +> write/query namespace is `db` (database), SQL is primary (**no Flux**). See +> [influxdb-2-vs-3.md](../influxdb-2-vs-3.md). + +## Cluster provisioning + +V3 is **cluster-only** (`create-db-cluster`; do not pass `--deployment-type`). Topology +is set by the parameter group (`InfluxDBV3Core` = single node, `InfluxDBV3Enterprise` = +multi-node). + +- **Cluster stuck in `CREATING` or `FAILED`** — run `get-db-cluster` to read `status`. + If `FAILED`: verify subnets and security groups are valid and in the same VPC, the + security group allows inbound on **8181**, then delete and recreate with corrected + parameters. Check CloudTrail for the `CreateDbCluster` event error details. +- **Passing the cluster name instead of the cluster ID** — `--db-cluster-id` expects the + service-generated `dbClusterId` (from the create response or `list-db-clusters`), not + the name you chose. Using the name returns `ResourceNotFoundException`. +- **Parameter group name rejected** — `--name` for `create-db-parameter-group` accepts + only `[a-zA-Z0-9]+` (no hyphens/underscores). Reference the group by its **ID** in + `--db-parameter-group-identifier`. + +## Writing / ingestion + +The native v3 write API (`/api/v3/write_lp`) returns these status codes (see the +[InfluxDB 3 write troubleshooting docs](https://docs.influxdata.com/influxdb3/core/write-data/troubleshoot/)): + +- **`400` Bad request** — some or all of the data was rejected. With partial writes + enabled (`accept_partial`, the default), valid points are still ingested and the + response body lists the rejected lines (up to 100). **Field type conflicts** surface + here: field types are inferred and **locked on first write**, so points whose field + types don't match existing data are rejected. Validate syntax (see + [line-protocol.md](../line-protocol.md)). +- **`401` Unauthorized** — missing/malformed `Authorization` header or a token without + write permission. +- **`404` Not found** — database not found; verify the `db` name. +- **`422` Unprocessable entity** - Writing the line protocol points would lead to the maximum number of databases, tables, columns, tags, or fields being exceeded. +- **`503` Service unavailable** — the server is temporarily rejecting writes; retry per + the `Retry-After` header. +- **Replica lag / slow writes** — caused by tiny writes from many concurrent + writers. Batch **5,000+ points per request** (see [`gotchas.md`](./gotchas.md)). + +## Querying (SQL / InfluxQL) + +Patterns and endpoints: [`query-guide.md`](./query-guide.md). + +- **`SELECT ... GROUP BY time(...)` fails** — that's InfluxQL syntax. In SQL use + `date_bin($__interval, time)` with `GROUP BY` on the bucketed column. +- **Flux query rejected** — V3 has **no Flux**. Rewrite as SQL or InfluxQL; there is no + automatic conversion (see [migrations.md](./migrations.md)). +- **InfluxQL function returns an error** — some InfluxQL functions aren't implemented in + V3, including the `SAMPLE()` selector and **all** technical/predictive-analysis + functions (e.g. `HOLT_WINTERS()`). The `SLIMIT`/`SOFFSET` clauses and cardinality + metaqueries are also unsupported. Test before relying on them; prefer SQL. See + [InfluxQL feature support](https://docs.influxdata.com/influxdb3/core/reference/influxql/feature-support/). +- **Queries time out or OOM under load** — increase the query memory pool + (`execMemPoolBytes`) and the number of DataFusion query threads (`dataFusionNumThreads`) + via the parameter group (see [`parameters.md`](./parameters.md)). + +## Schema, cardinality & tables + +See [`schema-design.md`](./schema-design.md) and [`gotchas.md`](./gotchas.md). + +- **Hitting the ~10,000-tables limit (across all databases)** — this is the default cap + (`--num-table-limit`, not exposed in the Timestream parameter group). Revisit your + table/measurement split first; raising it has compaction/query-performance implications. +- **Unexpected field-type errors on write** — see `400` above; the first write fixes the + type. + +## Parameters & tuning + +See [`parameters.md`](./parameters.md) and [`gotchas.md`](./gotchas.md). + +- **Parameter change didn't take effect** — parameter groups are **immutable**. Create a + new group and reassign it with `update-db-cluster --db-parameter-group-identifier + NEW_ID` (this reboots). There is no `UpdateDbParameterGroup` SDK operation. +- **Need more query parallelism / throughput** — there is **no IO-threads parameter** in + the v3 parameter group. Tune `dataFusionNumThreads` (DataFusion query threads) and the + related `dataFusionRuntime*` settings, and `execMemPoolBytes` for memory. + +## Processing engine, compaction & integrations + +- **OOM (`HashAggSpill` / Memory Exhausted) on Enterprise** — scheduled processing-engine + triggers run on **every** node. Target nodes with `node_spec` or scale up + (see [`gotchas.md`](./gotchas.md)). +- **Compactor node OOM / growing file counts** — monitor `system.parquet_files`; consider + a dedicated compactor node (Enterprise). +- **No QuickSight connector** — use **Grafana with Flight SQL / the InfluxDB SQL data + source** instead (see [`dashboard-guide.md`](./dashboard-guide.md)). + +## Health check + +`GET /health` is unauthenticated — use it to confirm the engine is reachable before +chasing auth or query issues: +```shell +curl https://:8181/health +``` + +## Core vs. Enterprise + +Timestream for InfluxDB Core has significant limitations compared to Enterprise. Specifically: + +| Core | Enterprise | +|-----------------------------------------|----------------------------------------------------------------------| +| Single node | Up to 15 total nodes | +| Default maximum query range of 72 hours | No limit on query range | +| Maximum of 5 databases | Maximum of 100 databases | +| Maximum of 2,000 total tables | Maximum of 10,000 total tables | diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/line-protocol.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/line-protocol.md new file mode 100644 index 0000000000..5ffe9b16f1 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/line-protocol.md @@ -0,0 +1,37 @@ +# Line Protocol + +InfluxDB, v2 and v3, uses line protocol to write data points. It is a text-based format that provides the measurement, tag set, field set, and timestamp of a data point. + +## Line Protocol Format + +``` +measurement,tag1=val1,tag2=val2 field1=1.0,field2="hello" timestamp +``` + +Rules: +- **Tags** — indexed key-value strings, used for filtering. Optional but recommended. +- **Fields** — non-indexed data values. At least one required per point. +- **Timestamp** — Unix epoch. Omit to use server time (not recommended for production). +- Sort tags alphabetically for best write compression. +- No spaces in tag keys, tag values, or field keys. Escape commas and spaces in values with `\`. + +Escaping rules: +| Location | Characters to escape | +|----------|---------------------| +| Measurement name | `,` and ` ` (space) | +| Tag keys & values | `,`, `=`, and ` ` (space) | +| Field keys | `,`, `=`, and ` ` (space) | +| Field string values | `"` (double-quote) — wrap value in `"` and escape internal quotes as `\"` | + +Example with escaping: `cpu,host=web\ server01,region=us\,east usage_idle=98.2` + +Field value types: +| Type | Syntax | Example | +|------|--------|---------| +| Float | bare number | `temp=72.3` | +| Integer | trailing `i` | `count=10i` | +| String | double-quoted | `status="ok"` | +| Boolean | `true`/`false` | `active=true` | + +Type is inferred on first write and locked — writing a different type to the same field causes an error and the conflicting point is dropped. + diff --git a/src/timestream-for-influxdb-mcp-server/kiro_power/steering/troubleshooting.md b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/troubleshooting.md new file mode 100644 index 0000000000..172e5cf8e4 --- /dev/null +++ b/src/timestream-for-influxdb-mcp-server/kiro_power/steering/troubleshooting.md @@ -0,0 +1,53 @@ +# Troubleshooting + +Common, cross-cutting issues when working with Amazon Timestream for InfluxDB. For +version- or topic-specific problems, use the cross-references below rather than +duplicating them here. Use the directory matching the user's engine — `influxdb3/` +or `influxdb2/`. + +## Where to look for specific issues + +| Symptom / area | See | +|---|---| +| Errors specific to InfluxDB 3 (clusters, SQL, processing engine) | [`influxdb3/troubleshooting.md`](./influxdb3/troubleshooting.md), [`influxdb3/gotchas.md`](./influxdb3/gotchas.md) | +| Errors specific to InfluxDB 2 (instances, Flux, read replicas) | [`influxdb2/troubleshooting.md`](./influxdb2/troubleshooting.md), [`influxdb2/gotchas.md`](./influxdb2/gotchas.md) | +| Query fails, returns no data, or is slow | [`influxdb3/query-guide.md`](./influxdb3/query-guide.md), [`influxdb2/query-guide.md`](./influxdb2/query-guide.md) (v2 has a **Query Performance** section) | +| Writes/ingest rejected, dropped points, type conflicts, replica lag | [`influxdb3/ingestion.md`](./influxdb3/ingestion.md), [`influxdb2/ingestion.md`](./influxdb2/ingestion.md) — see **Write Error Handling** (HTTP status codes) and **Batching** | +| Schema / cardinality problems (high memory, slow compaction, 10K-table limit) | [`influxdb3/schema-design.md`](./influxdb3/schema-design.md), [`influxdb2/schema-design.md`](./influxdb2/schema-design.md), plus the relevant `gotchas.md` | +| Parameter changes not taking effect, or tuning (V3) | [`influxdb3/parameters.md`](./influxdb3/parameters.md), plus `gotchas.md` | +| Token retrieval / which token to use | [influxdb-2-vs-3.md](./influxdb-2-vs-3.md) ("Retrieving tokens"), plus the relevant `onboarding.md` | +| Which version am I on / version-specific behavior | [influxdb-2-vs-3.md](./influxdb-2-vs-3.md) | +| Line protocol formatting errors | [line-protocol.md](./line-protocol.md) | + +## Bad connection / authorization + +The most common first-contact failures. Check, in order: + +1. **Wrong port.** By default, V2 listens on **8086**, V3 on **8181**. Hitting the wrong port looks + like a connection timeout or refused connection, not an auth error. +2. **Wrong, expired, or wrong-type token.** Retrieve or create the correct token — V2 via + the influx CLI (`influx auth create --operator`); V3 from the cluster's Secrets + Manager secret (`READONLY-InfluxDB-auth-parameters-`). Full steps: + [influxdb-2-vs-3.md](./influxdb-2-vs-3.md) → "Retrieving tokens". +3. **IAM does not gate the data plane.** The control plane uses SigV4/IAM, but + data-plane reads/writes are authorized only by the engine token — an IAM ReadOnly + policy will **not** block data-plane access (see the relevant `gotchas.md`). +4. **Network reachability.** Private instances/clusters require in-VPC access (or a + bastion/port-forward), and the security group must allow inbound on the configured + port. +5. **Namespace mismatch.** V2 requires the correct `org` + `bucket`; V3 requires the + correct `db`. A missing or incorrect namespace often surfaces as `404`/empty results + rather than an auth error. + +### Quick isolation: health check + +Both engines expose a `GET /health`. Use it to separate +"endpoint/network is down" from "auth, namespace, or payload is wrong": + +```shell +curl https://:/health +``` + +A healthy response (`OK`) means the engine is reachable, so the problem is most likely +auth (steps 2–3), networking to a private resource (step 5), or the namespace/payload — +not the endpoint itself. diff --git a/src/timestream-for-influxdb-mcp-server/power/POWER.md b/src/timestream-for-influxdb-mcp-server/power/POWER.md deleted file mode 100644 index d3e3c79632..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/POWER.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -name: "amazon-timestream-for-influxdb" -displayName: "Build a Time-Series Database with Amazon Timestream for InfluxDB" -description: "Query and analyze time-series data with Amazon Timestream for InfluxDB - built for IoT monitoring, DevOps metrics, real-time analytics, and high-cardinality data at scale. Deploy v2 and v3 instances, manage schemas, handle migrations and more." -keywords: ["influxdb","timestream","time-series","iot","observability","metrics","monitoring","devops","analytics","sql","telemetry"] -author: "AWS" ---- - -# Amazon Timestream for InfluxDB Power - -## Overview -The Amazon Timestream for InfluxDB Power provides access to Amazon Timestream for InfluxDB, a fully managed time-series database service compatible with InfluxDB 2 and InfluxDB 3. InfluxDB is purpose-built for high-ingest, high-cardinality workloads such as IoT telemetry, observability, metrics, and real-time analytics. - -This Power enables you to deploy and operate Timestream for InfluxDB instances and clusters, write and query time-series data using InfluxDB APIs and query languages, and manage schemas, retention, and migrations while leveraging AWS-managed scaling, durability, and security. - -- **Dual Version Support**: Deploy and manage both InfluxDB 2 and InfluxDB 3 instances and clusters - -- **Multi-Protocol Query Support**: Run Flux, InfluxQL, or SQL queries directly against your Timestream for InfluxDB instance - -- **Ingest Line Protocol**: Write high-throughput time-series data using InfluxDB's native line protocol - -- **Migration Support**: Handle migrations across InfluxDB versions - -- **Schema Management**: Create and manage buckets/databases, measurements/tables, and retention policies - - -## Available Steering Files - -- [x] `getting-started.md` - InfluxDB guidelines and operational Rules - - ALWAYS load before implementing schema changes or database operations - - MAY load when planning database application design - -- [ ] `influxdb-2-vs-3.md` - Highlights key differences between InfluxDB 2 and 3 - - ALWAYS load when there is ambiguity in InfluxDB versions - - ALWAYS load on migrations between InfluxDB 2 and 3 - -- [ ] `line-protocol.md` - Explains InfluxData's line protocol specification, with best practices and limitations - - SHOULD load when working with line protocol - -- [ ] `glossary.md` - Glossary of terms related to InfluxDB - - SHOULD load when there is vagueness in terminology or overlapping concepts - - -- [ ] `troubleshooting.md` - Common errors while working with InfluxDB and how to solve them - - SHOULD load on errors or when debugging - -- `influxdb3/` - - [ ] `developer-guide.md` - Contains guide for managing instances/clusters, queries/writes, example workflows - - [ ] `query-guide.md` - Query examples (SQL, InfluxQL) - - [ ] `troubleshooting.md` - Common errors in InfluxDB 3 and how to solve them - - [ ] `migrations.md` - - [ ] OSS (Core) -> Timestream for InfluxDB 3 - - [ ] Core -> Enterprise - - [ ] InfluxDB Cloud -> Timestream for InfluxDB 3 - - [ ] `dashboard-guide.md` - Generate grafana dashboards and visualizations - -- `influxdb2/` - - [ ] `developer-guide.md` - Contains guide for managing instances/clusters, queries/writes, example workflows - - [ ] `query-guide.md` - Query examples (Flux, InfluxQL) - - [ ] `troubleshooting.md` - Common errors in InfluxDB 2 and how to solve them - - [ ] `migrations` - - [ ] Timestream for InfluxDB (2) -> Timestream for InfluxDB 3 - - [ ] OSS -> Timestream for InfluxDB (2) - - [ ] `dashboard-guide.md` - Generate grafana dashboards and visualizations - - -## Available MCP Tools - -1. [Timestream for InfluxDB MCP Server](https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server#available-tools) - -The Timestream for InfluxDB MCP server provides tools for managing Amazon Timestream for InfluxDB clusters, instances, and parameter groups, along with InfluxDB-specific operations for writing data, querying with Flux, and managing buckets and organizations. - -2. [InfluxDB 3 MCP Server](https://github.com/influxdata/influxdb3_mcp_server?tab=readme-ov-file#available-tools) - -The InfluxDB 3 MCP Server provides tools for data operations (read/write), database lifecycle management (create/update/delete), schema inspection, and authentication token management across all InfluxDB versions (Core, Enterprise, Cloud). - - -## Configuration - -#### Timestream for InfluxDB MCP - -Prerequisites: -- A configured AWS profile -- (Optional) Provide instance (or cluster) details: - - `INFLUXDB_URL`: "https://your-influxdb-2-endpoint:8086" - - `INFLUXDB_TOKEN`: "your-influxdb-2-token" - - `INFLUXDB_ORG`: "your-influxdb-org" - -Note that this MCP server can be used to deploy both instances (V2) and clusters (V2, V3). - -#### InfluxDB 3 MCP - -Prerequisites: -- Set up the InfluxDB 3 MCP Server. [See here](https://github.com/influxdata/influxdb3_mcp_server?tab=readme-ov-file#2-integration-with-mcp-clients) for instructions. -- An existing InfluxDB 3 instance (or cluster). The following details are required: - - `INFLUX_DB_INSTANCE_URL`: "https://your-influxdb-3-endpoint:8181" - - `INFLUX_DB_TOKEN`: "your-influxdb-3-token" - - `INFLUX_DB_PRODUCT_TYPE`: "core" - -> See [mcp.json](./mcp.json) for an example configuration. - -## Additional Resources - -- [Timestream for InfluxDB 3](https://docs.aws.amazon.com/timestream/latest/developerguide/influxdb3.html) -- [Timestream for InfluxDB (InfluxDB 2)](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influxdb.html) -- [Code Samples Repository](https://github.com/awslabs/amazon-timestream-tools) - -#### MCP Servers -- [Amazon Timestream for InfluxDB MCP Server](https://github.com/awslabs/mcp/tree/main/src/timestream-for-influxdb-mcp-server) (Apache-2.0 license) -- [InfluxDB 3 MCP Server](https://github.com/influxdata/influxdb3_mcp_server/tree/main) (Apache-2.0 license) diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/glossary.md b/src/timestream-for-influxdb-mcp-server/power/steering/glossary.md deleted file mode 100644 index ca3c827692..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/glossary.md +++ /dev/null @@ -1 +0,0 @@ -# Glossary diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb-2-vs-3.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb-2-vs-3.md deleted file mode 100644 index 66f11829e4..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb-2-vs-3.md +++ /dev/null @@ -1 +0,0 @@ -# InfluxDB 2 vs 3 diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/dashboard-guide.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/dashboard-guide.md deleted file mode 100644 index 80001c064f..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/dashboard-guide.md +++ /dev/null @@ -1 +0,0 @@ -# Grafana Dashboards for InfluxDB 2 diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/migrations.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/migrations.md deleted file mode 100644 index 6c22a8a627..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/migrations.md +++ /dev/null @@ -1 +0,0 @@ -# Migrations to Timestream for InfluxDB diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/query-guide.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/query-guide.md deleted file mode 100644 index c6f0576d0d..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/query-guide.md +++ /dev/null @@ -1 +0,0 @@ -# InfluxDB 2 Query Guide diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/troubleshooting.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/troubleshooting.md deleted file mode 100644 index 9b07c48c26..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb2/troubleshooting.md +++ /dev/null @@ -1,4 +0,0 @@ -# Troubleshooting in InfluxDB 2 - -This file contains common additional errors encountered while working with InfluxDB 2 and guidelines for how to solve them. - diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/dashboard-guide.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/dashboard-guide.md deleted file mode 100644 index 0dfdf8a607..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/dashboard-guide.md +++ /dev/null @@ -1 +0,0 @@ -# Grafana Dashboards for InfluxDB 3 diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/development-guide.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/development-guide.md deleted file mode 100644 index a5e8cf3b0c..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/development-guide.md +++ /dev/null @@ -1,26 +0,0 @@ -# InfluxDB 3 Development Guide - -## Overview - -## Best Practices Guide - -- SHOULD never attempt FLUX queries - -## Tool Examples - -### Queries - -### Writes - -### Schema Operations - - -## Workflow Examples - -## Limitations - -- Core has significant restrictions compared to Enterprise; will need explicit guidelines in our steering docs - -## Troubleshooting - -See [troubleshooting.md](./troubleshooting.md). diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/migrations.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/migrations.md deleted file mode 100644 index d3e981737b..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/migrations.md +++ /dev/null @@ -1 +0,0 @@ -# Migrations to Timestream for InfluxDB 3 diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/query-guide.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/query-guide.md deleted file mode 100644 index 9b8e19bd67..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/query-guide.md +++ /dev/null @@ -1 +0,0 @@ -# InfluxDB 3 Query Guide diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/troubleshooting.md b/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/troubleshooting.md deleted file mode 100644 index d6ffcbe376..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/influxdb3/troubleshooting.md +++ /dev/null @@ -1,4 +0,0 @@ -# Troubleshooting in InfluxDB 3 - -This file contains common additional errors encountered while working with InfluxDB 3 and guidelines for how to solve them. - diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/line-protocol.md b/src/timestream-for-influxdb-mcp-server/power/steering/line-protocol.md deleted file mode 100644 index 4734bda6f8..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/line-protocol.md +++ /dev/null @@ -1 +0,0 @@ -# Line Protocol diff --git a/src/timestream-for-influxdb-mcp-server/power/steering/troubleshooting.md b/src/timestream-for-influxdb-mcp-server/power/steering/troubleshooting.md deleted file mode 100644 index 1663803a86..0000000000 --- a/src/timestream-for-influxdb-mcp-server/power/steering/troubleshooting.md +++ /dev/null @@ -1,10 +0,0 @@ -# Troubleshooting - -This file contains common errors encountered while working with InfluxDB and guidelines for how to solve them. - -For errors relating to a specific version, refer to the corresponding troubleshooting guide: - -- InfluxDB 3 Troubleshooting guide -- InfluxDB 2 Troubleshooting guide - -## Bad Connection/Authorization