Skip to content

Commit 02a1f7f

Browse files
atovpekobilly-the-fish
atovpeko
authored and
billy-the-fish
committed
Hypercore variable + cleanup (#4065)
1 parent c22fbbc commit 02a1f7f

22 files changed

+231
-248
lines changed

‎_partials/_hypercore-intro.md

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
$HYPERCORE_CAP is the $TIMESCALE_DB hybrid row-columnar storage engine, designed specifically for
2+
real-time analytics and powered by time-series data. The advantage of $HYPERCORE is its ability
3+
to seamlessly switch between row-oriented and column-oriented storage. This flexibility enables
4+
$CLOUD_LONG to deliver the best of both worlds, solving the key challenges in real-time analytics:
5+
6+
- High ingest throughput
7+
- Low-latency ingestion
8+
- Fast query performance
9+
- Efficient handling of data updates and late-arriving data
10+
- Streamlined data management
11+
12+
$HYPERCORE_CAP’s hybrid approach combines the benefits of row-oriented and column-oriented formats
13+
in each $SERVICE_LONG:
14+
15+
- **Fast ingest with $ROWSTORE**: new data is initially written to the $ROWSTORE, which is optimized for
16+
high-speed inserts and updates. This process ensures that real-time applications easily handle
17+
rapid streams of incoming data. Mutability—upserts, updates, and deletes happen seamlessly.
18+
19+
- **Efficient analytics with $COLUMNSTORE**: as the data **cools** and becomes more suited for
20+
analytics, it is automatically converted to the columnstore. This columnar format enables
21+
fast scanning and aggregation, optimizing performance for analytical workloads while also
22+
saving significant storage space.
23+
24+
- **Faster queries on compressed data in $COLUMNSTORE**: in the $COLUMNSTORE conversion, hypertable
25+
chunks are compressed by more than 90%, and organized for efficient, large-scale queries. Combined with [chunk skipping][chunk-skipping], this helps you save on storage costs and keeps your queries operating at lightning speed.
26+
27+
- **Full mutability with transactional semantics**: regardless of where data is stored,
28+
$HYPERCORE provides full ACID support. Like in a vanilla PostgreSQL database, inserts and updates
29+
to the $ROWSTORE and $COLUMNSTORE are always consistent, and available to queries as soon as they are
30+
completed.
31+
32+
![Hypercore workflow](https://assets.timescale.com/docs/images/hypercore-overview.png)
33+
34+
[chunk-skipping]: /use-timescale/:currentVersion:/hypertables/improve-query-performance/

‎_partials/_hypercore_manual_workflow.md

+8-8
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,37 @@
11
import EarlyAccess from "versionContent/_partials/_early_access_2_18_0.mdx";
22

3-
1. **Stop the jobs that are automatically adding chunks to the columnstore**
3+
1. **Stop the jobs that are automatically adding $CHUNKs to the $COLUMNSTORE**
44

5-
Retrieve the list of jobs from the [timescaledb_information.jobs][informational-views] view
6-
to find the job you need to [alter_job][alter_job].
5+
Retrieve the list of $JOBs from the [timescaledb_information.jobs][informational-views] view
6+
to find the $JOB you need to [alter_job][alter_job].
77

88
``` sql
99
SELECT alter_job(JOB_ID, scheduled => false);
1010
```
1111

12-
1. **Convert a chunk to update back to the rowstore**
12+
1. **Convert a $CHUNK to update back to the $ROWSTORE**
1313

1414
``` sql
1515
CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk');
1616
```
1717

18-
1. **Update the data in the chunk you added to the rowstore**
18+
1. **Update the data in the $CHUNK you added to the $ROWSTORE**
1919

2020
Best practice is to structure your [INSERT][insert] statement to include appropriate
21-
partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk:
21+
partition key values, such as the timestamp. TimescaleDB adds the data to the correct $CHUNK:
2222

2323
``` sql
2424
INSERT INTO metrics (time, value)
2525
VALUES ('2025-01-01T00:00:00', 42);
2626
```
2727

28-
1. **Convert the updated chunks back to the columnstore**
28+
1. **Convert the updated $CHUNKs back to the $COLUMNSTORE**
2929

3030
``` sql
3131
CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');
3232
```
3333

34-
1. **Restart the jobs that are automatically converting chunks to the columnstore**
34+
1. **Restart the $JOBs that are automatically converting $CHUNKs to the $COLUMNSTORE**
3535

3636
``` sql
3737
SELECT alter_job(JOB_ID, scheduled => true);

‎_partials/_hypercore_policy_workflow.md

+20-21
Original file line numberDiff line numberDiff line change
@@ -3,42 +3,41 @@
33

44
1. **Connect to your $SERVICE_LONG**
55

6-
In [$CONSOLE][services-portal] open an [SQL editor][in-console-editors]. You can also connect to your service using [psql][connect-using-psql].
6+
In [$CONSOLE][services-portal] open an [SQL editor][in-console-editors]. You can also connect to your $SERVICE_SHORT using [psql][connect-using-psql].
77

8-
1. **Enable columnstore on a hypertable**
8+
1. **Enable $COLUMNSTORE on a $HYPERTABLE**
99

10-
Create a [job][job] that automatically moves chunks in a hypertable to the columnstore at a specific time interval.
11-
By default, your table is `orderedby` the time column. For efficient queries on columnstore data, remember to
10+
By default, your table is `orderedby` the time column. For efficient queries on $COLUMNSTORE data, remember to
1211
`segmentby` the column you will use most often to filter your data:
1312

14-
* [Use `ALTER TABLE` for a hypertable][alter_table_hypercore]
13+
* [Use `ALTER TABLE` for a $HYPERTABLE][alter_table_hypercore]
1514
```sql
1615
ALTER TABLE crypto_ticks SET (
1716
timescaledb.enable_columnstore = true,
1817
timescaledb.segmentby = 'symbol');
1918
```
20-
* [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate]
19+
* [Use ALTER MATERIALIZED VIEW for a $CAGG][compression_continuous-aggregate]
2120
```sql
2221
ALTER MATERIALIZED VIEW assets_candlestick_daily set (
2322
timescaledb.enable_columnstore = true,
2423
timescaledb.segmentby = 'symbol' );
2524
```
26-
Before you say `huh`, a continuous aggregate is a specialized hypertable.
25+
Before you say `huh`, a $CAGG is a specialized $HYPERTABLE.
2726

28-
1. **Add a policy to convert chunks to the columnstore at a specific time interval**
27+
1. **Add a policy to convert $CHUNKs to the $COLUMNSTORE at a specific time interval**
2928

30-
For example, move yesterday's crypto trading data to the columnstore:
29+
Create a [$JOB][job] that automatically converts $CHUNKs in a $HYPERTABLE to the $COLUMNSTORE at a specific time interval. For example, convert yesterday's crypto trading data to the $COLUMNSTORE:
3130
``` sql
3231
CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d');
3332
```
3433
See [add_columnstore_policy][add_columnstore_policy].
3534
36-
1. **Check the columstore policy**
35+
1. **Check the $COLUMNSTORE policy**
3736
3837
1. View your data space saving:
3938
40-
When you convert data to the columnstore, as well as being optimized for analytics, it is compressed by more than
41-
90%. This saves on storage costs and keeps your queries operating at lightning speed. To see the amount of space
39+
When you convert data to the $COLUMNSTORE, as well as being optimized for analytics, it is compressed by more than
40+
90%. This helps you save on storage costs and keeps your queries operating at lightning speed. To see the amount of space
4241
saved:
4342
4443
``` sql
@@ -61,11 +60,11 @@
6160
```
6261
See [timescaledb_information.jobs][informational-views].
6362
64-
1. **Pause a columnstore policy**
63+
1. **Pause a $COLUMNSTORE policy**
6564
66-
If you need to modify or add a lot of data to a chunk in the columnstore, best practice is to stop any jobs moving
67-
chunks to the columnstore, [convert the chunk back to the rowstore][convert_to_rowstore], then modify the data.
68-
After the update, [convert the chunk to the columnstore][convert_to_columnstore] and restart the jobs.
65+
If you need to modify or add a lot of data to a $CHUNK in the $COLUMNSTORE, best practice is to stop any jobs moving
66+
$CHUNKs to the $COLUMNSTORE, [convert the $CHUNK back to the $ROWSTORE][convert_to_rowstore], then modify the data.
67+
After the update, [convert the $CHUNK to the $COLUMNSTORE][convert_to_columnstore] and restart the jobs.
6968
7069
``` sql
7170
SELECT * FROM timescaledb_information.jobs where
@@ -77,24 +76,24 @@
7776
```
7877
See [alter_job][alter_job].
7978
80-
1. **Restart a columnstore policy**
79+
1. **Restart a $COLUMNSTORE policy**
8180
8281
``` sql
8382
SELECT alter_job(JOB_ID, scheduled => true);
8483
```
8584
See [alter_job][alter_job].
8685
87-
1. **Remove a columnstore policy**
86+
1. **Remove a $COLUMNSTORE policy**
8887
8988
``` sql
9089
CALL remove_columnstore_policy('crypto_ticks');
9190
```
9291
See [remove_columnstore_policy][remove_columnstore_policy].
9392
94-
1. **Disable columnstore**
93+
1. **Disable $COLUMNSTORE**
9594
96-
If your table has chunks in the columnstore, you have to
97-
[convert the chunks back to the rowstore][convert_to_rowstore] before you disable the columnstore.
95+
If your table has $CHUNKs in the $COLUMNSTORE, you have to
96+
[convert the $CHUNKs back to the $ROWSTORE][convert_to_rowstore] before you disable the $COLUMNSTORE.
9897
``` sql
9998
ALTER TABLE crypto_ticks SET (timescaledb.enable_columnstore = false);
10099
```

‎_partials/_usage-based-storage-intro.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
$CLOUD_LONG charges are based on the amount of storage you use. You don't pay for
22
fixed storage size, and you don't need to worry about scaling disk size as your
3-
data grows - we handle it all for you. To reduce your data costs further,
4-
combine [Hypercore][hypercore], a [data retention policy][data-retention], and
3+
data grows—we handle it all for you. To reduce your data costs further,
4+
combine [$HYPERCORE][hypercore], a [data retention policy][data-retention], and
55
[tiered storage][data-tiering].
66

77
[hypercore]: /api/:currentVersion:/hypercore/

‎api/hypercore/add_columnstore_policy.md

+16-16
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,10 @@ import Since2180 from "versionContent/_partials/_since_2_18_0.mdx";
1414

1515
# add_columnstore_policy()
1616

17-
Create a [job][job] that automatically moves chunks in a hypertable to the columnstore after a
17+
Create a [job][job] that automatically moves chunks in a hypertable to the $COLUMNSTORE after a
1818
specific time interval.
1919

20-
You enable the columnstore a hypertable or continuous aggregate before you create a columnstore policy.
20+
You enable the $COLUMNSTORE a hypertable or continuous aggregate before you create a $COLUMNSTORE policy.
2121
You do this by calling `ALTER TABLE` for hypertables and `ALTER MATERIALIZED VIEW` for continuous aggregates.
2222

2323
To view the policies that you set or the policies that already exist,
@@ -27,11 +27,11 @@ see [informational views][informational-views], to remove a policy, see [remove_
2727

2828
## Samples
2929

30-
To create a columnstore job:
30+
To create a $COLUMNSTORE job:
3131

3232
<Procedure>
3333

34-
1. **Enable columnstore**
34+
1. **Enable $COLUMNSTORE**
3535

3636
* [Use `ALTER TABLE` for a hypertable][compression_alter-table]
3737
```sql
@@ -46,7 +46,7 @@ To create a columnstore job:
4646
timescaledb.segmentby = 'symbol' );
4747
```
4848

49-
1. **Add a policy to move chunks to the columnstore at a specific time interval**
49+
1. **Add a policy to move chunks to the $COLUMNSTORE at a specific time interval**
5050

5151
For example:
5252

@@ -70,7 +70,7 @@ To create a columnstore job:
7070
CALL add_columnstore_policy('cpu_weekly', INTERVAL '8 weeks');
7171
```
7272

73-
* Older than eight weeks and using the Hypercore table access method:
73+
* Older than eight weeks and using the $HYPERCORE table access method:
7474

7575
``` sql
7676
CALL add_columnstore_policy(
@@ -96,16 +96,16 @@ Calls to `add_columnstore_policy` require either `after` or `created_before`, bu
9696
<!-- vale Google.Acronyms = NO -->
9797
<!-- vale Vale.Spelling = NO -->
9898

99-
| Name | Type | Default | Required | Description |
100-
|--|--|--|--|--|
101-
| `hypertable` |REGCLASS| - | ✔ | Name of the hypertable or continuous aggregate to run this [job][job] on.|
102-
| `after` |INTERVAL or INTEGER|- | ✖ | Add chunks containing data older than `now - {after}::interval` to the columnstore. <br/> Use an object type that matchs the time column type in `hypertable`: <ul><li><b><code>TIMESTAMP</code>, <code>TIMESTAMPTZ</code>, or <code>DATE</code></b>: use an <code>INTERVAL</code> type.</li><li><b> Integer-based timestamps </b>: set an integer type using the [integer_now_func][set_integer_now_func].</li></ul> `after` is mutually exclusive with `created_before`. |
103-
| `created_before` |INTERVAL| NULL | ✖ | Add chunks with a creation time of `now() - created_before` to the columnstore. <br/> `created_before` is <ul><li>Not supported for continuous aggregates.</li><li>Mutually exclusive with `after`.</li></ul> |
104-
| `schedule_interval` |INTERVAL| 12 hours when [chunk_time_interval][chunk_time_interval] >= `1 day` for `hypertable`. Otherwise `chunk_time_interval` / `2`. | ✖ | Set the interval between the finish time of the last execution of this policy and the next start.|
105-
| `initial_start` |TIMESTAMPTZ| The interval from the finish time of the last execution to the [next_start][next-start].| ✖| Set the time this job is first run. This is also the time that `next_start` is calculated from.|
106-
| `timezone` |TEXT| UTC. However, daylight savings time(DST) changes may shift this alignment. | ✖ | Set to a valid time zone to mitigate DST shifting. If `initial_start` is set, subsequent executions of this policy are aligned on `initial_start`.|
107-
| `if_not_exists` |BOOLEAN| `false` | ✖ | Set to `true` so this job fails with a warning rather than an error if a columnstore policy already exists on `hypertable` |
108-
| `hypercore_use_access_method` | BOOLEAN | `NULL` | ✖ | Set to `true` to use hypercore table access metod. If set to `NULL` it will use the value from `timescaledb.default_hypercore_use_access_method`. |
99+
| Name | Type | Default | Required | Description |
100+
|--|--|--|--|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
101+
| `hypertable` |REGCLASS| - | ✔ | Name of the hypertable or continuous aggregate to run this [job][job] on. |
102+
| `after` |INTERVAL or INTEGER|- | ✖ | Add chunks containing data older than `now - {after}::interval` to the $COLUMNSTORE. <br/> Use an object type that matchs the time column type in `hypertable`: <ul><li><b><code>TIMESTAMP</code>, <code>TIMESTAMPTZ</code>, or <code>DATE</code></b>: use an <code>INTERVAL</code> type.</li><li><b> Integer-based timestamps </b>: set an integer type using the [integer_now_func][set_integer_now_func].</li></ul> `after` is mutually exclusive with `created_before`. |
103+
| `created_before` |INTERVAL| NULL | ✖ | Add chunks with a creation time of `now() - created_before` to the $COLUMNSTORE. <br/> `created_before` is <ul><li>Not supported for continuous aggregates.</li><li>Mutually exclusive with `after`.</li></ul> |
104+
| `schedule_interval` |INTERVAL| 12 hours when [chunk_time_interval][chunk_time_interval] >= `1 day` for `hypertable`. Otherwise `chunk_time_interval` / `2`. | ✖ | Set the interval between the finish time of the last execution of this policy and the next start. |
105+
| `initial_start` |TIMESTAMPTZ| The interval from the finish time of the last execution to the [next_start][next-start].| ✖| Set the time this job is first run. This is also the time that `next_start` is calculated from. |
106+
| `timezone` |TEXT| UTC. However, daylight savings time(DST) changes may shift this alignment. | ✖ | Set to a valid time zone to mitigate DST shifting. If `initial_start` is set, subsequent executions of this policy are aligned on `initial_start`. |
107+
| `if_not_exists` |BOOLEAN| `false` | ✖ | Set to `true` so this job fails with a warning rather than an error if a $COLUMNSTORE policy already exists on `hypertable` |
108+
| `hypercore_use_access_method` | BOOLEAN | `NULL` | ✖ | Set to `true` to use $HYPERCORE table access method. If set to `NULL` it will use the value from `timescaledb.default_hypercore_use_access_method`. |
109109

110110

111111
<!-- vale Google.Acronyms = YES -->

‎api/hypercore/alter_materialized_view.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
api_name: ALTER MATERIALIZED VIEW (Hypercore)
2+
api_name: ALTER MATERIALIZED VIEW (hypercore)
33
excerpt: Change an existing continuous aggregate
44
topics: [hypercore, continuous aggregates, columnstore,]
55
keywords: [hypercore, continuous aggregates, columnstore, ]
@@ -12,7 +12,7 @@ api:
1212
import Since2180 from "versionContent/_partials/_since_2_18_0.mdx";
1313
import EarlyAccess from "versionContent/_partials/_early_access_2_18_0.mdx";
1414

15-
# ALTER MATERIALIZED VIEW (Hypercore) <Tag type="community">Community</Tag>
15+
# ALTER MATERIALIZED VIEW ($HYPERCORE) <Tag type="community">Community</Tag>
1616

1717
`ALTER MATERIALIZED VIEW` statement can be used to modify some of the `WITH`
1818
clause [options][create_materialized_view] for the continuous aggregate view.

0 commit comments

Comments
 (0)