Skip to content

Apply feedback from documentation-website to PPL command docs #4997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RyanL1997 merged 11 commits into from
Jan 9, 2026
Merged

Apply feedback from documentation-website to PPL command docs #4997

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a3a8f74
Format PPL cmd/ docs for documentation-website.
kylehounslow Dec 15, 2025
05a83e9
apply feedback from documentation website to docs
ritvibhatt Dec 24, 2025
826b66d
fix links
ritvibhatt Jan 5, 2026
bac68f3
Merge branch 'main' into apply-docs-website-feedback
ritvibhatt Jan 5, 2026
57d82f8
fix links and example formats
ritvibhatt Jan 7, 2026
b129e16
fix top.md example format and patterns wildcard escaping
ritvibhatt Jan 7, 2026
ce93f50
fix jekyll tag conversion
ritvibhatt Jan 7, 2026
bac0202
update exporter script readme
ritvibhatt Jan 7, 2026
eae078b
fix join table formatting
ritvibhatt Jan 7, 2026
7b7733f
fix typos
ritvibhatt Jan 7, 2026
445b1fb
fix rex note
ritvibhatt Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 77 additions & 37 deletions docs/user/ppl/cmd/ad.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,76 @@
# ad (deprecated by ml command)

## Description
# ad (Deprecated)

The `ad` command applies Random Cut Forest (RCF) algorithm in the ml-commons plugin on the search result returned by a PPL command. Based on the input, the command uses two types of RCF algorithms: fixed-in-time RCF for processing time-series data, batch RCF for processing non-time-series data.
## Syntax
> **Warning**: The `ad` command is deprecated in favor of the [`ml` command](./ml.md).

## Fixed In Time RCF For Time-series Data
The `ad` command applies the Random Cut Forest (RCF) algorithm in the ML Commons plugin to the search results returned by a PPL command. The command provides two anomaly detection approaches:

ad [number_of_trees] [shingle_size] [sample_size] [output_after] [time_decay] [anomaly_rate] \<time_field\> [date_format] [time_zone] [category_field]
* number_of_trees: optional. Number of trees in the forest. **Default:** 30.
* shingle_size: optional. A shingle is a consecutive sequence of the most recent records. **Default:** 8.
* sample_size: optional. The sample size used by stream samplers in this forest. **Default:** 256.
* output_after: optional. The number of points required by stream samplers before results are returned. **Default:** 32.
* time_decay: optional. The decay factor used by stream samplers in this forest. **Default:** 0.0001.
* anomaly_rate: optional. The anomaly rate. **Default:** 0.005.
* time_field: mandatory. Specifies the time field for RCF to use as time-series data.
* date_format: optional. Used for formatting time_field. **Default:** "yyyy-MM-dd HH:mm:ss".
* time_zone: optional. Used for setting time zone for time_field. **Default:** "UTC".
* category_field: optional. Specifies the category field used to group inputs. Each category will be independently predicted.
- [Anomaly detection for time-series data](#anomaly-detection-for-time-series-data) using the fixed-in-time RCF algorithm
- [Anomaly detection for non-time-series data](#anomaly-detection-for-non-time-series-data) using the batch RCF algorithm

> **Note**: To use the `ad` command, `plugins.calcite.enabled` must be set to `false`.

## Syntax

The `ad` command has two different syntax variants, depending on the algorithm type.

### Anomaly detection for time-series data

Use this syntax to detect anomalies in time-series data. This method uses the fixed-in-time RCF algorithm, which is optimized for sequential data patterns.

The fixed-in-time RCF `ad` command has the following syntax:

```syntax
ad [number_of_trees] [shingle_size] [sample_size] [output_after] [time_decay] [anomaly_rate] <time_field> [date_format] [time_zone] [category_field]
```

### Parameters

The fixed-in-time RCF algorithm supports the following parameters.

| Parameter | Required/Optional | Description |
| --- | --- | --- |
| `time_field` | Required | The time field for RCF to use as time-series data. |
| `number_of_trees` | Optional | The number of trees in the forest. Default is `30`. |
| `shingle_size` | Optional | The number of records in a shingle. A shingle is a consecutive sequence of the most recent records. Default is `8`. |
| `sample_size` | Optional | The sample size used by the stream samplers in this forest. Default is `256`. |
| `output_after` | Optional | The number of points required by the stream samplers before results are returned. Default is `32`. |
| `time_decay` | Optional | The decay factor used by the stream samplers in this forest. Default is `0.0001`. |
| `anomaly_rate` | Optional | The anomaly rate. Default is `0.005`. |
| `date_format` | Optional | The format used for the `time_field` field. Default is `yyyy-MM-dd HH:mm:ss`. |
| `time_zone` | Optional | The time zone for the `time_field` field. Default is `UTC`. |
| `category_field` | Optional | The category field used to group input values. The predict operation is applied to each category independently. |

## Batch RCF For Non-time-series Data

### Anomaly detection for non-time-series data

Use this syntax to detect anomalies in data where the order doesn't matter. This method uses the batch RCF algorithm, which is optimized for independent data points.

The batch RCF `ad` command has the following syntax:

```syntax
ad [number_of_trees] [sample_size] [output_after] [training_data_size] [anomaly_score_threshold] [category_field]
* number_of_trees: optional. Number of trees in the forest. **Default:** 30.
* sample_size: optional. Number of random samples given to each tree from the training data set. **Default:** 256.
* output_after: optional. The number of points required by stream samplers before results are returned. **Default:** 32.
* training_data_size: optional. **Default:** size of your training data set.
* anomaly_score_threshold: optional. The threshold of anomaly score. **Default:** 1.0.
* category_field: optional. Specifies the category field used to group inputs. Each category will be independently predicted.
```

### Parameters

The batch RCF algorithm supports the following parameters.

| Parameter | Required/Optional | Description |
| --- | --- | --- |
| `number_of_trees` | Optional | The number of trees in the forest. Default is `30`. |
| `sample_size` | Optional | The number of random samples provided to each tree from the training dataset. Default is `256`. |
| `output_after` | Optional | The number of points required by the stream samplers before results are returned. Default is `32`. |
| `training_data_size` | Optional | The size of the training dataset. Default is the full dataset size. |
| `anomaly_score_threshold` | Optional | The anomaly score threshold. Default is `1.0`. |
| `category_field` | Optional | The category field used to group input values. The predict operation is applied to each category independently. |

## Example 1: Detecting events in New York City from taxi ridership data with time-series data

This example trains an RCF model and uses the model to detect anomalies in the time-series ridership data.
## Example 1: Detecting events in New York City taxi ridership time-series data

The following examples use the `nyc_taxi` dataset, which contains New York City taxi ridership data with fields including `value` (number of rides), `timestamp` (time of measurement), and `category` (time period classifications such as 'day' and 'night').

This example trains an RCF model and uses it to detect anomalies in time-series ridership data:

```ppl ignore
source=nyc_taxi
Expand All @@ -40,7 +79,7 @@ source=nyc_taxi
| where value=10844.0
```

Expected output:
The query returns the following results:

```text
fetched rows / total rows = 1/1
Expand All @@ -51,9 +90,10 @@ fetched rows / total rows = 1/1
+---------+---------------------+-------+---------------+
```

## Example 2: Detecting events in New York City from taxi ridership data with time-series data independently with each category

This example trains an RCF model and uses the model to detect anomalies in the time-series ridership data with multiple category values.
## Example 2: Detecting events in New York City taxi ridership time-series data by category

This example trains an RCF model and uses it to detect anomalies in time-series ridership data across multiple category values:

```ppl ignore
source=nyc_taxi
Expand All @@ -62,7 +102,7 @@ source=nyc_taxi
| where value=10844.0 or value=6526.0
```

Expected output:
The query returns the following results:

```text
fetched rows / total rows = 2/2
Expand All @@ -74,9 +114,10 @@ fetched rows / total rows = 2/2
+----------+---------+---------------------+-------+---------------+
```

## Example 3: Detecting events in New York City from taxi ridership data with non-time-series data

This example trains an RCF model and uses the model to detect anomalies in the non-time-series ridership data.
## Example 3: Detecting events in New York City taxi ridership non-time-series data

This example trains an RCF model and uses it to detect anomalies in non-time-series ridership data:

```ppl ignore
source=nyc_taxi
Expand All @@ -85,7 +126,7 @@ source=nyc_taxi
| where value=10844.0
```

Expected output:
The query returns the following results:

```text
fetched rows / total rows = 1/1
Expand All @@ -96,9 +137,10 @@ fetched rows / total rows = 1/1
+---------+-------+-----------+
```

## Example 4: Detecting events in New York City from taxi ridership data with non-time-series data independently with each category

This example trains an RCF model and uses the model to detect anomalies in the non-time-series ridership data with multiple category values.
## Example 4: Detecting events in New York City taxi ridership non-time-series data by category

This example trains an RCF model and uses it to detect anomalies in non-time-series ridership data across multiple category values:

```ppl ignore
source=nyc_taxi
Expand All @@ -107,7 +149,7 @@ source=nyc_taxi
| where value=10844.0 or value=6526.0
```

Expected output:
The query returns the following results:

```text
fetched rows / total rows = 2/2
Expand All @@ -119,6 +161,4 @@ fetched rows / total rows = 2/2
+----------+---------+-------+-----------+
```

## Limitations

The `ad` command can only work with `plugins.calcite.enabled=false`.
47 changes: 29 additions & 18 deletions docs/user/ppl/cmd/addcoltotals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,32 @@
# AddColTotals


# Description
# addcoltotals

The `addcoltotals` command computes the sum of each column and add a summary event at the end to show the total of each column. This command works the same way `addtotals` command works with row=false and col=true option. This is useful for creating summary reports with subtotals or grand totals. The `addcoltotals` command only sums numeric fields (integers, floats, doubles). Non-numeric fields in the field list are ignored even if its specified in field-list or in the case of no field-list specified.
The `addcoltotals` command computes the sum of each column and adds a summary row showing the total for each column. This command is equivalent to using `addtotals` with `row=false` and `col=true`, making it useful for creating summary reports with column totals.

# Syntax
The command only processes numeric fields (integers, floats, doubles). Non-numeric fields are ignored regardless of whether they are explicitly specified in the field list.

`addcoltotals [field-list] [label=<string>] [labelfield=<field>]`

- `field-list`: Optional. Comma-separated list of numeric fields to sum. If not specified, all numeric fields are summed.
- `labelfield=<field>`: Optional. Field name to place the label. If it specifies a non-existing field, adds the field and shows label at the summary event row at this field.
- `label=<string>`: Optional. Custom text for the totals row labelfield\'s label. Default is \"Total\".
## Syntax

# Example 1: Basic Example
The `addcoltotals` command has the following syntax:

The example shows placing the label in an existing field.
```syntax
addcoltotals [field-list] [label=<string>] [labelfield=<field>]
```

## Parameters

The `addcoltotals` command supports the following parameters.

| Parameter | Required/Optional | Description |
| --- | --- | --- |
| `<field-list>` | Optional | A comma-separated list of numeric fields to add. By default, all numeric fields are added. |
| `labelfield` | Optional | The field in which the label is placed. If the field does not exist, it is created and the label is shown in the summary row (last row) of the new field. |
| `label` | Optional | The text that appears in the summary row (last row) to identify the computed totals. When used with `labelfield`, this text is placed in the specified field in the summary row. Default is `Total`. |

### Example 1: Basic example

The following query places the label in an existing field:

```ppl
source=accounts
Expand All @@ -24,7 +35,7 @@ source=accounts
| addcoltotals labelfield='firstname'
```

Expected output:
The query returns the following results:

```text
fetched rows / total rows = 4/4
Expand All @@ -38,17 +49,17 @@ fetched rows / total rows = 4/4
+-----------+---------+
```

# Example 2: Adding column totals and adding a summary event with label specified.
## Example 2: Adding column totals with a custom summary label

The example shows adding totals after a stats command where final summary event label is \'Sum\' and row=true value was used by default when not specified. It also added new field specified by labelfield as it did not match existing field.
The following query adds totals after a `stats` command where the final summary event label is `Sum`. It also creates a new field specified by `labelfield` because this field does not exist in the data:

```ppl
source=accounts
| stats count() by gender
| addcoltotals `count()` label='Sum' labelfield='Total'
```

Expected output:
The query returns the following results:

```text
fetched rows / total rows = 3/3
Expand All @@ -61,9 +72,9 @@ fetched rows / total rows = 3/3
+---------+--------+-------+
```

# Example 3: With all options
## Example 3: Using all options

The example shows using addcoltotals with all options set.
The following query uses the `addcoltotals` command with all options set:

```ppl
source=accounts
Expand All @@ -73,7 +84,7 @@ source=accounts
| addcoltotals avg_balance, count label='Sum' labelfield='Column Total'
```

Expected output:
The query returns the following results:

```text
fetched rows / total rows = 4/4
Expand Down
Loading
Loading