Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: delete assets api #78

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,14 @@ Discover why users choose Compass as their main data discovery and lineage servi

Explore the following resources to get started with Compass:

- [Guides](https://goto.github.io/compass/docs/guides) provides guidance on ingesting and querying metadata from Compass.
- [Concepts](https://goto.github.io/compass/docs/concepts) describes all important Compass concepts.
- [Reference](https://goto.github.io/compass/docs/reference) contains details about configurations, metrics and other aspects of Compass.
- [Contribute](https://goto.github.io/compass/docs/contribute/contribution.md) contains resources for anyone who wants to contribute to Compass.
- [Guides](https://goto.github.io/compass/guides/ingestion) provides guidance on ingesting and querying metadata from Compass.
- [Concepts](https://goto.github.io/compass/concepts/overview) describes all important Compass concepts.
- [Reference](https://goto.github.io/compass/reference/api) contains details about configurations, metrics and other aspects of Compass.
- [Contribute](https://goto.github.io/compass/contribute/contributing) contains resources for anyone who wants to contribute to Compass.

## Installation

Install Compass on macOS, Windows, Linux, OpenBSD, FreeBSD, and on any machine. <br/>Refer this for [installations](https://goto.github.io/compass/docs/installation) and [configurations](https://goto.github.io/compass/docs/guides/configuration)
Install Compass on macOS, Windows, Linux, OpenBSD, FreeBSD, and on any machine. <br/>Refer this for [installations](https://goto.github.io/compass/installation) and [configurations](https://goto.github.io/compass/configuration)

#### Binary (Cross-platform)

Expand Down Expand Up @@ -204,7 +204,7 @@ elasticsearch cluster, set the value of `ES_TEST_SERVER_URL` to the URL of the e

Development of Compass happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving Compass.

Read our [contributing guide](https://goto.github.io/compass/docs/contribute/contribution.md) to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Compass.
Read our [contributing guide](https://goto.github.io/compass/contribute/contributing) to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Compass.

To help you get your feet wet and get you familiar with our contribution process, we have a list of [good first issues](https://github.com/goto/compass/labels/good%20first%20issue) that contain bugs which have a relatively limited scope. This is a great place to get started.

Expand Down
54 changes: 53 additions & 1 deletion docs/docs/reference/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Documentation of our Compass API with gRPC and gRPC-Gateway.

[More about Compass](https://goto.github.io/compass/)

## default
## Assets

### /v1beta1/assets

Expand Down Expand Up @@ -180,6 +180,51 @@ Delete a single asset with given ID
| 500 | Returned when theres is something wrong on the server side. | [Status](#status) |
| default | An unexpected error response. | [Status](#status) |

### /v1beta1/assets/delete-by-query
##### Summary

Delete assets by [query expression](https://expr-lang.org/).

##### Description

Delete all assets that match the given [query expression](https://expr-lang.org/).
The query expr at least must consist `refreshed_at`, `type`, and `service` identifiers.
`type` and `service` identifiers valid only if it's using equals (`==`) or `IN` operator, to prevent human error on deleting assets.
For example of the correct query:
```
refreshed_at <= "2023-12-12 23:59:59" && service in ["service-1", "service-2"] && type == "table"
```
```
refreshed_at <= (now() - duration('24h') && service == "service-1" && (type == "table" || data.foo != "bar")
```

The idea of query expr converter is convert `query_expr` to AST (Abstract Syntax Tree), then make it as SQL Query and Elasticsearch Query so can used as filter query on deletion process.
Currently, the expr query **already support most of the frequently used cases, except** ChainNode, SliceNode, CallNode, ClosureNode, PointerNode, VariableDeclaratorNode, MapNode, and PairNode.
For more contexts, please refer to [AST Node](https://github.com/expr-lang/expr/blob/master/ast/node.go) in expr-lang library and [Query Expr Converter](https://github.com/goto/compass/tree/main/pkg/query_expr) in Compass.
Example of **unsupported query for now** due to not directly produce a value is
```
service in filter(assets, .Service startsWith "T")
```

Complex query covered only if it directly produces a value, like `bool_identifier == !(findLast([1, 2, 3, 4], # > 2) == 4)` will produce `bool_identifier == false`.
However, **please do the best practice that try to simplify the query first** to makes readable and prevent unwanted things like errors or false positive result. Like example before, please write `false` instead of `!(findLast([1, 2, 3, 4], # > 2) == 4)`. You can use [expr-lang playground](https://expr-lang.org/playground) to simplify the query expr.


##### Parameters

| Name | Located in | Description | Required | Schema |
|------------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| ------ |
| query_expr | body | query expression based on [expr-lang](https://expr-lang.org/) to filtering the assets that wants to be deleted. `refreshed_at`, `type`, and `service` identifiers must exist in the query. The `type` and `service` must using equals (`==`) or `IN` operator | Yes | string |
| dry_run | body | if set to true, then deletion should not proceed and only return the affected rows. Else, will perform deletion in the background (default) | No | string |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we change the part saying:

if set to true, then deletion should not proceed and only return the affected rows

into

if set to true, deletion will not be executed but the number of rows matching the query is returned

also, need to put what is the default value to make it explicit. currently a bit confusing. an example:

(default: false) if set to true, ...


##### Responses

| Code | Description | Schema |
| ---- | ----------- |-----------------------------------------------|
| 200 | A successful response. | [DeleteAssetsResponse](#deleteassetsresponse) |
| 400 | Returned when the data that user input is wrong. | [Status](#status) |
| 500 | Returned when theres is something wrong on the server side. | [Status](#status) |
| default | An unexpected error response. | [Status](#status) |
### /v1beta1/assets/{id}/stargazers

#### GET
Expand Down Expand Up @@ -1580,6 +1625,13 @@ Request to be sent to create a tag's template
| ---- | ---- | ----------- | -------- |
| DeleteAssetResponse | object | | |

#### DeleteAssetsResponse

| Name | Type | Description | Required |
|---------------|---------|-----------------------------------------------------|----------|
| affected_rows | integer | the numbers of assets that match the given query | No |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to add column required in this response?



#### DeleteCommentResponse

| Name | Type | Description | Required |
Expand Down
Loading