notebooks/05-operations/05-compaction-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0cb3b009-ebde-4d56-9d59-a028d66d8309",
   "metadata": {},
   "source": [
    "# Change table data by using compaction\n",
    "<!--\n",
    "  ~ Licensed to the Apache Software Foundation (ASF) under one\n",
    "  ~ or more contributor license agreements.  See the NOTICE file\n",
    "  ~ distributed with this work for additional information\n",
    "  ~ regarding copyright ownership.  The ASF licenses this file\n",
    "  ~ to you under the Apache License, Version 2.0 (the\n",
    "  ~ \"License\"); you may not use this file except in compliance\n",
    "  ~ with the License.  You may obtain a copy of the License at\n",
    "  ~\n",
    "  ~   http://www.apache.org/licenses/LICENSE-2.0\n",
    "  ~\n",
    "  ~ Unless required by applicable law or agreed to in writing,\n",
    "  ~ software distributed under the License is distributed on an\n",
    "  ~ \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
    "  ~ KIND, either express or implied.  See the License for the\n",
    "  ~ specific language governing permissions and limitations\n",
    "  ~ under the License.\n",
    "  -->\n",
    "\n",
    "Through compaction, whether manual or running automatically, you can change the schema of a table, filter data out, and aggregate your data - whether that is the entire table, or just data of a particular time range.\n",
    "\n",
    "This tutorial demonstrates how to work with [compaction](https://druid.apache.org/docs/latest/data-management/compaction) to apply different dimension schemes and to apply transformations manually, though compaction can also run [automatically](https://druid.apache.org/docs/latest/data-management/automatic-compaction).\n",
    "\n",
    "In this tutorial you perform the following tasks:\n",
    "\n",
    "- Create a table using batch ingestion.\n",
    "- Run a compaction task to remove dimensions for a particular time period in the data.\n",
    "- Run a compaction task to remove all data that matches a particular criteria.\n",
    "- Run a task to change granularity of a data (rollup)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbdbf6ad-ca7b-40f5-8ca3-1070f4a3ee42",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "This tutorial works with Druid 30.0.0 or later.\n",
    "\n",
    "Before following this notebook, it's recommended to complete the notebook on [changing data layout with compaction](./04-compaction-partitioning.ipynb).\n",
    "\n",
    "Launch this tutorial and all prerequisites using the `druid-jupyter` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see the Learn Druid repository [readme](https://github.com/implydata/learn-druid).\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5007a243-b81a-4601-8f57-5b14940abbff",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "The following cells set up the notebook and learning environment ready for use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b769122-c5a4-404e-9ef8-9c0ebd97695a",
   "metadata": {},
   "source": [
    "### Set up a connection to Apache Druid\n",
    "\n",
    "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
    "\n",
    "If successful, the Druid version number will be shown in the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ec783b-df3f-4168-9be2-cdc6ad3e33c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import druidapi\n",
    "import os\n",
    "\n",
    "druid_headers = {'Content-Type': 'application/json'}\n",
    "\n",
    "if 'DRUID_HOST' not in os.environ.keys():\n",
    "    druid_host=f\"http://localhost:8888\"\n",
    "else:\n",
    "    druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
    "\n",
    "print(f\"Opening a connection to {druid_host}.\")\n",
    "druid = druidapi.jupyter_client(druid_host)\n",
    "display = druid.display\n",
    "sql_client = druid.sql\n",
    "status_client = druid.status\n",
    "\n",
    "status_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20a5a2ac-f153-4678-8672-e1ec83a7c309",
   "metadata": {},
   "source": [
    "### Import additional modules\n",
    "\n",
    "Run the following cell to import additional Python modules that you will use to call Druid APIs directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "139c87bb-266f-479b-99ad-90486aaf36fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "472589e4-1026-4b3b-bb79-eedabb2b44c4",
   "metadata": {},
   "source": [
    "## Create a table using batch ingestion\n",
    "\n",
    "Run the following cell to create a table using batch ingestion. Specific dimensions are selected from the source data.\n",
    "\n",
    "When completed, you'll see a description of the final table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f52a94fb-d2e4-403f-ab10-84d3af7bf2c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "table_name = 'example-wikipedia-datacompaction'\n",
    "\n",
    "sql='''\n",
    "REPLACE INTO \"''' + table_name + '''\" OVERWRITE ALL\n",
    "WITH \"ext\" AS (\n",
    "  SELECT *\n",
    "  FROM TABLE(\n",
    "    EXTERN(\n",
    "      '{\"type\":\"http\",\"uris\":[\"https://druid.apache.org/data/wikipedia.json.gz\"]}',\n",
    "      '{\"type\":\"json\"}'\n",
    "    )\n",
    "  ) EXTEND (\"isRobot\" VARCHAR, \"channel\" VARCHAR, \"timestamp\" VARCHAR, \"flags\" VARCHAR, \"isUnpatrolled\" VARCHAR, \"page\" VARCHAR, \"diffUrl\" VARCHAR, \"added\" BIGINT, \"comment\" VARCHAR, \"commentLength\" BIGINT, \"isNew\" VARCHAR, \"isMinor\" VARCHAR, \"delta\" BIGINT, \"isAnonymous\" VARCHAR, \"user\" VARCHAR, \"deltaBucket\" BIGINT, \"deleted\" BIGINT, \"namespace\" VARCHAR, \"cityName\" VARCHAR, \"countryName\" VARCHAR, \"regionIsoCode\" VARCHAR, \"metroCode\" BIGINT, \"countryIsoCode\" VARCHAR, \"regionName\" VARCHAR)\n",
    ")\n",
    "SELECT\n",
    "  TIME_PARSE(\"timestamp\") AS \"__time\",\n",
    "  \"namespace\",\n",
    "  \"page\",\n",
    "  \"user\",\n",
    "  \"channel\",\n",
    "  \"added\",\n",
    "  \"deleted\",\n",
    "  \"commentLength\",\n",
    "  \"isRobot\",\n",
    "  \"isAnonymous\",\n",
    "  \"regionIsoCode\",\n",
    "  \"countryIsoCode\"\n",
    "FROM \"ext\"\n",
    "PARTITIONED BY HOUR\n",
    "'''\n",
    "\n",
    "display.run_task(sql)\n",
    "sql_client.wait_until_ready(f'{table_name}')\n",
    "display.table(f'{table_name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c3a4eff-369b-416b-98f4-ddabcae59e9c",
   "metadata": {},
   "source": [
    "## Apply changes to data using compaction\n",
    "\n",
    "Compaction is a special type of native [Druid task](https://druid.apache.org/docs/latest/ingestion/tasks#all-task-types) that, like streaming ingestion, uses JSON specifications to define behaviors. Each contains:\n",
    "\n",
    "* An [ioConfig](https://druid.apache.org/docs/latest/data-management/manual-compaction#compaction-io-configuration), defining what the source data is for the job.\n",
    "* A [tuningConfig](https://druid.apache.org/docs/latest/ingestion/native-batch#tuningconfig), detailing specific controls.\n",
    "* And elements to control what happens to the data (as you would find in a [`dataSchema`](https://druid.apache.org/docs/latest/ingestion/ingestion-spec#dataschema) in streaming ingestion).\n",
    "  * The dimensions to put into the resulting data given in a `dimensionsSpec`.\n",
    "  * Any filters or calculations to do on the data as listed in the `transformSpec`.\n",
    "  * Any aggregation that should be done, as given in the `metricsSpec` when `rollup` is enabled.\n",
    " \n",
    "In the cells that follow you will see various examples of how to use the `dimensionsSpec` and `transformSpec` to affect table data as part of compaction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b93baea-be1d-4d3d-86a0-eac17ce79fd3",
   "metadata": {},
   "source": [
    "### Use dimensionsSpec to add or remove columns\n",
    "\n",
    "Amend the dimensions in the table at compaction time by using a [`dimensionsSpec`](https://druid.apache.org/docs/latest/data-management/manual-compaction#compaction-dimensions-spec). Options include removing (`dimensionExclusions`) or explicitly including (`dimensions`) specific columns.\n",
    "\n",
    "As each segment defines its own schema, use the SYS.SEGMENTS table to view the dimensions of the table in each partition by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "905520bd-c615-4070-b186-0e5bb2036cb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT DISTINCT\n",
    "    \"start\",\n",
    "    \"end\",\n",
    "    \"dimensions\"\n",
    "FROM sys.segments\n",
    "WHERE datasource = '{table_name}'\n",
    "ORDER BY 1\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e765785-d545-45e8-afa7-c552254d3236",
   "metadata": {},
   "source": [
    "The cell below will construct a compaction task specification that includes a [`dimensionsSpec`](https://druid.apache.org/docs/latest/data-management/manual-compaction#compaction-dimensions-spec) to remove specific columns.\n",
    "\n",
    "* The `ioConfig` is represented by `compaction_ioConfig`. It contains an `inputSpec` that has a restriction on the `interval` so that this task only affects data between 19:00 and 20:00.\n",
    "* The `granularitySpec` matches the original PARTITIONED BY.\n",
    "* The `dimensionsSpec` contains an explicit list of `dimensionExclusions` - these are what will be removed.\n",
    "\n",
    "Finally, the `compaction_spec` object uses these objects to create the final JSON compaction specification.\n",
    "\n",
    "Run the cell to print out the JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa4834b9-d940-4290-a365-b03830e7b012",
   "metadata": {},
   "outputs": [],
   "source": [
    "compaction_ioConfig_inputSpec = {\n",
    "    \"type\" : \"interval\",\n",
    "    \"interval\" : \"2016-06-27T19:00:00/PT1H\" }\n",
    "\n",
    "compaction_ioConfig = {\n",
    "    \"type\" : \"compact\",\n",
    "    \"inputSpec\" : compaction_ioConfig_inputSpec }\n",
    "\n",
    "compaction_granularitySpec = { \"segmentGranularity\" : \"HOUR\" }\n",
    "\n",
    "compaction_dimensionsSpec = {\n",
    "    \"dimensionExclusions\" : [ \"namespace\", \"isAnonymous\", \"user\" ] }\n",
    "\n",
    "compaction_spec = {\n",
    "    \"type\": \"compact\",\n",
    "    \"dataSource\": table_name,\n",
    "    \"ioConfig\": compaction_ioConfig,\n",
    "    \"granularitySpec\": compaction_granularitySpec,\n",
    "    \"dimensionsSpec\": compaction_dimensionsSpec\n",
    "}\n",
    "\n",
    "print(json.dumps(compaction_spec, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea4decc7-57d0-4c84-be4b-606580298969",
   "metadata": {},
   "outputs": [],
   "source": [
    "requests.post(f\"{druid_host}/druid/indexer/v1/task\", json.dumps(compaction_spec), headers=druid_headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "717c356f-c356-4ada-840c-ad6eeb6fc459",
   "metadata": {},
   "source": [
    "Run this cell below to follow along as the compaction task runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b83fdc93-5a6b-4bd8-aa94-fd185fff0b10",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT DISTINCT\n",
    "    \"start\",\n",
    "    \"end\",\n",
    "    \"dimensions\"\n",
    "FROM sys.segments\n",
    "WHERE datasource = '{table_name}'\n",
    "AND \"start\" LIKE '2016-06-27T1%'\n",
    "ORDER BY 1\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eead1d15-a329-4dd2-ad26-7a4d9eb8d3c0",
   "metadata": {},
   "source": [
    "When completed, you will see that the table no longer contains `namespace`, `isAnonymous`, or `user` between 1900 and 2000."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d4c313b-e68a-462e-8b15-89ff2694e8c8",
   "metadata": {},
   "source": [
    "### Use a transformSpec to filter out data\n",
    "\n",
    "Incorporate a [native filter](https://druid.apache.org/docs/latest/querying/filters) into a `transformSpec` during compaction to retain or remove rows that match a particular condition.\n",
    "\n",
    "Run the following cell to retrieve some data from the table between 1000 and 1100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccc033e1-af30-421d-89d3-ce2aef156cb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT channel,\n",
    "   COUNT(*) AS \"events\"\n",
    "FROM \"{table_name}\"\n",
    "WHERE TIME_IN_INTERVAL (\"__time\",'2016-06-27T10/PT1H')\n",
    "GROUP BY 1\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "055131e7-e696-4a2c-8988-8424244b5afd",
   "metadata": {},
   "source": [
    "Run the cell below to construct a compaction specification that only retains events where the `channel` is `#en.wikipedia`. Notice that the `interval` constrains this job to events in the table between 1000 and 1100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68d22722-db2a-4597-81d0-eea3899dfc2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "compaction_ioConfig_inputSpec = {\n",
    "    \"type\" : \"interval\",\n",
    "    \"interval\" : \"2016-06-27T10:00:00/PT1H\" }\n",
    "\n",
    "compaction_ioConfig = {\n",
    "    \"type\" : \"compact\",\n",
    "    \"inputSpec\" : compaction_ioConfig_inputSpec }\n",
    "\n",
    "compaction_granularitySpec = { \"segmentGranularity\" : \"HOUR\" }\n",
    "\n",
    "compaction_transformSpec = {\n",
    "    \"filter\":\n",
    "        {\n",
    "            \"type\": \"selector\",\n",
    "            \"dimension\": \"channel\",\n",
    "            \"value\": \"#en.wikipedia\"\n",
    "        }\n",
    "    }\n",
    "\n",
    "compaction_spec = {\n",
    "    \"type\": \"compact\",\n",
    "    \"dataSource\": table_name,\n",
    "    \"ioConfig\": compaction_ioConfig,\n",
    "    \"granularitySpec\": compaction_granularitySpec,\n",
    "    \"transformSpec\": compaction_transformSpec\n",
    "}\n",
    "\n",
    "print(json.dumps(compaction_spec, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c721bf8-24fb-492f-8eb9-1d51e9dc310d",
   "metadata": {},
   "source": [
    "Run the next cell to submit the compaction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71e4b07d-85f1-4427-bffa-adbc70377734",
   "metadata": {},
   "outputs": [],
   "source": [
    "requests.post(f\"{druid_host}/druid/indexer/v1/task\", json.dumps(compaction_spec), headers=druid_headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28e2b6c9-edf0-4db8-9e62-dc78fafe2b45",
   "metadata": {},
   "source": [
    "Run the cell below to follow along as Druid processes the table data.\n",
    "\n",
    "When completed, the time period selected _only_ contains rows for a particular channel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8969a6e8-3426-452d-8163-b020a5e232d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT channel,\n",
    "   COUNT(*) AS \"events\"\n",
    "FROM \"{table_name}\"\n",
    "WHERE TIME_IN_INTERVAL (\"__time\",'2016-06-27T10:00/PT1H')\n",
    "GROUP BY 1\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a644594e-5521-42ec-8e87-b15c80082ff8",
   "metadata": {},
   "source": [
    "### Aggregate table data using rollup\n",
    "\n",
    "Enable [`rollup`](https://druid.apache.org/docs/latest/data-management/compaction#rollup) inside the `granularitySpec` and add a [`metricsSpec`](https://druid.apache.org/docs/latest/ingestion/ingestion-spec#metricsspec) to apply a GROUP BY to table data through compaction.\n",
    "\n",
    "Run the following cell to create a compaction specification.\n",
    "\n",
    "* The `interval` is set to process only the first six hours of the data.\n",
    "* `rollup` is set to true to apply a GROUP BY on the data.\n",
    "* A `metricsSpec` is represented by the `compaction_metricsSpec` object, and sets up four metrics to be produced:\n",
    "    * Instead of the raw data, `added` becomes a SUM of the source values.\n",
    "    * `deleted` receives the same treatment as `added`, replacing the raw values with a SUM.\n",
    "    * A new column called `theta_user` is added - this contains an Apache Thetasketch of the underlying users for [approximate COUNT DISTINCT](https://druid.apache.org/docs/latest/querying/aggregations#approximate-aggregations) operations.\n",
    " \n",
    "In order to make the rollup ratio efficient:\n",
    "\n",
    "* `queryGranularity` is added to the `granularitySpec` so that the timestamp is changed to fifteen-minute precision.\n",
    "* Only specific dimensions are retained - these are specified using a `dimensionsSpec` list of `dimensions` to retain. Notice that `user` is no longer in the list of dimensions.\n",
    "\n",
    "Run the next cell to build the compaction specification and take a look at the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca28bc43-c264-42e5-96c3-ddbd8959b2d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "compaction_ioConfig_inputSpec = {\n",
    "    \"type\" : \"interval\",\n",
    "    \"interval\" : \"2016-06-27/PT6H\" }\n",
    "\n",
    "compaction_ioConfig = {\n",
    "    \"type\" : \"compact\",\n",
    "    \"inputSpec\" : compaction_ioConfig_inputSpec }\n",
    "\n",
    "compaction_granularitySpec = {\n",
    "    \"segmentGranularity\" : \"HOUR\",\n",
    "    \"rollup\" : \"true\",\n",
    "    \"queryGranularity\" : \"fifteen_minute\"}\n",
    "\n",
    "compaction_metricsSpec = [\n",
    "        { \"type\": \"doubleSum\", \"name\": \"added\", \"fieldName\": \"added\" },\n",
    "        { \"type\": \"doubleSum\", \"name\": \"deleted\", \"fieldName\": \"deleted\" },\n",
    "        { \"type\": \"thetaSketch\", \"name\": \"theta_user\", \"fieldName\": \"user\" }\n",
    "    ]\n",
    "\n",
    "compaction_dimensionsSpec = {\n",
    "    \"dimensions\": [\n",
    "        \"namespace\",\n",
    "        \"channel\",\n",
    "        \"isRobot\",\n",
    "        \"isAnonymous\",\n",
    "        \"regionIsoCode\",\n",
    "        \"countryIsoCode\"\n",
    "    ] }\n",
    "\n",
    "compaction_spec = {\n",
    "    \"type\": \"compact\",\n",
    "    \"dataSource\": table_name,\n",
    "    \"ioConfig\": compaction_ioConfig,\n",
    "    \"granularitySpec\": compaction_granularitySpec,\n",
    "    \"dimensionsSpec\": compaction_dimensionsSpec,\n",
    "    \"metricsSpec\": compaction_metricsSpec\n",
    "}\n",
    "\n",
    "print(json.dumps(compaction_spec, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a4959a6-3223-4d31-9950-a46218bfad17",
   "metadata": {},
   "source": [
    "Run the cell below to submit the compaction job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "251833eb-106c-48a6-adcd-43deef7a6165",
   "metadata": {},
   "outputs": [],
   "source": [
    "requests.post(f\"{druid_host}/druid/indexer/v1/task\", json.dumps(compaction_spec), headers=druid_headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3a7ab46-49ce-454c-9583-2845d0e7f583",
   "metadata": {},
   "source": [
    "Run the cell below as the process runs.\n",
    "\n",
    "When the compaction is finished you will see that up to 0600, the table contains multiple rows with the same timestamp. After this point, the precision of the table remains the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6daa6af-2ae0-4514-9d8d-6bdd7af75ad7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT\n",
    "  \"__time\",\n",
    "  COUNT(*) AS \"rows\"\n",
    "FROM \"{table_name}\"\n",
    "WHERE TIME_IN_INTERVAL(\"__time\", '2016-06-27T04:30/PT2H')\n",
    "GROUP BY 1\n",
    "LIMIT 10\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6592dc56-eaeb-40dc-ac69-a9af01c1eb38",
   "metadata": {},
   "source": [
    "Take a look at the actual data in the table by running the SQL below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5666aa61-0c6e-4f91-a9d9-861e92de0841",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT *\n",
    "FROM \"{table_name}\"\n",
    "WHERE TIME_IN_INTERVAL(\"__time\", 'PT16M/2016-06-27T06:01')\n",
    "ORDER BY __time DESC\n",
    "LIMIT 20\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44738d6d-cec2-40ad-aaba-998c758c63f4",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "Run the following cell to drop the table from the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8082b545-ba7f-4ede-bb6e-2a6dd62ba0d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "druid.datasources.drop(f\"{table_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54b8d5fe-ba85-4b5b-9669-0dd47dfbccd1",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* Compaction can be run manually or automatically.\n",
    "* Adjustments can be made to specific particular periods of time.\n",
    "* Data can be filtered out and the schema changed.\n",
    "* Rows can be aggregated and metrics emitted.\n",
    "\n",
    "## Learn more\n",
    "\n",
    "* Take a look at more options for [filters](https://druid.apache.org/docs/latest/querying/filters) in `transformSpec` in the documentation and the [native filters notebook](../02-ingestion/14-native-filters.ipynb).\n",
    "* Find the technical details in the documentation about the [`dimensionsSpec`](https://druid.apache.org/docs/latest/querying/dimensionspecs) and walk through examples in the [native dimensions](../02-ingestion/15-native-dimensions.ipynb) notebook.\n",
    "* Read the documentation on [native aggregations](https://druid.apache.org/docs/latest/querying/aggregations) you can add to the `metricsSpec`.\n",
    "* Understand the importance of approximation in the notebooks on [ranking](../03-query/02-approx-ranking.ipynb), [count distinct](../03-query/03-approx-count-distinct.ipynb), and [distribution](../03-query/04-approx-distribution.ipynb).\n",
    "* Learn more about compaction-time [`rollup`](https://druid.apache.org/docs/latest/data-management/compaction#rollup) in the documentation."
   ]
  }
 ],
 "metadata": {
  "execution": {
   "allow_errors": true,
   "timeout": 300
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}