notebooks/03-query/14-query-async-realtime.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0cb3b009-ebde-4d56-9d59-a028d66d8309",
   "metadata": {},
   "source": [
    "# Run a query offline on streaming, historical, and deep-storage data using the asynchronous query API\n",
    "\n",
    "<!--\n",
    "  ~ Licensed to the Apache Software Foundation (ASF) under one\n",
    "  ~ or more contributor license agreements.  See the NOTICE file\n",
    "  ~ distributed with this work for additional information\n",
    "  ~ regarding copyright ownership.  The ASF licenses this file\n",
    "  ~ to you under the Apache License, Version 2.0 (the\n",
    "  ~ \"License\"); you may not use this file except in compliance\n",
    "  ~ with the License.  You may obtain a copy of the License at\n",
    "  ~\n",
    "  ~   http://www.apache.org/licenses/LICENSE-2.0\n",
    "  ~\n",
    "  ~ Unless required by applicable law or agreed to in writing,\n",
    "  ~ software distributed under the License is distributed on an\n",
    "  ~ \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
    "  ~ KIND, either express or implied.  See the License for the\n",
    "  ~ specific language governing permissions and limitations\n",
    "  ~ under the License.\n",
    "  -->\n",
    "\n",
    "Druid provides two APIs to run SELECT queries: the interactive API and the asynchronous API.\n",
    "\n",
    "The interactive query API uses data pre-cached on Historical services and data arriving from event streams. However the asynchronous query API accesses data in deep storage in combination with streaming data.\n",
    "\n",
    "This tutorial focuses on using the asynchronous API to access data in [deep storage](https://druid.apache.org/docs/latest/api-reference/sql-api#query-from-deep-storage) in combination with data from [real-time ingestion](14-query-async-realtime.ipynb).\n",
    "\n",
    "To work through examples that focus on historical data, including examples of result pagination, see the notebook on [historical data query](21-query-async-historical).\n",
    "\n",
    "In this tutorial you perform the following tasks:\n",
    "\n",
    "- Generate three months of data and use batch ingestion to load it.\n",
    "- Set up stream ingestion for live data from generator.\n",
    "- Set up Retention rules to only hold 1 month of data in Historical cache.\n",
    "- Use synchronous and asynchronous queries to show timeline coverage of each case."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbdbf6ad-ca7b-40f5-8ca3-1070f4a3ee42",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "This tutorial works with Druid 29.0.0 or later.\n",
    "\n",
    "#### Run with Docker\n",
    "\n",
    "Launch this tutorial and all prerequisites using the `all-services` profile of the Docker Compose file for Jupyter-based Druid tutorials. For more information, see the Learn Druid repository [readme](https://github.com/implydata/learn-druid).\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5007a243-b81a-4601-8f57-5b14940abbff",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "The following cells set up the notebook and learning environment ready for use.\n",
    "\n",
    "### Set up and connect to the learning environment\n",
    "\n",
    "Run the next cell to set up the Druid Python client's connection to Apache Druid.\n",
    "\n",
    "If successful, the Druid version number will be shown in the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ec783b-df3f-4168-9be2-cdc6ad3e33c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import druidapi\n",
    "import os\n",
    "import requests\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "import time\n",
    "\n",
    "druid_headers = {'Content-Type': 'application/json'}\n",
    "\n",
    "if 'DRUID_HOST' not in os.environ.keys():\n",
    "    druid_host=f\"http://localhost:8888\"\n",
    "else:\n",
    "    druid_host=f\"http://{os.environ['DRUID_HOST']}:8888\"\n",
    "\n",
    "print(f\"Opening a connection to {druid_host}.\")\n",
    "druid = druidapi.jupyter_client(druid_host)\n",
    "display = druid.display\n",
    "sql_client = druid.sql\n",
    "status_client = druid.status\n",
    "\n",
    "status_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "764f1171-b44e-416e-83c6-6955fba9d17c",
   "metadata": {},
   "source": [
    "### Set up a connection to Apache Kafka\n",
    "\n",
    "Run the next cell to set up the connection to Apache Kafka."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ce6a49a-05ec-4f47-8dcf-e9816832e041",
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'KAFKA_HOST' not in os.environ.keys():\n",
    "   kafka_host=f\"http://localhost:9092\"\n",
    "else:\n",
    "    kafka_host=f\"{os.environ['KAFKA_HOST']}:9092\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dfaa6c5-4221-495a-b021-2b1b35b3b3fc",
   "metadata": {},
   "source": [
    "### Set up a connection to the data generator\n",
    "\n",
    "Run the next cell to set up the connection to the data generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9759f4e6-5326-43a3-a13f-b299ea0395a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "datagen_host = \"http://datagen:9999\"\n",
    "datagen_headers = {'Content-Type': 'application/json'}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "154c9339-3408-4597-89ca-6cc6902696a2",
   "metadata": {},
   "source": [
    "## Create a table using streaming ingestion\n",
    "\n",
    "In this section, you use the data generator to generate a stream of messages into a Apache Kafka topic and ingest it into a table in Druid.\n",
    "\n",
    "The data generator configuration here produces clickstream data to Kafka starting 12 days ago (`time_type`, set from `datagen_time_type`) for a duration of 10 days (`time`).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9302901-72b7-46be-8a5e-25a892afdc05",
   "metadata": {},
   "outputs": [],
   "source": [
    "datagen_topic = \"example-clickstream-async\"\n",
    "datagen_job = datagen_topic\n",
    "datagen_config = \"clickstream/clickstream.json\"\n",
    "datagen_time_type = (datetime.now() - timedelta(days=12)).strftime(\"%Y-%m-%d %H:%M:%S\")\n",
    "\n",
    "datagen_request = {\n",
    "    \"name\": datagen_job,\n",
    "    \"target\": { \"type\": \"kafka\", \"endpoint\": kafka_host, \"topic\": datagen_topic  },\n",
    "    \"config_file\": datagen_config,\n",
    "    \"time\":\"240h\",\n",
    "    \"time_type\": datagen_time_type,\n",
    "    \"concurrency\": 1\n",
    "}\n",
    "\n",
    "requests.post(f\"{datagen_host}/start\", json.dumps(datagen_request), headers=datagen_headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f9962e-b89b-44f8-970c-b8ec767a08bb",
   "metadata": {},
   "source": [
    "Run the following cell to wait until the data generator has finished publishing events to Kafka."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57fc8e9b-9e28-4092-8860-0c66896c53e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "datagen_done = False\n",
    "\n",
    "while not datagen_done:\n",
    "    result = requests.get(f\"{datagen_host}/status/{datagen_job}\").json()\n",
    "    if result[\"status\"] == 'COMPLETE':\n",
    "        datagen_done = True\n",
    "    else:\n",
    "        time.sleep(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d01aa96a-e115-4963-8711-73e16b2abf12",
   "metadata": {},
   "source": [
    "### Use streaming ingestion to populate the table\n",
    "\n",
    "Ingest data from an Apache Kafka topic into Apache Druid by submitting an [ingestion specification](https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html) to the [streaming ingestion supervisor API](https://druid.apache.org/docs/latest/api-reference/supervisor-api).\n",
    "\n",
    "Run the next cell to set up the [`ioConfig`](https://druid.apache.org/docs/latest/ingestion/ingestion-spec#ioconfig), [`tuningConfig`](https://druid.apache.org/docs/latest/ingestion/ingestion-spec#tuningconfig), and [`dataSchema`](https://druid.apache.org/docs/latest/ingestion/ingestion-spec#dataschema). Notice that the specification:\n",
    "\n",
    "* Begins reading from the beginning of the stream (`useEarliestOffset`) since this is the [first time](https://druid.apache.org/docs/latest/ingestion/kafka-ingestion#io-configuration) this topic has been read.\n",
    "* Has a primary partitioning (`segmentGranularity`) of `DAY`.\n",
    "* Uses automatic schema discovery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "816ea608-9def-4f35-b13a-6723d33618df",
   "metadata": {},
   "outputs": [],
   "source": [
    "ioConfig = {\n",
    "    \"type\": \"kafka\",\n",
    "    \"consumerProperties\": { \"bootstrap.servers\": kafka_host },\n",
    "    \"topic\": datagen_topic,\n",
    "    \"inputFormat\": { \"type\": \"json\" },\n",
    "    \"useEarliestOffset\": \"true\" }\n",
    "\n",
    "tuningConfig = { \"type\": \"kafka\" }\n",
    "\n",
    "table_name = datagen_topic\n",
    "\n",
    "dataSchema = {\n",
    "    \"dataSource\": table_name,\n",
    "    \"timestampSpec\": { \"column\": \"time\", \"format\": \"iso\" },\n",
    "    \"granularitySpec\": { \"rollup\": \"false\", \"segmentGranularity\": \"day\" },\n",
    "    \"dimensionsSpec\": { \"useSchemaDiscovery\" : \"true\"}\n",
    "    }\n",
    "\n",
    "ingestion_spec = {\n",
    "    \"type\": \"kafka\",\n",
    "    \"spec\": {\n",
    "        \"ioConfig\": ioConfig,\n",
    "        \"tuningConfig\": tuningConfig,\n",
    "        \"dataSchema\": dataSchema\n",
    "    }\n",
    "}\n",
    "\n",
    "requests.post(f\"{druid_host}/druid/indexer/v1/supervisor\", json.dumps(ingestion_spec), headers=druid_headers)\n",
    "sql_client.wait_until_ready(table_name, verify_load_status=False)\n",
    "display.table(table_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "966930ee-d178-459b-9897-b6412ed814bf",
   "metadata": {},
   "source": [
    "[Terminating or suspending](https://druid.apache.org/docs/latest/ingestion/supervisor#manage-a-supervisor) a supervisor will cause a [handoff](https://druid.apache.org/docs/latest/design/storage#indexing-and-handoff) from the indexers to deep storage, and from there to historicals.\n",
    "\n",
    "Run the following cell to pause execution of this notebook for 5 seconds before terminating the supervisor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eb6a35f-4c15-49a3-ab5c-5afb8aab8d6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(5)\n",
    "requests.post(f\"{druid_host}/druid/indexer/v1/supervisor/{table_name}/terminate\", headers=druid_headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dd5403-9724-4c2f-ab47-de7bf1cd5e29",
   "metadata": {},
   "source": [
    "Run the following cell to see how the table is currently loaded in the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9c3b105-dbe6-4807-8da1-2a3276f736c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT\n",
    "  a.\"start\",\n",
    "  a.\"end\",\n",
    "  c.\"server\",\n",
    "  c.\"tier\",\n",
    "  c.\"server_type\"\n",
    "FROM \"sys\".\"segments\" a\n",
    "LEFT JOIN \"sys\".\"server_segments\" b ON a.\"segment_id\" = b.\"segment_id\"\n",
    "LEFT JOIN \"sys\".\"servers\" c ON b.\"server\" = c.\"server\"\n",
    "WHERE \"datasource\" = '{table_name}'\n",
    "ORDER BY \"start\", \"tier\"\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce0729c4-7736-4b37-8fa0-ed84101ed40d",
   "metadata": {},
   "source": [
    "Run the cell above a number of times.\n",
    "\n",
    "You will see how the data remains advertised on the ingestion tasks (\"peons\"), is loaded to historicals, and then eventually stops being advertised on ingestion tasks.\n",
    "\n",
    "The default set of retention rules (`_default`) currently applies to the table, with the entire set of segments loaded to historicals."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca9cf99-ec07-4a40-9d14-47d982319c80",
   "metadata": {},
   "source": [
    "## Apply retention rules\n",
    "\n",
    "Force some of the data to be left on deep storage by changing the retention load rules for a table.\n",
    "\n",
    "The retention rules in the next cell ensure Druid loads data younger than 2 days (`period`)  to Historical services in the `_default_tier` tier. Druid applies the second rule to all remaining data which [leaves data on deep storage](https://druid.apache.org/docs/latest/querying/query-deep-storage#keep-segments-in-deep-storage-only):\n",
    "\n",
    "* `tieredReplicants` is empty.\n",
    "* `useDefaultTierForNull` is set to false.\n",
    "\n",
    "Run the cell to apply the rule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c922dc1-3b2b-4132-922d-e380407dd577",
   "metadata": {},
   "outputs": [],
   "source": [
    "retention_rules = [\n",
    "  {\n",
    "    \"type\": \"loadByPeriod\",\n",
    "    \"period\": \"P1W\",\n",
    "    \"tieredReplicants\": {\n",
    "      \"_default_tier\": 1\n",
    "    }\n",
    "  },\n",
    "  {\n",
    "    \"type\": \"loadForever\",\n",
    "    \"tieredReplicants\": {},\n",
    "    \"useDefaultTierForNull\": \"false\"\n",
    "  }\n",
    "]\n",
    "\n",
    "requests.post(f\"{druid_host}/druid/coordinator/v1/rules/{table_name}\", json.dumps(retention_rules), headers=druid_headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aabea50b-bb3a-496b-a3b0-2c256922a446",
   "metadata": {},
   "source": [
    "Now run the next cell to confirm the change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "deb19262-8f61-4ffb-be9d-e10d5ddc267b",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT\n",
    "  a.\"start\",\n",
    "  a.\"end\",\n",
    "  c.\"server\",\n",
    "  c.\"tier\"\n",
    "FROM \"sys\".\"segments\" a\n",
    "LEFT JOIN \"sys\".\"server_segments\" b ON a.\"segment_id\" = b.\"segment_id\"\n",
    "LEFT JOIN \"sys\".\"servers\" c ON b.\"server\" = c.\"server\"\n",
    "WHERE \"datasource\" = '{table_name}'\n",
    "ORDER BY \"start\", \"tier\"\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc00ce4b-c7b6-4353-912b-bbd60d60a1b9",
   "metadata": {},
   "source": [
    "Run the cell above until some segments are shown without a server and tier, indicating that those segments exist in the database, but are not cached on historicals for query through the interactive API."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05078320-e760-4556-9746-4937b0335ace",
   "metadata": {},
   "source": [
    "## Execute an asynchronous query\n",
    "\n",
    "Use the `/druid/v2/sql/statements` API endpoint to run asynchronous queries using MSQ engine.\n",
    "\n",
    "Run the following cell, which uses the `async_sql` method of the `druid_api` package to call the API and return the results. The `async_sql` method handles the [necessary steps](https://druid.apache.org/docs/latest/tutorials/tutorial-query-deep-storage#query-from-deep-storage) not only to submit the query, but to retrieve the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2824a1b7-b374-4464-b5b6-2dc17a75fcd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT\n",
    "  TIME_FLOOR(\"__time\",'P1D') AS \"period\",\n",
    "  COUNT(*) as \"events\"\n",
    "FROM \"{table_name}\"\n",
    "GROUP BY 1\n",
    "'''\n",
    "\n",
    "req = sql_client.sql_request(sql)\n",
    "result = sql_client.async_sql(req)\n",
    "result.wait_until_done()\n",
    "\n",
    "print(json.dumps(result.rows, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adbe3374-3487-4244-94ab-59c88995f0b5",
   "metadata": {},
   "source": [
    "Compare this to the faster set of results retrieved from the interactive Druid SQL API, which only includes segments that are cached on historicals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35ae2d8f-2510-4ff3-9662-e95bf05bcf11",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT\n",
    "  TIME_FLOOR(\"__time\",'P1D') AS \"period\",\n",
    "  COUNT(*) as \"events\"\n",
    "FROM \"{table_name}\"\n",
    "GROUP BY 1\n",
    "'''\n",
    "\n",
    "display.sql(sql)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69e92ec3-a8a9-4ba7-b35a-60cb83f5819c",
   "metadata": {},
   "source": [
    "To include results from streaming ingestion, include `includeSegmentSource` in the POST to the API.\n",
    "\n",
    "Run the following cell to resume the streaming of events into the Kafka topic and restart the supervisor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "682170c0-2fbe-4f7f-adfe-def6e15f62d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "datagen_time_type = 'REAL'\n",
    "\n",
    "datagen_request = {\n",
    "    \"name\": datagen_job,\n",
    "    \"target\": { \"type\": \"kafka\", \"endpoint\": kafka_host, \"topic\": datagen_topic  },\n",
    "    \"config_file\": datagen_config,\n",
    "    \"time\":\"15m\",\n",
    "    \"time_type\": datagen_time_type,\n",
    "    \"concurrency\": 100\n",
    "}\n",
    "\n",
    "requests.post(f\"{datagen_host}/start\", json.dumps(datagen_request), headers=datagen_headers)\n",
    "\n",
    "requests.post(f\"{druid_host}/druid/indexer/v1/supervisor\", json.dumps(ingestion_spec), headers=druid_headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc2d8cff-7775-4c48-9688-22e9eb81ff29",
   "metadata": {},
   "source": [
    "Submit the query to the asynchronous API again by running the next cell.\n",
    "\n",
    "This time, notice that `includeSegmentSource` is set in the context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8761d86c-98d8-4d99-b7ad-fa6c1a2e8aed",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql=f'''\n",
    "SELECT\n",
    "  TIME_FLOOR(\"__time\",'P1D') AS \"period\",\n",
    "  COUNT(*) as \"events\"\n",
    "FROM \"{table_name}\"\n",
    "GROUP BY 1\n",
    "'''\n",
    "\n",
    "req = sql_client.sql_request(sql)\n",
    "req.add_context(\"includeSegmentSource\", \"realtime\")\n",
    "result = sql_client.async_sql(req)\n",
    "result.wait_until_done()\n",
    "\n",
    "print(json.dumps(result.rows, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9733c153-9a66-4ef8-889a-fe4b6f62c185",
   "metadata": {},
   "source": [
    "The result from the query covers the full time-line of data available, the ingested data available in deep storage and data being streamed into the table continuously."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44738d6d-cec2-40ad-aaba-998c758c63f4",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "Run the following cell to stop the running data generator, delete the supervisor, reset the retention rules from the table, and drop the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8082b545-ba7f-4ede-bb6e-2a6dd62ba0d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "requests.post(f\"{datagen_host}/stop/{datagen_job}\", '').json()\n",
    "\n",
    "print(f\"Stop streaming generator: [{requests.post(f'{datagen_host}/stop/{datagen_job}','')}]\")\n",
    "print(f'Pause streaming ingestion: [{requests.post(f\"{druid_host}/druid/indexer/v1/supervisor/{datagen_topic}/suspend\",\"\")}]')\n",
    "\n",
    "print(f'Shutting down running tasks ...')\n",
    "\n",
    "tasks = druid.tasks.tasks(state='running', table=table_name)\n",
    "while len(tasks)>0:\n",
    "    for task in tasks:\n",
    "        print(f\"...stopping task [{task['id']}]\")\n",
    "        druid.tasks.shut_down_task(task['id'])\n",
    "    tasks = druid.tasks.tasks(state='running', table=table_name)\n",
    "\n",
    "print(f'Reset offsets for re-runnability: [{requests.post(f\"{druid_host}/druid/indexer/v1/supervisor/{datagen_topic}/reset\",\"\")}]')\n",
    "print(f'Terminate streaming ingestion: [{requests.post(f\"{druid_host}/druid/indexer/v1/supervisor/{datagen_topic}/terminate\",\"\")}]')\n",
    "\n",
    "retention_rules = []\n",
    "requests.post(f\"{druid_host}/druid/coordinator/v1/rules/{table_name}\", json.dumps(retention_rules), headers=druid_headers)\n",
    "\n",
    "print(f\"Drop table: [{druid.datasources.drop(table_name)}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54b8d5fe-ba85-4b5b-9669-0dd47dfbccd1",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* SELECT operations can run online (interactive API) or offline (asynchronous API).\n",
    "* The offline API can access data that has not been prefetched to historicals.\n",
    "* The same API can also access data currently being ingested from streams.\n",
    "\n",
    "## Learn more\n",
    "\n",
    "* Read about [using EXTERN to export data](https://druid.apache.org/docs/latest/multi-stage-query/reference#extern-to-export-to-a-destination).\n",
    "* See the [documentation](https://druid.apache.org/docs/latest/querying/query-deep-storage) and [tutorial](https://druid.apache.org/docs/latest/tutorials/tutorial-query-deep-storage) on querying from deep storage.\n",
    "* Read about [durable storage](https://druid.apache.org/docs/latest/multi-stage-query/reference#durable-storage) in the documentation, including how to [configure](https://druid.apache.org/docs/latest/operations/durable-storage) it.\n",
    "* See the [historical async query](21-query-async-historical.ipynb) notebook to examples of the API being used directly, and of using pagination in results.\n",
    "* See more retention load rules in the [load rules](20-tiering-historicals.ipynb) notebook."
   ]
  }
 ],
 "metadata": {
  "execution": {
   "allow_errors": true,
   "timeout": 300
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}