feat: Add Batch Scraping Documentation (#65)

Co-authored-by: Tom Kerkhove <[email protected]>

docs/scraping/batch-scraping.md

@@ -0,0 +1,43 @@
+# Using Batch Scraping with Azure Monitor API (Experimental)
+
+Using batch scraping mode, Promitor will fetch resource metrics in batches by using the Azure Monitor data-plane API instead of Azure Resource Manager (ARM) to remove throttling.
+
+> ⚠️ **The Azure Monitor data-plane API is not free, learn more [in the official pricing documentation](https://azure.microsoft.com/pricing/details/monitor/)**.
+
+Promitor deployments are able to monitor a greater set of unique resources and metrics, in part because its requests 
+go through dedicated APIs that allow for much greater throughput. However, there are important factors to consider(like cost) when
+deploying Promitor this way. 
+
+![Promitor Scraper with resource discovery](./../media/concepts/scraper-batch-scraping-mode.png)
+
+## Why Do Batch Scraping?
+The "traditional" scraping path goes through Azure Resource Management API, which enforces strict [rate limits](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling) unsuitable for the throughput needed to monitor large number of resources or metrics. Dedicated APIs for batch scraping aren't subject to the same ARM rate limiting mechanis, and thus allow for much greater throughput. 
+
+Additionally, there may be performance benefits associated with an overall lower number of networks requests. Though this benefit is only hypothetical for now.
+
+## Limitations  
+At the present moment, batch scraping can only target resources within the same region. Follow this [issue](https://github.com/tomkerkhove/promitor/issues/2555) for support across multiple regions
+
+Additionally, batch scraping does not work for LogAnalytics resources. Follow this [issue](https://github.com/tomkerkhove/promitor/issues/2556) for 
+suppor status. 
+
+## Cost Considerations 
+Under batch scraping mode, all declared metrics will be fetched through requests to
+[Azure Monitor API](https://learn.microsoft.com/en-us/rest/api/monitor/metrics-batch/batch?view=rest-monitor-2023-10-01). Billing for the API is calculated based on the number of requests.  
+
+Query Costs for metrics and logs are laid out [here](https://azure.microsoft.com/pricing/details/monitor/#pricing), under `Platform and custom metric queries (preview)` section. 
+
+To estimate amount of Azure Monitor API requests(queries) a Promitor deployment will make for a length of time `L`, one can use the formula 
+
+$$
+\small {\# of requests} = \text{number of resources} \times \text{number of metrics} \times \left(\frac{L}{\text{scrape interval * batch size}}\right)
+$$
+
+Consider an example deployment that monitors 10 metrics across the same set of 100 resources, with a scrape interval of 5 minutes and configured max batch size of 20. In a given day roughly
+100 * 10 * (1440/(5 * 20)) = 14400 requests will be made. The cost for that deployment can then be calculated using Azure Monitor's pricing chart
+
+## How Are Batches Built
+Batches are constructed using multiple constraints, such as the Azure Monitor metric within a batch must be the same. An important and configurable parameter is the max batch size. Azure Monitor currently(as of Oct 2024) imposes a max batch size of 50 per request, but users are free to impose an even lower limit. 
+
+## Runtime Configuration
+See [Runtime Configuration](runtime-configuration.md) page

docs/scraping/runtime-configuration.md

@@ -176,7 +176,7 @@ metricSinks:
       transformation: None # Optional. Default: None.
 ```
 
-#### What happens when metrics are unavailable for multi-dimensional metrics?
+### What happens when metrics are unavailable for multi-dimensional metrics?
 
 Promitor allows you to use one or more dimension(s) in metrics so that it will report all values.
 
@@ -353,7 +353,10 @@ The behavior of this can be configured to fit your needs:
 - `azureMonitor.integration.history.startingFromInHours` - Defines the amount of hours Promitor will use to define the starting point of the time window used for metric queries.
   - As an example, the default is 12 hours which means Promitor will fetch all metrics between now - 12 hours and now to find a matching metric. Typically this window can be very small but Promitor provides a margin by default to prevent problems for long aggregation periods. (Default: `12`)
 - `azureMonitor.integration.useAzureMonitorSdk` - In the newest release of Promitor, integration with Azure Monitor will use [Azure.Monitor](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/monitor/Azure.Monitor.Query) package under Azure SDK for .NET by default. This migration was needed because the [original SDK](https://github.com/Azure/azure-libraries-for-net) has been deprecated since 2022. 
-
+- `azureMonitor.integration.metricsBatching.enabled` - whether to scrape metrics batch scraping mode (Default: `False`) 
+- `azureMonitor.integration.metricsBatching.azureRegion` - Azure region of batch scrape resource targets. This is required for batch scraping to work
+- `azureMonitor.integration.metricsBatching.maxBatchSize` - Max number of resources allowed in a batch. Azure's limit is 50 but user can set it lower(Default: `50`) 
+
 Example:
 
 ```yaml
@@ -362,6 +365,10 @@ azureMonitor:
     informationLevel: Basic # Optional. Default: Basic
     isEnabled: false # Optional. Default: false
   integration:
+    metricsBatching:
+      enabled: true 
+      maxBatchSize: 25 
+      azureRegion: eastus
     useAzureMonitorSdk: true # Optional. Default: true
     history:
       startingFromInHours: 24 # Optional. Default: 12

mkdocs.yml

@@ -108,6 +108,10 @@ extra:
 extra_css:
 - stylesheets/extra.css
 
+extra_javascript:
+  - javascripts/mathjax.js
+  - https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
+
 # Extensions
 markdown_extensions:
   - admonition
@@ -122,6 +126,8 @@ markdown_extensions:
       emoji_generator: !!python/name:material.extensions.emoji.to_svg
   - toc:
       permalink: true
+  - pymdownx.arithmatex:
+      generic: true
   - pymdownx.highlight:
       anchor_linenums: true
   - pymdownx.inlinehilite
@@ -138,6 +144,7 @@ nav:
     - Overview: scraping/overview.md
     - Metric Labelling: scraping/labels.md
     - Runtime Configuration: scraping/runtime-configuration.md
+    - Use Batch Scraping (Experimental): scraping/batch-scraping.md
     - Available Providers:
       - Generic: scraping/providers/generic-azure-resource.md
       - Azure API Management: scraping/providers/api-management.md