Merge pull request #160 from Azure/feat/wildcard-filter-matching

Add wildcard filter matching and workspace sub-resource filtering

Author: petehauge

docs/commands/extract.md

@@ -95,17 +95,19 @@ For local development, `az login` is the simplest option. For CI/CD pipelines, u
 
 By default, `apiops extract` exports **all** resources from the APIM instance (34 resource types including APIs, products, backends, named values, tags, policies, and more).
 
-To extract only specific resources, pass a YAML filter file with `--filter`:
+To extract only specific resources, pass a YAML filter file with `--filter`. Filter entries support exact names and wildcard patterns (`*` for any characters, `?` for a single character):
 
 ```yaml
 # configuration.extractor.yaml
 apis:
   - echo-api
   - petstore-api
+  - 'prod-*'             # Wildcard: all APIs starting with prod-
 products:
   - starter
 backends:
   - backend-api
+  - '*-internal'        # Wildcard: all backends ending with -internal
 namedValues:
   - api-key
 tags:

docs/getting-started.md

@@ -61,6 +61,7 @@ Extract only the APIs you care about. Create a `filter.yaml`:
 apiNames:
   - pet-store-api
   - user-api
+  - 'staging-*'          # Wildcard: all APIs starting with staging-
 ```
 
 ```bash

docs/guides/filtering-resources.md

@@ -72,11 +72,47 @@ namedValues:
 - Simple fields must be an array of strings
 - `apis` and `workspaces` also accept nested object entries for sub-resource filtering (see below)
 - Names are matched case-insensitively against APIM resource names
+- Wildcard patterns are supported — `*` matches any characters, `?` matches a single character (see below)
+- Exact names and wildcard patterns can be mixed in the same array
 - An empty file extracts everything (same as no filter)
 - An empty array (`[]`) excludes ALL resources of that type
 
 ---
 
+## Wildcard Pattern Matching
+
+Filter entries support glob-style wildcard patterns for matching multiple resources by naming convention:
+
+| Wildcard | Meaning | Example |
+|----------|---------|---------|
+| `*` | Matches zero or more characters | `prod-*` matches `prod-api`, `prod-users` |
+| `?` | Matches exactly one character | `api-v?` matches `api-v1`, `api-v2` but not `api-v10` |
+
+### Examples
+
+```yaml
+apis:
+  - '*-test'          # All APIs ending with -test
+  - 'prod-*'          # All APIs starting with prod-
+  - '*-internal-*'    # All APIs containing -internal-
+  - 'v2-*-api'        # APIs following v2-{name}-api pattern
+  - 'echo-api'        # Exact names and patterns can be mixed
+
+products:
+  - 'test-*'          # All test products
+  - '*-starter'       # All starter tier products
+
+backends:
+  - 'backend-*-prod'  # All production backends
+
+namedValues:
+  - '*-connection-string'  # All connection string named values
+```
+
+Wildcard matching is case-insensitive, just like exact matching. Special characters in resource names (e.g., dots in `my.api.v1`) are treated literally — `my.api.*` matches `my.api.test` but not `myXapiXtest`.
+
+---
+
 ## Nested Sub-Resource Filtering
 
 ### API sub-resource filters
@@ -104,8 +140,6 @@ apis:
 
 ### Workspace sub-resource filters
 
-> **Note:** Workspace sub-resource filtering is parsed but not yet applied at runtime. Currently, including a workspace extracts all resources within it. This matches the Toolkit's configuration format for forward compatibility. See [#119](https://github.com/Azure/apiops-cli/issues/119) for tracking.
-
 The configuration format supports specifying which workspace-scoped resources to extract:
 
 ```yaml
@@ -121,7 +155,7 @@ workspaces:
   - team-b-workspace              # Simple: extract all resources
 ```
 
-Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`, `groups`, `loggers`, `namedValues`, `policyFragments`, `products`, `subscriptions`, `tags`, `versionSets`.
+Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`, `groups`, `loggers`, `namedValues`, `policyFragments`, `products`, `schemas`, `subscriptions`, `tags`, `versionSets`.
 
 ---
 
@@ -268,6 +302,20 @@ backends:
 
 There is no "exclude" syntax. To extract everything except certain resources, list all the resources you _do_ want. For large instances, it's often easier to extract everything and use `.gitignore` or separate branches to manage visibility.
 
+### Pattern-Based Team Filtering
+
+Using wildcard patterns to extract resources by naming convention:
+
+```yaml
+# configuration.extractor.yaml
+apis:
+  - 'team-payments-*'     # All APIs owned by the payments team
+namedValues:
+  - 'payments-*'          # All named values for payments
+backends:
+  - '*-payments-*'        # All backends related to payments
+```
+
 ---
 
 ## Tips
@@ -276,6 +324,7 @@ There is no "exclude" syntax. To extract everything except certain resources, li
 - **One filter per team** — In multi-team setups, each team maintains its own `configuration.extractor.yaml`
 - **Commit the filter file** — Keep it in version control alongside your artifacts so CI/CD pipelines can use it
 - **Case-insensitive matching** — Filter values are matched case-insensitively against APIM resource names
+- **Use wildcard patterns** — `*` and `?` patterns let you match resources by naming convention instead of listing each name individually
 - **Validate early** — The config loader validates filter entries and will throw `Failed to load filter config` on invalid YAML. Unknown top-level keys produce a warning.
 
 ---

src/lib/config-loader.ts

@@ -186,7 +186,7 @@ export async function loadFilterConfig(filePath: string): Promise<FilterConfig |
         if (Object.keys(subFilters).length > 0) {
           config.workspaceSubFilters = {};
           const validWsSubKeys = ['apis', 'backends', 'diagnostics', 'groups', 'loggers',
-            'namedValues', 'policyFragments', 'products', 'subscriptions', 'tags', 'versionSets'] as const;
+            'namedValues', 'policyFragments', 'products', 'schemas', 'subscriptions', 'tags', 'versionSets'] as const;
           for (const [wsName, sf] of Object.entries(subFilters)) {
             const wsSub: WorkspaceSubFilter = {};
             for (const wsField of validWsSubKeys) {

src/models/config.ts

@@ -44,6 +44,7 @@ export interface WorkspaceSubFilter {
   namedValues?: string[];
   policyFragments?: string[];
   products?: string[];
+  schemas?: string[];
   subscriptions?: string[];
   tags?: string[];
   versionSets?: string[];

src/services/filter-service.ts

@@ -199,12 +199,53 @@ export function extractRootApiName(name: string): string {
   return semiIndex >= 0 ? name.substring(0, semiIndex) : name;
 }
 
+/**
+ * Check if a pattern contains wildcard characters (* or ?).
+ */
+export function isWildcardPattern(pattern: string): boolean {
+  return pattern.includes('*') || pattern.includes('?');
+}
+
+/**
+ * Convert a glob-style wildcard pattern to a RegExp.
+ * Supports:
+ *  - `*` matches zero or more characters
+ *  - `?` matches exactly one character
+ * All other characters are escaped for literal matching.
+ * Matching is case-insensitive.
+ */
+export function wildcardToRegex(pattern: string): RegExp {
+  const escaped = pattern.replace(/[.+^${}()|[\]\\]/g, '\\$&');
+  const regexStr = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
+  return new RegExp(`^${regexStr}$`, 'i');
+}
+
+/** Warn once per pattern that looks likely to cause slow matching. */
+const warnedPatterns = new Set<string>();
+
+/**
+ * Match a string against a glob-style wildcard pattern (case-insensitive).
+ * Logs a warning for patterns with many wildcards that may be slow.
+ */
+export function wildcardMatch(pattern: string, text: string): boolean {
+  const starCount = (pattern.match(/\*/g) ?? []).length;
+  if (starCount > 4 && !warnedPatterns.has(pattern)) {
+    warnedPatterns.add(pattern);
+    logger.warn(
+      `Filter pattern "${pattern}" has ${starCount} wildcards and may be slow to evaluate`
+    );
+  }
+  return wildcardToRegex(pattern).test(text);
+}
+
 /**
  * Match a resource name against a filter allowlist.
  *
  * - undefined allowlist → include all (no filter for this type)
  * - empty array → include none
- * - non-empty array → case-insensitive match
+ * - non-empty array → case-insensitive exact match or wildcard pattern match
+ *
+ * Wildcard patterns use `*` (zero or more characters) and `?` (single character).
  */
 function matchesFilter(name: string, allowlist: string[] | undefined): boolean {
   if (allowlist === undefined) {
@@ -220,6 +261,9 @@ function matchesFilter(name: string, allowlist: string[] | undefined): boolean {
   const lowerRoot = extractRootApiName(lowerName);
 
   return allowlist.some((allowed) => {
+    if (isWildcardPattern(allowed)) {
+      return wildcardMatch(allowed, lowerName) || wildcardMatch(allowed, lowerRoot);
+    }
     const lowerAllowed = allowed.toLowerCase();
     return lowerName === lowerAllowed || lowerRoot === lowerAllowed;
   });

src/services/workspace-extractor.ts

@@ -9,12 +9,13 @@ import { IApimClient } from '../clients/iapim-client.js';
 import { IArtifactStore } from '../clients/iartifact-store.js';
 import { ApimServiceContext } from '../models/types.js';
 import { ResourceType, RESOURCE_TYPE_METADATA } from '../models/resource-types.js';
-import { FilterConfig } from '../models/config.js';
+import { FilterConfig, WorkspaceSubFilter } from '../models/config.js';
 import { extractResourceType, ExtractedResource } from './resource-extractor.js';
 import { extractApiResources, extractWorkspaceApiTags } from './api-extractor.js';
 import { extractProductResources, extractWorkspaceProductTags } from './product-extractor.js';
 import { logger } from '../lib/logger.js';
 import { getNamePart } from '../lib/resource-path.js';
+import { isWildcardPattern, wildcardMatch } from './filter-service.js';
 
 /**
  * Types that can exist at the workspace level, derived from RESOURCE_TYPE_METADATA.
@@ -58,17 +59,34 @@ export async function extractWorkspaces(
       logger.debug('Workspace filter is empty array — excluding all workspaces');
       return results;
     }
-    workspaceNames = filter.workspaces;
-  } else {
-    // No workspace filter defined — discover all workspaces
-    const discovered: string[] = [];
-    for await (const item of client.listResources(context, ResourceType.Workspace)) {
-      const name = item['name'];
-      if (typeof name === 'string') {
-        discovered.push(name);
+
+    const hasWildcards = filter.workspaces.some(isWildcardPattern);
+    if (hasWildcards) {
+      // Wildcard patterns require discovery so we can match against real names
+      const discovered = await discoverWorkspaceNames(client, context);
+      workspaceNames = discovered.filter((name) =>
+        filter.workspaces!.some((pattern) =>
+          isWildcardPattern(pattern)
+            ? wildcardMatch(pattern, name)
+            : pattern.toLowerCase() === name.toLowerCase()
+        )
+      );
+
+      // Warn about exact (non-wildcard) entries that didn't match any discovered workspace
+      for (const entry of filter.workspaces) {
+        if (!isWildcardPattern(entry)) {
+          const matched = discovered.some((d) => d.toLowerCase() === entry.toLowerCase());
+          if (!matched) {
+            logger.warn(`Workspace filter entry "${entry}" did not match any discovered workspace`);
+          }
+        }
       }
+    } else {
+      workspaceNames = filter.workspaces;
     }
-    workspaceNames = discovered;
+  } else {
+    // No workspace filter defined — discover all workspaces
+    workspaceNames = await discoverWorkspaceNames(client, context);
   }
 
   if (workspaceNames.length === 0) {
@@ -92,7 +110,8 @@ export async function extractWorkspaces(
     }
 
     const wsResult = await extractWorkspace(
-      client, store, context, wsName, outputDir, filter
+      client, store, context, wsName, outputDir,
+      resolveWorkspaceFilter(wsName, filter)
     );
     results.push(wsResult);
   }
@@ -218,3 +237,69 @@ async function extractWorkspace(
 
   return { workspaceName, resourceCount, errorCount };
 }
+
+/**
+ * Discover all workspace names from APIM.
+ */
+async function discoverWorkspaceNames(
+  client: IApimClient,
+  context: ApimServiceContext
+): Promise<string[]> {
+  const names: string[] = [];
+  for await (const item of client.listResources(context, ResourceType.Workspace)) {
+    const name = item['name'];
+    if (typeof name === 'string') {
+      names.push(name);
+    }
+  }
+  return names;
+}
+
+/**
+ * Resolve the effective FilterConfig for a workspace.
+ * If the workspace has a sub-filter in workspaceSubFilters, convert it to a FilterConfig.
+ * Otherwise return undefined (no filter = extract everything in the workspace).
+ */
+export function resolveWorkspaceFilter(
+  workspaceName: string,
+  filter?: FilterConfig
+): FilterConfig | undefined {
+  if (!filter?.workspaceSubFilters) {
+    return undefined;
+  }
+
+  // Case-insensitive lookup of workspace sub-filter
+  const lowerName = workspaceName.toLowerCase();
+  const matchingKey = Object.keys(filter.workspaceSubFilters).find(
+    (k) => k.toLowerCase() === lowerName
+  );
+
+  if (!matchingKey) {
+    return undefined;
+  }
+
+  const sub = filter.workspaceSubFilters[matchingKey];
+  return workspaceSubFilterToFilterConfig(sub);
+}
+
+/**
+ * Convert a WorkspaceSubFilter to a FilterConfig so the standard
+ * filter-service matching logic can be applied to workspace-scoped resources.
+ */
+function workspaceSubFilterToFilterConfig(sub: WorkspaceSubFilter): FilterConfig {
+  return {
+    apis: sub.apis,
+    apiSubFilters: sub.apiSubFilters,
+    backends: sub.backends,
+    diagnostics: sub.diagnostics,
+    groups: sub.groups,
+    loggers: sub.loggers,
+    namedValues: sub.namedValues,
+    policyFragments: sub.policyFragments,
+    products: sub.products,
+    schemas: sub.schemas,
+    subscriptions: sub.subscriptions,
+    tags: sub.tags,
+    versionSets: sub.versionSets,
+  };
+}

src/templates/configs/filter-config.ts

@@ -11,10 +11,12 @@ export function generateFilterConfig(): string {
 # For full format details and examples, see:
 # https://github.com/Azure/apiops-cli/blob/main/docs/guides/filtering-resources.md
 
-# Extract only specific APIs by name
+# Extract only specific APIs by name (or wildcard pattern)
 # apis:
 #   - echo-api
 #   - petstore-api
+#   - 'prod-*'                       # Wildcard: all APIs starting with prod-
+#   - '*-internal-*'                 # Wildcard: all APIs containing -internal-
 
 # Advanced: Filter API sub-resources (operations, diagnostics, schemas, releases)
 # apis:
@@ -23,6 +25,7 @@ export function generateFilterConfig(): string {
 #       operations:
 #         - get-pets
 #         - create-pet
+#         - 'list-*'                  # Wildcard: all operations starting with list-
 #       diagnostics:
 #         - applicationinsights
 #       schemas: []                   # Exclude all schemas
@@ -116,5 +119,9 @@ export function generateFilterConfig(): string {
 #   Example:
 #   gateways: []
 #   subscriptions: []
+# - Use * to match any characters: prod-* matches prod-api, prod-users
+# - Use ? to match a single character: api-v? matches api-v1, api-v2
+# - Exact names and wildcard patterns can be mixed in the same list
+# - All matching is case-insensitive
 `;
 }