feat: Fargate-based forensic analysis infrastructure by Stealinglight · Pull Request #10 · Stealinglight/Snapshot-Sleuth

Stealinglight · 2026-01-25T09:09:20Z

Summary

Migrates EC2-based forensic analysis to ephemeral AWS Fargate tasks for improved performance, security, and cost efficiency.

Key benefits:

Cold start reduction from 5-15 minutes to under 60 seconds
Ephemeral containers eliminate cross-case contamination
Pay-per-use cost model vs always-on EC2 instances

Changes

Infrastructure (CDK)

fargate-forensics-stack.ts - ECS cluster, ECR repos, task definitions with resource allocation matrix
forensics-vpc-stack.ts - VPC with endpoints for private networking (no NAT gateway)
snapshot-sleuth-stack.ts - Step Functions workflow for parallel tool orchestration

Forensic Tool Containers

forensics-base/ - Base image with SnapshotReader abstraction, heartbeat/progress emitter
yara-tool/ - YARA 4.5.0 scanner with rule management
clamav-tool/ - ClamAV malware scanner (optional/non-critical)
evidence-miner-tool/ - Artifact extraction (credentials, SSH keys, cron jobs)
log2timeline-tool/ - Timeline analysis with Plaso

Shared Libraries

types/fargate.ts - TypeScript types with MITRE ATT&CK mapping
config/resource-profiles.ts - Resource allocation matrix (CPU/memory/timeout by snapshot size)

Lambda

aggregation/index.ts - Result aggregation with normalized finding schema

CI/CD

docker-build.yml - GitHub Actions workflow with Trivy scanning and ECR push
scripts/build-forensics-images.sh - Local build script

Test plan

TypeScript compilation passes (bun run build)
Docker images build locally (./scripts/build-forensics-images.sh)
Deploy to staging environment
Run against demo scenarios (malware, exfil, privesc, clean)
Compare results with current EC2-based output
Performance benchmarking (cold start, total time)

Migrate EC2-based forensic analysis to ephemeral AWS Fargate tasks for improved performance, security, and cost efficiency. Key additions: - Forensic tool containers (YARA, ClamAV, EvidenceMiner, Log2Timeline) - Base forensics image with SnapshotReader abstraction for EBS access - Heartbeat and progress reporting via EventBridge - Step Functions workflow orchestrating parallel tool execution - VPC infrastructure with endpoints for private networking - Result aggregation Lambda with normalized finding schema - Resource allocation matrix scaling CPU/memory by snapshot size - CI workflow for Docker image builds with Trivy scanning - Shared TypeScript types and config for resource profiles Architecture highlights: - Ephemeral containers eliminate cross-case contamination - Strategy pattern enables future EBS Direct API migration - Critical/optional tool classification for error handling - MITRE ATT&CK mapping in normalized findings

Add g++ compiler to forensics-base image for building pytsk3 and libewf-python wheels that require C++14 support. Update CodeQL upload-sarif action from deprecated v2 to v3.

Copilot

Pull request overview

Migrates Snapshot Sleuth’s forensic execution from long-lived EC2 to an orchestrated, container-based workflow intended to run forensic tools as ephemeral tasks, with shared schemas/config and CI to build/publish tool images.

Changes:

Adds new CDK stacks to provision a private forensics VPC, ECS/Fargate resources, and a Step Functions workflow orchestration.
Introduces a Python “forensics-base” runtime plus multiple tool containers (YARA, ClamAV, Evidence Miner, Log2Timeline) and a TS Lambda result aggregation handler.
Adds shared TypeScript schemas/types and resource profile logic, plus CI + local scripts to build/push images and new HTML documentation.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 55 comments.

Show a summary per file

File	Description
scripts/build-forensics-images.sh	Local build/push script for forensic container images.
packages/shared/src/types/index.ts	Exports new shared Fargate forensic types.
packages/shared/src/types/fargate.ts	Adds normalized result/finding/workflow/task type definitions.
packages/shared/src/config/resource-profiles.ts	Adds CPU/memory/timeout resource allocation utilities per tool.
packages/shared/src/config/index.ts	Exports resource profile config helpers.
packages/lambda-ts/src/handlers/index.ts	Exposes new aggregation Lambda handler entrypoint.
packages/lambda-ts/src/handlers/aggregation/index.ts	Implements aggregation of per-tool results into a case summary in S3.
packages/lambda-py/yara-tool/tools/yara_tool.py	Implements YARA-based scanning tool logic.
packages/lambda-py/yara-tool/tools/init.py	Exports YaraTool package entrypoint.
packages/lambda-py/yara-tool/requirements.txt	Declares YARA tool Python dependencies.
packages/lambda-py/yara-tool/Dockerfile	Defines YARA tool container image.
packages/lambda-py/log2timeline-tool/tools/log2timeline_tool.py	Implements Plaso/Log2Timeline timeline extraction and analysis.
packages/lambda-py/log2timeline-tool/tools/init.py	Exports Log2TimelineTool package entrypoint.
packages/lambda-py/log2timeline-tool/requirements.txt	Declares Log2Timeline tool Python dependencies.
packages/lambda-py/log2timeline-tool/Dockerfile	Defines Log2Timeline tool container image.
packages/lambda-py/forensics-base/src/snapshot_reader.py	Adds snapshot reader abstraction and mounted EBS reader implementation.
packages/lambda-py/forensics-base/src/result_handler.py	Adds normalized result schema + S3 upload helpers for tool outputs.
packages/lambda-py/forensics-base/src/progress.py	Adds progress reporting integrated with heartbeat emission.
packages/lambda-py/forensics-base/src/heartbeat.py	Adds EventBridge heartbeat emitter for task progress monitoring.
packages/lambda-py/forensics-base/src/init.py	Exposes forensics-base library public API.
packages/lambda-py/forensics-base/requirements.txt	Declares forensics-base Python dependencies.
packages/lambda-py/forensics-base/entrypoint.py	Adds common container entrypoint wrapper to run tools + emit heartbeats + upload results.
packages/lambda-py/forensics-base/Dockerfile	Defines the shared forensic base image.
packages/lambda-py/evidence-miner-tool/tools/evidence_miner_tool.py	Implements artifact extraction/heuristic checks for credentials/persistence, etc.
packages/lambda-py/evidence-miner-tool/tools/init.py	Exports EvidenceMinerTool package entrypoint.
packages/lambda-py/evidence-miner-tool/requirements.txt	Declares Evidence Miner Python dependencies.
packages/lambda-py/evidence-miner-tool/Dockerfile	Defines Evidence Miner tool container image.
packages/lambda-py/clamav-tool/tools/clamav_tool.py	Implements ClamAV-based malware scanning tool logic.
packages/lambda-py/clamav-tool/tools/init.py	Exports ClamavTool package entrypoint.
packages/lambda-py/clamav-tool/requirements.txt	Declares ClamAV tool Python dependencies.
packages/lambda-py/clamav-tool/Dockerfile	Defines ClamAV tool container image.
packages/cdk/lib/stacks/snapshot-sleuth-stack.ts	Updates main stack to orchestrate Step Functions + ECS tasks + new Lambdas.
packages/cdk/lib/stacks/index.ts	Exports new stacks.
packages/cdk/lib/stacks/forensics-vpc-stack.ts	Adds isolated VPC + endpoints setup for private task networking.
packages/cdk/lib/stacks/fargate-forensics-stack.ts	Adds ECS cluster, ECR repos, and task definitions for tool execution.
packages/cdk/lib/constructs/storage.ts	Adds signatures bucket for tool rules/definitions and related helpers.
docs/Snapshot_Sleuth_Documentation.html	Adds architecture/product documentation page.
docs/2.0_Roadmap.html	Adds roadmap/architecture documentation page.
.github/workflows/docker-build.yml	Adds CI workflow to build/scan/push forensics Docker images.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-25T09:19:07Z

+    // Create or use Fargate stack
+    const fargateStack = props?.fargateStack ?? new FargateForensicsStack(this, 'FargateStack', {
+      projectName,
+      environment,
+      vpc: vpcStack.vpc,
+      securityGroup: vpcStack.forensicsSecurityGroup,
+      subnets: vpcStack.privateSubnets,
+      evidenceBucket: this.storage.evidenceBucket,
+      signaturesBucket: this.storage.signaturesBucket,
+      kmsKey: encryptionKey,
+    });


Similarly, this instantiates FargateForensicsStack (a cdk.Stack) inside another stack. This is likely to break synthesis/deployment unless it’s converted to a NestedStack or refactored into a Construct used by the parent stack.

Copilot · 2026-01-25T09:19:08Z

+      const taskDef = new ecs.FargateTaskDefinition(this, `${tool}TaskDef`, {
+        family: `${this.projectName}-${tool}-${this.envName}`,
+        cpu: config.baseCpu,
+        memoryLimitMiB: config.baseMemoryMb,
+        executionRole: this.executionRole,
+        taskRole,
+        // Enable EFS/EBS volume mounting
+        volumes: [
+          {
+            name: 'evidence',
+            // EBS volume will be configured at runtime via Step Functions
+          },
+        ],
+      });


The task definition declares a volumes: [{ name: 'evidence' }] entry without any EFS configuration. This creates an empty ephemeral Docker volume, not a mount of the snapshot/volume created in the workflow. As written, /mnt/evidence in the container won’t contain the snapshot data, so tools that expect a mounted filesystem will fail or scan nothing. Use a supported shared storage mechanism (e.g., EFS) or switch the reader implementation to an API-based approach that doesn’t require a mount.

Copilot · 2026-01-25T09:19:08Z

+          result: stepfunctions.Result.fromObject({
+            toolName: tool,
+            status: 'failed',
+          }),


In the Parallel branch error handler, the ${Tool}Failed Pass state uses result without a resultPath, so it replaces the entire branch state with { toolName, status }. Successful branches keep the full input plus taskResult. This makes the RunForensicTools output array heterogenous, which will complicate/ break downstream aggregation expecting a consistent ToolExecutionResult shape. Set an explicit resultPath (and/or outputPath) so both success and failure branches emit the same schema.

Suggested change

}),

}),

// Ensure failure branch preserves overall state shape and aligns with success resultPath

resultPath: '$.taskResult',

Copilot · 2026-01-25T09:19:08Z

+# ClamAV Tool Container
+# Extends forensics-base with ClamAV antivirus scanning
+
+FROM forensics-base:latest


Base image reference is inconsistent with the build pipeline: this uses FROM forensics-base:latest, but CI/local builds tag the base as snapshot-sleuth/forensics-base:<tag>. Align this FROM line with the actual base image name/tag (or ensure the base is also tagged as forensics-base:latest).

Suggested change

FROM forensics-base:latest

ARG FORENSICS_BASE_TAG=latest

FROM snapshot-sleuth/forensics-base:${FORENSICS_BASE_TAG}

Copilot · 2026-01-25T09:19:19Z

+        except Exception:
+            pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

except Exception as e:

self._logger.warning(

"Could not check file for credentials",

file_path=file_path,

error=str(e),

)

Copilot · 2026-01-25T09:19:19Z

+                except (OSError, PermissionError):
+                    pass
+
+            for f in files:
+                try:
+                    entry = self._path_to_entry(root_path / f, rel_root)
+                    file_entries.append(entry)
+                except (OSError, PermissionError):
+                    pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except (OSError, PermissionError):

pass

for f in files:

try:

entry = self._path_to_entry(root_path / f, rel_root)

file_entries.append(entry)

except (OSError, PermissionError):

pass

except (OSError, PermissionError) as exc:

# Best-effort traversal: skip directories we cannot access, but log for observability.

logger.debug(

"Skipping directory during snapshot walk due to access error",

path=str(root_path / d),

error=str(exc),

)

for f in files:

try:

entry = self._path_to_entry(root_path / f, rel_root)

file_entries.append(entry)

except (OSError, PermissionError) as exc:

# Best-effort traversal: skip files we cannot access, but log for observability.

logger.debug(

"Skipping file during snapshot walk due to access error",

path=str(root_path / f),

error=str(exc),

)

Copilot · 2026-01-25T09:19:19Z

+                try:
+                    entry = self._path_to_entry(root_path / f, rel_root)
+                    file_entries.append(entry)
+                except (OSError, PermissionError):


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-01-25T09:19:19Z

+            except OSError:
+                pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except OSError:

pass

except OSError as exc:

# Best-effort: if the symlink target cannot be read (for example due

# to permissions or a broken link), keep the default symlink_target

# value but log the failure for observability.

logger.debug(

"Failed to read symlink target",

path=str(path),

error=str(exc),

)

Copilot · 2026-01-25T09:19:20Z

+    curl -sSL https://github.com/VirusTotal/yara/archive/refs/tags/v${YARA_VERSION}.tar.gz | tar xz && \
+    cd yara-${YARA_VERSION} && \
+    ./bootstrap.sh && \
+    ./configure --enable-cuckoo --enable-magic --enable-dotnet && \
+    make && \
+    make install && \
+    ldconfig && \
+    cd / && rm -rf /tmp/yara-${YARA_VERSION}


The YARA Dockerfile downloads and builds YARA from a remote GitHub tarball via curl without any integrity verification, which opens a supply-chain risk if the download is tampered with or the transport/channel is compromised. An attacker who can influence the GitHub tarball or the network could inject arbitrary code into the built binary and gain code execution inside this container with the task’s IAM role and data access. Add strong integrity verification for the downloaded archive (e.g., pinned checksum or signature validation) or switch to a package/source distribution mechanism that provides built-in signature/checksum verification.

Remove unused TypeScript imports and prefix intentionally unused parameters with underscore to satisfy ESLint no-unused-vars rule.

…ilures These Python bindings for The Sleuth Kit and libewf were never imported or used anywhere in the codebase. They caused Docker build CI failures due to configure script errors during wheel compilation. The system-level packages (sleuthkit, libewf-dev) are already installed via apt-get.

github-advanced-security · 2026-01-25T09:30:07Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

The build-tools jobs run on separate runners from build-base, so the locally built forensics-base image isn't available. This exports the base image as an artifact and imports it in each tool build job.

BuildKit (used by docker/build-push-action) operates in an isolated context that cannot see images loaded into the local Docker daemon. Switching to plain docker build allows the tool builds to reference the forensics-base:latest image loaded from the artifact.

- yara: add curl and ca-certificates for downloading YARA source - log2timeline: add build tools (make, gcc, pkg-config, etc.) and -dev packages required to compile Plaso's Python extension dependencies

hashlib is a built-in Python 3 module and doesn't need a separate package. The hashlib-compat package doesn't exist on PyPI.

Copilot AI review requested due to automatic review settings January 25, 2026 09:09

Copilot started reviewing on behalf of Stealinglight January 25, 2026 09:09 View session

fix: resolve Docker build CI failures

d71fe6f

Add g++ compiler to forensics-base image for building pytsk3 and libewf-python wheels that require C++14 support. Update CodeQL upload-sarif action from deprecated v2 to v3.

Copilot AI reviewed Jan 25, 2026

View reviewed changes

Stealinglight added 2 commits January 25, 2026 01:20

fix: remove unused imports and variables to fix ESLint errors

d293fc8

Remove unused TypeScript imports and prefix intentionally unused parameters with underscore to satisfy ESLint no-unused-vars rule.

Stealinglight added 4 commits January 25, 2026 01:36

fix: share forensics-base image between CI jobs via artifact

32805bd

The build-tools jobs run on separate runners from build-base, so the locally built forensics-base image isn't available. This exports the base image as an artifact and imports it in each tool build job.

fix: add missing build dependencies to yara and log2timeline Dockerfiles

4d8fe1d

- yara: add curl and ca-certificates for downloading YARA source - log2timeline: add build tools (make, gcc, pkg-config, etc.) and -dev packages required to compile Plaso's Python extension dependencies

fix: remove non-existent hashlib-compat package from yara requirements

351196c

hashlib is a built-in Python 3 module and doesn't need a separate package. The hashlib-compat package doesn't exist on PyPI.

Stealinglight merged commit 0eb73aa into main Jan 25, 2026
9 checks passed

Stealinglight deleted the feature/fargate-forensics-migration branch January 25, 2026 19:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Fargate-based forensic analysis infrastructure - #10

feat: Fargate-based forensic analysis infrastructure#10
Stealinglight merged 8 commits into
mainfrom
feature/fargate-forensics-migration

Stealinglight commented Jan 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

Copilot AI Jan 25, 2026

Uh oh!

github-advanced-security AI commented Jan 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-          }),
+          }),
+          // Ensure failure branch preserves overall state shape and aligns with success resultPath
+          resultPath: '$.taskResult',

	FROM forensics-base:latest
	ARG FORENSICS_BASE_TAG=latest
	FROM snapshot-sleuth/forensics-base:${FORENSICS_BASE_TAG}

-        except Exception:
-            pass
+        except Exception as e:
+            self._logger.warning(
+                "Could not check file for credentials",
+                file_path=file_path,
+                error=str(e),
+            )

-                except (OSError, PermissionError):
-                    pass
-            for f in files:
-                try:
-                    entry = self._path_to_entry(root_path / f, rel_root)
-                    file_entries.append(entry)
-                except (OSError, PermissionError):
-                    pass
+                except (OSError, PermissionError) as exc:
+                    # Best-effort traversal: skip directories we cannot access, but log for observability.
+                    logger.debug(
+                        "Skipping directory during snapshot walk due to access error",
+                        path=str(root_path / d),
+                        error=str(exc),
+                    )
+            for f in files:
+                try:
+                    entry = self._path_to_entry(root_path / f, rel_root)
+                    file_entries.append(entry)
+                except (OSError, PermissionError) as exc:
+                    # Best-effort traversal: skip files we cannot access, but log for observability.
+                    logger.debug(
+                        "Skipping file during snapshot walk due to access error",
+                        path=str(root_path / f),
+                        error=str(exc),
+                    )

-            except OSError:
-                pass
+            except OSError as exc:
+                # Best-effort: if the symlink target cannot be read (for example due
+                # to permissions or a broken link), keep the default symlink_target
+                # value but log the failure for observability.
+                logger.debug(
+                    "Failed to read symlink target",
+                    path=str(path),
+                    error=str(exc),
+                )

Conversation

Stealinglight commented Jan 25, 2026

Summary

Changes

Infrastructure (CDK)

Forensic Tool Containers

Shared Libraries

Lambda

CI/CD

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

github-advanced-security AI commented Jan 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants