diff --git a/docs/Advanced/automated.md b/docs/Advanced/automated.md index f4a1375..df0f633 100644 --- a/docs/Advanced/automated.md +++ b/docs/Advanced/automated.md @@ -1,7 +1,7 @@ # Automated Detection ## Important Notice ⚠️ -This section has been withdrawn from version 2.0.3 for further tuning. Please refer to the updated documentation for the latest information and guidelines regarding automated detection. +This section has been withdrawn from version 1.2.0 for further tuning. Please refer to the updated documentation for the latest information and guidelines regarding automated detection. Instead of running manually you can configure Tempo to run on a schedule using the following commands. diff --git a/docs/Advanced/finetune.md b/docs/Advanced/finetune.md index 162fd01..407c0f9 100644 --- a/docs/Advanced/finetune.md +++ b/docs/Advanced/finetune.md @@ -12,7 +12,8 @@ Our baseline model provides robust performance across many use cases. However, f - Start by testing the baseline model on a representative subset of your data using the [Evaluation function](#performance-evaluation) - Collect performance metrics: - Accuracy rate - - Cohen’s Kappa + - F1 Score + - Recall 2. Decision Criteria for Fine-Tuning - Consider fine-tuning if: @@ -50,22 +51,20 @@ You can evaluate model performance using the `CALL MODEL_OPTIMIZATION.EVALUATE_P CALL MODEL_OPTIMIZATION.EVALUATE_PERFORMANCE(); ``` -### **Interpretation of Kappa Score:** -| **Kappa Score (κ)** | **Level of Agreement** | -|----------------------|------------------------| -| < 0.0 | Poor (Worse than chance) | -| 0.0 – 0.20 | Slight agreement | -| 0.21 – 0.40 | Fair agreement | -| 0.41 – 0.60 | Moderate agreement | -| 0.61 – 0.80 | Substantial agreement | -| 0.81 – 1.00 | Almost perfect agreement | +You would obtain a table of performance metrics including F1 Score, Recall, and Accuracy: + +| Metric | Value | +|----------|-------| +| Accuracy | 0.95 | +| F1 Score | 0.92 | +| Recall | 0.90 | # Fine-tuning the Model If you want to increase the accuracy by tuning the model to your own network you can use the following commands. ```sql -CALL model_optimization.tune_model(); +CALL MODEL_OPTIMIZATION.TUNE_MODEL(); ``` Purpose: Updates model based on tuning log data from the reference page as shown in the screenshot @@ -74,7 +73,7 @@ Purpose: Updates model based on tuning log data from the reference page as shown ### Model Rollback ```sql -CALL INFRA_CONTROLS.MODEL_ROLLBACK(version:int); +CALL INFRA_CONTROLS.MODEL_ROLLBACK(int); ``` Removes the specified version of the model and its metadata from the app. diff --git a/docs/assets/DeepDive-onprem.png b/docs/assets/DeepDive-onprem.png new file mode 100644 index 0000000..e6fa510 Binary files /dev/null and b/docs/assets/DeepDive-onprem.png differ diff --git a/docs/assets/ForensicQuery-onprem.png b/docs/assets/ForensicQuery-onprem.png new file mode 100644 index 0000000..a0ba71f Binary files /dev/null and b/docs/assets/ForensicQuery-onprem.png differ diff --git a/docs/assets/ForensicQuery.png b/docs/assets/ForensicQuery.png index dd5ec19..31fde76 100644 Binary files a/docs/assets/ForensicQuery.png and b/docs/assets/ForensicQuery.png differ diff --git a/docs/assets/ForensicSettings-onprem.png b/docs/assets/ForensicSettings-onprem.png new file mode 100644 index 0000000..bfa89e0 Binary files /dev/null and b/docs/assets/ForensicSettings-onprem.png differ diff --git a/docs/assets/ForensicSettings.png b/docs/assets/ForensicSettings.png new file mode 100644 index 0000000..00daa0b Binary files /dev/null and b/docs/assets/ForensicSettings.png differ diff --git a/docs/assets/ForensichatExecConsole-onprem.png b/docs/assets/ForensichatExecConsole-onprem.png new file mode 100644 index 0000000..acd4ccf Binary files /dev/null and b/docs/assets/ForensichatExecConsole-onprem.png differ diff --git a/docs/assets/ForensichatExecConsole.png b/docs/assets/ForensichatExecConsole.png new file mode 100644 index 0000000..738d984 Binary files /dev/null and b/docs/assets/ForensichatExecConsole.png differ diff --git a/docs/assets/forensichatHome-onprem.png b/docs/assets/forensichatHome-onprem.png new file mode 100644 index 0000000..5f8be74 Binary files /dev/null and b/docs/assets/forensichatHome-onprem.png differ diff --git a/docs/assets/forensichatHome.png b/docs/assets/forensichatHome.png index d12f9fb..1bdf16f 100644 Binary files a/docs/assets/forensichatHome.png and b/docs/assets/forensichatHome.png differ diff --git a/docs/assets/reference_page.png b/docs/assets/reference_page.png index 30d1126..bb91c39 100644 Binary files a/docs/assets/reference_page.png and b/docs/assets/reference_page.png differ diff --git a/docs/assets/tempo_permissions.png b/docs/assets/tempo_permissions.png index 0860ad6..f0da1be 100644 Binary files a/docs/assets/tempo_permissions.png and b/docs/assets/tempo_permissions.png differ diff --git a/docs/assets/threatoverview-onprem.png b/docs/assets/threatoverview-onprem.png new file mode 100644 index 0000000..cc61296 Binary files /dev/null and b/docs/assets/threatoverview-onprem.png differ diff --git a/docs/forensichat.md b/docs/forensichat.md deleted file mode 100644 index 5bf9a9c..0000000 --- a/docs/forensichat.md +++ /dev/null @@ -1,78 +0,0 @@ -# Forensichat Guide - -## Overview -Forensichat is a cybersecurity analysis platform integrated within Tempo that helps security analysts investigate network traffic and identify potential threats. It leverages deep learning technology to analyze patterns and anomalies in network data, providing deeper insights than traditional rule-based systems. - -![Forensichat Home Interface](./assets/forensichatHome.png) - -## Prerequisites -Before using Forensichat, ensure the Tempo app has been properly initialized and that you have granted the necessary permissions as outlined in the Snowflake Quickstart Guide. You will also need to run the inference as shown in the Quickstart Guide in order for there to be any data to analyze. - -## 1. Accessing Forensichat - -To access and use Forensichat, follow these steps: - -1. Navigate to the Tempo app in your Snowflake instance. Generally under data products > apps > Tempo. -2. The Forensichat interface will load, showing the available features and options. - -## 2. Using the Forensic Analysis Assistant - -The Forensic Analysis Assistant allows you to perform natural language queries against your network data: - -![Forensic Analysis Assistant](./assets/ForensicQuery.png) - -```sql --- To use the Forensic Analysis Assistant, simply enter your query in the text field --- No SQL command is needed as this is done through the UI -``` - -### Purpose -This feature enables security analysts to ask questions about network behavior and potential security incidents in natural language, making advanced forensic analysis more accessible. - -## 3. Available Features - -Forensichat includes several built-in capabilities: - -1. **MITRE ATT&CK Framework Analysis**: Categorize threats according to industry standards, mapping detected anomalies to known adversarial tactics and techniques. - -2. **Network Traffic Visualization**: Monitor traffic patterns over time to identify unusual activity that may indicate a security incident. - -3. **Threat Overview**: Get a high-level view of potential security incidents, showing metrics like: - - Unique IP Connections - - Total Network Events - - Potential Anomalies - - MITRE ATT&CK Framework Distribution - - ![Threat Overview](./assets/threatoverview.png) - -4. **Deep Investigation**: Perform detailed forensic analysis on specific connections that have been flagged as suspicious. - - ![Deep Dive Analysis](./assets/DeepDive.png) - -## 4. Usage Toggles - -Forensichat provides several filtering options that can be enabled or disabled according to your analysis needs: - -![Forensichat Usage Toggles](./assets/toggles.png) - -1. **Ignore Unclassified MITRE Tactics**: Filter out events that couldn't be mapped to MITRE tactics. -2. **Ignore Unclassified Events**: Filter out events that couldn't be classified. -3. **Ignore Benign Flows**: Filter out network flows classified as normal. -4. **Ignore Anomalous Flows**: Filter out flows classified as anomalous (useful when focusing on specific known patterns). - -## 5. Creating New Queries - -To create a new analysis query: - -1. Click the **New Query** button in the Forensic Analysis Assistant interface. -2. Enter your question in natural language (e.g., "What suspicious activities occurred between 192.168.1.10 and external IPs last week?"). -3. Click **Analyze Query** to process your request. - -![Creating a New Query](./assets/ForensicQuery.png) - -## Notes -- Forensichat works best when MITRE classification has been run on your data (using `CALL THREAT_INTELLIGENCE.MITRE_TACTIC_CLASSIFICATION()` as outlined in the [MITRE Classification Guide](./mitreclass.md)). -- The effectiveness of analysis depends on the quality and quantity of data available to the system. -- For detailed investigation of specific anomalies, use the sequence ID with `CALL INSPECT.INVESTIGATE_SEQUENCE(sequence_id:int)` as outlined in the Deep Dive Analysis section of the [Snowflake Quickstart Guide](./snowflake.md). -- For a complete list of available commands, refer to the [Snowflake Command Reference](./snow_commandRef.md). -- If using your own data, ensure it includes the required features as specified in the [SF Features Guide](./SF_features.md). diff --git a/docs/mitreclass.md b/docs/mitreclass.md index a0d1190..52919d8 100644 --- a/docs/mitreclass.md +++ b/docs/mitreclass.md @@ -34,6 +34,14 @@ CALL THREAT_INTELLIGENCE.MITRE_TACTIC_CLASSIFICATION(); ### Purpose This procedure analyzes known anomalous logs and maps them to relevant MITRE ATT&CK tactics and techniques for deeper security insights. +### Viewing Classification Results + +After running the classification procedure, you can view the results by querying the output table: + +```sql +SELECT * FROM TEMPO.THREAT_INTELLIGENCE.MITRE_TACTICS_MAPPINGS +``` + ## Notes - Ensure proper reference assigning for table access. - Classification results will be available in your output table for further investigation. diff --git a/docs/onPrem.md b/docs/onPrem.md new file mode 100644 index 0000000..eb874ce --- /dev/null +++ b/docs/onPrem.md @@ -0,0 +1,117 @@ +# Install + +## Overview +Tempo is a modern solution that leverages deep learning for network threat detection. By moving beyond conventional rule-based approaches, Tempo offers a more sophisticated and adaptable way to identify and respond to security events. Tempo is uniquely able to see attacks that others cannot, using a Deep Learning model that analyzes network and flow logs to detect various attacks, mapping them to MITRE ATT&CK for integration with your SIEM, SOC, and Threat Response systems. + +## Prerequisites for On-Premises Deployment + +To deploy Tempo on-premises, you'll need: +- Docker installed and configured +- Helm v3+ installed +- Kubernetes cluster configured (kubectl access) with GPUs +- deeptempo license key with appropriate permissions +- zip file with the installer (provided by DeepTempo) + +## On-Premises Deployment Instructions + +1. **Prepare Environment**: + - For testing, you can use Minikube with GPU support: + ```bash + # Install Minikube + curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 + sudo install minikube-linux-amd64 /usr/local/bin/minikube + + # Start minikube with GPU support + minikube start --driver docker --container-runtime docker --gpus all + minikube addons enable nvidia-device-plugin + ``` + +2. **Deploy Using Script**: + ```bash + ./deploy.sh + ``` + +3. **Exposing the Service**: + - If running in a test environment with Minikube: + ```bash + kubectl port-forward service/tempo-nginx 32598:80 --address 0.0.0.0 & + ``` + +## Using Your Own Data + +1. Navigate to the **Data Sources** section from the left menu +2. Select a data source or use demo data for quick analysis + - Demo data allows you to explore the platform's capabilities with sample NetFlow data (CIC netflow dataset) + - You can upload your network logs through the Data Sources interface or connect to an S3 bucket +3. Ensure the data is formatted correctly for proper field identification +4. Start the analysis process by clicking "Run Inference" + +## Required Data Features + +Your dataset must include the following features for proper analysis: + +| Feature | Description | +|---------|-------------| +| timestamp | String datetime when flow started (e.g., "2017-03-07 08:55:58") | +| flow_dur | The duration of the flow in seconds | +| src_ip | A unique identifier of the source device initiating the flow | +| dest_ip | A unique identifier of the destination device receiving the flow | +| src_port | Source port | +| dest_port | Destination port | +| fwd_bytes | Total number of payload bytes, sent from source to destination | +| bwd_bytes | Total number of payload bytes, sent from destination to source | +| total_fwd_pkts | Total number of packets sent from source to destination | +| total_bwd_pkts | Total number of packets sent from destination to source | +| label | Required if fine-tuning. Indicates if flow is suspicious (1) or normal (0) | + +These features can be exported from network monitoring tools such as NetFlow, Wireshark, Zeek, SolarWinds, or AWS/GCP Flow logs. + +## System Configuration + +Access the Settings section to configure Tempo according to your needs: + +### System Status +Monitor the operational status of various components: +- Processing Module +- Vector Database +- Inference Server + +All components should show as "Online" for proper system functionality. + +### Application Settings + +#### Analysis Filters +Configure how Tempo processes and displays data: +- **Ignore Unclassified MITRE Tactics**: Filter out events that couldn't be mapped to MITRE tactics +- **Ignore Benign Network Flows**: Hide normal network activities +- **Ignore Anomalous Flows**: Filter out anomalies (useful when focusing on specific patterns) + +#### Display Settings +- **Show Timestamps**: Toggle timestamp display in results + +### Reset Application +If needed, you can reset the application to its default state, clearing all loaded data and settings. + +**Note**: This action cannot be undone. + +## Troubleshooting On-Premises Deployment + +If you encounter issues: + +- **Components Showing Offline**: Check network connectivity and resource allocation +- **No Data Appearing**: Verify data source configuration and format +- **Analysis Not Working**: Ensure all prerequisites are installed and properly configured +- **Performance Issues**: Check system resources, particularly GPU availability for on-premises deployments + +For some useful commands: +1. Verify Docker credentials: `docker login ghcr.io -u ` +2. Check Kubernetes connection: `kubectl get nodes` +3. Review Helm release status: `helm status tempo -n default` +4. Check pod status: `kubectl get pods -n default` +5. View pod logs: `kubectl logs -n default` + +## Additional Resources + +For more information and assistance: +- **MITRE ATT&CK Framework**: [attack.mitre.org](https://attack.mitre.org/) +- **Support**: [support@deeptempo.ai](mailto:support@deeptempo.ai) \ No newline at end of file diff --git a/docs/onPremUse.md b/docs/onPremUse.md new file mode 100644 index 0000000..d7d387a --- /dev/null +++ b/docs/onPremUse.md @@ -0,0 +1,86 @@ +# Guide + +## Overview +Forensichat is a cybersecurity analysis platform integrated within Tempo that helps security analysts investigate network traffic and identify potential threats. It leverages deep learning technology to analyze patterns and anomalies in network data, providing deeper insights than traditional rule-based systems. + +![Forensichat Home Interface](./assets/forensichatHome-onprem.png) + +## Prerequisites +You will also need to run the inference as shown here in order for there to be any data to analyze. + +## 1. Accessing the UI + +To access and use Tempo, follow these steps: + +1. Navigate to the Tempo app URL. This would have been configured by you during install. +2. The Tempo interface will load, showing the available features and options. + + +## 2. Running inference: + +- Select your data source from the data sources tab and follow the steps in the UI to add your data source. +- Once the data source is added run the inference step to proccess it. + + +![alt text](assets/ForensichatExecConsole-onprem.png) + + +**Monitor Status** + - Watch the status indicator above the button: + - ⏳ *Running* (auto-polls every 5 s) + - ✅ *Completed* + - ❌ *Failed* + +**View Results** + - Once completed, you can view the resuts in the *Threat Overview* and *Deep Investigation* tabs. + +## 3. Using the Forensic Analysis Assistant + +The Forensic Analysis Assistant allows you to perform natural language queries against your network data: +This feature enables security analysts to ask questions about network behavior and potential security incidents in natural language, making advanced forensic analysis more accessible. + +![Forensic Analysis Assistant](./assets/ForensicQuery-onprem.png) + +To create a new analysis query: + +1. Click the **New Query** button in the Forensic Analysis Assistant interface. +2. Enter your question in natural language (e.g., "What suspicious activities occurred between 192.168.1.10 and external IPs last week?"). +3. Click **Analyze Query** to process your request. + +`Sample Query: Show me top 10 similar events sequence id 982b5a35-d289-46f7-8adb-6aea0936b1c2` + +## 4. Available Features + +Forensichat includes several built-in capabilities: + +1. **MITRE ATT&CK Framework Analysis**: Categorize threats according to industry standards, mapping detected anomalies to known adversarial tactics and techniques. + +2. **Network Traffic Visualization**: Monitor traffic patterns over time to identify unusual activity that may indicate a security incident. + +3. **Threat Overview**: Get a high-level view of potential security incidents, showing metrics like: + - Unique IP Connections + - Total Network Events + - Potential Anomalies + - MITRE ATT&CK Framework Distribution + + ![Threat Overview](./assets/threatoverview-onprem.png) + +4. **Deep Investigation**: Perform detailed forensic analysis on specific connections that have been flagged as suspicious. + + ![Deep Dive Analysis](./assets/DeepDive-onprem.png) + + +## 5. Settings and Filters +The Tempo application provides customizable filtering and display options to tailor forensic investigations to specific analytical needs. + +![App Settings](./assets/ForensicSettings-onprem.png) + +The following toggles allow users to control which types of events or flows are shown in the visualization and downstream analyses: + +| Toggle | Description | +|--------|-------------| +| **Ignore Unclassified MITRE Tactics** | Hides events that could not be mapped to any MITRE ATT&CK tactic, enabling users to concentrate on recognized adversarial behavior patterns. | +| **Ignore Benign Network Flows** | Excludes network flows flagged as benign or normal, reducing noise in the data and helping to isolate potentially suspicious traffic. | +| **Ignore Anomalous Flows** | Removes network flows labeled as anomalous. This can be useful when users want to focus only on confirmed patterns or known behavior baselines. | + + diff --git a/docs/snowCMD.md b/docs/snowCMD.md new file mode 100644 index 0000000..1a4e9d4 --- /dev/null +++ b/docs/snowCMD.md @@ -0,0 +1,98 @@ +# Tempo Snowflake Worksheets + +## Overview +Tempo is a modern solution that leverages deep learning for network threat detection. By moving beyond conventional rule-based approaches, Tempo offers a more sophisticated and adaptable way to identify and respond to security events. This guide focuses on using Tempo via Snowflake's command-line interface (CLI) and SQL worksheets. + +## Snowflake worksheet Configuration and Setup + +### 1. Select Database + +From the top of the worksheet, use the "Select Databases" dropdown to attach your database. If you are using demo data, select the option with TEMPO at the beginning of its name. + +### 2. Resource Management + +In a new Snowflake worksheet, you need to set up the necessary procedures. You can add these commands one by one or include them all in a single worksheet. + +#### Initialize Application Resources +```sql +CALL INFRA_CONTROLS.CREATE_RESOURCES(); +``` + +**Purpose**: Initializes the application by loading required model weights and configuration using the granted permissions for warehouse and compute pool creation and task management. + +**Note**: It is recommended that you run this command separately before running the complete worksheet, as it can take some time for resources to spin up. If you are the account admin, you can monitor resources using `SHOW COMPUTE POOLS IN ACCOUNT;`. Once the compute pools are idle, you may continue with the rest of the worksheet. + +### 3. Detection + +#### Run Static Inference +```sql +CALL STATIC_DETECTION.ANOMALY_DETECTION(TRUE); +``` + +**Parameters**: +- `True/False`: Specifies whether to include optional MITRE tactic mappings in the classified anomalies. Set to TRUE to include the mappings, or FALSE to exclude them. + +**Purpose**: This parameter toggles whether MITRE strategy mappings are added to the identified anomalies during the inference phase or if only the anomaly detection task runs. + +## Using Your Own Data with Snowflake Worksheets + +To run Tempo on your own data: + +```sql +-- Initialize Application Resources +CALL INFRA_CONTROLS.CREATE_RESOURCES(); + +-- Run Static Inference (set to True to include MITRE tactic mappings) +CALL STATIC_DETECTION.ANOMALY_DETECTION(TRUE); +``` + +## Advanced Analysis Commands + +One unique capability of Tempo is that potential incidents are tied back to underlying data sequences. Every sequence used by Tempo is assigned a unique ID and is available for additional analysis. You can see this Sequence ID with every anomaly identified. + +```sql +CALL INSPECT.INVESTIGATE_SEQUENCE('sequence_id'); +``` + +**Parameters**: +- `sequence_id`: Identifier of the sequence to analyze (varchar). This ID can be used for deeper investigation on suspicious interactions. + +**Purpose**: This procedure returns the raw data points used in the composition of a given sequence. + +If you ran the inference with MITRE tactic mappings turned on, you can dive even deeper using the MITRE ATT&CK Classification documentation. + +## Required Data Features + +Your dataset must include the following features for proper analysis: + +| Feature | Description | +|---------|-------------| +| timestamp | String datetime when flow started (e.g., "2017-03-07 08:55:58") | +| flow_dur | The duration of the flow in seconds | +| src_ip | A unique identifier of the source device initiating the flow | +| dest_ip | A unique identifier of the destination device receiving the flow | +| src_port | Source port | +| dest_port | Destination port | +| fwd_bytes | Total number of payload bytes, sent from source to destination | +| bwd_bytes | Total number of payload bytes, sent from destination to source | +| total_fwd_pkts | Total number of packets sent from source to destination | +| total_bwd_pkts | Total number of packets sent from destination to source | +| label | Required if fine-tuning. Indicates if flow is suspicious (1) or normal (0) | + +These features can be exported from network monitoring tools such as NetFlow, Wireshark, Zeek, SolarWinds, or AWS/GCP Flow logs. + +## Troubleshooting Snowflake Deployment + +If you encounter issues: + +1. Check warehouse and compute pool status using `SHOW COMPUTE POOLS IN ACCOUNT;` +2. Verify permissions are properly configured +3. Review SQL command syntax for any errors +4. Ensure the management console is accessible via the Streamlit dashboard + +## Additional Resources + +For more information and assistance: +- **Snowflake Marketplace**: Find the Tempo app in the Snowflake Marketplace or directly access it [Here](https://app.snowflake.com/marketplace/listing/GZTYZOYXHP3/deeptempo-cybersecurity-tempo) +- **MITRE ATT&CK Framework**: [attack.mitre.org](https://attack.mitre.org/) +- **Support**: [support@deeptempo.ai](mailto:support@deeptempo.ai) \ No newline at end of file diff --git a/docs/snowUI.md b/docs/snowUI.md new file mode 100644 index 0000000..33ab018 --- /dev/null +++ b/docs/snowUI.md @@ -0,0 +1,127 @@ +# Setup + +## Overview +Tempo is a modern solution that leverages deep learning for network threat detection. By moving beyond conventional rule-based approaches, Tempo offers a more sophisticated and adaptable way to identify and respond to security events. Tempo uses a Deep Learning model that analyzes network and flow logs to detect various attacks, mapping them to MITRE ATT&CK for integration with your SIEM, SOC, and Threat Response systems. + +## Finding and Installing Tempo in Snowflake + +In the Snowflake Marketplace, you can find the Tempo app or directly access it [Here](https://app.snowflake.com/marketplace/listing/GZTYZOYXHP3/deeptempo-cybersecurity-tempo). + +### Prerequisites for Snowflake Deployment +- Grant Tempo necessary permissions including creating warehouses and compute pools +- Configure proper database access + +### Selecting Storage for Tempo Deployment + +If you want to run Tempo on your own data, follow these steps to select the correct storage before launching the app: + +1. Click the **Add** button next to the **on Incident Inference Logs** section +2. In the popup window, click **+Select Data** +3. From the dropdown menu, find and select the appropriate table +4. Click **Save** to confirm your selection + +If you would like to use demo data, you can skip this step. + +Once Tempo is running, a management interface launches which will help you monitor Tempo. + +## Tempo User Interface Navigation + +The Tempo interface includes several main sections accessible from the navigation menu on the left: + +- **Home**: Dashboard with overview and welcome information +- **Data Sources**: Configure and manage data input sources +- **Threat Overview**: High-level analysis of your network security posture +- **Forensic Analysis**: Detailed investigation capabilities +- **Deep Investigation**: Advanced analysis of specific threats +- **Settings**: System configuration and preferences + +## Gaining Insights with the UI + +### Threat Overview + +The Threat Overview section provides a comprehensive view of your network security status: + +#### MITRE ATT&CK Framework Distribution + +Tempo maps detected threats to the MITRE ATT&CK framework, providing a distribution chart that shows: + +- **Initial Access**: Events indicating initial compromise attempts +- **Collection**: Data gathering activities +- **Impact**: Actions aimed at manipulation, interruption, or destruction +- **Credential Access**: Attempts to steal credentials +- **Privilege Escalation**: Efforts to gain higher-level permissions +- **Discovery**: Network and system discovery activities +- **Exfiltration**: Data theft attempts +- **Lateral Movement**: Movement between systems in the network +- **Reconnaissance**: Information gathering activities + +### Detailed Network Forensics + +The Forensic Analysis section allows for in-depth investigation of network activities: + +1. Select source and destination IPs to filter events +2. View total connections, forward bytes, and backward bytes +3. Analyze the distribution of MITRE ATT&CK phases +4. Examine detailed occurrences of each attack phase + +For specific network connections, you can: +- Filter by IP addresses to focus on particular communication paths +- View detailed statistics about data transferred +- Identify the predominant attack techniques used + +### Using the Forensichat Interface + +Forensichat is Tempo's AI-powered analysis assistant that enables natural language interaction with your security data: + +#### Creating New Queries +1. Click the **New Query** button +2. Enter your question in natural language, such as: + `Show me the top 20 similar events to sequence id 002dfbd6-df9a-4eed-a41f-4d707a54c183` +3. Click **Analyze Query** to process your request + +#### Viewing Query Results +Results are displayed in a table format showing relevant information such as: +- Sequence IDs +- Source and destination IPs +- Data transfer volumes +- MITRE ATT&CK classifications + +#### Forensic Insights +Tempo provides insights based on similarity search in high-dimensional embeddings constructed from your logs. For any suspicious sequence, you can view similar events to identify patterns and potential attack campaigns. + +## Required Data Features + +Your dataset must include the following features for proper analysis: + +| Feature | Description | +|---------|-------------| +| timestamp | String datetime when flow started (e.g., "2017-03-07 08:55:58") | +| flow_dur | The duration of the flow in seconds | +| src_ip | A unique identifier of the source device initiating the flow | +| dest_ip | A unique identifier of the destination device receiving the flow | +| src_port | Source port | +| dest_port | Destination port | +| fwd_bytes | Total number of payload bytes, sent from source to destination | +| bwd_bytes | Total number of payload bytes, sent from destination to source | +| total_fwd_pkts | Total number of packets sent from source to destination | +| total_bwd_pkts | Total number of packets sent from destination to source | +| label | Required if fine-tuning. Indicates if flow is suspicious (1) or normal (0) | + +These features can be exported from network monitoring tools such as NetFlow, Wireshark, Zeek, SolarWinds, or AWS/GCP Flow logs. + +## Troubleshooting + +If you encounter issues: + +1. Check the System Status page to ensure all components are online +2. Verify data source configurations +3. Check warehouse and compute pool status using `SHOW COMPUTE POOLS IN ACCOUNT;` +4. Verify permissions are properly configured +5. Ensure the management console is accessible via the Streamlit dashboard + +## Additional Resources + +For more information and assistance: +- **Snowflake Marketplace**: Find the Tempo app in the Snowflake Marketplace or directly access it [Here](https://app.snowflake.com/marketplace/listing/GZTYZOYXHP3/deeptempo-cybersecurity-tempo) +- **MITRE ATT&CK Framework**: [attack.mitre.org](https://attack.mitre.org/) +- **Support**: [support@deeptempo.ai](mailto:support@deeptempo.ai) \ No newline at end of file diff --git a/docs/snowUseage.md b/docs/snowUseage.md new file mode 100644 index 0000000..6fae2ed --- /dev/null +++ b/docs/snowUseage.md @@ -0,0 +1,114 @@ +# Guide + +## Overview +Forensichat is a cybersecurity analysis platform integrated within Tempo that helps security analysts investigate network traffic and identify potential threats. It leverages deep learning technology to analyze patterns and anomalies in network data, providing deeper insights than traditional rule-based systems. + +![Forensichat Home Interface](./assets/forensichatHome.png) + +## Prerequisites +If using Snowflake, ensure the Tempo app has been properly initialized and that you have granted the necessary permissions as outlined in the Snowflake Setup Guide. + +If you want to run Tempo on your own data, follow these steps to select the correct storage before launching the app: + +![reference page](./assets/reference_navi.gif) + +If you would like to use this demo data, please *skip this step* and continue: + +## 1. Accessing The UI + +To access and use Tempo, follow these steps: + +1. Navigate to the Tempo app in your Snowflake instance. Generally under data products > apps > Tempo. +2. The Tempo interface will load, showing the available features and options. + + +## 2. Executing Procedures throught the UI: +This interface enables users to run predefined Tempo procedures directly from the UI inplace of a Snowflake worksheet. + + +> **⚠️ Note:** +> - **Inference Procedures** (Incident Classification, Mitre Classification) will default to the demo data tables unless you explicitly configure which table to target under Reference Pages. +> - **Performance Evaluation** and **Model Fine-Tuning** **require** you to explicitly set the target evaluation or tuning table on the reference page—they have no demo defaults. +> +> Ensure you have selected or created the correct source tables before executing these procedures, or the command will fail. + +![alt text](assets/ForensichatExecConsole.png) + + +1. **Select Your Procedure** + - Click the **Select Procedure** dropdown. + - Choose the desired command (e.g. “Investigate Sequence 🔎”, “Incident Classification 🛡️”, etc.). + +2. **(Investigate Sequence only) Enter Sequence ID** + - If you picked **Investigate Sequence 🔎**, a text field appears. + - Type or paste the 29-character NetFlow Sequence ID and press **Enter**. + +3. **(Incident Classification only) Toggle MITRE Mapping** + - If you picked **Incident Classification 🛡️**, use the **Include MITRE Classification** switch to turn ATT&CK mapping on or off. + +4. **Execute the Command** + - Click **Execute Command** to launch the procedure. + - The button is disabled until all required inputs are valid or until run is completed. + +5. **Monitor Status** + - Watch the status indicator above the button: + - ⏳ *Running* (auto-polls every 5 s) + - ✅ *Completed* + - ❌ *Failed* + +6. **View Results** + - Once completed, scroll down to see the output table for your selected procedure: + - **Investigate Sequence** shows raw NetFlow rows. + - **Incident/Mitre Classification** displays inference logs. + - **Performance Evaluation** lists recent metrics. + - **Model Fine-Tuning** returns a summary message. + +## 3. Using the Forensic Analysis Assistant + +The Forensic Analysis Assistant allows you to perform natural language queries against your network data: + +![Forensic Analysis Assistant](./assets/ForensicQuery.png) + +`Sample Query: Show me top 10 similar events sequence id 982b5a35-d289-46f7-8adb-6aea0936b1c2` + +``` +-- To use the Forensic Analysis Assistant, simply enter your query in the text field +-- No SQL command is needed as this is done through the UI +``` + +### Purpose +This feature enables security analysts to ask questions about network behavior and potential security incidents in natural language, making advanced forensic analysis more accessible. + +## 4. Available Features + +Forensichat includes several built-in capabilities: + +1. **MITRE ATT&CK Framework Analysis**: Categorize threats according to industry standards, mapping detected anomalies to known adversarial tactics and techniques. + +2. **Network Traffic Visualization**: Monitor traffic patterns over time to identify unusual activity that may indicate a security incident. + +3. **Threat Overview**: Get a high-level view of potential security incidents, showing metrics like: + - Unique IP Connections + - Total Network Events + - Potential Anomalies + - MITRE ATT&CK Framework Distribution + + ![Threat Overview](./assets/threatoverview.png) + +4. **Deep Investigation**: Perform detailed forensic analysis on specific connections that have been flagged as suspicious. + + ![Deep Dive Analysis](./assets/DeepDive.png) + +## 5. Settings and Filters +The Tempo application provides customizable filtering and display options to tailor forensic investigations to specific analytical needs. + +![App Settings](./assets/ForensicSettings.png) + +The following toggles allow users to control which types of events or flows are shown in the visualization and downstream analyses: + +| Toggle | Description | +|--------|-------------| +| **Ignore Unclassified MITRE Tactics** | Hides events that could not be mapped to any MITRE ATT&CK tactic, enabling users to concentrate on recognized adversarial behavior patterns. | +| **Ignore Benign Network Flows** | Excludes network flows flagged as benign or normal, reducing noise in the data and helping to isolate potentially suspicious traffic. | +| **Ignore Anomalous Flows** | Removes network flows labeled as anomalous. This can be useful when users want to focus only on confirmed patterns or known behavior baselines. | +| **Ignore Unclassified Events** | Filters out events that were not categorized during the detection pipeline, simplifying the visual workspace. | diff --git a/docs/snow_commandRef.md b/docs/snow_commandRef.md index cf6d5fa..db53f16 100644 --- a/docs/snow_commandRef.md +++ b/docs/snow_commandRef.md @@ -13,15 +13,23 @@ Required Permissions: Warehouse, compute pool, and task management access ### Run Inference ```sql -CALL STATIC_DETECTION.ANOMALY_DETECTION(with_mitre:boolean); +CALL STATIC_DETECTION.ANOMALY_DETECTION(boolean); ``` Parameters: -- `True/False`: Specifies whether to include optional MITRE tactic mappings in the classified anomalies. Set to TRUE to include the mappings, or False to exclude them. +- `True or False`: Specifies whether to include optional MITRE tactic mappings in the classified anomalies. Set to TRUE to include the mappings, or False to exclude them. Purpose: Executes inference on specified service data +#### Viewing Classification Results + +After running the classification procedure, you can view the results by querying the output table: + +```sql +SELECT * FROM STATIC_DETECTION.ANOMALOUS_EVENTS +``` + ### Deep Dive Analysis ```sql -CALL INSPECT.INVESTIGATE_SEQUENCE(sequence_id:varchar); +CALL INSPECT.INVESTIGATE_SEQUENCE(varchar); ``` Parameters: - `sequence_id`: Identifier of the sequence to analyze (varchar) @@ -34,6 +42,14 @@ CALL THREAT_INTELLIGENCE.MITRE_TACTIC_CLASSIFICATION(); ``` Purpose: Classifies known anomlies to the corresponding MITRE ATT&CK technique +#### Viewing Classification Results + +After running the classification procedure, you can view the results by querying the output table: + +```sql +SELECT * FROM TEMPO.THREAT_INTELLIGENCE.MITRE_TACTICS_MAPPINGS +``` + ## Model Optimization ### Model Evaluation @@ -52,7 +68,7 @@ Purpose: Updates model based on tuning log data from the reference page ### Model Rollback ```sql -CALL INFRA_CONTROLS.MODEL_ROLLBACK(version:int); +CALL INFRA_CONTROLS.MODEL_ROLLBACK(int); ``` Parameters: - `version`: The integer version number of the model to be rolled back. diff --git a/docs/snowflake.md b/docs/snowflake.md index e597f38..b310248 100644 --- a/docs/snowflake.md +++ b/docs/snowflake.md @@ -60,11 +60,11 @@ It is recommended that you run this command before running the sheet as a whole. ### Run Static Inference ```sql -CALL STATIC_DETECTION.ANOMALY_DETECTION(with_mitre:boolean); +CALL STATIC_DETECTION.ANOMALY_DETECTION(boolean); ``` **Parameters:** -- `True/False`: Specifies whether to include optional MITRE tactic mappings in the classified anomalies. Set to TRUE to include the mappings, or False to exclude them. +- `True or False`: Specifies whether to include optional MITRE tactic mappings in the classified anomalies. Set to TRUE to include the mappings, or False to exclude them. #### Purpose: This parameter toggles whether MITRE strategy mappings are added to the identified anomalies during the inference phase or if only the anomaly detection task runs. @@ -73,15 +73,13 @@ This parameter toggles whether MITRE strategy mappings are added to the identifi One unique capability of Tempo is that potential incidents are tied back to underlying data sequences. Every sequence used by Tempo is assigned a unique ID and is available for additional analysis. You can see this Sequence ID with every anomaly identified. ```sql -CALL INSPECT.INVESTIGATE_SEQUENCE(sequence_id:varchar); +CALL INSPECT.INVESTIGATE_SEQUENCE(varchar); ``` **Parameters:** - `sequence_id`: Identifier of the sequence to analyze (varchar). This ID can be used down the road if any anomalies are detected to run deeper investigation on suspicious interactions. #### Purpose: This procedure returns the raw data points used in the composition of a given sequence. -Note: If running on demo data let's use 2 as the id (valid IDs 1-1200) - If you ran the inference with MITRE tactic mappings turned on you can dive even deeper using the [MITRE ATT&CK Classification doc here](/docs/mitreclass.md) ## Notes diff --git a/docusaurus.config.js b/docusaurus.config.js index 5c42a23..fbff739 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -81,7 +81,7 @@ const config = { }, items: [ { - to: '/docs/snowflake', + to: '/docs/snowUI', label: 'Snowflake Quickstart', position: 'left', }, diff --git a/sidebars.js b/sidebars.js index adac0a3..a9e74d0 100644 --- a/sidebars.js +++ b/sidebars.js @@ -14,20 +14,37 @@ /** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */ const sidebars = { tutorialSidebar: [ - 'snowflake', + { type: 'category', - label: 'Advanced', + label: 'On-Premise', items: [ - 'Advanced/automated', - 'Advanced/finetune', + 'onPrem', + 'onPremUse', ], }, - 'SF_features', - 'mitreclass', - 'snow_commandRef', - 'forensichat', + { + type: 'category', + label: 'Snowflake', + items: [ + 'snowUI', + 'snowUseage', + { + type: 'category', + label: 'Worksheets', + items: [ + 'snowCMD', + 'SF_features', + 'mitreclass', + 'snow_commandRef', + 'Advanced/automated', + 'Advanced/finetune', + ], + }, + ], + }, + { type: 'category', label: 'Visualizations',