Add Streamlit-Based AI Dataset Validator Tool with Real-Time File Upload and Anomaly Detection by Stanley1414 · Pull Request #101 · Redback-Operations/redback-data-warehouse

Stanley1414 · 2025-05-23T02:20:17Z

This pull request introduces a Streamlit-based AI Dataset Validator tool with the following core functionalities:

🔹 Real-Time File Upload Support
Users can upload datasets (CSV, Excel, JSON) via an intuitive web interface. The tool handles live data uploads for quick inspection and feedback.

🔹 Automatic Format Conversion
Uploaded files are automatically converted to a standardized structure suitable for processing and analysis.

🔹 Anomaly Detection & Correction
Built-in validation checks identify missing values, outliers, and inconsistent entries. Where possible, the tool provides automated suggestions or corrections.

🔹 ETL Pipeline Integration
The tool integrates with the existing data warehouse ETL pipeline to improve data quality at ingestion.

🔹 README and Documentation Updates
Project documentation updated to reflect usage instructions and architecture overview.

This tool significantly enhances the data preparation workflow for the Redback Data Warehouse project by empowering users to validate and sanitize datasets before ingestion.

Please review and advise on improvements or testing feedback.

…al runs, total distance, avg speed, avg pace) added; saved cleaned data and stats.

…sing steps

Added requirement gathering document for Project 3, including team discussions, stakeholder inputs, and data specifications.

github-actions · 2025-05-23T02:20:42Z

🔒 Security Scan Results

🔒 Security Scan Results
=========================

Bandit Scan Results:
-------------------
Run started:2025-05-23 02:20:40.040383

Test results:
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: ./Core DW Infrastructure/app/streamlitdw_fe.py:9:0
8	import datetime
9	import subprocess
10	import pandas as pd

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/app/streamlitdw_fe.py:115:19
114	    try:
115	        response = requests.post("http://dp-logstash:5044", json=log_data)
116	        if response.status_code != 200:

--------------------------------------------------
>> Issue: [B607:start_process_with_partial_path] Starting a process with a partial executable path
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b607_start_process_with_partial_path.html
   Location: ./Core DW Infrastructure/app/streamlitdw_fe.py:169:17
168	    try:
169	        result = subprocess.run(
170	            ["python", "etl_pipeline.py", file_name, preprocessing_option], 
171	            check=True, 
172	            stdout=subprocess.PIPE, 
173	            stderr=subprocess.PIPE, 
174	            text=True
175	        )
176	        st.success("ETL pipeline executed successfully.")

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b603_subprocess_without_shell_equals_true.html
   Location: ./Core DW Infrastructure/app/streamlitdw_fe.py:169:17
168	    try:
169	        result = subprocess.run(
170	            ["python", "etl_pipeline.py", file_name, preprocessing_option], 
171	            check=True, 
172	            stdout=subprocess.PIPE, 
173	            stderr=subprocess.PIPE, 
174	            text=True
175	        )
176	        st.success("ETL pipeline executed successfully.")

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/app/streamlitdw_fe.py:186:19
185	        api_url = f"http://{api_url_base}/list-files?bucket={bucket}"
186	        response = requests.get(api_url)
187	        if response.status_code == 200:

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/app/streamlitdw_fe.py:200:19
199	        params = {"bucket": bucket, "project": project, "filename": filename}  # Avoid re-adding the project folder
200	        response = requests.get(api_url, params=params)
201	        st.write(f"API URL: {api_url}, Params: {params}, Status Code: {response.status_code}")  # added logs

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/dremio-api/api.py:21:20
20	def get_dremio_token():
21	    auth_response = requests.post(f'{dremio_url}/apiv2/login', json={'userName': dremio_username, 'password': dremio_password})
22	    auth_response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/dremio-api/api.py:32:15
31	    }
32	    response = requests.post(f'{dremio_url}/api/v3/sql', headers=headers, json={'sql': sql})
33	    response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/dremio-api/api.py:46:19
45	    while True:
46	        response = requests.get(f'{dremio_url}/api/v3/job/{job_id}', headers=headers)
47	        response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/dremio-api/api.py:55:15
54	    # Fetch the query results
55	    response = requests.get(f'{dremio_url}/api/v3/job/{job_id}/results', headers=headers)
56	    response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Core DW Infrastructure/dremio-api/api.py:66:15
65	    }
66	    response = requests.get(f'{dremio_url}/api/v3/catalog', headers=headers)
67	    response.raise_for_status()

--------------------------------------------------
>> Issue: [B104:hardcoded_bind_all_interfaces] Possible binding to all interfaces.
   Severity: Medium   Confidence: Medium
   CWE: CWE-605 (https://cwe.mitre.org/data/definitions/605.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b104_hardcoded_bind_all_interfaces.html
   Location: ./Core DW Infrastructure/dremio-api/api.py:100:17
99	    port = int(os.getenv('FLASK_RUN_PORT', 5000))
100	    app.run(host='0.0.0.0', port=port)

--------------------------------------------------
>> Issue: [B104:hardcoded_bind_all_interfaces] Possible binding to all interfaces.
   Severity: Medium   Confidence: Medium
   CWE: CWE-605 (https://cwe.mitre.org/data/definitions/605.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b104_hardcoded_bind_all_interfaces.html
   Location: ./Core DW Infrastructure/flask/flaskapi_dw.py:86:17
85	if __name__ == '__main__':
86	    app.run(host='0.0.0.0', port=5000)  # Running on port 5000 IMPORTANT

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: ./File Upload Service/app/streamlitdw_fe.py:9:0
8	import datetime
9	import subprocess
10	import pandas as pd

--------------------------------------------------
>> Issue: [B607:start_process_with_partial_path] Starting a process with a partial executable path
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b607_start_process_with_partial_path.html
   Location: ./File Upload Service/app/streamlitdw_fe.py:58:17
57	    try:
58	        result = subprocess.run(
59	            ["python", "etl_pipeline.py", file_name, preprocessing_option], 
60	            check=True, 
61	            stdout=subprocess.PIPE, 
62	            stderr=subprocess.PIPE, 
63	            text=True
64	        )
65	        st.success("ETL pipeline executed successfully.")

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b603_subprocess_without_shell_equals_true.html
   Location: ./File Upload Service/app/streamlitdw_fe.py:58:17
57	    try:
58	        result = subprocess.run(
59	            ["python", "etl_pipeline.py", file_name, preprocessing_option], 
60	            check=True, 
61	            stdout=subprocess.PIPE, 
62	            stderr=subprocess.PIPE, 
63	            text=True
64	        )
65	        st.success("ETL pipeline executed successfully.")

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./File Upload Service/app/streamlitdw_fe.py:75:19
74	        api_url = f"http://10.137.0.149:5000/list-files?bucket={bucket}"  # Updated
75	        response = requests.get(api_url)
76	        if response.status_code == 200:

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./File Upload Service/app/streamlitdw_fe.py:91:19
90	        params = {"bucket": bucket, "project": project, "filename": filename}  # Avoid re-adding the project folder
91	        response = requests.get(api_url, params=params)
92	        st.write(f"API URL: {api_url}, Params: {params}, Status Code: {response.status_code}")  # added logs

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: ./File Upload Service/app/streamlitdw_fe_mt.py:8:0
7	import datetime
8	import subprocess  # For triggering ETL pipeline
9	import pandas as pd  # Added for dataframe functionality

--------------------------------------------------
>> Issue: [B607:start_process_with_partial_path] Starting a process with a partial executable path
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b607_start_process_with_partial_path.html
   Location: ./File Upload Service/app/streamlitdw_fe_mt.py:58:8
57	        # Run the ETL script as a subprocess
58	        subprocess.run(["python", "etl_pipeline.py", preprocessing_option], check=True)
59	        st.success("ETL pipeline executed successfully.")

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b603_subprocess_without_shell_equals_true.html
   Location: ./File Upload Service/app/streamlitdw_fe_mt.py:58:8
57	        # Run the ETL script as a subprocess
58	        subprocess.run(["python", "etl_pipeline.py", preprocessing_option], check=True)
59	        st.success("ETL pipeline executed successfully.")

--------------------------------------------------
>> Issue: [B104:hardcoded_bind_all_interfaces] Possible binding to all interfaces.
   Severity: Medium   Confidence: Medium
   CWE: CWE-605 (https://cwe.mitre.org/data/definitions/605.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b104_hardcoded_bind_all_interfaces.html
   Location: ./File Upload Service/flask/flaskapi_dw.py:86:17
85	if __name__ == '__main__':
86	    app.run(host='0.0.0.0', port=5000)  # Running on port 5000 IMPORTANT

--------------------------------------------------
>> Issue: [B104:hardcoded_bind_all_interfaces] Possible binding to all interfaces.
   Severity: Medium   Confidence: Medium
   CWE: CWE-605 (https://cwe.mitre.org/data/definitions/605.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b104_hardcoded_bind_all_interfaces.html
   Location: ./MongoDB_Connection/Project1/main.py:12:35
11	    debug_mode = os.environ.get('FLASK_DEBUG', 'False').lower() == 'true'
12	    app.run(debug=debug_mode, host='0.0.0.0')

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Flask-api/api.py:21:20
20	def get_dremio_token():
21	    auth_response = requests.post(f'{dremio_url}/apiv2/login', json={'userName': dremio_username, 'password': dremio_password})
22	    auth_response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Flask-api/api.py:32:15
31	    }
32	    response = requests.post(f'{dremio_url}/api/v3/sql', headers=headers, json={'sql': sql})
33	    response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Flask-api/api.py:46:19
45	    while True:
46	        response = requests.get(f'{dremio_url}/api/v3/job/{job_id}', headers=headers)
47	        response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Flask-api/api.py:55:15
54	    # Fetch the query results
55	    response = requests.get(f'{dremio_url}/api/v3/job/{job_id}/results', headers=headers)
56	    response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Flask-api/api.py:66:15
65	    }
66	    response = requests.get(f'{dremio_url}/api/v3/catalog', headers=headers)
67	    response.raise_for_status()

--------------------------------------------------
>> Issue: [B104:hardcoded_bind_all_interfaces] Possible binding to all interfaces.
   Severity: Medium   Confidence: Medium
   CWE: CWE-605 (https://cwe.mitre.org/data/definitions/605.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b104_hardcoded_bind_all_interfaces.html
   Location: ./Structured Dremio Solution/Flask-api/api.py:100:17
99	    port = int(os.getenv('FLASK_RUN_PORT', 5000))
100	    app.run(host='0.0.0.0', port=port)

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Script/pipeline.py:63:20
62	try:
63	    auth_response = requests.post(f'{dremio_url}/apiv2/login', json={'userName': username, 'password': password})
64	    auth_response.raise_for_status()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Script/pipeline.py:115:23
114	    try:
115	        sql_response = requests.post(f'{dremio_url}/api/v3/sql', headers=headers, json={'sql': command})
116	        sql_response.raise_for_status()

--------------------------------------------------
>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
   Severity: Medium   Confidence: Low
   CWE: CWE-89 (https://cwe.mitre.org/data/definitions/89.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b608_hardcoded_sql_expressions.html
   Location: ./Structured Dremio Solution/Script/pipeline.py:168:12
167	    placeholders = ', '.join(['?' for _ in data[0]])
168	    query = f"INSERT INTO {table_name} VALUES ({placeholders})"
169	    cursor = conn.cursor()

--------------------------------------------------
>> Issue: [B113:request_without_timeout] Call to requests without timeout
   Severity: Medium   Confidence: Low
   CWE: CWE-400 (https://cwe.mitre.org/data/definitions/400.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b113_request_without_timeout.html
   Location: ./Structured Dremio Solution/Script/pipeline.py:182:19
181	    try:
182	        response = requests.get(url)
183	        response.raise_for_status()

--------------------------------------------------
>> Issue: [B108:hardcoded_tmp_directory] Probable insecure usage of temp file/directory.
   Severity: Medium   Confidence: Medium
   CWE: CWE-377 (https://cwe.mitre.org/data/definitions/377.html)
   More Info: https://bandit.readthedocs.io/en/1.8.3/plugins/b108_hardcoded_tmp_directory.html
   Location: ./pre-processing/pre-processing.py:177:29
176	
177	            temp_file_path = f'/tmp/{obj.object_name}'
178	

--------------------------------------------------

Code scanned:
	Total lines of code: 1963
	Total lines skipped (#nosec): 0
	Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 0

Run metrics:
	Total issues (by severity):
		Undefined: 0
		Low: 9
		Medium: 25
		High: 0
	Total issues (by confidence):
		Undefined: 0
		Low: 19
		Medium: 6
		High: 9
Files skipped (0):

Dependency Check Results:
-----------------------

✅ No critical security issues detected.

The code has passed all critical security checks.

ben-AI-cybersec · 2025-06-30T15:18:47Z

should be in a subdirectory, not the root of the repository. Closing PR

Stanley1414 added 3 commits May 5, 2025 11:55

Integrated data aggregation into ETL pipeline: weekly statistics (tot…

9ee5019

…al runs, total distance, avg speed, avg pace) added; saved cleaned data and stats.

Updated README: Garmin Run Data – ETL Pipeline update and data proces…

e519940

…sing steps

Add files via upload

37b8eb4

Added requirement gathering document for Project 3, including team discussions, stakeholder inputs, and data specifications.

ben-AI-cybersec closed this Jun 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Streamlit-Based AI Dataset Validator Tool with Real-Time File Upload and Anomaly Detection#101

Add Streamlit-Based AI Dataset Validator Tool with Real-Time File Upload and Anomaly Detection#101
Stanley1414 wants to merge 3 commits intoRedback-Operations:mainfrom
Stanley1414:main

Stanley1414 commented May 23, 2025

Uh oh!

github-actions bot commented May 23, 2025

Uh oh!

ben-AI-cybersec commented Jun 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

Stanley1414 commented May 23, 2025

Uh oh!

github-actions bot commented May 23, 2025

🔒 Security Scan Results

Uh oh!

ben-AI-cybersec commented Jun 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments