Skip to content

Commit 916f33f

Browse files
manmolecularmanmolecular
authored
Add Tornado-based web-server. Support REST API methods. Implement task manager. (#41)
* Modify environment files * Move saver to another module * Add SQLite database handling * Add Tornado server * Add main server runner * Move saver to another module * Add multiprocessing * Add docker-compose and postgres * Update database handling * Update environment files * Remove unused imports from main.py * Update database handlers * Update task manager; handle task status * Fix task spawner * Little cosmetic fixes, format files * Format with Black * Add environment variables * Add limit parameter to the task list handler * Don't initialize task spawner, it's not required * Remove single case wrapper from the main.py * Add Docker support for server * Fix makefile: spaces to tabs * Format with Black * Cosmetic: add new line to dockerignore * Add healthcheckers * Add CLI interface * Add color logging * Update gitignore * Add parents for scripts in logging * Fix makefile * Add screenshots * Update README.md * Update README.md
1 parent 8b56d5b commit 916f33f

38 files changed

+1360
-126
lines changed

‎.dockerignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*
2+
.*
3+
!src
4+
!server.py
5+
!requirements.txt
6+
!docker/server/wait-for-it.sh

‎.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Custom
22
.idea/
33
.DS_Store
4-
results/*
4+
results/*.json
55

66
# Byte-compiled / optimized / DLL files
77
__pycache__/

‎Makefile

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
.PHONY: up up_build test clean
2+
3+
up:
4+
docker-compose up -d
5+
6+
up_build:
7+
docker-compose up -d --build --force-recreate
8+
9+
tests:
10+
python3 -W ignore:ResourceWarning -m unittest discover -v -b
11+
12+
clean:
13+
docker-compose down

‎README.md

Lines changed: 151 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,60 @@
1-
# osint-framework
2-
:eyes: All-in-one OSINT/RECON Swiss Knife
1+
# OSINT Framework
2+
3+
[![Required OS](https://img.shields.io/badge/OS-Linux%20based-blue)](https://en.wikipedia.org/wiki/Linux)
4+
[![Python3 Version](https://img.shields.io/badge/python-3.7%2B-blue)](https://www.python.org/downloads/)
5+
[![License](https://img.shields.io/badge/license-GPL--2.0-blue)](/LICENSE)
6+
[![Code Style](https://img.shields.io/badge/code%20style-black-000000)](https://github.com/psf/black)
7+
[![Last Commit](https://img.shields.io/github/last-commit/osint-dev-team/osint-framework)](https://github.com/osint-dev-team/osint-framework)
8+
9+
<p align="center">
10+
:fork_and_knife: All-in-one OSINT/RECON Swiss Knife
11+
</p>
12+
13+
<p align="center">
14+
<img src="/assets/screenshots/logo.png?raw=true" alt="OSINT Framework Logo" width="50%" height="50%" />
15+
</p>
16+
17+
## Screenshots
18+
19+
<div align="center">
20+
<img src="/assets/screenshots/cli.png?raw=true" alt="OSINT Framework CLI interface">
21+
<p align="center"><i>CLI interface</i></p>
22+
</div>
23+
324

425
## Installing
526
```bash
6-
virtualenv -p python3 venv
7-
(or python3 -m venv venv)
27+
virtualenv -p python3 venv (or python3 -m venv venv)
28+
source venv/bin/activate
829
pip3 install -r requirements.txt
930
```
1031

11-
## Running as a framework
12-
_First of all: provide some arguments in the `main.py` file to collect information based on your data (WIP now, will be improved later)_
13-
14-
To run the framework:
32+
## Testing
1533
```bash
16-
python3 main.py example_scenario.yaml
34+
make tests
1735
```
18-
To run the tests:
36+
37+
## Running
38+
### As a framework
39+
To run the framework with a command-line interface:
1940
```bash
20-
chmod +x run_tests.sh
21-
./run_tests.sh
41+
python3 cli.py -h
2242
```
43+
### As a REST API web service
44+
45+
<div align="center">
46+
<img src="/assets/screenshots/docker.png?raw=true" alt="OSINT Framework Docker usage">
47+
</div>
2348

24-
## Running as a separated module
49+
To run the framework as a web service via docker and docker-compose:
50+
```bash
51+
make up
52+
```
53+
or
54+
```bash
55+
docker-compose up
56+
```
57+
## As a separated module
2558
Basic:
2659
```python3
2760
python3 -m src.scripts.<category>.<name> any_arguments_here
@@ -39,6 +72,111 @@ Example output:
3972
4073
```
4174

75+
## REST API web service usage
76+
1. Create the task:
77+
```http
78+
POST /api/tasks/create HTTP/1.1
79+
Host: localhost:8888
80+
Content-Type: application/json
81+
82+
[
83+
{
84+
"case": "base",
85+
"name": "testname-profile",
86+
"description": "Base example for 'testname' user profile",
87+
"kwargs": {
88+
"username": "testname",
89+
"email": "[email protected]",
90+
"fullname": "Test Name"
91+
}
92+
},
93+
{
94+
"case": "osint",
95+
"name": "johndoe-profile",
96+
"description": "Osint example for 'johndoe' user profile",
97+
"kwargs": {
98+
"username": "johndoe",
99+
"email": "[email protected]",
100+
"fullname": "John Doe"
101+
}
102+
},
103+
{
104+
"case": "recon",
105+
"name": "facebook-website",
106+
"description": "Recon example for 'facebook.com' website",
107+
"kwargs": {
108+
"url": "https://facebook.com"
109+
}
110+
},
111+
{
112+
"case": "recon",
113+
"name": "vk-website",
114+
"description": "Recon example for 'vk.com' website",
115+
"kwargs": {
116+
"url": "https://vk.com"
117+
}
118+
},
119+
{
120+
"case": "recon",
121+
"name": "mail-website",
122+
"description": "Recon example for 'mail.ru' website",
123+
"kwargs": {
124+
"url": "https://mail.ru"
125+
}
126+
},
127+
{
128+
"case": "recon",
129+
"name": "8-8-8-8-host",
130+
"description": "Recon example for '8.8.8.8' host",
131+
"kwargs": {
132+
"ip": "8.8.8.8"
133+
}
134+
},
135+
{
136+
"case": "recon",
137+
"name": "92-63-64-162-host",
138+
"description": "Recon example for '92.63.64.162' host",
139+
"kwargs": {
140+
"ip": "92.63.64.162"
141+
}
142+
},
143+
{
144+
"case": "recon",
145+
"name": "13-91-95-74-host",
146+
"description": "Recon example for '13.91.95.74' host",
147+
"kwargs": {
148+
"ip": "13.91.95.74"
149+
}
150+
},
151+
{
152+
"case": "recon",
153+
"name": "87-240-190-78-host",
154+
"description": "Recon example for '87.240.190.78' host",
155+
"kwargs": {
156+
"ip": "87.240.190.78"
157+
}
158+
},
159+
{
160+
"case": "osint",
161+
"name": "phone-check",
162+
"description": "check information about the phone number",
163+
"kwargs": {
164+
"phone": 89138111111
165+
}
166+
}
167+
]
168+
```
169+
2. Check tasks status:
170+
```http
171+
GET /api/tasks/list HTTP/1.1
172+
Host: localhost:8888
173+
```
174+
3. Get the results when the task is done:
175+
```http
176+
GET /api/results?task_id=<YOUR_TASK_ID> HTTP/1.1
177+
Host: localhost:8888
178+
```
179+
42180
## Create your own script
43181
Use the following structure:
44182
1. Create your own module directory in the following way:

‎assets/screenshots/cli.png

2.21 MB
Loading

‎assets/screenshots/docker.png

46.8 KB
Loading

‎assets/screenshots/logo.png

65.3 KB
Loading

‎cli.py

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/usr/bin/env python3
2+
3+
"""
4+
Main runner.
5+
"""
6+
7+
from logging import basicConfig, INFO
8+
9+
from src.cli.handlers.files import FileManager
10+
from src.cli.interface.arguments import parse_args
11+
from src.cli.interface.validators import check_arg_length, check_py_version
12+
from src.cli.interface.opener import show_opener
13+
from src.core.runner.manager import CaseManager
14+
from pathlib import Path
15+
16+
17+
basicConfig(level=INFO)
18+
19+
20+
if __name__ == "__main__":
21+
# fmt: off
22+
23+
show_opener()
24+
check_py_version()
25+
check_arg_length()
26+
scenario = str(parse_args().scenario)
27+
28+
if scenario.endswith("json"):
29+
cases = FileManager.load_json_scenario(scenario)
30+
else:
31+
cases = FileManager.load_yaml_scenario(scenario)
32+
33+
manager = CaseManager(cases)
34+
results = list(manager.multi_case_runner())
35+
36+
FileManager.save_results(results, name=Path(scenario).stem)
37+
38+
# fmt: on

‎docker-compose.yml

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
version: "3.8"
2+
3+
services:
4+
db:
5+
container_name: osint-framework-db
6+
image: postgres:alpine
7+
environment:
8+
POSTGRES_HOST_AUTH_METHOD: trust
9+
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-osint_framework}
10+
POSTGRES_USER: ${POSTGRES_USER:-osint_framework}
11+
PGDATA: /data/postgres
12+
healthcheck:
13+
test: pg_isready -U osint_framework
14+
interval: 30s
15+
timeout: 5s
16+
retries: 5
17+
volumes:
18+
- postgres:/data/postgres
19+
networks:
20+
- postgres
21+
ports:
22+
- "5432:5432"
23+
server:
24+
container_name: osint-framework-server
25+
image: osint-framework-server:1.0
26+
depends_on:
27+
- db
28+
environment:
29+
POSTGRES_DATABASE: ${POSTGRES_DATABASE:-osint}
30+
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-osint_framework}
31+
POSTGRES_USER: ${POSTGRES_USER:-osint_framework}
32+
POSTGRES_HOST: ${POSTGRES_HOST:-osint-framework-db}
33+
POSTGRES_PORT: ${POSTGRES_PORT:-5432}
34+
build:
35+
context: .
36+
target: osint-framework-server
37+
dockerfile: docker/server/Dockerfile
38+
healthcheck:
39+
test: curl --fail -s http://localhost:8888/api/health || exit 1
40+
interval: 30s
41+
timeout: 5s
42+
retries: 5
43+
ports:
44+
- "8888:8888"
45+
networks:
46+
- postgres
47+
networks:
48+
postgres:
49+
driver: bridge
50+
volumes:
51+
postgres:

‎docker/server/Dockerfile

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
FROM python:3.8-alpine as osint-framework-server
2+
3+
LABEL org.label-schema.name="OSINT Framework Server" \
4+
org.label-schema.description="OSINT Framework Server" \
5+
org.label-schema.license="GPL-2.0"
6+
7+
COPY . /app/
8+
COPY docker/server/wait-for-it.sh /app/wait-for-it.sh
9+
WORKDIR /app
10+
RUN apk add --no-cache --virtual .build_deps build-base libffi-dev gcc musl-dev && \
11+
apk add --no-cache postgresql-dev bash curl && \
12+
pip install --no-cache-dir -r requirements.txt && \
13+
apk del .build_deps && \
14+
chmod +x wait-for-it.sh
15+
16+
EXPOSE 8888
17+
ENTRYPOINT ["./wait-for-it.sh", "-t", "5", "osint-framework-db:5432", "--", "python3", "server.py"]

0 commit comments

Comments
 (0)