|
1 | | -# osint-framework |
2 | | -:eyes: WIP: All-in-one OSINT Tools |
| 1 | +# OSINT Framework |
| 2 | + |
| 3 | +[](https://en.wikipedia.org/wiki/Linux) |
| 4 | +[](https://www.python.org/downloads/) |
| 5 | +[](/LICENSE) |
| 6 | +[](https://github.com/psf/black) |
| 7 | +[](https://github.com/osint-dev-team/osint-framework) |
| 8 | + |
| 9 | +<p align="center"> |
| 10 | + :fork_and_knife: All-in-one OSINT/RECON Swiss Knife |
| 11 | +</p> |
| 12 | + |
| 13 | +<p align="center"> |
| 14 | + <img src="/assets/screenshots/logo.png?raw=true" alt="OSINT Framework Logo" width="50%" height="50%" /> |
| 15 | +</p> |
| 16 | + |
| 17 | +## Screenshots |
| 18 | + |
| 19 | +<div align="center"> |
| 20 | + <img src="/assets/screenshots/cli.png?raw=true" alt="OSINT Framework CLI interface"> |
| 21 | + <p align="center"><i>CLI interface</i></p> |
| 22 | +</div> |
| 23 | + |
| 24 | + |
| 25 | +## Installing |
| 26 | +```bash |
| 27 | +virtualenv -p python3 venv (or python3 -m venv venv) |
| 28 | +source venv/bin/activate |
| 29 | +pip3 install -r requirements.txt |
| 30 | +``` |
| 31 | + |
| 32 | +## Testing |
| 33 | +```bash |
| 34 | +make tests |
| 35 | +``` |
| 36 | + |
| 37 | +## Running |
| 38 | +### As a framework |
| 39 | +To run the framework with a command-line interface: |
| 40 | +```bash |
| 41 | +python3 cli.py -h |
| 42 | +``` |
| 43 | +### As a REST API web service |
| 44 | + |
| 45 | +<div align="center"> |
| 46 | + <img src="/assets/screenshots/docker.png?raw=true" alt="OSINT Framework Docker usage"> |
| 47 | +</div> |
| 48 | + |
| 49 | +To run the framework as a web service via docker and docker-compose: |
| 50 | +```bash |
| 51 | +make up |
| 52 | +``` |
| 53 | +or |
| 54 | +```bash |
| 55 | +docker-compose up |
| 56 | +``` |
| 57 | +## As a separated module |
| 58 | +Basic: |
| 59 | +```python3 |
| 60 | +python3 -m src.scripts.<category>.<name> any_arguments_here |
| 61 | +``` |
| 62 | +Example command: |
| 63 | +```bash |
| 64 | +python3 -m src.scripts.other.user_greeting JohnDoe |
| 65 | +``` |
| 66 | +Example output: |
| 67 | +``` |
| 68 | +{'message': "Successfully finished! (args: (), kwargs: {'username': " |
| 69 | + "'johndoe'})", |
| 70 | + 'result': 'Hello, JohnDoe!', |
| 71 | + 'status': 'success'} |
| 72 | +
|
| 73 | +``` |
| 74 | + |
| 75 | +## REST API web service usage |
| 76 | +1. Create the task: |
| 77 | +```http |
| 78 | +POST /api/tasks/create HTTP/1.1 |
| 79 | +Host: localhost:8888 |
| 80 | +Content-Type: application/json |
| 81 | +
|
| 82 | +[ |
| 83 | + { |
| 84 | + "case": "base", |
| 85 | + "name": "testname-profile", |
| 86 | + "description": "Base example for 'testname' user profile", |
| 87 | + "kwargs": { |
| 88 | + "username": "testname", |
| 89 | + |
| 90 | + "fullname": "Test Name" |
| 91 | + } |
| 92 | + }, |
| 93 | + { |
| 94 | + "case": "osint", |
| 95 | + "name": "johndoe-profile", |
| 96 | + "description": "Osint example for 'johndoe' user profile", |
| 97 | + "kwargs": { |
| 98 | + "username": "johndoe", |
| 99 | + |
| 100 | + "fullname": "John Doe" |
| 101 | + } |
| 102 | + }, |
| 103 | + { |
| 104 | + "case": "recon", |
| 105 | + "name": "facebook-website", |
| 106 | + "description": "Recon example for 'facebook.com' website", |
| 107 | + "kwargs": { |
| 108 | + "url": "https://facebook.com" |
| 109 | + } |
| 110 | + }, |
| 111 | + { |
| 112 | + "case": "recon", |
| 113 | + "name": "vk-website", |
| 114 | + "description": "Recon example for 'vk.com' website", |
| 115 | + "kwargs": { |
| 116 | + "url": "https://vk.com" |
| 117 | + } |
| 118 | + }, |
| 119 | + { |
| 120 | + "case": "recon", |
| 121 | + "name": "mail-website", |
| 122 | + "description": "Recon example for 'mail.ru' website", |
| 123 | + "kwargs": { |
| 124 | + "url": "https://mail.ru" |
| 125 | + } |
| 126 | + }, |
| 127 | + { |
| 128 | + "case": "recon", |
| 129 | + "name": "8-8-8-8-host", |
| 130 | + "description": "Recon example for '8.8.8.8' host", |
| 131 | + "kwargs": { |
| 132 | + "ip": "8.8.8.8" |
| 133 | + } |
| 134 | + }, |
| 135 | + { |
| 136 | + "case": "recon", |
| 137 | + "name": "92-63-64-162-host", |
| 138 | + "description": "Recon example for '92.63.64.162' host", |
| 139 | + "kwargs": { |
| 140 | + "ip": "92.63.64.162" |
| 141 | + } |
| 142 | + }, |
| 143 | + { |
| 144 | + "case": "recon", |
| 145 | + "name": "13-91-95-74-host", |
| 146 | + "description": "Recon example for '13.91.95.74' host", |
| 147 | + "kwargs": { |
| 148 | + "ip": "13.91.95.74" |
| 149 | + } |
| 150 | + }, |
| 151 | + { |
| 152 | + "case": "recon", |
| 153 | + "name": "87-240-190-78-host", |
| 154 | + "description": "Recon example for '87.240.190.78' host", |
| 155 | + "kwargs": { |
| 156 | + "ip": "87.240.190.78" |
| 157 | + } |
| 158 | + }, |
| 159 | + { |
| 160 | + "case": "osint", |
| 161 | + "name": "phone-check", |
| 162 | + "description": "check information about the phone number", |
| 163 | + "kwargs": { |
| 164 | + "phone": 89138111111 |
| 165 | + } |
| 166 | + } |
| 167 | +] |
| 168 | +``` |
| 169 | +2. Check tasks status: |
| 170 | +```http |
| 171 | +GET /api/tasks/list HTTP/1.1 |
| 172 | +Host: localhost:8888 |
| 173 | +``` |
| 174 | +3. Get the results when the task is done: |
| 175 | +```http |
| 176 | +GET /api/results?task_id=<YOUR_TASK_ID> HTTP/1.1 |
| 177 | +Host: localhost:8888 |
| 178 | +``` |
| 179 | + |
| 180 | +## Create your own script |
| 181 | +Use the following structure: |
| 182 | +1. Create your own module directory in the following way: |
| 183 | +``` |
| 184 | +/src/scripts/<choose_your_category_here>/<your_script_name>/<script_files> |
| 185 | +``` |
| 186 | +2. Provide the following structure of your script directory: |
| 187 | +``` |
| 188 | +your_script_name |
| 189 | +├── requirements.txt - provide required libraries |
| 190 | +├── __init__.py - use this module to set the default parent directory (you can copy this file from any other script) |
| 191 | +├── __main__.py - use this module to provide some basic interface to use your script as a module (the same as if __name__ == "__main__") |
| 192 | +├── module.py - use this module to describe the basic logic of your module (you can import it in the __main__.py to provide interface) |
| 193 | +└── test_module.py - use this module for unittest tests |
| 194 | +``` |
| 195 | +3. Create the `__init__.py` file. An example of the `__init__.py` boilerplate structure can be seen below: |
| 196 | +```python3 |
| 197 | +import sys |
| 198 | +from pathlib import Path |
| 199 | + |
| 200 | +__root_dir = Path(__file__).parents[4] |
| 201 | +sys.path.append(str(__root_dir)) |
| 202 | + |
| 203 | +``` |
| 204 | +4. Create the `__main__.py` file. An example of the `__main__.py` boilerplate structure can be seen below: |
| 205 | +```python3 |
| 206 | +#!/usr/bin/env python3 |
| 207 | + |
| 208 | +from pprint import pprint |
| 209 | +from sys import argv |
| 210 | + |
| 211 | +from src.core.utils.module import run_module |
| 212 | +from .module import Runner |
| 213 | + |
| 214 | +result = run_module(Runner, args=argv, arg_name="username", arg_default="johndoe") |
| 215 | +pprint(result) |
| 216 | +``` |
| 217 | +5. Create the module itself. An example of the basic `module.py` file can be seen below: |
| 218 | +```python3 |
| 219 | +#!/usr/bin/env python3 |
| 220 | + |
| 221 | +# Import any required runner |
| 222 | +# 1. OsintRunner - for OSINT scripts |
| 223 | +# 2. ReconRunner - for RECON scripts |
| 224 | +# 3. BaseRunner - for out-of-scope scripts ("other") |
| 225 | +from src.core.base.osint import OsintRunner, BaseRunner, ReconRunner, PossibleKeys |
| 226 | + |
| 227 | +# Import 'ScriptResponse' to return good responses from the module, like |
| 228 | +# 1. ScriptResponse.success - if everything is good |
| 229 | +# 2. ScriptResponse.error - if everything is bad |
| 230 | +from src.core.utils.response import ScriptResponse |
| 231 | + |
| 232 | +# Validate your named arguments. For example, this validator |
| 233 | +# will raise 'KeyError' if you will try to put 'hostname' argument |
| 234 | +# into the 'OsintRunner' runner, and so on |
| 235 | +from src.core.utils.validators import validate_kwargs |
| 236 | + |
| 237 | +# You can use OsintRunner, ReconRunner or BaseRunner as the base class |
| 238 | +class Runner(OsintRunner): |
| 239 | + """ |
| 240 | + Basic script example |
| 241 | + """ |
| 242 | + |
| 243 | + # Define required arguments here |
| 244 | + required = ["my_argument"] |
| 245 | + |
| 246 | + def __init__(self, logger: str = __name__): |
| 247 | + """ |
| 248 | + Re-init base class instance with this function. |
| 249 | + Simply put, you need to provide proper logger name |
| 250 | + to the parent class, so please, save this structure for |
| 251 | + the init function. |
| 252 | + :param logger: logger to use (name of _this_ runner by default) |
| 253 | + """ |
| 254 | + super(Runner, self).__init__(logger) |
| 255 | + |
| 256 | + # Validate input arguments (if you need some validation) |
| 257 | + @validate_kwargs(PossibleKeys.KEYS) |
| 258 | + def run(self, *args, **kwargs) -> ScriptResponse.success or ScriptResponse.error: |
| 259 | + """ |
| 260 | + The main '.run()' function to run your script. |
| 261 | + Note: this function is always synchronous, without any |
| 262 | + async/await init. You can use 'asyncio.run(...)' here, |
| 263 | + but don't put any 'async' before function definition |
| 264 | + :param args: args that you provide (not used for now) |
| 265 | + :param kwargs: kwargs that you provide (required to run something!) |
| 266 | + :return: ScriptResponse message (error or success) |
| 267 | + """ |
| 268 | + argument = kwargs.get("my_argument", "Arguments were not provided!") |
| 269 | + ... |
| 270 | + return ScriptResponse.success(message=f"Script finished with argument {argument}") |
| 271 | +``` |
| 272 | +6. For `test_module.py` you can use any required tests (as you wish). A test case for your module is required to keep the project clean. |
0 commit comments