Skip to content

Commit 8721ae4

Browse files
avanbrunt-cbavanbrunt-cb
authored
Merge pull request #48 from carbonblack/develop
Release v1.0-alpha.1
2 parents 9af6b34 + ed72e0b commit 8721ae4

File tree

85 files changed

+107105
-2
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

85 files changed

+107105
-2
lines changed

‎.coveragerc

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[run]
2+
source = src/cbc_binary_toolkit
3+
[report]
4+
exclude_lines =
5+
pragma: no cover
6+
if __name__ == .__main__.:

‎.gitignore

+17
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,8 @@ coverage.xml
4646
*.cover
4747
.hypothesis/
4848
.pytest_cache/
49+
*log.txt
50+
persist_test.db
4951

5052
# Translations
5153
*.mo
@@ -97,8 +99,23 @@ venv.bak/
9799
# Rope project settings
98100
.ropeproject
99101

102+
# Eclipse/PyDev
103+
/.project
104+
/.pydevproject
105+
/.settings
106+
107+
# macOS
108+
.DS_Store
109+
100110
# mkdocs documentation
101111
/site
102112

103113
# mypy
104114
.mypy_cache/
115+
116+
# Local configuration
117+
config/binary-analysis-config.yaml
118+
119+
.coverage
120+
codeship.aes
121+
env

‎Dockerfile

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
from python:3
2+
MAINTAINER [email protected]
3+
4+
COPY . /app
5+
WORKDIR /app
6+
7+
RUN pip install -r requirements.txt
8+
RUN ln -s /usr/local/lib/python3.8/site-packages/usr/local/lib/libyara.so /usr/local/lib/libyara.so

‎MANIFEST.in

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
include src/cbc_binary_toolkit_examples/engine/yara_local/example_rule.yara

‎README.md

+137-2
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,137 @@
1-
# cb-binary-analysis
2-
Binary Analysis SDK for the Carbon Black Cloud
1+
[![Codeship Status for carbonblack/cb-binary-analysis](https://app.codeship.com/projects/6a7a91c0-2a8b-0138-4f71-1610ceb87095/status?branch=develop)](https://app.codeship.com/projects/384255)
2+
[![Coverage Status](https://coveralls.io/repos/github/carbonblack/cb-binary-analysis/badge.svg?branch=develop&t=rhX4tc)](https://coveralls.io/github/carbonblack/cb-binary-analysis?branch=develop)
3+
# Carbon Black Cloud Binary Toolkit
4+
5+
#### \*\*Disclaimer: This is an ALPHA release\*\*
6+
7+
**Latest Version:** 1.0a1
8+
<br>
9+
**Release Date:** 05/11/2020
10+
11+
The Carbon Black Cloud Binary Toolkit provides a system of processing incoming SHA256 hashes by integrating with the Unified Binary Store (UBS) on the Carbon Black Cloud (CBC).
12+
13+
14+
## Recent updates
15+
16+
View the latest release notes [here](https://github.com/carbonblack/cbc-binary-toolkit/releases).
17+
18+
19+
## License
20+
21+
Use of the Carbon Black API is governed by the license found in [LICENSE](LICENSE).
22+
23+
## Support
24+
25+
1. View all API and integration offerings on the [Developer Network](https://developer.carbonblack.com) along with reference documentation, video tutorials, and how-to guides.
26+
2. Use the [Developer Community Forum](https://community.carbonblack.com/) to discuss issues and get answers from other API developers in the Carbon Black Community.
27+
3. Create a github issue for bugs and change requests. Formal [Carbon Black Support](http://carbonblack.com/resources/support/) coming with v1.0.
28+
29+
## Requirements
30+
31+
The Carbon Black Cloud Binary Toolkit is design to work on Python 3.6 and above.
32+
33+
All requirements are installed as part of `pip install` or if you're planning on pushing changes to the Carbon Black Cloud Binary Toolkit, the following can be used after cloning the repo `pip install requirements.txt`
34+
35+
### Python Packages
36+
* argparse
37+
* cbapi
38+
* python-dateutil
39+
* pyyaml
40+
* requests
41+
* schema
42+
* yara-python
43+
44+
### Carbon Black Cloud
45+
* Enterprise EDR
46+
47+
## Getting Started
48+
49+
There are two ways to use the Carbon Black Cloud Binary Toolkit. You can either run the Binary Analysis Tool using out-of-the-box functionality, or you can use the Toolkit to develop your own tool for processing binaries.
50+
51+
52+
First you will need to install the Binary Toolkit with the following command:
53+
```
54+
pip install cbc-binary-toolkit
55+
```
56+
57+
### Running Binary Analysis tool
58+
59+
The cbc-binary-analysis tool provides out-of-the-box builtin resources for processing binaries and managing the analysis results. For more information see the [User Guide](https://github.com/carbonblack/cbc-binary-toolkit/wiki/User-Guide) wiki page.
60+
61+
```
62+
usage: cbc-binary-analysis [-h] [-c CONFIG]
63+
[-ll {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
64+
{analyze,restart,clear} ...
65+
66+
positional arguments:
67+
{analyze,restart,clear}
68+
Binary analysis commands
69+
analyze Analyze a list of hashes by command line or file
70+
restart Restart a failed job and pick up where the job crashed
71+
or exited
72+
clear Clear cache of analyzed hashes. All or by timestamp
73+
74+
optional arguments:
75+
-h, --help show this help message and exit
76+
-c CONFIG, --config CONFIG
77+
Location of the configuration file (default .../carbonblackcloud/binary-toolkit/binary-analysis-config.yaml.example)
78+
-ll {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
79+
The base log level (default INFO)
80+
```
81+
82+
**Note: Run --help on any of the commands for up to date arguments**
83+
84+
85+
### Using the Toolkit to develop your own tools
86+
87+
The following python code snippet will allow you to begin developing with the Carbon Black Cloud Binary toolkit. For more information see the [Developer Guide](https://github.com/carbonblack/cbc-binary-toolkit/wiki/Developer-Guide) wiki page.
88+
```
89+
from cbc_binary_toolkit import *
90+
```
91+
92+
93+
## Developing Improvements for the Carbon Black Cloud Binary Toolkit
94+
95+
If you want to provide additional examples, fix a bug, or add a feature to the Toolkit the following steps will get you started.
96+
97+
### Installing for Toolkit development
98+
99+
You will need to fork the repo in order to create pull requests when submitting code for review. For details on forking a repo, see [here](https://help.github.com/en/github/getting-started-with-github/fork-a-repo)
100+
101+
```
102+
git clone https://github.com/{fork-name}/cbc-binary-toolkit
103+
cd cbc-binary-toolkit
104+
pip install requirements.txt
105+
```
106+
107+
108+
### Running the Toolkit tests
109+
110+
To check if your code changes didn't break any use cases the following command will run all the tests:
111+
```
112+
pytest
113+
Optional args:
114+
-s Logs streamed to stdout
115+
-k {test or file} Selectively runs test matching string or file
116+
```
117+
118+
### Development Flow
119+
120+
To begin a code change start by creating a branch off of the develop branch.
121+
```
122+
git checkout develop
123+
git checkout -b {branch-name}
124+
```
125+
126+
When the feature or bug fix is finished you will need to create a pull request to the CarbonBlack repo, the following will push your changes to Github.
127+
```
128+
git push {remote} {branch-name}
129+
```
130+
131+
If your branch is behind the develop branch then you will need to rebase.
132+
```
133+
git checkout {branch-name}
134+
git rebase develop
135+
```
136+
137+
Note if your develop branch is out of sync with the CarbonBlack repo then you will need to sync your fork. For information on syncing your fork, see [here](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork)

‎VERSION

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
1.0a1

‎bin/linters.sh

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/sh
2+
3+
set -e
4+
5+
echo 'Running flake8....'
6+
flake8 --docstring-convention google src/cbc_binary_toolkit/*.py
7+
flake8 --docstring-convention google src/tests/*.py

‎bin/tests_n_reports.sh

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#! /bin/bash
2+
3+
set -e
4+
5+
echo 'Running tests....'
6+
coverage run -m pytest
7+
8+
echo 'Running report and sending to coveralls....'
9+
coverage report -m
10+
coveralls

‎codeship-services.yml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
testing:
2+
build:
3+
dockerfile: Dockerfile
4+
encrypted_env_file:
5+
- env.encrypted

‎codeship-steps.yml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- name: linters
2+
service: testing
3+
command: bin/linters.sh
4+
- name: tests
5+
service: testing
6+
command: bin/tests_n_reports.sh

‎config/binary-analysis-config.yaml.example

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
id: cbc_binary_toolkit
2+
version: 0.0.1
3+
carbonblackcloud:
4+
url: https://defense-dev01.cbdtest.io
5+
api_token: ABCDEFGHIJKLMNOPQRSTUVWXYZ/1234567890
6+
org_key: abcdefgh
7+
ssl_verify: True
8+
expiration_seconds: 3600
9+
database:
10+
_provider: cbc_binary_toolkit.state.builtin.Persistor
11+
location: ":memory:"
12+
engine:
13+
name: engine-name
14+
feed_id: 578fTEQBNWXHCPGIXD4RA
15+
type: local
16+
_provider: example.engine

‎env.encrypted

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
codeship:v2
2+
syxgwyyjEE1w+MuEoUTyqbe9npoNpNkMGQOgwERtWg5PM3Gjv+xsBaTyJcY5WjRYB0OW8Jv3Psr5Jpn5yq26fGmFlh8VuFFXagGQYppCAlww8CGoyUSaWvQoxVMqFUp84fa4HGvRBA==

‎pytest.ini

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
[pytest]
2+
log_cli = True
3+
log_cli_level = FATAL
4+
markers =
5+
incremental: Fails out for incremental tests which depend on previous steps

‎requirements.txt

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Package dependencies
2+
argparse
3+
cbapi
4+
python-dateutil
5+
pyyaml
6+
requests
7+
schema
8+
yara-python
9+
10+
# Dev dependencies
11+
flake8
12+
flake8-docstrings
13+
coverage
14+
coveralls
15+
pytest

‎setup.cfg

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[metadata]
2+
description-file = README.md
3+
4+
[bdist_wheel]
5+
universal = 1
6+
7+
[flake8]
8+
ignore = F400, D415, D212, E722
9+
exclude = *__init__.py,
10+
max-doc-length = 120
11+
max-line-length = 120
12+
docstring-convention=google

‎setup.py

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
#!/usr/bin/env python3
2+
"""Python setup script for PIP packaging"""
3+
from __future__ import unicode_literals
4+
import os
5+
from setuptools import setup, find_packages
6+
7+
8+
def read(fname):
9+
"""Process files for configuration"""
10+
return open(os.path.join(os.path.dirname(__file__), fname)).read()
11+
12+
13+
install_reqs = [
14+
"argparse",
15+
"cbapi",
16+
"python-dateutil",
17+
"pyyaml",
18+
"requests",
19+
"schema",
20+
"yara-python"
21+
]
22+
23+
setup(
24+
name="cbc_binary_toolkit",
25+
version=read("VERSION"),
26+
url="https://developer.carbonblack.com/",
27+
license="MIT",
28+
author="VMware Carbon Black",
29+
author_email="[email protected]",
30+
description="The VMware Carbon Black Cloud Binary Toolkit provides useful tools to process "
31+
"binaries and upload IOCs to your Feeds",
32+
long_description=read("README.md"),
33+
long_description_content_type='text/markdown',
34+
platforms="any",
35+
classifiers=[
36+
"Development Status :: 3 - Alpha",
37+
"Intended Audience :: Developers",
38+
"Intended Audience :: System Administrators",
39+
"Operating System :: OS Independent",
40+
"Programming Language :: Python :: 3.6",
41+
"Programming Language :: Python :: 3.7",
42+
"Programming Language :: Python :: 3.8",
43+
"Topic :: Security",
44+
"Topic :: Software Development :: Libraries :: Python Modules",
45+
],
46+
install_requires=install_reqs,
47+
package_dir={'': 'src'},
48+
packages=find_packages(where="src", exclude=["tests.*", "tests"]),
49+
include_package_data=True,
50+
entry_points={"console_scripts": ["cbc-binary-analysis = cbc_binary_toolkit_examples.tools.analysis_util:main"]},
51+
data_files=[("carbonblackcloud/binary-toolkit", ["config/binary-analysis-config.yaml.example"])]
52+
)

‎src/cbc_binary_toolkit/__init__.py

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from .ingestion_component import IngestionComponent
2+
from .deduplication_component import DeduplicationComponent
3+
from .engine_results import EngineResults
4+
5+
from .errors import InitializationError

0 commit comments

Comments
 (0)