Skip to content

Commit 268efb1

Browse files
rvilgalysrvilgalys
authored
Merge pull request #42 from google/container
Container -> Master
2 parents 030b4b9 + 8bb8e7f commit 268efb1

File tree

25 files changed

+1843
-652
lines changed

25 files changed

+1843
-652
lines changed

‎.github/workflows/docker_build.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
name: Docker build
2+
3+
on: [push]
4+
5+
jobs:
6+
docker-build:
7+
# we are using ubuntu as it ships with docker pre-installed
8+
runs-on: ubuntu-latest
9+
steps:
10+
- uses: actions/checkout@v3
11+
# Our Dockerfile runs all of our Go Tests
12+
- name: Build and Test Docker Image
13+
run: docker build --tag wr-container .

‎.github/workflows/go_tests.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Go Test & Build
2+
3+
on: [push]
4+
5+
jobs:
6+
go-test-build:
7+
runs-on: ubuntu-latest
8+
steps:
9+
- uses: actions/checkout@v3
10+
11+
- name: Set up Go
12+
uses: actions/setup-go@v3
13+
with:
14+
go-version: 1.19
15+
16+
- name: Install Dependencies
17+
run: go install ./
18+
19+
- name: Run Go Tests
20+
run: go test -v ./
21+
22+
- name: Build wrlookup
23+
run: go build -v -o wrlookup ./cmd/wrlookup/main.go
24+
25+
- name: Build wrserver
26+
run: go build -v -o wrserver ./cmd/wrserver/main.go

‎Dockerfile‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
FROM golang:1.19 as build
2+
3+
WORKDIR /go/src/webrisk
4+
5+
# Cache go.mod to pre-download dependencies
6+
COPY go.mod go.sum ./
7+
RUN go mod download && go mod verify
8+
9+
COPY . .
10+
11+
# Analyze our Go code and run all tests. To speed up building, such as during
12+
# development, consider commenting out these lines.
13+
RUN go vet -v
14+
RUN go test -v
15+
16+
RUN CGO_ENABLED=0 go build -o /go/bin/wrserver cmd/wrserver/main.go
17+
18+
FROM gcr.io/distroless/static-debian11 as wrserver
19+
20+
COPY --from=build /go/bin/wrserver /
21+
22+
# The APIKEY Environmental Variable should be passed in at runtime. Example:
23+
# docker run -e APIKEY=XXXXXXXXXXXXXXXXXXXXX -p 8080:8080 <container label>
24+
ENTRYPOINT [ "/wrserver"]

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,4 +199,4 @@
199199
distributed under the License is distributed on an "AS IS" BASIS,
200200
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201201
See the License for the specific language governing permissions and
202-
limitations under the License.
202+
limitations under the License.

‎README.md‎

Lines changed: 216 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -1,92 +1,252 @@
1-
# Reference Implementation for the Usage of Google Cloud WebRisk APIs (Beta)
1+
# Web Risk Client App | Container & Go
22

3-
The `webrisk` Go package can be used with the
4-
[Google Cloud WebRisk APIs (Beta)](https://cloud.google.com/web-risk/)
5-
to access the Google Cloud WebRisk lists of unsafe web resources. Inside the
6-
`cmd` sub-directory, you can find two programs: `wrlookup` and `wrserver`. The
7-
`wrserver` program creates a proxy local server to check URLs and a URL
8-
redirector to redirect users to a warning page for unsafe URLs. The `wrlookup`
9-
program is a command line service that can also be used to check URLs.
3+
[Web Risk](https://cloud.google.com/web-risk) is the enterprise version of
4+
Google's [Safe Browsing API](https://safebrowsing.google.com/) that protects 5
5+
Billion devices globally from dangerous URLs including phishing, malware,
6+
unwanted software, and social engineering.
107

11-
This **README.md** is a quickstart guide on how to build, deploy, and use the
12-
WebRisk Go package. It can be used out-of-the-box. The GoDoc and API
13-
documentation provide more details on fine tuning the parameters if desired.
8+
This client implements the Web Risk [Update API](https://cloud.google.com/web-risk/docs/update-api),
9+
which allows for URLs to be checked for badness via privacy-preserving and
10+
low-latency API. It works out-of-the-box via either Docker or Go.
1411

12+
This README provides a quickstart guide to running a client either with Docker
13+
or as Go binaries. It also serves as a reference implementation of the API. The
14+
GoDoc and API documentation in the `.go` source files provide more details on
15+
fine-tuning the parameters if desired.
1516

16-
# How to Build
17+
Supported clients:
1718

18-
To download and install from the source, run the following command:
19+
- `wrserver` runs a thin HTTP client that can query URLs via a POST request or
20+
a redirection endpoint that diverts bad URLs to a warning page. This is the
21+
client wrapped by Docker.
22+
- `wrlookup` is a command line service that takes URLs from `STDIN` and outputs results to `STDOUT`. It can
23+
accept multiple URLs at a time on separate lines.
24+
25+
Supported blocklists:
26+
27+
- `MALWARE`
28+
- `UNWANTED_SOFTWARE`
29+
- `SOCIAL_ENGINEERING`
30+
- [`SOCIAL_ENGINEERING_EXTENDED_COVERAGE`](https://cloud.google.com/web-risk/docs/extended-coverage)
31+
32+
The client is originally forked from the [Safebrowsing Go Client](https://github.com/google/safebrowsing).
33+
34+
# Enable Web Risk
35+
36+
To begin using Web Risk, you will need a [GCP](https://cloud.google.com/)
37+
Account and a project to work in.
38+
39+
1. Enable the [Web Risk API](https://console.cloud.google.com/marketplace/product/google/webrisk.googleapis.com).
40+
41+
2. [Create an API Key](https://console.cloud.google.com/apis/credentials).
42+
43+
3. [Enable Billing](https://console.cloud.google.com/billing) for your account
44+
and make sure it's linked to your project.
45+
46+
# Install Docker and/or Go
47+
48+
To use the Container App, you will need [Docker](https://www.docker.com/). To
49+
compile binaries from source or run tests install [Go](https://go.dev/).
50+
51+
# Docker Quickstart (recommended)
52+
53+
We have included a Dockerfile to accelerate and simplify onboarding. This
54+
container wraps the `wrserver` binary detailed [below](#using-wrserver).
55+
56+
## Clone and Build Container
57+
58+
Building the container is straightforward.
59+
60+
First, clone this repo into a local directory.
61+
62+
```
63+
git clone https://github.com/google/webrisk && cd webrisk
64+
```
65+
66+
Build the container. This will run all tests before compiling `wrserver` into
67+
a distroless container.
68+
```
69+
docker build --tag wr-container .
70+
```
71+
72+
## Run Container
73+
74+
We supply the `APIKEY` as an environmental variable to the container at runtime
75+
so that the API Key is not revealed as part of the docker file or in `docker ps`.
76+
This example also provides a port binding.
77+
78+
```
79+
docker run -e APIKEY=XXXXXXXXXXXXXXXXXXXXXXX -p 8080:8080 wr-container
80+
```
81+
82+
`wrserver` defaults to port 8080, but you can bind any port on the host machine.
83+
See the [Docker documentation](https://docs.docker.com/config/containers/container-networking/)
84+
for details.
85+
86+
See [Using `wrserver`](#using-wrserver) below for how to query URLs or use the
87+
redirection endpoint.
88+
89+
# Go Binary Quickstart | `wrlookup` example
90+
91+
The Go Client can be compiled and run directly without Docker. In this example
92+
we will use that to run the `wrlookup` binary that takes URLs from `STDIN` and
93+
outputs to `STDOUT`.
94+
95+
Before compiling from source you should [install Go](https://go.dev/doc/install)
96+
and have some familiarity with Go development. See [here](https://go.dev/doc/tutorial/getting-started)
97+
for a good place to get started.
98+
99+
## Clone Source & Install Dependencies
100+
101+
To download and install this branch from the source, run the following commands.
102+
103+
First clone this repo into a local directory and switch to the webrisk
104+
directory.
105+
106+
```
107+
git clone https://github.com/google/webrisk && cd webrisk
108+
```
109+
110+
Next, install dependencies.
19111

20112
```
21-
go get github.com/google/webrisk
113+
go install .
22114
```
23115

24-
The programs below execute from your `$GOPATH/bin` folder.
25-
Add that to your `$PATH` for convenience:
116+
## Build and Execute `wrlookup`
117+
118+
After installing dependencies, you can build and run `wrlookup`
26119

27120
```
28-
export PATH=$PATH:$GOPATH/bin
121+
go build -o wrlookup cmd/wrlookup/main.go
29122
```
30123

31-
The program expects an API key as a parameter, export it with the following
32-
command for later use:
124+
Run the binary and supply an API key.
33125

34126
```
35-
export APIKEY=Your Api Key
127+
./wrlookup -apikey=XXXXXXXXXXXXXXXXXXXXXXX
36128
```
37129

38-
# Proxy Server
130+
You should see some output similar to below as `wrlookup` starts up.
39131

40-
The `wrserver` server binary runs a WebRisk API lookup proxy that allows
41-
users to check URLs via a simple JSON API.
132+
```
133+
webrisk: 2023/01/27 19:36:46 database.go:110: no database file specified
134+
webrisk: 2023/01/27 19:36:53 database.go:384: database is now healthy
135+
webrisk: 2023/01/27 19:36:53 webrisk_client.go:492: Next update in 30m29s
136+
```
42137

43-
1. Once the Go environment is setup, run the following command with your API key:
138+
`wrlookup` will take any URLs from `STDIN`. Test your configuration with a sample:
44139

45-
```
46-
go get github.com/google/webrisk/cmd/wrserver
47-
wrserver -apikey $APIKEY
48-
```
140+
```
141+
http://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html #input
142+
Unsafe URL: [SOCIAL_ENGINEERING_EXTENDED_COVERAGE] # output
143+
```
49144

50-
With the default settings this will start a local server at **127.0.0.1:8080**.
145+
# Using `wrserver`
51146

52-
2. The server also uses an URL redirector (listening on `/r`) to show an interstitial for anything marked unsafe.
53-
If the URL is safe, the client is automatically redirected to the target. Else, an interstitial warning page is shown as recommended by Web Risk.
54-
Try these URLs:
147+
`wrserver` runs a WebRisk API lookup proxy that allows users to check URLs via
148+
a simple JSON API. This local API will use the API key supplied by the Docker
149+
container or the command line that runs the binary.
55150

56-
```
57-
127.0.0.1:8080/r?url=http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/MALWARE/URL/
58-
127.0.0.1:8080/r?url=http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/SOCIAL_ENGINEERING/URL/
59-
127.0.0.1:8080/r?url=http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/UNWANTED_SOFTWARE/URL/
60-
127.0.0.1:8080/r?url=http://www.google.com/
61-
```
151+
First start the `wrserver` by either running the container or binary.
62152

63-
3. The server also has a lightweight implementation of the API v4 threatMatches endpoint.
64-
To use the local proxy server to check a URL, send a POST request to `127.0.0.1:8080/v1beta1/uris:search` with the following JSON body:
153+
To run in Docker:
65154

66-
```json
67-
{
68-
"uri":"http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/MALWARE/URL/",
69-
"threatTypes":[
70-
"MALWARE"
71-
]
72-
}
73-
```
155+
```
156+
docker run -e APIKEY=XXXXXXXXXXXXXXXXXXXXXXX -p 8080:8080 <container_name>
157+
```
74158

75-
# Command-Line Lookup
159+
To run from a CLI, compile as [`wrlookup`](#build-and-execute-wrlookup) above
160+
and run:
76161

77-
The `wrlookup` command-line binary is another example of how the Go Safe
78-
Browsing library can be used to protect users from unsafe URLs. This
79-
command-line tool filters unsafe URLs piped via STDIN. Example usage:
162+
```
163+
./wrserver -apikey=XXXXXXXXXXXXXXXXXXXXXXX
164+
```
165+
166+
With the default settings this will start a local server at **0.0.0.0:8080**.
167+
168+
The server has a lightweight implementation of a
169+
[Web Risk Lookup API](https://cloud.google.com/web-risk/docs/lookup-api)-like
170+
endpoint at `v1/uris:search`. To use the local endpoint to check a URL, send a
171+
POST request to `0.0.0.0:8080/v1/uris:search` with the a JSON body similar to
172+
the following.
173+
174+
```json
175+
{
176+
"uri":"http://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html"
177+
}
178+
```
179+
180+
A sample cURL command:
80181

81182
```
82-
$ go get github.com/google/webrisk/cmd/wrlookup
83-
$ echo "http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/MALWARE/URL/" | wrlookup -apikey=$APIKEY
183+
curl -H 'Content-Type: application/json' \
184+
-d '{"uri":"http://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html"}' \
185+
-X POST '0.0.0.0:8080/v1/uris:search'
84186
```
85187

188+
See [Sample URLs](#sample-urls) below to test the different blocklists.
189+
190+
`wrserver` also serves a URL redirector listening on `/r?url=...` which will
191+
show an interstitial for anything marked unsafe.
192+
193+
If the URL is safe, the client is automatically redirected to the target.
194+
Otherwise an interstitial warning page is shown as recommended by Web Risk.
195+
196+
Try some sample URLs:
197+
198+
```
199+
http://0.0.0.0:8080/r?url=https://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html
200+
http://0.0.0.0:8080/r?url=https://testsafebrowsing.appspot.com/s/malware.html
201+
http://0.0.0.0:8080/r?url=https://www.google.com/
202+
```
203+
204+
### Differences from Web Risk Lookup API
205+
206+
There are two significant differences between this local endpoint and the
207+
public [`v1/uris:search` endpoint](https://cloud.google.com/web-risk/docs/lookup-api):
208+
209+
- The public endpoint accepts `GET` requests instead of `POST` requests.
210+
- The local `wrserver` endpoint uses the privacy-preserving and lower latency
211+
[Update API](https://cloud.google.com/web-risk/docs/update-api) making it better
212+
suited for higher-demand use cases.
213+
214+
# Sample URLs
215+
216+
For testing the blocklists, you can use the following URLs:
217+
218+
- Phishing / Social Engineering: https://testsafebrowsing.appspot.com/s/phishing.html
219+
- Malware: https://testsafebrowsing.appspot.com/s/malware.html
220+
- Unwanted Software: https://testsafebrowsing.appspot.com/s/unwanted.html
221+
- Social Engineering Extended Coverage: https://testsafebrowsing.appspot.com/s/social_engineering_extended_coverage.html
222+
223+
# Troubleshooting
224+
225+
## 4XX Errors
226+
227+
If you start the client without proper credentials or project set up, you will
228+
see an error similar to what is shown below on startup:
229+
230+
```
231+
webrisk: 2023/01/27 19:36:13 database.go:217: ListUpdate failure (1): webrisk: unexpected server response code: 400
232+
```
233+
234+
For 400 errors, this usually means the API key is incorrect or was not supplied
235+
correctly.
236+
237+
For 403 errors, this could mean the Web Risk API is not enabled for your project
238+
**or** your project does not have Billing enabled.
239+
240+
# About the Social Engineering Extended Coverage List
241+
242+
This is a newer blocklist that includes a greater range of risky URLs that
243+
are not included in the Safebrowsing blocklists shipped to most browsers.
244+
The extended coverage list offers significantly more coverage, but may have
245+
a higher number of false positives. For more details, see [here](https://cloud.google.com/web-risk/docs/extended-coverage).
86246

87-
# WebRisk System Test
247+
## WebRisk System Test
88248
To perform an end-to-end test on the package with the WebRisk backend,
89-
run the following command after exporting your API key:
249+
run the following command after exporting your API key as $APIKEY:
90250

91251
```
92252
go test github.com/google/webrisk -v -run TestWebriskClient

0 commit comments

Comments
 (0)