Skip to content

Commit dd04555

Browse files
xuyongzhe-ncsujuagargi
xuyongzhe-ncsu
and
juagargi
authored
Domain owner + PCA + Policy Log (#3)
* first * change name * update_pca_policyLog * new Policy Log; Refactoring will be done in monday * Interaction between PCA and Policy log * Interaction between PCA and Policy log * Update README.md * remove executables * clean test files * refactor * refactored version * add some certs * clean up * redesign return structure * Update README.md * update images * modifies according to suggestions * modifies according to suggestions * Some instructions in README * Update README.md * clean exec and test files * clean up * change to lower cases * delete pkg * readd pkg * fix go.mod issue * use relative path * fix small bugs * final clean up * Update README.md * Update README.md * remove unwanted files * modify according to feedback * remove test data * clean up * update readme * go fmt ./... * go mod tidy * move api documentation to docs * rename docs to doc * remove spurious file * start using stretchr/testify in our unit tests * simplify SPT definition and Equal operation * simplify NewPCA * Add Makefile. Remove opaque scripts/* and replace with makefile rules. Binary files will go to bin/ Temporary file created in structure_test lives in a temporary dir and it is deleted before finishing the test (under normal conditions). * update .gitignore * simplify policylog/server/logsigner/config and its test * added two TODOs a the future fix * remove another unnecessary testdata * modification according to feedback * modify make file * add config file * unit test for verifier * minimize log server code * more comments; fix missing err handling * map server components; more comments * fix small bugs * batched smt * batched smt * map server interface * use newer lib * fix bug; deleted nodes will be removed from db * fix db leak * start implementing map server * go mod tidy * clean up; more comments * refactor for PR * small bugs * delete .DS_Store added by mistake * fix import files * fix go.mod * fix directory structure in README * add make test target * typo tire->trie * check error in test: avoid panic * no sudo to run the integration test * remove strange targets from makefile * fix doc string of public func; fix TODO string * fmt.Errorf(,err with %w not %s * rename file with typo * fix makefile after removing one target * refactor according to PR * modify scripts * add start mysql.server * log picker * go mod tidy * use the same error * fix small bugs in domain.go * stress test; more complete tests; larger batch writes size; * DONEgit add .! DO NOT read the code(messy), I will refactor tomorrow; Add PC and PC related funcs * basically done; I will refactor tomorrow * small change * more comments Co-authored-by: Juan A. Garcia Pardo <[email protected]>
1 parent cabf63a commit dd04555

File tree

93 files changed

+10167
-14
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

93 files changed

+10167
-14
lines changed

‎.gitignore

+1-8
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,8 @@
11
# Binaries for programs and plugins
2-
*.exe
3-
*.exe~
4-
*.dll
5-
*.so
6-
*.dylib
2+
bin/*
73

84
# Test binary, built with `go test -c`
95
*.test
106

117
# Output of the go coverage tool, specifically when used with LiteIDE
128
*.out
13-
14-
# Dependency directories (remove the comment below to include it)
15-
# vendor/

‎Makefile

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
.PHONY: all clean test policy_log
2+
3+
all: build_policy_log build_integration_test build_benchmark
4+
5+
clean:
6+
@rm -f bin/*
7+
8+
test:
9+
@go test ./...
10+
11+
build_policy_log:
12+
@go build -o bin/logserver_exec cmd/logserver/logserver_exec.go
13+
@go build -o bin/logsigner_exec cmd/logsigner/logsigner_exec.go
14+
15+
setup_db: start_db create_smt_database create_log_database create_tables
16+
17+
start_db:
18+
@mysql.server start
19+
20+
create_smt_database:
21+
@mysql -u root -e "CREATE SCHEMA IF NOT EXISTS \`map\`;"
22+
23+
create_tables:
24+
@mysql -u root -e "CREATE TABLE IF NOT EXISTS \`map\`.\`cacheStore\` (\`key\` VARCHAR(64) NOT NULL,\`value\` VARCHAR(2048) NOT NULL, PRIMARY KEY (\`key\`));"
25+
@mysql -u root -e "CREATE TABLE IF NOT EXISTS \`map\`.\`deleteTest\` (\`key\` VARCHAR(64) NOT NULL,\`value\` VARCHAR(2048) NOT NULL, PRIMARY KEY (\`key\`));"
26+
@mysql -u root -e "CREATE TABLE IF NOT EXISTS \`map\`.\`domainEntries\` (\`key\` VARCHAR(64) NOT NULL,\`value\` LONGTEXT NOT NULL, PRIMARY KEY (\`key\`));"
27+
@mysql -u root -e "CREATE TABLE IF NOT EXISTS \`map\`.\`updatedDomains\` (\`domainHash\` VARCHAR(64) NOT NULL, PRIMARY KEY (\`domainHash\`));"
28+
29+
create_log_database:
30+
@./scripts/reset_db/resetdb.sh
31+
32+
33+
34+
build_integration_test:
35+
@go build -o ./bin/policylog_interaction ./tests/intergration_tests/policylog_interaction
36+
@go build -o ./bin/domainowner_pca_policlog_interaction ./tests/intergration_tests/domainowner_pca_policlog_interaction
37+
@go build -o ./bin/mapserver ./tests/intergration_tests/mapserver
38+
@go build -o ./bin/smt ./tests/intergration_tests/smt
39+
@go build -o ./bin/log_picker ./tests/intergration_tests/log_picker
40+
41+
drop_cacheTable:
42+
@mysql -u root -e "DROP TABLE map.deleteTest;"
43+
44+
run_integration_test:
45+
@./scripts/integration_tests.sh
46+
47+
build_benchmark:
48+
@go build -o ./bin/log_benchmark ./tests/benchmark/logserver_benchmark
49+
@go build -o ./bin/smt_benchmark ./tests/benchmark/smt_benchmark
50+
51+
run_log_benchmark:
52+
@./scripts/log_benchmark.sh
53+
54+
run_smt_benchmark:
55+
@./bin/smt_benchmark

‎README.md

+68-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,68 @@
1-
# fpki
1+
# FPKI
2+
3+
## Features
4+
5+
- Issuance and logging of RPC (Root Policy Certificate)
6+
- Issuance of SPT (Signed Policy Certificate) using RPC
7+
- Verification of RPC using SPT
8+
- Consistency verification of the log
9+
10+
## Dir structure
11+
12+
.
13+
├── cert # certificates for testing. Eg. cert for PCA
14+
├── cmd # Executables. Log server and log signer
15+
├── config # Config file for PCA and policy log
16+
├── doc # FPKI design
17+
│   ├── images
18+
│   └── performance_test_output # Some results of the policy log performance
19+
├── pkg # Packages of the project
20+
├── scripts # Scripts to clean and build the executables
21+
└── tests # Some intergration tests. Unit tests are in the individual pkg
22+
23+
24+
## Overview
25+
![Alt text](doc/images/overview.png?raw=true"Overview")
26+
27+
The figure above illustrates the components so far. The domain owner sends RCSR to the PCA, and PCA signs the RCSR to generate an RPC. Then the PCA sends the RPC to the policy log to get an SPT. The log verifier will verify the SPT and the consistency between the old tree head and the newest tree head.
28+
29+
### Policy log
30+
Trillian is used in the policy log.
31+
32+
The policy log is the most complex component so far. It consists of four sub-components:
33+
- **(Log server)** Log server is responsible for receiving and sending responses. However, it does not generate proof of inclusion. It is similar to a user interface, which handles the RPC request and distributes the result.
34+
- **(Log signer)** Log signer is responsible for adding the new leaves, generating the new tree head, and the proof of inclusion for every added leaf.
35+
- **(Log client)** Log client is responsible for sending the new leaves to the log server and retrieving information from the log server.
36+
- **(Admin client)** Admin client is responsible for managing trees in the log server. For example, create a new tree or delete an existing tree.
37+
38+
Within the policy log, the log client and admin client only communicate with the log server via grpc. Log signer only communicates with the log server, and the communication is internal, so we don't have access to it. For other components outside the policy log, they can only have access to the log client to add leaves of fetch proof of inclusion. Admin client should only be accessed internally.
39+
40+
## How to run the integration tests
41+
There are two integration tests which require the setup of Trillian server.
42+
43+
Download trillian and setup database (more information on https://github.com/google/trillian#mysql-setup)
44+
```
45+
git clone https://github.com/google/trillian.git
46+
cd scripts
47+
./resetdb.sh
48+
```
49+
Download the FPKI
50+
51+
Conpile the executable
52+
```
53+
make
54+
make test_folders
55+
```
56+
57+
Open two terminals, run "cmd/logserver_exec" and "cmd/logsigner_exec"
58+
59+
Run the tests:
60+
```
61+
cd fpki
62+
go test ./...
63+
```
64+
65+
66+
67+
68+

‎bin/.keepme

Whitespace-only changes.

‎cmd/logserver/logserver_exec.go

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
package main
2+
3+
import (
4+
"github.com/netsec-ethz/fpki/pkg/policylog/server/logserver"
5+
)
6+
7+
func main() {
8+
logserver.CreateLogServer("./config/logserver_config.json")
9+
}

‎cmd/logsigner/logsigner_exec.go

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
package main
2+
3+
import (
4+
"github.com/netsec-ethz/fpki/pkg/policylog/server/logsigner"
5+
)
6+
7+
func main() {
8+
logsigner.CreateLogSigner("./config/logsigner_config.json")
9+
}

‎config/adminclient_config.json

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"rpcMaxWaitingTimeInSec": 10,
3+
"hashStrategy": "RFC6962_SHA256",
4+
"displayName": "policy_log",
5+
"description": "tree for policy log",
6+
"maxRootDuration": 10,
7+
"maxReceiveMessageSize": 10000,
8+
"logAddress": "localhost:8090",
9+
"OutPutPath": "./file_exchange/policylog"
10+
}

‎config/certs/server_key.pem

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
-----BEGIN RSA PRIVATE KEY-----
2+
MIIEowIBAAKCAQEArrrQ5MN4mdcp5XouqmcmPG489eRtbkIn9elKOCDLgpA9OFAS
3+
KM26Vskm0jwR9unrVE8NXXdRbotQfVpL7iAPGOPfoSglBXKmiAdmRG0idw6+xRlp
4+
ffgHE3CDhNnz1tpVXBTE+U84f48v+sVd1gnK4oA/uT7X7D6vO5cHK1M9rmpo+SiK
5+
lcYSHvF19/qgiwF9cc1z3ug6M4SciqEbUNdW1R3BSW+9ulTZluT4Hbml4C8hkktN
6+
9zlHUpWdHzH1NlcRqzObBp7ZvB/OrKh8iA0WBXLXNzlBdB9EXSHjqJcI/sKn0Zf/
7+
5RO9QYT8wjDDbj8H+4+/wRd2q8Y10yQomIy6WQIDAQABAoIBAHOQ2C0WJCQMLYe7
8+
ojXOd3msp+EyrjYox2lcXVx2RGQcpoEiZIBcYYyZCnYuiOs4mA9xIbPaASAOQLRI
9+
DsR2q8WYogZjGOlbhU2fJ0LGs+7u77pJLn7El1mCJ1qtFRvb2G1Ix2blwMaNm/xO
10+
odu5KllY3czWBdYZN6lAINvE+JzsZ9/OLSEO7zvbEThfe2FQO8u4oz0/7EilPU+F
11+
pBH4gXSYi2He8Ngasyg5K5CpZp6WcQgL/iP4F+xNkDxo6UNQ0KJyfNelOnZvX35M
12+
JJyQa6ewnhfBKrmNIw0/RzRIdOm3NoNlpkwlepRxYyeNoZfBKE96yF6+IIVkhZkv
13+
lIRFzKkCgYEA5EOmHHwK4nXRCau2KfE0JGE68Fo+x6m5UImpuuO9Vik/Ca4icJc4
14+
3kIJcVm8emzwXmKgjp3ZU6OJ0c+H0vNMsIN8BI9DCmMf2A9dAsET7H5na1fvloRK
15+
hu11Rps+jVpAIPG4SHlJAqbCJXl5uuwHPMciOHlkvFvsg/hLE7g8CbcCgYEAw/Xu
16+
dNyIf9vnhMOEPMPOwkAQxxJBqKIwRwC5df8/SNLumrlZf7KmjUk7jwXq3Y/3LC34
17+
yaxICIqS6F5jJWQE3RcdYBha1zprc8yakCIPkXLABu/jTX0IG0SYHfthtn0Xu9v2
18+
kwqg1usPDc1n09TBWuqz/9LaAb3TOk7bmWgvnG8CgYEAhe9DqncGE/a5XwlH67hB
19+
tg7u/FSXfaQSvjToSGaUk9fDyzRHtPSmQ5NP4xiaRONi7p7EQqpP+Il+toMk2Rw1
20+
JWUhAbPlzFR5PNdW5eTQgfVxTUTdqd2ZtS3joDg7LcmE00/Bp5SyyEF8rWxnhswz
21+
OvBFHgnsl7EASDcKJln1sGUCgYBus50tOXSHPpSQ5N7scPV5uTWBMGgVOuKpCdi7
22+
tyFqI3cVw0WzJRp/0HIr/vzKg8zvkVztmTzsTlRlnwTFl7ZDrA41Piz7T45qUZiz
23+
efpvgXLHuHchTFPrC5ofveZacFqbDs1CDgwEva+4bWSNP7CnitV63QO5XiJY2zml
24+
dT0IUwKBgHKA1JAcslRM7/2ctyIC7ocifVdOZHUDBpYNFzDAAIWFTMOGEf1CPXQl
25+
RWBtTicGNC2Rs+UQfTcS9S72qcElJQk+P11L8KoFoqrYZYD8Ff0Gmo456GXTCZf6
26+
JGyRL6+owmzb2d4jJVBytNrxG2/yGwcYFIKR/fB5p0O3fzPLzG4b
27+
-----END RSA PRIVATE KEY-----

‎config/logclient_config.json

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"TreeId": 55555555555,
3+
"RPCAddress": "localhost:8090",
4+
"MaxReceiveMessageSize": 10000,
5+
"OutPutPath": "./file_exchange/policylog",
6+
"RPCPath": "./file_exchange/pcaoutput/rpc",
7+
"NumOfWorker": 15
8+
}

‎config/logserver_config.json

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"RpcEndpoint": "localhost:8090",
3+
"HttpEndpoint": "localhost:8091",
4+
"HealthzTimeout": 5,
5+
"EtcdService": "trillian-logserver",
6+
"EtcdHTTPService": "trillian-logserver-http",
7+
"QuotaSystem": "mysql",
8+
"StorageSystem": "mysql",
9+
"TreeGCEnabled": true,
10+
"TreeDeleteThresholdInHour": 168,
11+
"TreeDeleteMinRunIntervalInHour": 4
12+
}

‎config/logsigner_config.json

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{
2+
"RpcEndpoint": "localhost:8092",
3+
"HttpEndpoint": "localhost:8093",
4+
"TlsCertFile": "",
5+
"TlsKeyFile": "",
6+
"HealthzTimeoutInSec": 5,
7+
"QuotaSystem": "mysql",
8+
"StorageSystem": "mysql",
9+
"ConfigFile": "",
10+
"SequencerIntervalFlagInMillSec": 100,
11+
"BatchSizeFlag": 1000,
12+
"NumSeqFlag": 10,
13+
"SequencerGuardWindowFlag": 0,
14+
"ForceMaster": true,
15+
"EtcdHTTPService": "trillian-logsigner-http",
16+
"LockDir": "/test/multimaster",
17+
"QuotaIncreaseFactor": 0,
18+
"PreElectionPauseInSec": 1,
19+
"MasterHoldIntervalInSec": 60,
20+
"MasterHoldJitterInSec": 120,
21+
"CpuProfile": "",
22+
"MemProfile": ""
23+
}

‎config/pca_config.json

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"CAName": "pca",
3+
"KeyPath": "./config/certs/server_key.pem",
4+
"OutputPath": "./file_exchange/pcaoutput",
5+
"PolicyLogOutputPath": "./file_exchange/policylog"
6+
}

‎doc/NewDesignFinal.pdf

1.77 MB
Binary file not shown.

‎doc/User_Manual.pdf

42.1 KB
Binary file not shown.

‎doc/images/overview.png

334 KB
Loading

‎doc/performance_test_output/result

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
For 10000 leaves
2+
3+
queue leaves succeed!
4+
8.434761293s
5+
fetch proofs succeed!
6+
8.656676005s
7+
8+
AVG: 0.8 ms for adding, 0.8 ms for querying the proof
9+
10+
11+
Every second 500 leaves
12+
Every minute 30000 leaves
13+
Every hour 1.8 million leaves
14+
15+
16+
There are 367 millions domains in total(does not change every year, even declines a little last year). Assume update frequency is 4 per year. Daily updates will be 8 millions(one RPC, one PC). It takes 4 hours to append and retrieve the daily updates. Even my lap top can do the job for all the domains -:)

0 commit comments

Comments
 (0)