feat: added support for multiple cassette storage backends (AWS S3) (#…

…108) * added s3file > WriteFile * localstack + add tests * localstack + add tests * add tests for s3file ReadFile * update CI to spin up localstack container * add tests for S3Storage NotExists * Added Recipe: VCR with a custom RequestMatcher to README * added example5 for AWS storage * bump to v14 * doc: spread the word about AWS S3 storage support
seborama · Jul 29, 2023 · 2861b4c · 2861b4c
1 parent 88b23dd
commit 2861b4c
Show file tree

Hide file tree

Showing 40 changed files with 582 additions and 80 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -23,14 +23,29 @@ jobs:
 
   test:
     name: Test
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
 
+      - name: Start LocalStack
+        run: |
+          pip install localstack awscli-local[ver1] # install LocalStack cli and awslocal
+          docker pull localstack/localstack         # Make sure to pull the latest version of the image
+          localstack start -d                       # Start LocalStack in the background
+          
+          echo "Waiting for LocalStack startup..."  # Wait 30 seconds for the LocalStack container
+          localstack wait -t 30                     # to become ready before timing out 
+          echo "Startup complete"          
+ 
       - name: Set up Go
         uses: actions/setup-go@v2
         with:
           go-version: 1.17
 
       - name: Test
         run: make test
+        env:
+          LOCALSTACK_ENDPOINT: http://localhost:4566
+          AWS_DEFAULT_REGION: eu-west-1
+          AWS_ACCESS_KEY_ID: test
+          AWS_SECRET_ACCESS_KEY: test
diff --git a/.gitignore b/.gitignore
@@ -32,3 +32,10 @@ _testmain.go
 *.prof
 /coverage.out
 /bin
+
+/volume
+
+.env
+.direnv
+.envrc
+
diff --git a/README.md b/README.md
@@ -24,12 +24,12 @@
     <img src="https://github.com/seborama/govcr/actions/workflows/codeql-analysis.yml/badge.svg?branch=master" alt="govcr">
   </a>
 
-  <a href="https://pkg.go.dev/github.com/seborama/govcr/v13">
+  <a href="https://pkg.go.dev/github.com/seborama/govcr/v14">
     <img src="https://img.shields.io/badge/godoc-reference-blue.svg" alt="govcr">
   </a>
 
-  <a href="https://goreportcard.com/report/github.com/seborama/govcr/v13">
-    <img src="https://goreportcard.com/badge/github.com/seborama/govcr/v13" alt="govcr">
+  <a href="https://goreportcard.com/report/github.com/seborama/govcr/v14">
+    <img src="https://goreportcard.com/badge/github.com/seborama/govcr/v14" alt="govcr">
   </a>
 </p>
 
@@ -58,6 +58,7 @@ This project is an adaptation for Google's Go / Golang programming language.
   - [Recipe: VCR with encrypted cassette - custom nonce generator](#recipe-vcr-with-encrypted-cassette---custom-nonce-generator)
   - [Recipe: Cassette decryption](#recipe-cassette-decryption)
   - [Recipe: Changing cassette encryption](#recipe-changing-cassette-encryption)
+  - [Recipe: VCR with cassette storage on AWS S3](#recipe-vcr-with-cassette-storage-on-aws-s3)
   - [Recipe: VCR with a custom RequestMatcher](#recipe-vcr-with-a-custom-requestmatcher)
   - [Recipe: VCR with a replaying Track Mutator](#recipe-vcr-with-a-replaying-track-mutator)
   - [Recipe: VCR with a recording Track Mutator](#recipe-vcr-with-a-recording-track-mutator)
@@ -97,15 +98,15 @@ We use a "relaxed" request matcher because `example.com` injects an "`Age`" head
 ## Install
 
 ```bash
-go get github.com/seborama/govcr/v13@latest
+go get github.com/seborama/govcr/v14@latest
 ```
 
 For all available releases, please check the [releases](https://github.com/seborama/govcr/releases) tab on github.
 
 And your source code would use this import:
 
 ```go
-import "github.com/seborama/govcr/v13"
+import "github.com/seborama/govcr/v14"
 ```
 
 For versions of **govcr** before v5 (which don't use go.mod), use a dependency manager to lock the version you wish to use (perhaps v4)!
@@ -119,7 +120,7 @@ go get gopkg.in/seborama/govcr.v4
 
 ## Glossary of Terms
 
-**VCR**: Video Cassette Recorder. In this context, a VCR refers to the engine and data that this project provides. A VCR is both an HTTP recorder and player. When you use a VCR, HTTP requests are replayed from previous recordings (**tracks** saved in **cassette** files on the filesystem). When no previous recording exists for the request, it is performed live on the HTTP server, after what it is saved to a **track** on the **cassette**.
+**VCR**: Video Cassette Recorder. In this context, a VCR refers to the engine and data that this project provides. A VCR is both an HTTP recorder and player. When you use a VCR, HTTP requests are replayed from previous recordings (**tracks** saved in **cassette** files, on the filesystem or in AWS S3, etc). When no previous recording exists for the request, it is performed live on the HTTP server, after what it is saved to a **track** on the **cassette**.
 
 **cassette**: a sequential collection of **tracks**. This is in effect a JSON file.
 
@@ -135,7 +136,9 @@ go get gopkg.in/seborama/govcr.v4
 
 **govcr** is a wrapper around the Go `http.Client`. It can record live HTTP traffic to files (called "**cassettes**") and later replay HTTP requests ("**tracks**") from them instead of live HTTP calls.
 
-The code documentation can be found on [godoc](https://pkg.go.dev/github.com/seborama/govcr/v13).
+Cassette files can be stored on the filesystem or on a cloud storage service (AWS S3), etc.
+
+The code documentation can be found on [godoc](https://pkg.go.dev/github.com/seborama/govcr/v14).
 
 When using **govcr**'s `http.Client`, the request is matched against the **tracks** on the '**cassette**':
 
@@ -438,7 +441,7 @@ vcr := govcr.NewVCR(
 The command is located in the `cmd/govcr` folder, to install it:
 
 ```bash
-go install github.com/seborama/govcr/v13/cmd/govcr@latest
+go install github.com/seborama/govcr/v14/cmd/govcr@latest
 ```
 
 Example usage:
@@ -467,6 +470,23 @@ err := vcr.SetCipher(
 
 [(toc)](#table-of-content)
 
+### Recipe: VCR with cassette storage on AWS S3
+
+At time of creating a new VCR with **govcr**, provide an initialised S3 client:
+
+```go
+// See TestExample5 in tests for fully working example.
+s3Client := /* ... */
+s3f := fileio.NewAWS(s3Client)
+
+vcr := govcr.NewVCR(govcr.
+    NewCassetteLoader(exampleCassetteName5).
+    WithStore(s3f),
+)
+```
+
+[(toc)](#table-of-content)
+
 ### Recipe: VCR with a custom RequestMatcher
 
 This example shows how to handle situations where a header in the request needs to be ignored, in this case header `X-Custom-Timestamp` (or the **track** would not match and hence would not be replayed).

diff --git a/cassette/cassette.go b/cassette/cassette.go
@@ -14,12 +14,12 @@ import (
 	"github.com/google/uuid"
 	"github.com/pkg/errors"
 
-	"github.com/seborama/govcr/v13/cassette/track"
-	"github.com/seborama/govcr/v13/compression"
-	cryptoerr "github.com/seborama/govcr/v13/encryption/errors"
-	govcrerr "github.com/seborama/govcr/v13/errors"
-	"github.com/seborama/govcr/v13/fileio"
-	"github.com/seborama/govcr/v13/stats"
+	"github.com/seborama/govcr/v14/cassette/track"
+	"github.com/seborama/govcr/v14/compression"
+	cryptoerr "github.com/seborama/govcr/v14/encryption/errors"
+	govcrerr "github.com/seborama/govcr/v14/errors"
+	"github.com/seborama/govcr/v14/fileio"
+	"github.com/seborama/govcr/v14/stats"
 )
 
 // Cassette contains a set of tracks.
@@ -38,7 +38,7 @@ type FileIO interface {
 	MkdirAll(path string, perm os.FileMode) error
 	ReadFile(name string) ([]byte, error)
 	WriteFile(name string, data []byte, perm os.FileMode) error
-	IsNotExist(err error) bool
+	NotExist(name string) (bool, error)
 }
 
 const (
@@ -69,13 +69,13 @@ func WithCrypter(crypter Crypter) Option {
 }
 
 // WithStore provides a dedicated storage engine for the cassette data.
-func WithStore(crypter Crypter) Option {
+func WithStore(store FileIO) Option {
 	return func(k7 *Cassette) {
-		if k7.crypter != nil {
-			log.Println("notice: setting a crypter but another one had already been registered - this is incorrect usage")
+		if k7.store != nil {
+			log.Println("notice: setting a storer but another one had already been registered - this is incorrect usage")
 		}
 
-		k7.crypter = crypter
+		k7.store = store
 	}
 }
 
@@ -315,11 +315,14 @@ func (k7 *Cassette) readCassette(cassetteName string) ([]byte, error) {
 		k7.store = &fileio.OSFile{}
 	}
 
+	if notExist, err := k7.store.NotExist(cassetteName); err != nil {
+		return nil, errors.Wrap(err, "failed to check cassette existence")
+	} else if notExist {
+		return nil, nil // not found, return nil data
+	}
+
 	data, err := k7.store.ReadFile(cassetteName)
 	if err != nil {
-		if k7.store.IsNotExist(err) {
-			return nil, nil // not found, return nil data
-		}
 		return nil, errors.Wrap(err, "failed to read cassette data from source")
 	}
 
@@ -431,7 +434,7 @@ func LoadCassette(cassetteName string, opts ...Option) *Cassette {
 
 	data, err := k7.readCassette(cassetteName)
 	if err != nil {
-		panic(fmt.Sprintf("unable to invalid / load corrupted cassette '%s': %+v", cassetteName, err))
+		panic(fmt.Sprintf("unable to load invalid / corrupted cassette '%s': %+v", cassetteName, err))
 	}
 
 	if data != nil {

diff --git a/cassette/cassette_test.go b/cassette/cassette_test.go
@@ -8,9 +8,9 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/seborama/govcr/v13/cassette"
-	"github.com/seborama/govcr/v13/cassette/track"
-	"github.com/seborama/govcr/v13/encryption"
+	"github.com/seborama/govcr/v14/cassette"
+	"github.com/seborama/govcr/v14/cassette/track"
+	"github.com/seborama/govcr/v14/encryption"
 )
 
 func Test_cassette_GzipFilter(t *testing.T) {

diff --git a/cassette/cassette_wb_test.go b/cassette/cassette_wb_test.go
@@ -5,7 +5,7 @@ import (
 
 	"github.com/stretchr/testify/assert"
 
-	"github.com/seborama/govcr/v13/stats"
+	"github.com/seborama/govcr/v14/stats"
 )
 
 func Test_cassette_NumberOfTracks_PanicsWhenNoCassette(t *testing.T) {

diff --git a/cassette/track/http_test.go b/cassette/track/http_test.go
@@ -9,7 +9,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/seborama/govcr/v13/cassette/track"
+	"github.com/seborama/govcr/v14/cassette/track"
 )
 
 func TestRequest_Clone(t *testing.T) {

diff --git a/cassette/track/mutator_test.go b/cassette/track/mutator_test.go
@@ -10,7 +10,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/seborama/govcr/v13/cassette/track"
+	"github.com/seborama/govcr/v14/cassette/track"
 )
 
 func Test_Mutator_On(t *testing.T) {

diff --git a/cassette/track/track.go b/cassette/track/track.go
@@ -9,7 +9,7 @@ import (
 
 	"github.com/pkg/errors"
 
-	trkerr "github.com/seborama/govcr/v13/cassette/track/errors"
+	trkerr "github.com/seborama/govcr/v14/cassette/track/errors"
 )
 
 // Track is a recording (HTTP Request + Response) in a cassette.

diff --git a/cassette/track/track_test.go b/cassette/track/track_test.go
@@ -13,7 +13,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/seborama/govcr/v13/cassette/track"
+	"github.com/seborama/govcr/v14/cassette/track"
 )
 
 func TestTrack_ToHTTPResponse(t *testing.T) {

diff --git a/cmd/govcr/main.go b/cmd/govcr/main.go
@@ -7,8 +7,8 @@ import (
 
 	"github.com/pkg/errors"
 
-	"github.com/seborama/govcr/v13/cassette"
-	"github.com/seborama/govcr/v13/encryption"
+	"github.com/seborama/govcr/v14/cassette"
+	"github.com/seborama/govcr/v14/encryption"
 )
 
 func main() {

diff --git a/concurrency_test.go b/concurrency_test.go
@@ -15,8 +15,8 @@ import (
 	"github.com/pkg/errors"
 	"github.com/stretchr/testify/require"
 
-	"github.com/seborama/govcr/v13"
-	"github.com/seborama/govcr/v13/stats"
+	"github.com/seborama/govcr/v14"
+	"github.com/seborama/govcr/v14/stats"
 )
 
 func TestConcurrencySafety(t *testing.T) {

diff --git a/controlpanel.go b/controlpanel.go
@@ -3,8 +3,8 @@ package govcr
 import (
 	"net/http"
 
-	"github.com/seborama/govcr/v13/cassette/track"
-	"github.com/seborama/govcr/v13/stats"
+	"github.com/seborama/govcr/v14/cassette/track"
+	"github.com/seborama/govcr/v14/stats"
 )
 
 // ControlPanel holds the parts of a VCR that can be interacted with.

diff --git a/controlpanel_wb_test.go b/controlpanel_wb_test.go
@@ -6,7 +6,7 @@ import (
 
 	"github.com/stretchr/testify/assert"
 
-	"github.com/seborama/govcr/v13/cassette/track"
+	"github.com/seborama/govcr/v14/cassette/track"
 )
 
 func TestControlPanel_SetRecordingMutators(t *testing.T) {

diff --git a/docker-compose.yaml b/docker-compose.yaml
@@ -0,0 +1,17 @@
+version: "3.8"
+
+services:
+  localstack:
+    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
+    image: localstack/localstack
+    ports:
+      - "127.0.0.1:4566:4566"            # LocalStack Gateway
+      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
+    environment:
+      - DEBUG=${DEBUG-}
+      - DOCKER_HOST=unix:///var/run/docker.sock
+    volumes:
+      - "${LOCALSTACK_VOLUME_DIR:-/tmp/lima/govcr}:/var/lib/localstack"
+      - "/var/run/docker.sock:/var/run/docker.sock"
+
+
diff --git a/encryption/.study/rsa.go b/encryption/.study/rsa.go
@@ -11,7 +11,7 @@ import (
 	"github.com/pkg/errors"
 	"golang.org/x/crypto/ssh"
 
-	cryptoerr "github.com/seborama/govcr/v13/encryption/errors"
+	cryptoerr "github.com/seborama/govcr/v14/encryption/errors"
 )
 
 // nolint: deadcode

diff --git a/encryption/aesgcm.go b/encryption/aesgcm.go
@@ -6,7 +6,7 @@ import (
 
 	"github.com/pkg/errors"
 
-	cryptoerr "github.com/seborama/govcr/v13/encryption/errors"
+	cryptoerr "github.com/seborama/govcr/v14/encryption/errors"
 )
 
 // NewAESGCMWithRandomNonceGenerator creates a new Cryptor initialised with an

diff --git a/encryption/aesgcm_test.go b/encryption/aesgcm_test.go
@@ -6,7 +6,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/seborama/govcr/v13/encryption"
+	"github.com/seborama/govcr/v14/encryption"
 )
 
 func TestCryptor_AESGCM(t *testing.T) {

diff --git a/encryption/chacha20poly1305_test.go b/encryption/chacha20poly1305_test.go
@@ -6,7 +6,7 @@ import (
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
-	"github.com/seborama/govcr/v13/encryption"
+	"github.com/seborama/govcr/v14/encryption"
 )
 
 func TestCryptor_ChaCha20Poly1305(t *testing.T) {

diff --git a/examples/Example1_test.go b/examples/Example1_test.go
@@ -6,8 +6,8 @@ import (
 
 	"github.com/stretchr/testify/assert"
 
-	"github.com/seborama/govcr/v13"
-	"github.com/seborama/govcr/v13/stats"
+	"github.com/seborama/govcr/v14"
+	"github.com/seborama/govcr/v14/stats"
 )
 
 const exampleCassetteName1 = "temp-fixtures/TestExample1.cassette.json"

diff --git a/examples/Example2_test.go b/examples/Example2_test.go
@@ -6,7 +6,7 @@ import (
 	"testing"
 	"time"
 
-	"github.com/seborama/govcr/v13"
+	"github.com/seborama/govcr/v14"
 )
 
 const exampleCassetteName2 = "temp-fixtures/TestExample2.cassette.json"

diff --git a/examples/Example3_test.go b/examples/Example3_test.go
@@ -7,8 +7,8 @@ import (
 	"testing"
 
 	"github.com/google/uuid"
-	"github.com/seborama/govcr/v13"
-	"github.com/seborama/govcr/v13/cassette/track"
+	"github.com/seborama/govcr/v14"
+	"github.com/seborama/govcr/v14/cassette/track"
 	"github.com/stretchr/testify/require"
 )
-Original file line number
+Diff line change
@@ Expand Up / @@ -32,3 +32,10 @@ _testmain.go @@
     *.prof
     /coverage.out
     /bin
+    /volume
+    .env
+    .direnv
+    .envrc