Skip to content

Commit 3ee1cbe

Browse files
Feat/jobs snapshot instance (#86)
* feat(jobs): add example to snapshot instance volumes * fix string * update main readme * add improvements * Update jobs/instances-snapshot/README.md Co-authored-by: Ramy <[email protected]> --------- Co-authored-by: Ramy <[email protected]>
1 parent 81241ab commit 3ee1cbe

File tree

7 files changed

+207
-0
lines changed

7 files changed

+207
-0
lines changed

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ Table of Contents:
8181
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------- |
8282
| **[Serverless Jobs Hello World](jobs/terraform-hello-world/README.md)** <br/> An example of building a container image and running it as a Serverless Job using Terraform. | N/A | [Terraform]-[Console] |
8383
| **[Serverless MLOps](jobs/ml-ops/README.md)** <br/> An example of running a Serverless Machine Leaning workflow. | Python | [Terraform]-[Console]-[CLI] |
84+
| **[Auto Snapshot Instances](jobs/instances-snapshot/README.md)** <br/> Use Serverless Jobs to create snapshots of your instances | Go | [Console] |
8485

8586
### 💬 Messaging and Queueing
8687

jobs/instances-snapshot/Dockerfile

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Using apline/golang image
2+
FROM golang:1.22-alpine
3+
4+
# Set destination for COPY
5+
WORKDIR /app
6+
7+
# Copy required files
8+
COPY go.mod ./
9+
COPY go.sum ./
10+
COPY *.go ./
11+
12+
# Build the executable
13+
RUN go build -o /jobs-snapshot
14+
15+
# Run the executable
16+
CMD [ "/jobs-snapshot" ]

jobs/instances-snapshot/README.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Serverless Jobs for automatic Instance snapshot
2+
3+
This project shows how it's possible to automate tasks using Serverless Jobs.
4+
5+
This example is very simple, it generates snapshots of your desired Instance.
6+
7+
# Set-up
8+
9+
## Requirements
10+
11+
- Scaleway Account
12+
- Docker daemon running to build the image
13+
- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
14+
15+
## Step 1 : Build and push to Container registry
16+
17+
Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
18+
with Containers. So first, using your terminal reach this folder and run the following commands:
19+
20+
```shell
21+
# First command is to login to container registry, you can find it in Scaleway console
22+
docker login rg.fr-par.scw.cloud/jobs-snapshot -u nologin --password-stdin <<< "$SCW_SECRET_KEY"
23+
24+
# Here we build the image to push
25+
docker build -t rg.fr-par.scw.cloud/jobs-snapshot/jobs-snapshot:v1 .
26+
27+
# Push the image online to be used on Serverless Jobs
28+
docker push rg.fr-par.scw.cloud/jobs-snapshot/jobs-snapshot:v1
29+
```
30+
31+
Note about Serverless Containers versus Serverless Jobs. As we do not expose a web server and we do not
32+
require features such as auto-scaling, Serverless Jobs are perfect for this use case.
33+
34+
To check if everyting is ok, on the Console you can verify if your tag is present in Container Registry.
35+
36+
## Step 2: Creating the Job Definition
37+
38+
On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par
39+
40+
1. On Container image, select the image you created in the step before.
41+
1. You can set the image name to something clear like `jobs-snapshot` too.
42+
1. For the region you can select the one you prefer :)
43+
1. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory.
44+
1. To schedule your job for example every night at 2am, you can set the cron to `0 2 * * *`.
45+
1. Important: advanced option, you need to set the following environment variables:
46+
47+
- `INSTANCE_ID`: grab the instance ID you want to create snapshots from
48+
- `INSTANCE_ZONE`: you need to give the ZONE of you instance, like `fr-par-2`
49+
- `SCW_ACCESS_KEY`: your access key
50+
- `SCW_SECRET_KEY`: your secret key
51+
- `SCW_DEFAULT_ORGANIZATION_ID`: your organzation ID
52+
53+
1. Click "create job"
54+
55+
## Step 3: Run the job
56+
57+
On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
58+
59+
# Possible improvements
60+
61+
You can exercice by adding the following features:
62+
63+
- Instead of managing a single instance, make it account wide
64+
- Add disk backups
65+
- Add alerts if something goes wrong
66+
67+
# Additional content
68+
69+
- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/)
70+
- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/)
71+
- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/)
72+
- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/)

jobs/instances-snapshot/go.mod

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
module github.com/scaleway/serverless-examples/jobs/instances-snapshot
2+
3+
go 1.22.2
4+
5+
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.21
6+
7+
require gopkg.in/yaml.v2 v2.4.0 // indirect

jobs/instances-snapshot/go.sum

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
2+
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
3+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.21 h1:yWfiTPwYxB0l5fGMhl/G+liULugVIHD9AU77iNLrURQ=
4+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.21/go.mod h1:fCa7OJZ/9DRTnOKmxvT6pn+LPWUptQAmHF/SBJUGEcg=
5+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
6+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
7+
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
8+
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=

jobs/instances-snapshot/main.go

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"os"
6+
7+
"github.com/scaleway/scaleway-sdk-go/api/instance/v1"
8+
"github.com/scaleway/scaleway-sdk-go/scw"
9+
)
10+
11+
func main() {
12+
fmt.Println("creating snapshot of instance...")
13+
14+
// Create a Scaleway client with credentials from environment variables.
15+
client, err := scw.NewClient(
16+
// Get your organization ID at https://console.scaleway.com/organization/settings
17+
scw.WithDefaultOrganizationID(os.Getenv("SCW_DEFAULT_ORGANIZATION_ID")),
18+
19+
// Get your credentials at https://console.scaleway.com/iam/api-keys
20+
scw.WithAuth(os.Getenv("SCW_ACCESS_KEY"), os.Getenv("SCW_SECRET_KEY")),
21+
22+
// Get more about our availability zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/
23+
scw.WithDefaultRegion(scw.RegionFrPar),
24+
)
25+
if err != nil {
26+
panic(err)
27+
}
28+
29+
// Create SDK objects for Scaleway Instance product
30+
instanceAPI := instance.NewAPI(client)
31+
32+
if err := createSnapshots(instanceAPI); err != nil {
33+
panic(err)
34+
}
35+
}
36+
37+
func createSnapshots(instanceAPI *instance.API) error {
38+
gotInstance, err := instanceAPI.GetServer(&instance.GetServerRequest{
39+
ServerID: os.Getenv("INSTANCE_ID"),
40+
Zone: scw.Zone(os.Getenv("INSTANCE_ZONE")),
41+
})
42+
if err != nil {
43+
return fmt.Errorf("error while getting instance %w", err)
44+
}
45+
46+
for _, volume := range gotInstance.Server.Volumes {
47+
snapshotResp, err := instanceAPI.CreateSnapshot(&instance.CreateSnapshotRequest{
48+
Name: volume.Name + RandomString(4),
49+
VolumeID: &volume.ID,
50+
VolumeType: instance.SnapshotVolumeTypeBSSD,
51+
Zone: scw.Zone(os.Getenv("INSTANCE_ZONE")),
52+
})
53+
if err != nil {
54+
return fmt.Errorf("error while creating snapshopt %w", err)
55+
}
56+
fmt.Println("created snapshot ", snapshotResp.Snapshot.ID)
57+
}
58+
59+
return nil
60+
}
61+
62+
func init() {
63+
if os.Getenv("SCW_DEFAULT_ORGANIZATION_ID") == "" {
64+
panic("missing SCW_DEFAULT_ORGANIZATION_ID")
65+
}
66+
67+
if os.Getenv("SCW_ACCESS_KEY") == "" {
68+
panic("missing SCW_ACCESS_KEY")
69+
}
70+
71+
if os.Getenv("SCW_SECRET_KEY") == "" {
72+
panic("missing SCW_SECRET_KEY")
73+
}
74+
75+
if os.Getenv("INSTANCE_ID") == "" {
76+
panic("missing INSTANCE_ID")
77+
}
78+
79+
if os.Getenv("INSTANCE_ZONE") == "" {
80+
panic("missing INSTANCE_ZONE")
81+
}
82+
}

jobs/instances-snapshot/string.go

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
package main
2+
3+
import "crypto/rand"
4+
5+
// RandomString is used to generate a random string containing upper and lower characters
6+
// + number, of size n.
7+
func RandomString(n int) string {
8+
const letters = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
9+
10+
bytes := make([]byte, n)
11+
12+
if _, err := rand.Read(bytes); err != nil {
13+
panic(err)
14+
}
15+
16+
for i, b := range bytes {
17+
bytes[i] = letters[b%byte(len(letters))]
18+
}
19+
20+
return string(bytes)
21+
}

0 commit comments

Comments
 (0)