Skip to content

Commit dac09ce

Browse files
BemilieBemilie
authored
feat(tests): node examples use offline testing (#35)
* tests: update cors-node example to use serverless local testing * tests: update typescript-with-node example to use serverless local testing * tests: update node-upload-file-s3-multipart example to use serverless local testing * tests: update postgre-sql-node example to use serverless local testing * tests: update image-labelling-node example to use serverless local testing * tests: update image-transform-node example to use serverless local testing * style: always use sh to stay consistent for code in README (node examples only)
1 parent 16aa570 commit dac09ce

File tree

21 files changed

+355
-80
lines changed

21 files changed

+355
-80
lines changed

‎functions/cors-node/README.md‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,66 @@ This function shows how to handle preflight CORS requests that will be sent by a
44

55
The example uses totally permissive, open CORS, you may want to modify this to make it more secure.
66

7-
## Setup
7+
## Requirements
8+
9+
This example assumes you are familiar with how serverless functions work. If needed, you can check [Scaleway's official documentation](https://www.scaleway.com/en/docs/serverless/functions/quickstart/)
10+
11+
This example uses the Scaleway Serverless Framework Plugin. Please set up your environment with the requirements stated in the [Scaleway Serverless Framework Plugin](https://github.com/scaleway/serverless-scaleway-functions) before trying out the example.
812

9-
This example uses the [Scaleway Serverless Framework Plugin](https://github.com/scaleway/serverless-scaleway-functions).
13+
Additionnaly it uses the [serverless-functions-node](https://github.com/scaleway/serverless-functions-node) library for local testing.
1014

11-
Once this is set up, you can run:
15+
## Context
1216

13-
```console
17+
The function of the example allows most CORS requests. For more documentation on how to control access for CORS requests, see [HTTP response headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#the_http_response_headers)
18+
19+
## Setup
20+
21+
Once your environment is set up, you can run:
22+
23+
```sh
1424
npm install
1525

26+
NODE_ENV=test node handler.js
27+
```
28+
29+
This will launch a local server, allowing you to test the function. For that, you can run in another terminal:
30+
31+
The result should be similar to:
32+
33+
```sh
34+
HTTP/1.1 200 OK
35+
access-control-allow-origin: *
36+
access-control-allow-headers: *
37+
access-control-allow-methods: *
38+
access-control-expose-headers: *
39+
content-type: text/plain
40+
content-length: 35
41+
Date: Mon, 17 Apr 2023 07:03:06 GMT
42+
Connection: keep-alive
43+
Keep-Alive: timeout=72
44+
server: envoy
45+
46+
This is allowing most CORS requests
47+
```
48+
49+
You can also check the result of your function in a browser. It should be "This is allowing most CORS requests".
50+
51+
## Deploy and run
52+
53+
Finally, if the test succeeded, you can deploy your function with:
54+
55+
```sh
1656
serverless deploy
1757
```
1858

1959
Then, from the given URL, you can run:
2060

21-
```console
61+
```sh
2262
# Options request
2363
curl -i -X OPTIONS <function URL>
2464

2565
# Get request
2666
curl -i -X GET <function URL>
2767
```
68+
69+
When invoking this function, the output should be similar to the one obtained when testing locally.

‎functions/cors-node/handler.js‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,10 @@ const handleCorsVeryPermissive = (event, context, cb) => {
3535
};
3636

3737
export { handleCorsPermissive, handleCorsVeryPermissive };
38+
39+
/* This is used to test locally and will not be executed on Scaleway Functions */
40+
if (process.env.NODE_ENV === 'test') {
41+
import("@scaleway/serverless-functions").then(scw_fnc_node => {
42+
scw_fnc_node.serveHandler(handleCorsPermissive, 8080);
43+
});
44+
}

‎functions/cors-node/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
"license": "ISC",
99
"dependencies": {},
1010
"devDependencies": {
11+
"@scaleway/serverless-functions": "^1.0.1",
1112
"serverless-scaleway-functions": "^0.4.4"
1213
},
1314
"description": ""

‎functions/image-labelling-node/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
node_modules/
2+
package-lock.json
3+
.serverless/

‎functions/image-labelling-node/README.md‎

Lines changed: 36 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,23 @@
11
# Image labeling example with TensorFlow.js
22

3+
This example shows how to label an RGB image in an S3 bucket using serverless functions.
4+
35
## Requirements
46

57
This example assumes that you are familiar with:
6-
* how serverless functions work. If needed, you can check [Scaleway official documentation](https://www.scaleway.com/en/docs/serverless/functions/quickstart/).
7-
* how S3 object storage works, and especially how to create a bucket and upload files within a bucket. Please refer to scaleway's documentation [here](https://www.scaleway.com/en/docs/storage/object/quickstart/).
8-
* how to create and get user API access and secret keys using IAM. Please refer to IAM documentation [here](https://www.scaleway.com/en/docs/identity-and-access-management/iam/concepts/).
8+
9+
* how serverless functions work. If needed, you can check [Scaleway official documentation](https://www.scaleway.com/en/docs/serverless/functions/quickstart/).
10+
* how S3 object storage works, and especially how to create a bucket and upload files within a bucket. Please refer to scaleway's documentation [here](https://www.scaleway.com/en/docs/storage/object/quickstart/).
11+
* how to create and get user API access and secret keys using IAM. Please refer to IAM documentation [here](https://www.scaleway.com/en/docs/identity-and-access-management/iam/concepts/).
912

1013
This example uses the Scaleway Serverless Framework Plugin. Please set up your environment with the requirements stated in the [Scaleway Serverless Framework Plugin](https://github.com/scaleway/serverless-scaleway-functions) before trying out the example.
1114

15+
Additionnaly it uses the [serverless-functions-node](https://github.com/scaleway/serverless-functions-node) library for local testing.
1216

1317
## Context
1418

1519
This example shows how to label an RGB image in an S3 bucket using serverless functions. The example uses a pre-trained ready-to-use model from [TensorFlow.js](https://www.tensorflow.org/js/models). The model is called `mobilenet` and can be used on server or client side. It returns three labels of an image with their respective prediction probabilities (namely, logits). Check `mobilenet` Github repository [here](https://github.com/tensorflow/tfjs-models/tree/master/mobilenet).
1620

17-
1821
## Description
1922

2023
The function gets an RGB image (jpg, jpeg, or png formats) from an S3 bucket, transforms it into a TensorFlow-compatible object (namely, a tensor), and then applys a pretrained `mobilenet` model to return the labels and their respective logits in a json format.
@@ -29,40 +32,40 @@ Create an S3 bucket and upload an RGB image (jpg, jpeg or png format) within thi
2932

3033
### Fill environment variables
3134

32-
Fill your environment variables within `serverless.yml` file:
35+
Ensure to create a bucket and have the following secrets variables available in your environment (to be able to test locally) and within `serverless.yml` file (to be able to deploy):
3336

34-
```
37+
```yml
3538
secret:
3639
USER_ACCESS_KEY: <bucket scw access key>
3740
USER_SECRET_KEY: <bucket scw access key id>
38-
S3_ENDPOINT_URL: http://s3.fr-par.scw.cloud
41+
S3_ENDPOINT_URL: s3.fr-par.scw.cloud
3942
```
4043
4144
### Install npm modules
4245
4346
Once your environment is set up, you can install `npm` dependencies from `package.json` file using:
4447

45-
```
48+
```sh
4649
npm install
4750
```
4851

49-
### Deploy and call the function
52+
### Test locally
5053

51-
Then deploy your function and get its URL using:
54+
Once your environment is set up, you can test your function locally with:
5255

53-
```
54-
serverless deploy
56+
```sh
57+
NODE_ENV=test node handler.js
5558
```
5659

57-
From the given function URL, you can get image labels in a `json` format:
60+
This will launch a local server, allowing you to test the function. For that, you can run in another terminal:
5861

59-
```
60-
curl -X GET "<function URL>/?sourceBucket=<source bucket name>&sourceKey=<file name within bucket>" | jq
62+
```sh
63+
curl -X GET "http://localhost:8080?sourceBucket=<source bucket name>&sourceKey=<filename within bucket>"
6164
```
6265

63-
The result should be similar to:
66+
The output should be similar to:
6467

65-
```
68+
```json
6669
{
6770
"labels": [
6871
{
@@ -79,7 +82,22 @@ The result should be similar to:
7982
}
8083
]
8184
}
82-
8385
```
8486

8587
You can also check the result of your function in a browser. It should return the same result.
88+
89+
### Deploy and call the function
90+
91+
Finally, if the test succeeded, you can deploy your function with:
92+
93+
```sh
94+
serverless deploy
95+
```
96+
97+
Then, from the given URL, you can run:
98+
99+
```sh
100+
curl -X GET "<function url>?sourceBucket=<source bucket name>&sourceKey=<filename within bucket>"
101+
```
102+
103+
When invoking this function, the output should be similar to the one obtained when testing locally.

‎functions/image-labelling-node/handler.js‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,16 +9,12 @@ const userSecretKey = process.env.USER_SECRET_KEY;
99

1010
async function handle(event, context, callback) {
1111

12-
// useful for function logging
13-
console.log(event)
14-
console.log(context)
15-
1612
const sourceBucket = event.queryStringParameters.sourceBucket;
1713
const sourceKey = event.queryStringParameters.sourceKey;
1814

1915
const response = {
2016
statusCode: 200,
21-
headers: { "Content-Type": ["application/json"] },
17+
headers: { "Content-Type": "application/json" },
2218
body: {
2319
labels: await classifyImage(imageToTensor(await getImageFromS3(sourceBucket, sourceKey, s3EndpointUrl, userAccessKey, userSecretKey)))
2420
}
@@ -28,7 +24,6 @@ async function handle(event, context, callback) {
2824

2925
};
3026

31-
3227
async function getImageFromS3(sourceBucket, sourceKey, s3EndpointUrl, userAccessKey, userSecretKey) {
3328

3429
const s3 = new S3({
@@ -90,3 +85,10 @@ async function classifyImage(imageTensor) {
9085
}
9186

9287
export { handle }
88+
89+
/* This is used to test locally and will not be executed on Scaleway Functions */
90+
if (process.env.NODE_ENV === 'test') {
91+
import("@scaleway/serverless-functions").then(scw_fnc_node => {
92+
scw_fnc_node.serveHandler(handle, 8080);
93+
});
94+
}

‎functions/image-labelling-node/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
"aws-sdk": "^2.1314.0"
1616
},
1717
"devDependencies": {
18+
"@scaleway/serverless-functions": "^1.0.1",
1819
"serverless-scaleway-functions": "^0.4.5"
1920
},
2021
"description": "",

‎functions/image-transform-node/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
node_modules/
2+
package-lock.json
3+
.serverless/

‎functions/image-transform-node/BucketScan.js‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ const https = require("https");
33
const { isNull } = require("util");
44

55
// Get info from env variables
6-
76
const SOURCE_BUCKET = process.env.SOURCE_BUCKET;
87
const S3_ENDPOINT_URL = process.env.S3_ENDPOINT_URL;
98
const ID = process.env.ACCESS_KEY_ID;
@@ -78,3 +77,10 @@ exports.handle = async (event, context, callback) => {
7877
}
7978
});
8079
};
80+
81+
/* This is used to test locally and will not be executed on Scaleway Functions */
82+
if (process.env.NODE_ENV === 'test') {
83+
import("@scaleway/serverless-functions").then(scw_fnc_node => {
84+
scw_fnc_node.serveHandler(exports.handle, 8080);
85+
});
86+
}

‎functions/image-transform-node/ImageTransform.js‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,14 @@ const AWS = require("aws-sdk");
33
const sharp = require("sharp");
44

55
// Get connexion information from secret environment variables
6-
const srcBucket = process.env.SOURCE_BUCKET;
6+
const SOURCE_BUCKET = process.env.SOURCE_BUCKET;
77
const S3_ENDPOINT_URL = process.env.S3_ENDPOINT_URL;
88
const ID = process.env.ACCESS_KEY_ID;
99
const SECRET = process.env.ACCESS_KEY;
10-
const dstBucket = process.env.DESTINATION_BUCKET;
10+
const DEST_BUCKET = process.env.DESTINATION_BUCKET;
11+
const RESIZED_WIDTH = process.env.RESIZED_WIDTH;
1112

12-
let width = parseInt(process.env.RESIZED_WIDTH, 10);
13+
let width = parseInt(RESIZED_WIDTH, 10);
1314
if (width < 1 || width > 1000) {
1415
width = 200;
1516
}
@@ -49,7 +50,7 @@ exports.handle = async (event, context, callback) => {
4950
// Download the image from the S3 source bucket.
5051
try {
5152
const params = {
52-
Bucket: srcBucket,
53+
Bucket: SOURCE_BUCKET,
5354
Key: srcKey,
5455
};
5556
var origimage = await s3.getObject(params).promise();
@@ -71,7 +72,7 @@ exports.handle = async (event, context, callback) => {
7172
// Upload the image to the destination bucket
7273
try {
7374
const destparams = {
74-
Bucket: dstBucket,
75+
Bucket: DEST_BUCKET,
7576
Key: dstKey,
7677
Body: buffer,
7778
ContentType: "image",
@@ -83,11 +84,11 @@ exports.handle = async (event, context, callback) => {
8384
}
8485
console.log(
8586
"Successfully resized " +
86-
srcBucket +
87+
SOURCE_BUCKET +
8788
"/" +
8889
srcKey +
8990
" and uploaded to " +
90-
dstBucket +
91+
DEST_BUCKET +
9192
"/" +
9293
dstKey
9394
);
@@ -99,10 +100,17 @@ exports.handle = async (event, context, callback) => {
99100
"Image : " +
100101
srcKey +
101102
" has successfully been resized and pushed to the bucket " +
102-
dstBucket,
103+
DEST_BUCKET,
103104
}),
104105
headers: {
105106
"Content-Type": "application/json",
106107
},
107108
};
108109
};
110+
111+
/* This is used to test locally and will not be executed on Scaleway Functions */
112+
if (process.env.NODE_ENV === 'test') {
113+
import("@scaleway/serverless-functions").then(scw_fnc_node => {
114+
scw_fnc_node.serveHandler(exports.handle, 8081);
115+
});
116+
}

0 commit comments

Comments
 (0)