Update everything

Author: AntonKozlov

.gitignore

@@ -1,4 +1,6 @@
+/jdk
 /target
 /aws-lambda-rie
-/jdk
 /cr
+/log.json
+/response.json

README.md

@@ -1,96 +1,137 @@
-# Blank function (Java)
+# AWS Lambda on Java with CRaC
 
-![Architecture](/sample-apps/blank-java/images/sample-blank-java.png)
+This is an example of a Java AWS Lambda that uses Project CRaC to reduce the cost and time of the first function invocation.
 
-The project source includes function code and supporting resources:
+Based on https://github.com/awsdocs/aws-lambda-developer-guide/tree/main/sample-apps/blank-java.
 
-- `src/main` - A Java function.
-- `src/test` - A unit test and helper classes.
-- `template.yml` - An AWS CloudFormation template that creates an application.
-- `build.gradle` - A Gradle build file.
-- `pom.xml` - A Maven build file.
-- `1-create-bucket.sh`, `2-build-layer.sh`, etc. - Shell scripts that use the AWS CLI to deploy and manage the application.
+In this example the function will be packed as the [container image](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-images.html#gettingstarted-images-package) together with JDK CRaC runtime and the CRaC image.
 
-Use the following instructions to deploy the sample application.
+The CRaC image with the snapshot of JVM and application states will be preparied with the help of [AWS Runtime Interface Emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator).
 
-# Requirements
-- [Java 8 runtime environment (SE JRE)](https://www.oracle.com/java/technologies/javase-downloads.html)
-- [Gradle 5](https://gradle.org/releases/) or [Maven 3](https://maven.apache.org/docs/history.html)
-- The Bash shell. For Linux and macOS, this is included by default. In Windows 10, you can install the [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to get a Windows-integrated version of Ubuntu and Bash.
-- [The AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) v1.17 or newer.
+The deployment scheme will be:
+* Create the container image with the function
+* Create the CRaC image:
+  * Run the function in the local environment
+  * Optionally, warm-up the function
+  * Trigger checkpoint to dump the image
+* Create the container image with the function and the image
+* Deploy the container to AWS
 
-If you use the AWS CLI v2, add the following to your [configuration file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) (`~/.aws/config`):
+![Deployment](./deployment.png)
+
+* [Instruction](#instruction)
+  * [Build](#build)
+  * [Checkpoint](#checkpoint)
+  * [Deploy](#deploy)
+    * [Test locally](#test-locally)
+    * [Deploy to AWS](#deploy-to-aws)
+* [Techical details](#techical-details)
+
+## Instruction
+
+The repo contain a helper script to perform routine operations.
+Its intended to be "executable documetnation", so the reader is invited to look to the script
+
+The step zero fetches necessary dependencies:
 
 ```
-cli_binary_format=raw-in-base64-out
+./crac-steps.sh s00_init
 ```
 
-This setting enables the AWS CLI v2 to load JSON events from a file, matching the v1 behavior.
+### Build
 
-# Setup
-Download or clone this repository.
+The function needs to use a fork of [AWS Lambda Java Libraries](https://github.com/CRaC/aws-lambda-java-libs), `pom.xml` was changed to:
 
-    $ git clone https://github.com/awsdocs/aws-lambda-developer-guide.git
-    $ cd aws-lambda-developer-guide/sample-apps/blank-java
+```
+    <dependency>
+      <groupId>io.github.crac.com.amazonaws</groupId>
+      <artifactId>aws-lambda-java-runtime-interface-client</artifactId>
+      <version>1.0.0</version>
+    </dependency>
+```
 
-To create a new bucket for deployment artifacts, run `1-create-bucket.sh`.
+Build the app and create the container image for checkpoint (`crac-lambda-checkpoint`):
+```
+./crac-steps.sh s01_build
+```
 
-    blank-java$ ./1-create-bucket.sh
-    make_bucket: lambda-artifacts-a5e491dbb5b22e0d
+### Checkpoint
+
+Run the container for checkpoint on the local machine:
+
+```
+./crac-steps.sh s02_start_checkpoint
+```
+
+At this point you may provide a workload to warm-up your function, served on port `8080`.
+
+Then trigger checkpoint from another terminal:
+```
+./crac-steps.sh s03_checkpoint
+```
 
-To build a Lambda layer that contains the function's runtime dependencies, run `2-build-layer.sh`. Packaging dependencies in a layer reduces the size of the deployment package that you upload when you modify your code.
+The former function process should exit, leaving the CRaC image in the `cr` directory.
 
-    blank-java$ ./2-build-layer.sh
+Create the final container image with the CRaC image (`crac-lambda-restore`):
 
-# Deploy
+```
+./crac-steps.sh s04_prepare_restore
+```
 
-To deploy the application, run `3-deploy.sh`.
+### Deploy
 
-    blank-java$ ./3-deploy.sh
-    BUILD SUCCESSFUL in 1s
-    Successfully packaged artifacts and wrote output template to file out.yml.
-    Waiting for changeset to be created..
-    Successfully created/updated stack - blank-java
+Now you can deploy `crac-lambda-restore` as the container function to the AWS.
 
-This script uses AWS CloudFormation to deploy the Lambda functions and an IAM role. If the AWS CloudFormation stack that contains the resources already exists, the script updates it with any changes to the template or function code.
+You may also check the function locally with RIE.
 
-You can also build the application with Maven. To use maven, add `mvn` to the command.
+#### Test locally
 
-    java-basic$ ./3-deploy.sh mvn
-    [INFO] Scanning for projects...
-    [INFO] -----------------------< com.example:blank-java >-----------------------
-    [INFO] Building blank-java-function 1.0-SNAPSHOT
-    [INFO] --------------------------------[ jar ]---------------------------------
-    ...
+You can also test the container locally:
 
-# Test
-To invoke the function, run `4-invoke.sh`.
+```
+./crac-steps.sh s05_local_restore
+```
 
-    blank-java$ ./4-invoke.sh
-    {
-        "StatusCode": 200,
-        "ExecutedVersion": "$LATEST"
-    }
+From another terminal:
+```
+./crac-steps.sh "post hi"
+```
 
-Let the script invoke the function a few times and then press `CRTL+C` to exit.
+To test cold start, drop file system caches by:
+```
+./crac-steps.sh make_cold_local
+```
 
-The application uses AWS X-Ray to trace requests. Open the [X-Ray console](https://console.aws.amazon.com/xray/home#/service-map) to view the service map.
+#### Deploy to AWS
 
-![Service Map](/sample-apps/blank-java/images/blank-java-servicemap.png)
+The helper script can deploy the container, assuming there is a function named `crac-test` configured to use image `crac-test`.
 
-Choose a node in the main function graph. Then choose **View traces** to see a list of traces. Choose any trace to view a timeline that breaks down the work done by the function.
+Initialize the evironment:
+```
+eval $(./crac-steps.sh s06_init_aws)
+```
 
-![Trace](/sample-apps/blank-java/images/blank-java-trace.png)
+Push the container and update the function:
+```
+./crac-steps.sh s07_deploy_aws
+```
 
-Finally, view the application in the Lambda console.
+To test cold start, reconfigure the function by:
+```
+./crac-steps.sh make_cold_aws
+```
 
-*To view the application*
-1. Open the [applications page](https://console.aws.amazon.com/lambda/home#/applications) in the Lambda console.
-2. Choose **blank-java**.
+## Techical details
 
-  ![Application](/sample-apps/blank-java/images/blank-java-application.png)
+AWS environment restricts features and capabilities for functions in containers:
+* the file system is read-only, except `/tmp`
+* setting PID/TIDs for being restored process, `ptrace`, and some other syscalls that are usually required by CRIU are not allowed
+* storage performance is not great
 
-# Cleanup
-To delete the application, run `5-cleanup.sh`.
+To overcome storage limits we unpack libjvm.so to /tmp at the start of the container.
+Same is done for CRaC image after it's added to the container image.
 
-    blank$ ./5-cleanup.sh
+CRaC's build of CRIU attempts to set the PID/TIDs on restore first, in case this functionality is presented.
+When this fails in AWS, it attempts to align the next PID by creating empty do-nothing processes/threads. 
+As in the vanilla CRIU, there is strict requirement for PID/TIDs to be available.
+To ensure Java PID/TIDs won't intersect with another processes on restore, checkpoint script ensures Java PID/TIDs are created with a reasonable offset, see [the script](./checkpoint.cmd.sh#L8).

checkpoint.cmd.sh

@@ -12,4 +12,5 @@ exec /aws-lambda-rie /bin/bash -c '\
 		-XX:CRaCCheckpointTo=/cr \
 		-cp /function:/function/lib/* \
 		-Dcom.amazonaws.services.lambda.runtime.api.client.NativeClient.libsBase=/function/lib/ \
+		--add-opens java.base/java.util=ALL-UNNAMED \
 		com.amazonaws.services.lambda.runtime.api.client.AWSLambda $0' "$@"

crac-steps.sh

@@ -1,6 +1,9 @@
 #!/bin/bash
 
-CONTAINERNAME=crac
+LAMBDA_NAME=crac-test
+LAMBDA_IMAGE=crac-test
+
+CRAC_VERSION=17-crac+2
 
 IOLIM=60m
 DEV=/dev/nvme0n1
@@ -10,26 +13,26 @@ CPU=0.88
 iolim() { IOLIM=$1; }
   cpu() {   CPU=$1; }
 
-s00_init() {
-
-	if [ -z $JAVA_HOME ]; then
-	       echo "No	JAVA_HOME specified"
-	       return 1
-	fi
-
+dojlink() {
+	local JDK=$1
 	rm -rf jdk
-	cp -r $JAVA_HOME jdk
+	$JDK/bin/jlink --bind-services --output jdk --module-path $JDK/jmods --add-modules java.base,jdk.unsupported,java.sql
+	# XXX
+	cp $JDK/lib/criu jdk/lib/
+}
+
+s00_init() {
+	curl -LO https://github.com/CRaC/openjdk-builds/releases/download/$CRAC_VERSION/jdk$CRAC_VERSION.tar.gz
+	tar axf jdk$CRAC_VERSION.tar.gz
+	dojlink jdk$CRAC_VERSION
 
-	curl -L -o aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/download/v1.3/aws-lambda-rie-$(uname -m) 
+	curl -L -o aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/download/v1.3/aws-lambda-rie-$(uname -m)
 	chmod +x aws-lambda-rie
 }
 
-dojlink() {
-	$JAVA_HOME/bin/jlink --bind-services --output jdk --module-path $JAVA_HOME/jmods --add-modules java.base,jdk.unsupported,java.sql
-}
 
 s01_build() {
-	mvn compile dependency:copy-dependencies -DincludeScope=runtime
+	mvn clean compile dependency:copy-dependencies -DincludeScope=runtime
 	docker build -t crac-lambda-checkpoint -f Dockerfile.checkpoint .
 }
 
@@ -70,7 +73,7 @@ s04_prepare_restore() {
 	docker build -t crac-lambda-restore -f Dockerfile.restore .
 }
 
-dropcache() {
+make_cold_local() {
         sync
         echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
 }
@@ -85,11 +88,13 @@ local_test() {
 		--device-write-bps $DEV:$IOLIM \
 		--cpus $CPU \
 		--entrypoint '' \
+		--security-opt seccomp=$PWD/seccomp.json \
 		"$@"
 }
 
 s05_local_restore() {
-	local_test crac-lambda-restore \
+	local_test \
+		crac-lambda-restore \
 		/aws-lambda-rie /bin/bash /restore.cmd.sh
 }
 
@@ -102,43 +107,41 @@ local_baseline() {
 			example.Handler::handleRequest
 }
 
+ltest() {
+	local_test \
+		-v /home:/home \
+		-v $PWD/logdir:/tmp/log \
+		crac-lambda-restore \
+		/bin/bash $PWD/restore.cmd.sh
+}
+
 s06_init_aws() {
 	ACCOUNT=$(aws sts get-caller-identity | jq -r '.Account')
 	echo export ACCOUNT=$ACCOUNT
 	REGION=$(aws configure get region)
 	echo export REGION=$REGION
 	ECR=$ACCOUNT.dkr.ecr.$REGION.amazonaws.com
 	echo export ECR=$ECR
-	REMOTEIMG=$ECR/crac-test
+	REMOTEIMG=$ECR/$LAMBDA_IMAGE
 	echo export REMOTEIMG=$REMOTEIMG
 	aws ecr get-login-password | docker login --username AWS --password-stdin $ECR 1>&2
 }
 
 s07_deploy_aws() {
         docker tag crac-lambda-restore $REMOTEIMG
         docker push $REMOTEIMG
-}
-
-init_lambda() {
-	if ! [ $LAMBDANAME ]; then
-		echo "Provide LAMBDANAME= preconfigured by a container image: \
-https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-images.html#gettingstarted-images-function" >&2
-		exit 1
-	fi
-}
 
-update_lambda() {
         local digest=$(docker inspect -f '{{ index .RepoDigests 0 }}' $REMOTEIMG)
-        aws lambda update-function-code --function-name $LAMBDANAME --image $digest
-        aws lambda wait function-updated --function-name $LAMBDANAME
+        aws lambda update-function-code --function-name $LAMBDA_NAME --image $digest
+        aws lambda wait function-updated --function-name $LAMBDA_NAME
 }
 
-s08_invoke_lambda() {
+s08_invoke_aws() {
 	rm -f response.json log.json
 
 	aws lambda invoke  \
 		--cli-binary-format raw-in-base64-out \
-		--function-name $LAMBDANAME \
+		--function-name $LAMBDA_NAME \
 		--payload "$(< event.json) " \
 		--log-type Tail \
 		response.json \
@@ -148,11 +151,12 @@ s08_invoke_lambda() {
 	jq -r .LogResult < log.json | base64 -d
 }
 
-coldstart_lambda() {
-	local mem=$(aws lambda get-function-configuration --function-name $LAMBDANAME | jq -r '.MemorySize')
+make_cold_aws() {
+	local mem=$(aws lambda get-function-configuration --function-name $LAMBDA_NAME | jq -r '.MemorySize')
 	local min=256
 	local max=512
-	aws lambda update-function-configuration --function-name $LAMBDANAME --memory-size $(($min + (($mem + 1) % ($max - $min))))
+	aws lambda update-function-configuration --function-name $LAMBDA_NAME --memory-size $(($min + (($mem + 1) % ($max - $min))))
+	aws lambda wait function-updated --function-name $LAMBDA_NAME
 }
 
 for i; do

deployment.png

@@ -12,11 +12,6 @@
     <maven.compiler.target>1.8</maven.compiler.target>
   </properties>
   <dependencies>
-    <dependency>
-      <groupId>io.github.crac</groupId>
-      <artifactId>org-crac</artifactId>
-      <version>0.1.0</version>
-    </dependency>
     <dependency>
       <groupId>io.github.crac.com.amazonaws</groupId>
       <artifactId>aws-lambda-java-runtime-interface-client</artifactId>

pom.xml

@@ -2,6 +2,9 @@
 
 [ -d /tmp/sub/jdk ] && exit 0
 
+# ensure umask is consistent between environments
+umask 0002
+
 mkdir -p /tmp/sub/jdk/lib/server/
 lz4 -d /jdk/lib/server/libjvm.so.lz4 /tmp/sub/jdk/lib/server/libjvm.so
-ln -s -t /tmp/sub/jdk/lib /jdk/lib/*
+ln -s -t /tmp/sub/jdk/lib /jdk/lib/* 2>/dev/null

prepare-jdk.cmd.sh

@@ -2,7 +2,6 @@
 
 /prepare-jdk.cmd.sh
 
-lz4 -d /cr.tar.lz4  - | tar x -C /tmp/
+lz4 -q -d /cr.tar.lz4  - | tar x -C /tmp/
 
 /jdk/bin/java -XX:CRaCRestoreFrom=/tmp/cr
-# exec /jdk/lib/criu restore -W . --shell-job --action-script /jdk/lib/action-script -D /tmp/cr/ -v1 --exec-cmd -- /jdk/lib/wait