Echo chef-client.log when a node fails to bootstrap

This helps with investigating bootstrap failures in cases CloudWatch logs are not available.

Older CINC was outputting logs to STDOUT even when --logfile is specified. i.e. CINC was outputting logs to both STDOUT and the log file.

Sometime in CINC 16, it stopped outputting logs to STDOUT when --logfile is specified. There are no clear changelog items. But there are some discussions around it:
https://discourse.chef.io/t/chef-log-not-printing-for-console-in-chef-client-16-6-14/17983

Author: hanwen-cluster

CHANGELOG.md

@@ -4,6 +4,9 @@ CHANGELOG
 3.14.0
 ------
 
+**ENHANCEMENTS**
+- Echo chef-client log when a node fails to bootstrap. This helps with investigating bootstrap failures in cases CloudWatch logs are not available.
+
 **CHANGES**
 - Ubuntu 20.04 is no longer supported.
 - Support prioritized and capacity-optimized-prioritized Allocation Strategy

cli/src/pcluster/resources/compute_node/user_data.sh

@@ -70,6 +70,10 @@ write_files:
       function error_exit
       {
         echo "Bootstrap failed with error: $1"
+        # Echo chef-client.log to console logs for debugging
+        echo "========== BEGIN CHEF-CLIENT.LOG =========="
+        cat /var/log/chef-client.log || echo "chef-client.log not found"
+        echo "========== END CHEF-CLIENT.LOG ==========="
         # wait logs flush before signaling the failure
         sleep 10
         # TODO: add possibility to override this behavior and keep the instance for debugging
@@ -173,6 +177,10 @@ MIME-Version: 1.0
 function error_exit
 {
   echo "Timed-out when bootstrapping instance"
+  # Echo chef-client.log to console logs for debugging
+  echo "========== BEGIN CHEF-CLIENT.LOG =========="
+  cat /var/log/chef-client.log || echo "chef-client.log not found"
+  echo "========== END CHEF-CLIENT.LOG ==========="
   sleep 10  # Allow logs to propagate
   shutdown -h now
   exit 1

cli/src/pcluster/resources/head_node/user_data.sh

@@ -64,6 +64,10 @@ MIME-Version: 1.0
 
 function error_exit
 {
+  # Echo chef-client.log to console logs for debugging
+  echo "========== BEGIN CHEF-CLIENT.LOG =========="
+  cat /var/log/chef-client.log || echo "chef-client.log not found"
+  echo "========== END CHEF-CLIENT.LOG ==========="
   # wait logs flush before signaling the failure
   sleep 10
   # trim the error message because there is a size limit of 4096 bytes for cfn-signal

cli/src/pcluster/resources/login_node/user_data.sh

@@ -57,6 +57,10 @@ write_files:
       function error_exit
       {
         echo "Bootstrap failed with error: $1"
+        # Echo chef-client.log to console logs for debugging
+        echo "========== BEGIN CHEF-CLIENT.LOG =========="
+        cat /var/log/chef-client.log || echo "chef-client.log not found"
+        echo "========== END CHEF-CLIENT.LOG ==========="
         # wait logs flush before signaling the failure
         sleep 10
         # TODO: add possibility to override this behavior and keep the instance for debugging
@@ -145,6 +149,10 @@ MIME-Version: 1.0
 function error_exit
 {
   echo "Timed-out when bootstrapping instance"
+  # Echo chef-client.log to console logs for debugging
+  echo "========== BEGIN CHEF-CLIENT.LOG =========="
+  cat /var/log/chef-client.log || echo "chef-client.log not found"
+  echo "========== END CHEF-CLIENT.LOG ==========="
   sleep 10  # Allow logs to propagate
   shutdown -h now
   exit 1