Instructions for getting coredumps in Kubernetes

Instructions for doing coredumps in DigitalOcean Kubernetes

1. From your computer, get into a node as root using nsenter_node.sh:

#!/bin/sh
set -x

node=${1}
nodeName=$(kubectl get node ${node} -o template --template='{{index .metadata.labels "kubernetes.io/hostname"}}') 
nodeSelector='"nodeSelector": { "kubernetes.io/hostname": "'${nodeName:?}'" },'
podName=${USER}-nsenter-${node}

kubectl run ${podName:?} --restart=Never -it --rm --image overriden --overrides '
{
  "spec": {
    "hostPID": true,
    "hostNetwork": true,
    '"${nodeSelector?}"'
    "tolerations": [{
        "operator": "Exists"
    }],
    "containers": [
      {
        "name": "nsenter",
        "image": "alexeiled/nsenter:2.34",
        "command": [
          "/nsenter", "--all", "--target=1", "--", "su", "-"
        ],
        "stdin": true,
        "tty": true,
        "securityContext": {
          "privileged": true
        }
      }
    ]
  }
}' --attach "$@"

eg. ./nsenter_node.sh pool-default-gp-4-csih3

2. before touching anything in the remote shell, note down the default values of ulimit -c and /proc/sys/kernel/core_pattern

3. enable coredumps:
   ulimit -c unlimited

4. set-up s3fs; see https://github.com/s3fs-fuse/s3fs-fuse. follow ubuntu/debian install instructions and mounting instructions for 3rd-party S3 providers

5. create dump script in /dump.sh:

#!/bin/sh

cat - > /mountpoint/coredump

5. instruct the system to use that script for writing coredump:
   echo "|/dump.sh" > /proc/sys/kernel/core_pattern

6. kill -SEGV <pid> where pid is pid from the host, not from the container namespace! run ps -aux or top from the host, not from the container shells!

7. revert things back as they were; for do-kube-1.21.11-do.1:
   ulimit -c 0
echo "core" > /proc/sys/kernel/core_pattern