Uvicorn throwing 137 exit code

[sunil-cred]

Problem Context

Our interservice Rest API calls started failing suddenly a few days back, so when we looked into this we found the error as ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) initially so we thought this is related to our services not being able to full fill all the requests so we looked into the Kubernetes stats where we found that some of our pods are getting unavailable suddenly
when we looked into pods details we found that

Reason: ERROR
Exit code: 137

initially, we thought this is related to memory allocation to the pods but then we have allocated ample memory to these pods and the nodes hosting them. So we started digging deep then we found this on StackOverflow that process is sending this error code which is being displayed by the pod.

So we tested and found that when the application is not able to create more subprocess or threads it throws this exit code so we tested by running this in a simple API in quart and found that the quart application is throwing the 137 exit code

async def func():
while True:
subprocess.Popen(['ls'])
return "OK", 200
can you please help on this as we are stuck on this why quart might exit with this exit code.

I have also raised this with the Quart Team (https://github.com/pgjones/quart/issues/144)

Please help on this ASAP.

[Kludex]

How can I help here? 🤔

The link is incorrect...

[offbynull]

@Kludex Here's the actual link: pallets/quart#144

FWIW I'm also experiencing this a similar issue.

[iudeen]

Error 137 is raised by the Python process when it doesn't have the resources to run normally. In your case I'm not sure what the bottleneck is, are you setting memory limits at pod level? What's your CPU limit, also, the sample code you have mentioned, is it similar to the actual scenario, that is, having a sub process Popen?