This week ive met an application that are being deployed as Pod in OCP but having a very unique behaviour, it keeps giving below error every one and a while.
[0m[5585.146s][warning][os,thread] Failed to start thread "Unknown thread"
- pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[5585.147s][warning][os,thread] Failed to start the native thread for java.lang.Thread "HandshakeCompletedNotify-Thread"
[5586.153s][warning][os,thread] Failed to start thread "Unknown thread"
- pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[5586.154s][warning][os,thread] Failed to start the native thread for java.lang.Thread "HandshakeCompletedNotify-Thread"
[0m[5589.672s][warning][os,thread] Failed to start thread "Unknown thread"
- pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[5589.673s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-4944-thread-1"
[31m06:57:23,949
ERROR [io.undertow.request] (default task-34) UT005023: Exception handling request to /actuator/health: java.lang.OutOfMemoryError:
unable to create native thread: possibly out of memory or process/resource limits reached
It seems that once this error happens, Pod will never recover from this condition. So Openshift need to find a way to handle this situation.
One workaround which i found is by utilizing Kubernetes Liveness Probe, which will detect application’s healthness.
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 3
periodSeconds: 4
successThreshold: 1
failureThreshold: 2
For this configuration I am setting a 4 seconds delay between request and will wait for 3 seconds for reply from the corresponding Pod. And if Pod are unable to response to Openshift’s request for two times, Openshift will force terminate the Pod assuming that the Pod is in an unhealthy state.
This strategy makes applications restart quite often in a day, but at least it will be healthy again after being restarted forcefully.