[memory_checker] cgroup memory usage file 'memory.usage_in_bytes' does not exist on device

[volodymyrsamotiy]

Description
Very rarely, during config reload or reboot, below error is observed:

ERR memory_checker: [memory_checker] cgroup memory usage file '/sys/fs/cgroup/memory/docker/3d4452045e8e85223825a45a913eb9ae041494be5702fa5b4c9059cd7d0b8b0b/memory.usage_in_bytes' of container '3d4452045e8e85223825a45a913eb9ae041494be5702fa5b4c9059cd7d0b8b0b' does not exist on device! Exiting ...

It happens on shutdown and it is not harmful in any way except just error in syslog.

It looks like race condition between memory_checker and docker shutdown flow.

memory_checker is running periodically and checks memory usage.
It checks memory for containers using this file '/sys/fs/cgroup/memory/docker/.../memory.usage_in_bytes'.
It could happen that container is already down which means 'memory.usage_in_bytes' file does not exist anymore.
But memory_checker was executed in parallel so it didn't know yet container is down, so still checked the file and printed an error that there is no such file.

Steps to reproduce the issue:

1. Execute config reload or reboot (please note, as mentioned above reproduce rate is low)

Describe the results you received:
Very rarely, during config reload or reboot, below error is observed:

ERR memory_checker: [memory_checker] cgroup memory usage file '/sys/fs/cgroup/memory/docker/3d4452045e8e85223825a45a913eb9ae041494be5702fa5b4c9059cd7d0b8b0b/memory.usage_in_bytes' of container '3d4452045e8e85223825a45a913eb9ae041494be5702fa5b4c9059cd7d0b8b0b' does not exist on device! Exiting ...

Describe the results you expected:
No memory_checker related error during config reload and reboot.

[dgsudharsan]

@qiluo-msft Can you please investigate this issue?

[arlakshm]

@qiluo-msft to check