Upstart: init process consumes 100% cpu and big amount of memory

Recently I had issue with Upstart service. By incident service generated plenty (dozens of megabytes per minute) of logs on STDOUT. After few hours of execution we have seen 100% Init process CPU consumption and memory usage approaching to 2GB! Then application frozen in arbitrary state. It was interesting that last log file size in /var/log/upstart was 2147483646 bytes, that is MAX_INT-1.

Command dmesg returned following:

[73216.335000] init: Failed to write to log file /var/log/upstart/myservice.log 
[73216.335019] init: Failed to write to log file /var/log/upstart/myservice.log

It seems problem is described in For some reason it froze during log rotation and since then all messages was buffered in init process memory.

So check amount of logs your service produces on STDOUT/STDERR. We just had for debug purposes logs generated by log4j redirected to console too. So we disabled console output for production and problem disappeared.

Tags:  Upstart  Ubuntu