Upstart: init process consumes 100% cpu and big amount of memory

Recently I had issue with Upstart service. By incident service generated plenty (dozens of megabytes per minute) of logs on STDOUT. After few hours of execution we have seen 100% Init process CPU consumption and memory usage approaching to 2GB! Then application frozen in arbitrary state. It was interesting that last log file size in /var/log/upstart was 2147483646 bytes, that is MAX_INT-1.

Command dmesg returned following:

[73216.335000] init: Failed to write to log file /var/log/upstart/myservice.log 
[73216.335019] init: Failed to write to log file /var/log/upstart/myservice.log
...

It seems problem is described in https://paperairoplane.net/?p=725. For some reason it froze during log rotation and since then all messages was buffered in init process memory.

So check amount of logs your service produces on STDOUT/STDERR. We just had for debug purposes logs generated by log4j redirected to console too. So we disabled console output for production and problem disappeared.

Tags:  Upstart  Ubuntu 
Last blog articles:
Mým fuckupům
Ubuntu: Login loop
Leaving LinkedIn
LinkedIn: Odcházení
git gui: Add pull and stash commands

Tags:
Java Ubuntu česky Gradle personal Software-architecture DIY children React Eclipse DisplayLink AI Spring JavaScript AspectJ neuron SSH LXC JavaFX Immutables Gnome3 GIT FatJar Edison technologies startup science procrastination pokusy podnikání linux lambda kvas ellipsis destilace chemie alkohol VisualVM Upstart Tomcat Selenium REST PrimeFaces Log4j Jooq Jackson JSF Debug DI CSS

Last tweets: