Upstart: init process consumes 100% cpu and big amount of memory

Recently I had issue with Upstart service. By incident service generated plenty (dozens of megabytes per minute) of logs on STDOUT. After few hours of execution we have seen 100% Init process CPU consumption and memory usage approaching to 2GB! Then application frozen in arbitrary state. It was interesting that last log file size in /var/log/upstart was 2147483646 bytes, that is MAX_INT-1.

Command dmesg returned following:

[73216.335000] init: Failed to write to log file /var/log/upstart/myservice.log 
[73216.335019] init: Failed to write to log file /var/log/upstart/myservice.log
...

It seems problem is described in https://paperairoplane.net/?p=725. For some reason it froze during log rotation and since then all messages was buffered in init process memory.

So check amount of logs your service produces on STDOUT/STDERR. We just had for debug purposes logs generated by log4j redirected to console too. So we disabled console output for production and problem disappeared.

Tags:  Upstart  Ubuntu 
Last blog articles:
Leaving LinkedIn
LinkedIn: Odcházení
git gui: Add pull and stash commands
Bosch Silence Plus + code E24
Using LXC virtualization for testing server installation

Tags:
Java Ubuntu Gradle česky personal Software-architecture DIY children React Eclipse DisplayLink AI Spring JavaScript AspectJ neuron SSH LXC JavaFX Immutables Gnome3 GIT FatJar Edison technologies science procrastination pokusy lambda kvas ellipsis destilace chemie alkohol VisualVM Upstart Tomcat Selenium REST PrimeFaces Log4j Jooq Jackson JSF Debug DI CSS

Last tweets: