Entries tagged 'logging'

i have had this blog post about moving from logs to metrics open for a while, since i know one of the weak points in our systems right now is some pretty basic stuff like logging and monitoring. and then jwz ran into a problem with logging errors from php-fpm and what it reminded me about is how logs need to carry enough context so you can pull the threads together from something like a single request.

i have not wrapped my head around the idea of just using metrics, because that sounds like rather a lot of data to be storing. maybe i’m just an on-prem brain in a cloud world.

justin mason flagged this article about "The log/event processing pipeline you can't have" a while back, and it has been on my mind ever since. our digital infrastructure is split across a few machines (virtual and not) and i often wish that i had a more cohesive way of collecting logs and doing even minimally interesting things with them.

i think the setup there is probably overkill for what i want, but i love the philosophy behind it. small, simple tools that fit together in an old-school unix way.

i set up an instance of graylog to play with a state-of-the-art log management tool, and it is actually pretty nice. the documentation around it is kind of terrible right now because the latest big release broke a lot of the recipes for processing logs.

right now, the path i am using for getting logs from nginx in a docker container to graylog involves nginx outputting JSON that gets double-encoded. it’s all very gross.

i think i am having a hard time finding the correct tooling for the gap between “i run everything on a single box” and “i have a lot of VC money to throw at an exponentially scalable system”. (while also avoiding AWS.)

(the very first post to this blog was the same ren & stimpy reference as the title of this post.)