While Scout provides powerful monitoring tools (try it now!) mastering logging is an awesome complement to these skills. In this post, we’ll see how to create readable, actionable logs for both humans and machines. You’ll improve your logging strategy, drastically reduce troubleshooting time, and put yourself in the best possible position for maximum observability.

 As a starting example, let’s take this error log:

No such module as sqlalchemy?! (You could swear you’ve installed sqlalchemy. Yes, you checked, and did.) Can’t load a plugin? And what is a “dialect” here? 

So, you do what any seasoned developer would do and ask ChatGPT. It turns out that the DATABASE_URL was slightly wrong for this use case. It suggests a one-line code change, and that works!

…but should we have had to ask an AI to parse a log message? Shouldn’t the message be clear already? If ChatGPT knows, surely the software reporting the error should also know?

Yes, it should – and it should tell you, too!

The psychology of logs

Another example: it’s 11 p.m. on a Friday, and you’re trying to figure out why production has stalled. You look through CloudWatch and see something like this: 

Here are some options if faced with such a log dump in a high-stress situation:

1. Manual analysis: You could try to piece together the sequence of events. However, this is time-consuming and error-prone, especially under pressure.
2. Grep and prayer: You might frantically search for specific error messages, hoping to stumble upon the root cause.
3. Shotgun debugging: In desperation, you might start restarting services randomly, hoping it fixes the issue without understanding why.

What is not so obvious is, well, how to properly troubleshoot this issue. Detached from the actual scenario, perhaps you can understand what is happening. But in a real-life crisis situation, when the engineer is tired (and maybe unfamiliar with this particular issue) things aren’t so clear.

Whenever information needs to be shared in a stressful situation, that information has to be clear, concise, and actionable. When an engineer is already under pressure, possibly sleep-deprived, and racing against time, asking them to parse through a wall of text and piece together a narrative sets them up for failure.

Several negative outcomes are possible:

1. Cognitive overload can lead to increased error rates, decision paralysis, and situations where engineers might latch onto familiar patterns or issues, potentially missing the root cause.
2. The impact of poorly designed logs extends beyond the immediate incident. It can affect the entire organizational culture around incident response.
3. Ambiguous logs also make it harder for new team members to learn from past incidents, slowing down their onboarding and reducing the team’s overall resilience.

Let’s talk about have to solve this.

#1: Treat logs as first-class citizens

Logs are often poorly thought out because no one treats them as a key part of the code. They’re often an afterthought, hastily added just before deployment or, worse, in response to a production issue. This approach fundamentally misunderstands the role and importance of logs in an application’s lifecycle.

Logs are not just a debugging tool but the core of your observability. They tell the story of your application over time, capturing its behavior, performance, and health. In many ways, logs are the closest thing we have to a time machine in software development, allowing us to travel back to specific moments and understand what happened.

Yet, despite their crucial role, logs are frequently neglected in the development process. They’re rarely reviewed in code reviews with the same rigor as functional code, seldom tested or validated, and often the first thing to be cut when deadlines loom. This neglect stems from a misconception that logs are separate from an application’s core functionality.

Treating logs as first-class citizens in your codebase means:

• Designing them alongside your application architecture
• Evolving them as your application evolves
• Reviewing and refining them regularly
• Testing them to ensure they provide value
• Considering their performance impact and storage requirements

By elevating logs to first-class status, we acknowledge their importance in the development process and ensure they receive the attention they deserve.

#2: Consider logs as deliverables

Viewing logs as deliverables emphasizes their value beyond the development team. They’re used by operations teams, customer support, and even business analysts. They inform decisions about system health, user behavior, and business metrics. In many organizations, logs feed into monitoring systems, alerting mechanisms, and data analytics pipelines. They’re a critical part of the entire software ecosystem.

As deliverables, logs should be:

• Planned and designed with intention for various stakeholders
• Developed to meet specific requirements and use cases
• Tested for accuracy and usefulness in real-world scenarios
• Maintained and updated over time to reflect changing needs
• Documented for ease of use and understanding by different teams

Logs are for life–the life of your application, the life of your system, and often, the professional lives of the people working with your software. They’re not just for debugging today’s issues but for understanding tomorrow’s challenges and opportunities.

#3: Make logs readable

There are multiple needles to thread when it comes to making logs readable: 

1. We want to make logs readable to humans (but that doesn’t mean long sentences of explanations). 
2. We need the information to be clear and concise, so our 11pm engineer can understand the problem and fix it. 
3. We also need to think about how machines will read these errors. Monitoring and alerting tools need to monitor and be on alert. This is made more complicated if the logs are written for human eyes only. 

So, for humans and machines, the first consideration must be structure. If we look back at our poor logs from earlier, we can see they lack this:

Yes, these logs have a lot of detail, but they are tough to parse (for both humans and machines). A better option would be to structure each message within a format such as JSON or XML:

This is immediately more machine-parseable and human-readable! It also contains more information while remaining (somewhat) concise. It contains a clear hierarchy and consistent fields and key-value pairs, which makes scanning easier. It also has flexibility (like the data, which allows for additional context specific to each event type). Plus, we now have a trace_id for connecting related events across services.

The second consideration is context. The above “good” version is better because it contains more information from a business context. In particular, the “query_snippet”: “SELECT * FROM large_table WHERE…” is helpful for engineers trying to isolate the problem.

However, as we said, logs should be viewed as a deliverable beyond engineers. Thus, a data object could contain more information depending on the end user for this particular log. 

There is currently an RFC, RFC 9457, in front of the IETF, which makes this problem explicit for HTTP errors. It suggests adding problem details to HTTP errors to give developers more context about the errors they get from servers and applications.

The exact “problem details” concept can be used within all logs. Your logs must include timestamps, levels, and some form of ID to help tools segment, analyze, and visualize them. You might also want to include metadata and tags for easier filtering and aggregation. For instance, perhaps you’re instrumenting a data pipeline for analysts or data engineers. In that case, a log might look like this:

This structure offers essential information for understanding the error, its scope, and its impact on data analysis tasks without including excessive technical details that might not be relevant to a data analyst’s role:

1. An explicit error identifier “DATA_PIPELINE_FAILURE“, which describes the issue succinctly.
2. A timestamp that allows for pinpointing when the error occurred.
3. Relevant error details – pipeline_id identifies which specific data pipeline failed, and others go into more detail.
4. The impact field shows which analytics are affected.

Solving the problem with logs

All of the above sounds excellent, obvious …and difficult.

As much as logs are first-class citizens and logs are deliverables, the truth is that they don’t feel like that, so they get relegated to the backburner within the development process. How can we make it so developers can see the benefits of good logs?

#1: Start small and implement structured logging in a single, critical service. This allows teams to experience the benefits firsthand without overwhelming them with a massive overhaul.

#2: Create templates. Develop log templates for common scenarios, making it easier for developers to implement good logging practices consistently. Here’s a template for that:

#3: Automate where possible. Use linters and static analysis tools to enforce logging standards in your codebase.

#4: Make it part of the review process. Log quality should be included as a criterion in code reviews, treating it with the same importance as functional code.

#5: Measure the impact. Track metrics like mean time to resolution (MTTR) before and after implementing better logging practices. Concrete improvements can help justify the effort.

By treating logs with the respect they deserve, we make our systems more observable and our incidents more manageable and create a more empathetic environment for future maintainers of our code.

🔍 Thanks for reading! And another thing: At Scout, we’ve got the perfect monitoring setup to complement your logging skills  – give us a try with our 14-day free trial!