Unlike the frontend, where a user interface provides a visual representation of the project's state, logs serve as a window into the inner workings of a backend system. In the context of backend development, where logs primarily consist of text, understanding 'what,' 'when,' and 'how' to log is crucial for gaining clear insights into the backend's operational state.

It is imperative for logs to convey the whole picture without overwhelming the observer with information; this requires presenting concise and quarriable data.

In this piece, I aim to share insights gained from my recent experience in deploying logs for a Python-based REST API backend service using FastAPI and AWS CloudWatch. While the 'how to log' aspect is relatively straightforward, there is often a scarcity of discussion regarding 'what' to log and 'when' to log. This article seeks to address these critical questions.

What to log?

Understanding what information is being logged and how it is structured is crucial, as it simplifies the process of finding the right information and ensures that logs are both readable and actionable.

When it comes to the information itself, it's best to anticipate what you might need in the future:

• Keep the data in your logs concise and informative. Minimizing unnecessary data in the log stream prevents clutter and makes it easier to locate the information you need.
• While it's often advised to log only critical information like fatal errors and exceptions, my experience led me to consider additional user journey logs for analytics. This decision was influenced by the app's growing user base and the need for insights into user actions.

So I ended up logging the following type of information in my Fast Api service - raised exceptions with tracebacks and exception information, user state and journey logs for each incoming request.

Logging data in JSON format enhances traceability and simplifies log stream queries. This log format, which I adopted, allowed me to gather all the important details in a single log message.

{
  "correlation_id": "",
  "user_id": "",
  "log_message": "",
  "file_name": "",
  "function_name": "",
  "line_number": "",
  "log_level": "",
  "timestamp": "",
}

• correlation_id: A unique identifier that connects related log messages, making it easier to trace the flow of a specific operation or request. It is common across all logs for a single request making it easy to link all logs for a particular request.
• user_id: The identifier of the user or system component associated with the log entry, providing context for who triggered the action.
• log_message: A concise description of the event or action being logged, offering a clear and informative account of what occurred.
• log_level: The log level indicates the severity or importance of the log message. It helps categorize log entries, such as "info," "warning," "error," or "debug," making it easier to filter and prioritize logs based on their significance.
• timestamp: A human-readable timestamp that provides the exact moment when the log entry was generated. Human-readable timestamps are essential because they allow anyone reading the logs to quickly and easily grasp when an event occurred.