Handlers

QueueListenerHandler

A simple QueueHandler subclass implementation utilizing QueueListener for configured handlers. This is helpful for detaching the logger handlers from the main threads, which reduces the risk of getting blocked, for example, when using slower handlers such as smtp, file, or socket handlers.

Example Usage

An example YAML configuration file utilizing QueueListenerHandler

logging.yaml

version: 1
objects:
  queue:
    class: queue.Queue
    maxsize: 1000
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
  file_handler:
    class: logging.FileHandler
    filename: test_logger.log
    formatter: simple
  queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
      - cfg://handlers.console
      - cfg://handlers.file_handler
    queue: cfg://objects.queue
loggers:
  test_logger:
    level: DEBUG
    handlers:
      - queue_handler
    propagate: no
root:
  level: NOTSET
  handlers:
    - console

Just load the configuration file and start logging.

test_logger.py

import logging.config

import yaml

with open("logging.yaml", "r") as config_file:
    logging.config.dictConfig(yaml.safe_load(config_file.read()))

logger = logging.getLogger("test_logger")

logger.debug("This is a debug log")
logger.info("This is an info log")
logger.warning("This is an warning log")
logger.error("This is an error log")
logger.critical("This is a critical log")

Optional Params

A queue object must be passed since the handler does not set a default queue implementation. Set maxsize: -1 to make the queue unlimited.

Module Members

class logging_.handlers.queue_listener_handler.QueueListenerHandler(queue: Any, handlers: Any, respect_handler_level: bool = True, auto_run: bool = True)

Bases: Handler

QueueListenerHandler class for managing a queue listener with configured handlers.

This class sets up a queue listener logger handler with customizable configurations. Inspired by Rob Blackbourn’s article: https://rob-blackbourn.medium.com/how-to-use-python-logging-queuehandler-with-dictconfig-1e8b1284e27a.

It intentionally subclasses logging.Handler and re-implements the small QueueHandler behaviour (prepare/enqueue/emit) instead of subclassing logging.handlers.QueueHandler. Since Python 3.12, logging.config.dictConfig special-cases every QueueHandler subclass and takes over its construction (creating the QueueListener itself and rejecting the class-style queue specifier used here). Subclassing Handler directly avoids that intercept, so the same YAML configuration works identically on Python 3.8-3.15.

Example configuration:

# logging.yaml
version: 1
objects:
  queue:
    class: queue.Queue
    maxsize: -1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
  file_handler:
    class: logging.FileHandler
    filename: test_logger.log
    formatter: simple
  queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
    - cfg://handlers.console
    - cfg://handlers.file_handler
    queue: cfg://objects.queue

__init__(queue: Any, handlers: Any, respect_handler_level: bool = True, auto_run: bool = True)

Instantiates QueueListenerHandler object.

A simple QueueHandler-like implementation utilizing QueueListener for configured handlers. This is helpful for detaching your logger handlers from the main processing threads, which reduces the risk of getting blocked, for example, when using slower handlers such as smtp, file, or socket handlers.

Parameters:

• queue – A queue instance passed from configuration.

• handlers – A list of handlers passed from configuration.

• respect_handler_level – Flag for overriding logging levels specified in handlers. Default: True.

• auto_run – Flag for starting the queue listener automatically. Default: True.

prepare(record: LogRecord) → LogRecord

Prepares a record for queuing.

Mirrors logging.handlers.QueueHandler.prepare: formats the record and removes unpickleable items so the record can safely cross the queue to the listener thread.

Parameters:

record – A logging.LogRecord object.

Returns:

A copy of the record, safe to enqueue.

enqueue(record: LogRecord)

Enqueues a record on the queue using put_nowait.

Parameters:

record – A logging.LogRecord object.

__module__ = 'logging_.handlers.queue_listener_handler'