Handlers

QueueListenerHander

A simple QueueHandler subclass implementation utilizing QueueListener for configured handlers. This is helpful for detaching the logger handlers from the main threads, which reduces the risk of getting blocked, for example, when using slower handlers such as smtp, file, or socket handlers.

Example Usage

An example YAML configuration file utilizing QueueListenerHander

logging.yaml
version: 1
objects:
  queue:
    class: queue.Queue
    maxsize: 1000
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
  file_handler:
    class: logging.FileHandler
    filename: test_logger.log
    formatter: simple
  queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
      - cfg://handlers.console
      - cfg://handlers.file_handler
    queue: cfg://objects.queue
loggers:
  test_logger:
    level: DEBUG
    handlers:
      - queue_handler
    propagate: no
root:
  level: NOTSET
  handlers:
    - console

Just load the configuration file and start logging.

test_logger.py
import logging.config
import yaml

with open("logging.yaml", "r") as config_file:
    logging.config.dictConfig(yaml.safe_load(config_file.read()))

logger = logging.getLogger("test_logger")

logger.debug("This is a debug log")
logger.info("This is an info log")
logger.warning("This is an warning log")
logger.error("This is an error log")
logger.critical("This is a critical log")

Optional Params

A queue object must be passed since the handler does not set a default queue implementation. Set maxsize: -1 to make the queue unlimited.

Module Members

class logging_.handlers.queue_listener_handler.QueueListenerHandler(queue: Any, handlers: Any, respect_handler_level: bool = True, auto_run: bool = True)

Bases: logging.handlers.QueueHandler

QueueListenerHandler class for managing a queue listener with configured handlers.

This class sets up a queue listener logger handler with customizable configurations. Inspired by Rob Blackbourn’s article: https://rob-blackbourn.medium.com/how-to-use-python-logging-queuehandler-with-dictconfig-1e8b1284e27a

Example configuration:

# logging.yaml
version: 1
objects:
  queue:
    class: queue.Queue
    maxsize: -1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout
  file_handler:
    class: logging.FileHandler
    filename: test_logger.log
    formatter: simple
  queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
    - cfg://handlers.console
    - cfg://handlers.file_handler
   queue: cfg://objects.queue
__init__(queue: Any, handlers: Any, respect_handler_level: bool = True, auto_run: bool = True)

Instantiates QueueListenerHandler object.

A simple QueueHandler subclass implementation utilizing QueueListener for configured handlers. This is helpful for detaching ypur logger handlers from the main processing threads, which reduces the risk of getting blocked, for example, when using slower handlers such as smtp, file, or socket handlers.

Parameters
  • queue – A queue instance passed from configuration.

  • handlers – A list of handlers passed from configuration.

  • respect_handler_level – Flag for overriding logging levels specified in handlers. Default: True.

  • auto_run – Flag for starting the queue listener automatically. Default: True.

__module__ = 'logging_.handlers.queue_listener_handler'