Sunday 31 October 2010

logutils: Using recent logging features with older Python versions

Python’s development is a process of continuous improvement, and the standard library of course undergoes changes as new versions of Python are released. In the logging package, there have been numerous improvements in recent months:

  • Dictionary-based configuration (PEP 391)
  • QueueHandler for working with multiprocessing and in-process queues

In addition, there have been a few bits of utility code which have been posted here:

Many of the enhancements have been added as new features which will emerge in Python 3.2, but cannot be added to Python 2.X because there are no Python 2.X branches where new features can be added. Python 2.7, the last 2.X release, is in maintenance mode and will only accept bug-fixes which maintain backwards compatibility.

Although Python 3.2+ is the way forward for Python development, many users will be constrained to use 2.X for some considerable time yet, mostly because libraries they depend on have not yet been ported to 3.X, but sometimes also because their codebase is too large to consider porting to 3.X, when viewed from a cost/benefit viewpoint, or perhaps because their code needs to remain workable under 2.X and they cannot maintain two separate codebases.

For those users who are forced to stick with Python 2.X but who want to use some of the features added recently to logging, the logutils project offers one way in which they can access these features. This is available via PyPI so it can be installed using pip or easy_install. The package is currently at release 0.2 and contains the following:

  • Dictionary-based configuration (PEP 391) which is broadly compatible with Python 2.7 and 3.2.
  • The LoggerAdapter class is easier to subclass.
  • The QueueHandler and QueueListener classes are available for use with in-process, multiprocessing and other types of queue, as described in this post.
  • The TestHandler and Matcher classes are available for use in unit tests, as described in this post.
  • An enhanced HTTPHandler allows the use of secure connections and credentials for logging in to websites.

The package needs Python 2.4 or later. Some features are not available in Python 2.4, due to limitations of that Python version. For example, str.format is only available from Python 2.6 onwards;  LoggerAdapter relies on a feature added in Python 2.5, namely the extra parameter for adding additional context to logs; and named handlers are not available for Python 2.X versions < 2.7 and Python 3.X versions < 3.2. (As a consequence of this, incremental handler configuration is not available on the versions where named handlers are not available.)

Project downloads are here. You can post issues here (needs a Google account). Documentation is available here.


  1. It might be a good idea to recommend that developers use

    import logutils as logging

    in their code when using logutils. This means that in the future if the code is eventually ported to Python 3.2+ it's a simple one-line change to make it use the stdlib version.

  2. @kisielk: Thanks for the suggestion. The reasons I didn't do this were:

    1. logutils is not a drop-in replacement for logging, and the chances are high that you'd need to also import logging where you imported logutils.

    2. There are some things in logutils which will are not planned to be added to logging (e.g. the Matcher, BraceMessage and DollarMessage classes). So there is the likelihood that many users will continue to import both packages.