pyramid_tm

Overview

pyramid_tm is a package which allows Pyramid requests to join the active transaction as provided by the Python transaction package. (See the documentation for the transaction package for an explanation of what “joining the active transaction” means).

Installation

Install using pip, e.g. (within a virtualenv):

$ pip install pyramid_tm

Setup

Once pyramid_tm is installed, you must use the config.include mechanism to include it into your Pyramid project’s configuration. In your Pyramid project’s __init__.py:

1
2
config = Configurator(.....)
config.include('pyramid_tm')

Or use the pyramid.includes configuration setting in your .ini file:

1
2
[app:myapp]
pyramid.includes = pyramid_tm

After the package is included, whenever a new request enters the application, a new transaction is associated with that request.

Note

When the repoze.tm or repoze.tm2 middleware is in the WSGI pipeline, pyramid_tm becomes inactive.

transaction Usage

At the beginning of a request a new transaction is started using the request.tm.begin() function. Once the request has finished all of its works (ie views have finished running), a few checks are tested:

  1. Did some a transaction.doom() cause the transaction to become “doomed”? if so, request.tm.abort().
  2. Did an exception occur in the underlying code? if so, request.tm.abort()
  3. If the tm.commit_veto configuration setting was used, did the commit veto callback, called with the response generated by the application, return a result that evaluates to True? if so, request.tm.abort().

If none of these checks calls request.tm.abort() then the transaction is instead committed using request.tm.commit().

By itself, this transaction machinery doesn’t do much. It is up to third-party code to join the active transaction to benefit. See repoze.filesafe for an example of how files creation can be committed or rolled back based on transaction and the pyramid_mailer package to see how you can prevent emails from being sent until a transaction succeeds. ZODB database connections are automatically joined to the transaction, as well as SQLAlchemy connections which are configured with zope.sqlalchemy.register(session) from the zope.sqlalchemy package.

Savepoints

When using sessions / data managers joined to the transaction, it’s important to synchronize changes across those managers. This means that it’s usually incorrect to use your backend’s session lifecycle functions directly such as sqlalchemy.orm.Session.begin_nested. Instead, synchronize a savepoint across all joined data managers via sp = request.tm.savepoint(). The savepoint can be rolled back via sp.rollback(). For example:

def my_view(request):
    sp = request.tm.savepoint()
    try:
        page = WikiPage()
        page.id = 5  # maybe the id 5 violates a unique constraint
        request.dbsession.add(page)
        request.dbsession.flush()
    except sqlalchemy.exc.IntegrityError:
        # page already exists!
        sp.rollback()
    # continue with or without the data added in the try-clause
    ...

Note

Not every data manager supports savepoints and as such some changes may not be able to be rolled back.

Error Handling

pyramid_tm is positioned OVER the EXCVIEW tween. The implication of this is that the transaction may still be open and alive during the execution of your exception views. This is not guaranteed. If you write an exception view that expects an open transaction then you should declare your intent using the tm_active=True view predicate otherwise it may be executed later in the pipeline after the transaction has already been completed. For example:

from pyramid.view import exception_view_config

log = __import__('logging').getLogger(__name__)

@exception_view_config(Exception, tm_active=True)
def transactional_error_view(exc, request):
    # depending on your AuthenticationPolicy the authenticated
    # userid likely requires a lookup in your database which would
    # require an active transaction
    if request.authenticated_userid is not None:
        log.exception('authenticated user caused an exception')
    else:
        log.exception('unknown user caused an exception')
    response = request.response
    response.status_code = 500
    return response

@exception_view_config(Exception)
def default_error_view(exc, request):
    log.exception('unknown user caused an exception')
    response = request.response
    response.status_code = 500
    return response

In the above example, transactional_error_view will be invoked only when an exception occurs during the pyramid_tm lifecycle. Otherwise, default_error_view will be invoked as a fallback.

The transaction created and completed by pyramid_tm should be used for operations directly related to processing the request. Very often it is desirable to perform operations on the database and other backends in a failure scenario. This should be done using a separate transaction / connection, possibly in autocommit mode. Do not use request.tm and request.dbsession and such for these cases as the work added to that transaction is expected to be aborted upon any failures.

Retries

pyramid_tm ships with support for pyramid_retry which is an execution policy that will retry requests when they fail with exceptions marked as retryable. By default, retrying is turned off. In order to turn it on you must update your app’s configuration:

from pyramid.config import Configurator

def main(global_config, **settings):
    config = Configurator(settings=settings)
    config.include('pyramid_retry')
    config.include('pyramid_tm')

Finally, ensure that your application’s settings have retry.attempts set to a value greater than 1.

When the transaction manager calls the downstream handler, if the handler raises a retryable exception, pyramid_tm will mark the exception as retryable by pyramid_retry. The execution policy will detect a retryable error and create a new copy of the request with new state.

Retryable exceptions include ZODB.POSException.ConflictError, and certain exceptions raised by various data managers, such as psycopg2.extensions.TransactionRollbackError, cx_Oracle.DatabaseError where the exception’s code is 8877. Any exception which inherits from transaction.interfaces.TransientError will be marked as retryable.

Read more about retrying requests in the pyramid_retry documentation.

Custom Transaction Managers

By default pyramid_tm will use the threadlocal transaction.manager to associate one transaction manager per thread. If you wish to override this and provide your own transaction manager you can create your own manager hook that will return the manager it should use.

1
2
3
4
import transaction

def manager_hook(request):
    return transaction.TransactionManager(explicit=True)

To enable this hook, add it as the tm.manager_hook setting in your app.

1
2
3
4
5
6
7
from pyramid.config import Configurator

def app(global_conf, **settings):
    settings['tm.manager_hook'] = manager_hook
    config = Configurator(settings=settings)
    config.include('pyramid_tm')
    # ...

This specific example, using an explicit mode non-threadlocal manager, is highly recommended and is shipped as pyramid_tm.explicit_manager(). Simply set tm.manager_hook = pyramid_tm.explicit_manager in your settings to enable it.

The current transaction manager being used for any particular request can always be accessed on the request as request.tm so long as it is accessed while the pyramid_tm tween is active. If you try to access request.tm outside of the tween or during a request in which pyramid_tm was disabled, request.tm will raise an AttributeError.

Note

It is recommended to use a custom transaction manager with explicit=True, as in the example above, instead of the threadlocal transaction.manager to give greater control over the transaction’s lifecycle and to weed out potential bugs in your application. For example, you may have some parts of your app that access the manager after it has already been committed. This will open an implicit transaction that is never committed, and will even hang around until a subsequent request aborts the implicit transaction. Instead, if you set explicit=True, any code affecting the manager outside of the lifecycle of the transaction will cause an error and will be noticed quickly.

Adding an Activation Hook

It may not always be desirable to have every request managed by the transaction manager automatically. It is possible to configure pyramid_tm with an “activate” hook. The callback function receives the request. It can then examine it and return False if the transaction manager should be disabled for that request.

1
2
3
4
5
6
def activate_hook(request):
    if request.path_info.startswith('/long-poll'):
        # Allow the long-poll class to manage its own connections to avoid
        # long-lived transactions.
        return False
    return True

To enable this hook, add it as the tm.activate_hook setting in your app.

1
2
3
4
5
6
7
from pyramid.config import Configurator

def app(global_conf, **settings):
    settings['tm.activate_hook'] = activate_hook
    config = Configurator(settings=settings)
    config.include('pyramid_tm')
    # ...

Or via PasteDeploy:

1
2
[app:myapp]
tm.activate_hook = myapp.activate_hook

In either configuration the value for tm.activate_hook is a dotted Python name.

Adding a Commit Veto Hook

It is possible to configure pyramid_tm with a “commit veto” hook. The commit veto hook receives the request and the response. It can examine both of them, and return True if the transaction should be vetoed. If the transaction is vetoed, it will be aborted instead of committed. By default, pyramid_tm does not configure a commit veto into the system; you must do it explicitly.

pyramid_tm contains a pyramid_tm.default_commit_veto() that is suitable for use when you want to abort when the response’s status code indicates non-success or if you’d like to signal that the transaction should be aborted or committed using a response header. The default commit veto vetoes a commit if the status code starts with 4 or 5 or there is a X-Tm response header with a value that does not equal commit.

1
2
3
4
5
def default_commit_veto(request, response):
    xtm = response.headers.get('x-tm')
    if xtm is not None:
        return xtm != 'commit'
    return response.status.startswith(('4', '5'))

If you’d like to use this commit veto in your system, you can do it via Python:

1
2
3
4
5
6
7
from pyramid.config import Configurator

def app(global_conf, **settings):
    settings['tm.commit_veto'] = 'pyramid_tm.default_commit_veto'
    config = Configurator(settings=settings)
    config.include('pyramid_tm')
    # ...

Or via PasteDeploy:

1
2
[app:myapp]
tm.commit_veto = pyramid_tm.default_commit_veto

If you’d like to use a different “commit veto” callback, create a function with the same signature (request, response) and return value (True or False), then pass a tm.commit_veto key/value pair in your settings which points at the Python dotted name of this commit veto.

Via Python:

1
2
3
4
5
6
from pyramid.config import Configurator

def app(global_conf, settings):
    settings['tm.commit_veto'] = 'my.package.commit_veto'
    config = Configurator(settings=settings)
    config.include('pyramid_tm')

Via PasteDeploy:

1
2
[app:myapp]
tm.commit_veto = my.package.commit_veto

In the PasteDeploy example, the path is a dotted Python name, where the dots separate module and package names, and the colon separates a module from its contents. In the above example, the code would be implemented as a “commit_veto” function which lives in the “package” submodule of the “my” package.

View Predicates

pyramid_tm registers a view predicate named tm_active which accepts a value of True or False. This can be useful for declaring intent when defining exception views that require access to the transaction controlled by pyramid_tm. For specific examples, see Error Handling.

If the request is manually completed via request.tm.abort() or request.tm.commit(), this predicate may be incorrect depending on the specific transaction manager being used. After completing a transaction controlled by the transaction manager in explicit mode it is necessary to invoke request.tm.begin() to start a new one or any subsequent uses of the transaction manager will fail.

Explicit Tween Configuration

Note that the transaction manager is a Pyramid “tween”, and it can be used in the explicit tween list if its implicit position in the tween chain is incorrect (see the output of ptweens):

[app:myapp]
pyramid.tweens = someothertween
                 pyramid_tm.tm_tween_factory
                 pyramid.tweens.excview_tween_factory

It usually belongs directly above the “pyramid.tweens.excview_tween_factory” entry in the `` ptweens`` output, and will attempt to sort there by default as the result of having config.include('pyramid_tm') invoked.

Avoid Accessing the Authentication Policy

By default the tween will access pyramid.request.Request.unauthenticated_userid in order to annotate the transaction with information about the user. This can be turned off by setting the ini option tm.annotate_user = false.

More Information

Reporting Bugs / Development Versions

Visit http://github.com/Pylons/pyramid_tm to download development or tagged versions.

Visit http://github.com/Pylons/pyramid_tm/issues to report bugs.

Changes

2.2 (2017-07-03)

Backward Incompatibilities

  • This is a backward-incompatible change for anyone using the tm.commit_veto hook. Anyone else is unaffected.

    The tm.commit_veto hook will now be consulted for any squashed exceptions instead of always aborting. Previously, if an exception was handled by an exception view, the transaction would always be aborted. Now, the commit_veto can inspect request.exception and the generated response to determine whether to commit or abort.

    The new behavior when using the pyramid_tm.default_commit_veto is that a squashed exception may be committed if either of the following conditions are true:

    • The response contains the x-tm header set to commit.
    • The response’s status code does not start with 4 or 5.

    In most cases the response would result in 4xx or 5xx exception and would be aborted - this behavior remains the same. However, if the squashed exception rendered a response that is 3xx or 2xx (such as raising pyramid.httpexceptions.HTTPFound), then the transaction will be committed instead of aborted.

    See https://github.com/Pylons/pyramid_tm/pull/65

2.1 (2017-06-07)

  • On Pyramid >= 1.7 any errors raised from pyramid_tm invoking request.tm.abort and request.tm.commit will be caught and used to lookup and execute an exception view to return an error response. This exception view will be executed with an inactive transaction manager. See https://github.com/Pylons/pyramid_tm/pull/61

2.0 (2017-04-11)

Major Features

Backward Incompatibilities

  • The tm.attempts setting has been removed and retry support has been moved into a new package named pyramid_retry. If you want retry support then please look at that library for more information about installing and enabling it. See https://github.com/Pylons/pyramid_tm/pull/55
  • The pyramid_tm tween has been moved over the EXCVIEW tween. If you have any hacks in your application that are opening a new transaction inside your exception views then it’s likely you will want to remove them or re-evaluate when upgrading. See https://github.com/Pylons/pyramid_tm/pull/55
  • Drop support for Pyramid < 1.5.

Minor Features

  • Support for Python 3.6.

1.1.1 (2016-11-21)

1.1.0 (2016-11-19)

  • Support transaction 2.x.
  • The transaction’s request path and userid are now coerced to unicode by first decoding as utf-8 and falling back to latin-1. If the userid does not conform to these restrictions then set tm.annotate_user = no in your settings. See https://github.com/Pylons/pyramid_tm/pull/50

1.0.2 (2016-11-18)

1.0.1 (2016-10-24)

  • Removes the AttributeError when request.tm is accessed outside the tween. It turns out this broke subrequests as well as pshell and pyramid.paster.bootstrapp CLI scripts, especially when using the global transaction manager which can be tracked outside of the tween. See https://github.com/Pylons/pyramid_tm/pull/48

1.0 (2016-09-12)

  • Drop Python 2.6, 3.2 and 3.3 support.
  • Add Python 3.5 support.
  • Subtle bugs can occur if you use the transaction manager during a request in which pyramid_tm is disabled via an activate_hook. To combat these types of errors, attempting to access request.tm will now raise an AttributeError when pyramid_tm is inactive. See https://github.com/Pylons/pyramid_tm/pull/46

0.12.1 (2015-11-25)

  • Fix compatibility with 1.2 and 1.3 again. This wasn’t fully fixed in the 0.12 release as the tween was relying on request properties working (which they do not inside tweens in older versions). See https://github.com/Pylons/pyramid_tm/pull/39

0.12 (2015-05-20)

0.11 (2015-02-04)

0.10 (2015-01-06)

0.9 (2014-12-30)

0.8 (2014-11-12)

0.7 (2012-12-30)

  • Write unauthenticated userid and request.path_info as transaction metadata via t.setUser and t.note respectively during a commit.

0.6 (2012-12-26)

  • Disuse the confusing and bug-ridden generator-plus-context-manager “attempts” mechanism from the transaction package for retrying retryable exceptions (e.g. ZODB ConflictError). Use a simple while loop plus a counter and imperative logic instead.

0.5 (2012-06-26)

Bug Fixes

  • When a non-retryable exception was raised as the result of a call to transaction.manager.commit, the exception was not reraised properly. Symptom: an unrecoverable exception such as Unsupported: Storing blobs in <somestorage> is not supported. would be swallowed inappropriately.

0.4 (2012-03-28)

Bug Fixes

Testing

  • No longer tested under Python 2.5 by tox.ini (and therefore no longer tested under 2.5 by the Pylons Jenkins server). The package may still work under 2.5, but automated tests will no longer show breakage when it changes in ways that break 2.5 support.
  • Squash test deprecation warnings under Python 3.2.

0.3 (2011-09-27)

Features

  • The transaction manager has been converted to a Pyramid 1.2 “tween” (instead of an event subscriber). It will be slotted directly “below” the exception view handler, meaning it will have a chance to handle exceptions before they are turned into responses. This means it’s best to “raise HTTPFound(...)” instead of “return HTTPFound(...)” if you want an HTTP exception to abort the transaction.
  • The transaction manager will now retry retryable exceptions (such as a ZODB conflict error) if tm.attempts is configured to be more than the default of 1. See the Retrying section of the documentation.
  • Python 3.2 compatibility (requires Pyramid 1.3dev+).

Backwards Incompatibilities

  • Incompatible with Pyramid < 1.2a1. Use pyramid_tm version 0.2 if you need compatibility with an older Pyramid installation.

  • The default_commit_veto commit veto callback is no longer configured into the system by default. Use tm.commit_veto = pyramid_tm.default_commit_veto in the deployment settings to add it. This is for parity with repoze.tm2, which doesn’t configure in a commit veto by default either.

  • The default_commit_veto no longer checks for the presence of the X-Tm-Abort header when attempting to figure out whether the transaction should be aborted (although it still checks for the X-Tm header). Use version 0.2 or a custom commit veto function if your application depends on the X-Tm-Abort header.

  • A commit veto is now called with two arguments: request and response. The request is the webob request that caused the transaction manager to become active. The response is the response returned by the Pyramid application. This call signature is incompatible with older versions. The call signature of a pyramid_tm 0.2 and older commit veto accepted three arguments: environ, status, and headers. If you’re using a custom commit_veto function, you’ll need to either convert your existing function to use the new calling convention or use a wrapper to make it compatible with the new calling convention. Here’s a simple wrapper function (bwcompat_commit_veto_wrapper) that will allow you to use your existing custom commit veto function:

    def bwcompat_commit_veto_wrapper(request, response):
    return my_custom_commit_veto(request.environ, response.status,
                                 response.headerlist)

Deprecations

  • The pyramid_tm.commit_veto configuration setting is now canonically spelled as tm.commit_veto. The older spelling will continue to work, but may raise a deprecation error when used.

0.2 (2011-07-18)

  • A new header X-Tm is now honored by the default_commit_veto commit veto hook. If this header exists in the headerlist, its value must be a string. If its value is commit, the transaction will be committed regardless of the status code or the value of X-Tm-Abort. If the value of the X-Tm header is abort (or any other string value except commit), the transaction will be aborted, regardless of the status code or the value of X-Tm-Abort.

0.1 (2011-02-23)

  • Initial release, based on repoze.tm2