pyramid_retry

pyramid_retry is an execution policy for Pyramid that wraps requests and can retry them a configurable number of times under certain "retryable" error conditions before indicating a failure to the client.

Warning

This package will only work with Pyramid 1.9 and newer.

Installation

Stable release

To install pyramid_retry, run this command in your terminal:

$ pip install pyramid_retry

If you don't have pip installed, this Python installation guide can guide you through the process.

From sources

The sources for pyramid_retry can be downloaded from the Github repo.

$ git clone https://github.com/Pylons/pyramid_retry.git

Once you have a copy of the source, you can install it with:

$ pip install -e .

Usage

Activate pyramid_retry by including it in your application:

def main(global_config, **settings):
    config = Configurator(settings=settings)
    config.include('pyramid_retry')
    # ...
    config.add_route('home', '/')

By default pyramid_retry will register an instance of pyramid_retry.RetryableExecutionPolicy() as an execution policy in your application using the retry.attempts setting as the maximum number of attempts per request. The default number of attempts is 3. This number is configurable in your application's .ini file as follows:

[app:main]
# ...
retry.attempts = 3

The policy will handle any requests that fail because the application raised an instance of pyramid_retry.RetryableException or another exception implementing the pyramid_retry.IRetryableError interface.

The below, very contrived example, shows conceptually what's going on when a request is retried. The failing_view is executed initially and for the final attempt the recovery_view is executed.

@view_config(route_name='home')
def failing_view(request):
    raise RetryableException

@view_config(route_name='home', is_last_attempt=True, renderer='string')
def recovery_view(request):
    return 'success'

Of course you probably wouldn't write actual code that expects to fail like this. More realistically you may use a library like pyramid_tm to translate certain transactional errors marked as "transient" into retryable errors.

Custom Retryable Errors

The simple approach to marking errors as retryable is to simply catch the error and raise a pyramid_retry.RetryableException instead:

from pyramid_retry import RetryableException
import requests

def view(request):
    try:
        response = requests.get('https://www.google.com')
    except requests.Timeout:
        raise RetryableException

This will work but if this is the last attempt then the failed request will not actually be retried and on top of that the original exception is lost.

A better approach is to preserve the original exception and simply mark it as retryable using the pyramid_retry.IRetryableError marker interface:

from pyramid_retry import mark_error_retryable
import requests
import zope.interface

# mark requests.Timeout errors as retryable
mark_error_retryable(requests.Timeout)

def view(request):
    response = requests.get('https://www.google.com')

Per-Request Attempts

It may be desirable to override the attempts per-request. For example, if one endpoint on the system cannot afford to make a copy of the request via request.make_body_seekable() then the activate hook can be used to set attempts= on that endpoint.

def activate_hook(request):
    if request.path == '/upload':
        return 1  # disable retries on this endpoint

config.add_settings({'retry.activate_hook': activate_hook})

The activate_hook should return a number >= 1 or None. If None then the policy will fallback to the retry.attempts setting.

View Predicates

When the library is included in your application it registers two new view predicates which are especially useful on exception views to determine when to handle certain errors.

retryable_error=[True/False] will match the exception view only if the exception is both an retryable error and there are remaining attempts in which the request would be retried. See pyramid_retry.RetryableErrorPredicate for more information.

last_retry_attempt=[True/False] will match only if, when the view is executed, there will not be another attempt for this request. See pyramid_retry.LastAttemptPredicate for more information.

Receiving Retry Notifications

The pyramid_retry.IBeforeRetry event can be subscribed to receive a callback with the request and environ prior to the pipeline being completely torn down. This can be very helpful if any state is stored on the environ itself that needs to be reset prior to the retry attempt.

from pyramid.events import subscriber
from pyramid_retry import IBeforeRetry

@subscriber(IBeforeRetry)
def retry_event(event):
    print(f'A retry is about to occur due to {event.exception}.')

The exception attribute indicates the exception that triggered the retry. The exception may come from either request.exception if it was caught and a response was rendered, or it may come from an uncaught exception.

Caveats

  • In order to guarantee that a request can be retried it must make the body seekable. This is done via request.make_body_seekable(). Generally the body is loaded directly from environ['wsgi.input'] which is controlled by the WSGI server. However to make the body seekable it is copied into a seekable wrapper. In some cases this can lead to a very large copy operation before the request is executed.
  • pyramid_retry does not copy the environ or make any attempt to restore it to its original state before retrying a request. This means anything stored on the environ will persist across requests created for that environ.

Indices and tables