Adding Tests

We will now add tests for the models and views as well as a few functional tests in a new tests package. Tests ensure that an application works, and that it continues to work when changes are made in the future.

Test harness

The project came bootstrapped with some tests and a basic harness. These are located in the tests package at the top-level of the project. It is a common practice to put tests into a tests package alongside the application package, especially as projects grow in size and complexity. A useful convention is for each module in the application to contain a corresponding module in the tests package. The test module would have the same name with the prefix test_.

The harness consists of the following setup:

• The project.optional-dependencies stanza of pyproject.toml - controls the dependencies installed when testing. When the list is changed, it is necessary to re-run $VENV/bin/pip install -e ".[testing]" to ensure the new dependencies are installed.

• The tool.pytest.ini_options stanza of pyproject.toml controls basic pytest configuration, including where to find the tests. We have configured pytest to search for tests in the application package and in the tests package.

• The tool.coverage.run stanza of pyproject.toml controls coverage config. In our setup, it works with the pytest-cov plugin that we use via the --cov options to the pytest command.

• The testing.ini file is a mirror of development.ini and production.ini that contains settings used for executing the test suite. Most importantly, it contains the database connection information used by tests that require the database.

• The tests/conftest.py file defines the core fixtures available throughout our tests. The fixtures are explained in more detail in the following sections. Open tests/conftest.py and follow along.

Session-scoped test fixtures

• app_settings - the settings dict parsed from the testing.ini file that would normally be passed by pserve into your app's main function.

• dbengine - initializes the database. It's important to start each run of the test suite from a known state, and this fixture is responsible for preparing the database appropriately. This includes deleting any existing tables, running migrations, and potentially even loading some fixture data into the tables for use within the tests.

• app - the Pyramid WSGI application, implementing the pyramid.interfaces.IRouter interface. Most commonly this would be used for functional tests.

Per-test fixtures

• tm - a transaction.TransactionManager object controlling a transaction lifecycle. Generally other fixtures would join to the tm fixture to control their lifecycle and ensure they are aborted at the end of the test.

• dbsession - a sqlalchemy.orm.session.Session object connected to the database. The session is scoped to the tm fixture. Any changes made will be aborted at the end of the test.

• testapp - a webtest.TestApp instance wrapping the app and is used to sending requests into the application and return full response objects that can be inspected. The testapp is able to mutate the request environ such that the dbsession and tm fixtures are injected and used by any code that's touching request.dbsession and request.tm. The testapp maintains a cookiejar, so it can be used to share state across requests, as well as the transaction database connection.

• app_request - a pyramid.request.Request object that can be used for more lightweight tests versus the full testapp. The app_request can be passed to view functions and other code that need a fully functional request object.

• dummy_request - a pyramid.testing.DummyRequest object that is very lightweight. This is a great object to pass to view functions that have minimal side-effects as it'll be fast and simple.

• dummy_config — a pyramid.config.Configurator object used as configuration by dummy_request. Useful for mocking configuration like routes and security policies.

Modifying the fixtures

We're going to make a few application-specific changes to the test harness. It's always good to come up with patterns for things that are done often to avoid lots of boilerplate.

• Initialize the cookiejar with a CSRF token. Remember our application is using pyramid.csrf.CookieCSRFStoragePolicy.

• testapp.get_csrf_token() - every POST/PUT/DELETE/PATCH request must contain the current CSRF token to prove to our app that the client isn't a third-party. So we want an easy way to grab the current CSRF token and add it to the request.

• testapp.login(params) - many pages are only accessible by logged in users so we want a simple way to login a user at the start of a test.

Update tests/conftest.py to look like the following, adding the highlighted lines:

import alembic
import alembic.config
import alembic.command
import os
from pyramid.paster import get_appsettings
from pyramid.scripting import prepare
from pyramid.testing import DummyRequest, testConfig
import pytest
import transaction
from webob.cookies import Cookie
import webtest

from tutorial import main
from tutorial import models
from tutorial.models.meta import Base


def pytest_addoption(parser):
    parser.addoption('--ini', action='store', metavar='INI_FILE')

@pytest.fixture(scope='session')
def ini_file(request):
    # potentially grab this path from a pytest option
    return os.path.abspath(request.config.option.ini or 'testing.ini')

@pytest.fixture(scope='session')
def app_settings(ini_file):
    return get_appsettings(ini_file)

@pytest.fixture(scope='session')
def dbengine(app_settings, ini_file):
    engine = models.get_engine(app_settings)

    alembic_cfg = alembic.config.Config(ini_file)
    Base.metadata.drop_all(bind=engine)
    alembic.command.stamp(alembic_cfg, None, purge=True)

    # run migrations to initialize the database
    # depending on how we want to initialize the database from scratch
    # we could alternatively call:
    # Base.metadata.create_all(bind=engine)
    # alembic.command.stamp(alembic_cfg, "head")
    alembic.command.upgrade(alembic_cfg, "head")

    yield engine

    Base.metadata.drop_all(bind=engine)
    alembic.command.stamp(alembic_cfg, None, purge=True)

@pytest.fixture(scope='session')
def app(app_settings, dbengine):
    return main({}, dbengine=dbengine, **app_settings)

@pytest.fixture
def tm():
    tm = transaction.TransactionManager(explicit=True)
    tm.begin()
    tm.doom()

    yield tm

    tm.abort()

@pytest.fixture
def dbsession(app, tm):
    session_factory = app.registry['dbsession_factory']
    return models.get_tm_session(session_factory, tm)

class TestApp(webtest.TestApp):
    def get_cookie(self, name, default=None):
        # webtest currently doesn't expose the unescaped cookie values
        # so we're using webob to parse them for us
        # see https://github.com/Pylons/webtest/issues/171
        cookie = Cookie(' '.join(
            '%s=%s' % (c.name, c.value)
            for c in self.cookiejar
            if c.name == name
        ))
        return next(
            (m.value.decode('latin-1') for m in cookie.values()),
            default,
        )

    def get_csrf_token(self):
        """
        Convenience method to get the current CSRF token.

        This value must be passed to POST/PUT/DELETE requests in either the
        "X-CSRF-Token" header or the "csrf_token" form value.

        testapp.post(..., headers={'X-CSRF-Token': testapp.get_csrf_token()})

        or

        testapp.post(..., {'csrf_token': testapp.get_csrf_token()})

        """
        return self.get_cookie('csrf_token')

    def login(self, params, status=303, **kw):
        """ Convenience method to login the client."""
        body = dict(csrf_token=self.get_csrf_token())
        body.update(params)
        return self.post('/login', body, **kw)

@pytest.fixture
def testapp(app, tm, dbsession):
    # override request.dbsession and request.tm with our own
    # externally-controlled values that are shared across requests but aborted
    # at the end
    testapp = TestApp(app, extra_environ={
        'HTTP_HOST': 'example.com',
        'tm.active': True,
        'tm.manager': tm,
        'app.dbsession': dbsession,
    })

    # initialize a csrf token instead of running an initial request to get one
    # from the actual app - this only works using the CookieCSRFStoragePolicy
    testapp.set_cookie('csrf_token', 'dummy_csrf_token')

    return testapp

@pytest.fixture
def app_request(app, tm, dbsession):
    """
    A real request.

    This request is almost identical to a real request but it has some
    drawbacks in tests as it's harder to mock data and is heavier.

    """
    with prepare(registry=app.registry) as env:
        request = env['request']
        request.host = 'example.com'

        # without this, request.dbsession will be joined to the same transaction
        # manager but it will be using a different sqlalchemy.orm.Session using
        # a separate database transaction
        request.dbsession = dbsession
        request.tm = tm

        yield request

@pytest.fixture
def dummy_request(tm, dbsession):
    """
    A lightweight dummy request.

    This request is ultra-lightweight and should be used only when the request
    itself is not a large focus in the call-stack.  It is much easier to mock
    and control side-effects using this object, however:

    - It does not have request extensions applied.
    - Threadlocals are not properly pushed.

    """
    request = DummyRequest()
    request.host = 'example.com'
    request.dbsession = dbsession
    request.tm = tm

    return request

@pytest.fixture
def dummy_config(dummy_request):
    """
    A dummy :class:`pyramid.config.Configurator` object.  This allows for
    mock configuration, including configuration for ``dummy_request``, as well
    as pushing the appropriate threadlocals.

    """
    with testConfig(request=dummy_request) as config:
        yield config

Unit tests

We can test individual APIs within our codebase to ensure they fulfill the expected contract that the rest of the application expects. For example, we'll test the password hashing features we added to the tutorial.models.User object.

Create tests/test_user_model.py such that it appears as follows:

from tutorial import models


def test_password_hash_saved():
    user = models.User(name='foo', role='bar')
    assert user.password_hash is None

    user.set_password('secret')
    assert user.password_hash is not None

def test_password_hash_not_set():
    user = models.User(name='foo', role='bar')
    assert not user.check_password('secret')

def test_correct_password():
    user = models.User(name='foo', role='bar')
    user.set_password('secret')
    assert user.check_password('secret')

def test_incorrect_password():
    user = models.User(name='foo', role='bar')
    user.set_password('secret')
    assert not user.check_password('incorrect')

Integration tests

We can directly execute the view code, bypassing Pyramid and testing just the code that we've written. These tests use dummy requests that we'll prepare appropriately to set the conditions each view expects, such as adding dummy data to the session. We'll be using dummy_config to configure the necessary routes, as well as setting the security policy as pyramid.testing.DummySecurityPolicy to mock dummy_request.identity.

Update tests/test_views.py such that it appears as follows:

from pyramid.testing import DummySecurityPolicy
import sqlalchemy as sa

from tutorial import models


def makeUser(name, role):
    return models.User(name=name, role=role)


def setUser(config, user):
    config.set_security_policy(
        DummySecurityPolicy(identity=user)
    )

def makePage(name, data, creator):
    return models.Page(name=name, data=data, creator=creator)

class Test_view_wiki:
    def _callFUT(self, request):
        from tutorial.views.default import view_wiki
        return view_wiki(request)

    def _addRoutes(self, config):
        config.add_route('view_page', '/{pagename}')

    def test_it(self, dummy_config, dummy_request):
        self._addRoutes(dummy_config)
        response = self._callFUT(dummy_request)
        assert response.location == 'http://example.com/FrontPage'

class Test_view_page:
    def _callFUT(self, request):
        from tutorial.views.default import view_page
        return view_page(request)

    def _makeContext(self, page):
        from tutorial.routes import PageResource
        return PageResource(page)

    def _addRoutes(self, config):
        config.add_route('edit_page', '/{pagename}/edit_page')
        config.add_route('add_page', '/add_page/{pagename}')
        config.add_route('view_page', '/{pagename}')

    def test_it(self, dummy_config, dummy_request, dbsession):
        # add a page to the db
        user = makeUser('foo', 'editor')
        page = makePage('IDoExist', 'Hello CruelWorld IDoExist', user)
        dbsession.add_all([page, user])

        # create a request asking for the page we've created
        self._addRoutes(dummy_config)
        dummy_request.context = self._makeContext(page)

        # call the view we're testing and check its behavior
        info = self._callFUT(dummy_request)
        assert info['page'] is page
        assert info['content'] == (
            '<div class="document">\n'
            '<p>Hello <a href="http://example.com/add_page/CruelWorld">'
            'CruelWorld</a> '
            '<a href="http://example.com/IDoExist">'
            'IDoExist</a>'
            '</p>\n</div>\n'
        )
        assert info['edit_url'] == 'http://example.com/IDoExist/edit_page'

class Test_add_page:
    def _callFUT(self, request):
        from tutorial.views.default import add_page
        return add_page(request)

    def _makeContext(self, pagename):
        from tutorial.routes import NewPage
        return NewPage(pagename)

    def _addRoutes(self, config):
        config.add_route('add_page', '/add_page/{pagename}')
        config.add_route('view_page', '/{pagename}')

    def test_get(self, dummy_config, dummy_request, dbsession):
        setUser(dummy_config, makeUser('foo', 'editor'))
        self._addRoutes(dummy_config)
        dummy_request.context = self._makeContext('AnotherPage')
        info = self._callFUT(dummy_request)
        assert info['pagedata'] == ''
        assert info['save_url'] == 'http://example.com/add_page/AnotherPage'

    def test_submit_works(self, dummy_config, dummy_request, dbsession):
        dummy_request.method = 'POST'
        dummy_request.POST['body'] = 'Hello yo!'
        dummy_request.context = self._makeContext('AnotherPage')
        setUser(dummy_config, makeUser('foo', 'editor'))
        self._addRoutes(dummy_config)
        self._callFUT(dummy_request)
        page = dbsession.scalars(
            sa.select(models.Page).where(models.Page.name == 'AnotherPage')
        ).one()
        assert page.data == 'Hello yo!'

class Test_edit_page:
    def _callFUT(self, request):
        from tutorial.views.default import edit_page
        return edit_page(request)

    def _makeContext(self, page):
        from tutorial.routes import PageResource
        return PageResource(page)

    def _addRoutes(self, config):
        config.add_route('edit_page', '/{pagename}/edit_page')
        config.add_route('view_page', '/{pagename}')

    def test_get(self, dummy_config, dummy_request, dbsession):
        user = makeUser('foo', 'editor')
        page = makePage('abc', 'hello', user)
        dbsession.add_all([page, user])

        self._addRoutes(dummy_config)
        dummy_request.context = self._makeContext(page)
        info = self._callFUT(dummy_request)
        assert info['pagename'] == 'abc'
        assert info['save_url'] == 'http://example.com/abc/edit_page'

    def test_submit_works(self, dummy_config, dummy_request, dbsession):
        user = makeUser('foo', 'editor')
        page = makePage('abc', 'hello', user)
        dbsession.add_all([page, user])

        self._addRoutes(dummy_config)
        dummy_request.method = 'POST'
        dummy_request.POST['body'] = 'Hello yo!'
        setUser(dummy_config, user)
        dummy_request.context = self._makeContext(page)
        response = self._callFUT(dummy_request)
        assert response.location == 'http://example.com/abc'
        assert page.data == 'Hello yo!'

Functional tests

We'll test the whole application, covering security aspects that are not tested in the unit and integration tests, like logging in, logging out, checking that the basic user cannot edit pages that it didn't create but the editor user can, and so on.

Update tests/test_functional.py such that it appears as follows:

import pytest
import transaction

from tutorial import models


basic_login = dict(login='basic', password='basic')
editor_login = dict(login='editor', password='editor')

@pytest.fixture(scope='session', autouse=True)
def dummy_data(app):
    """
    Add some dummy data to the database.

    Note that this is a session fixture that commits data to the database.
    Think about it similarly to running the ``initialize_db`` script at the
    start of the test suite.

    This data should not conflict with any other data added throughout the
    test suite or there will be issues - so be careful with this pattern!

    """
    tm = transaction.TransactionManager(explicit=True)
    with tm:
        dbsession = models.get_tm_session(app.registry['dbsession_factory'], tm)
        editor = models.User(name='editor', role='editor')
        editor.set_password('editor')
        basic = models.User(name='basic', role='basic')
        basic.set_password('basic')
        page1 = models.Page(name='FrontPage', data='This is the front page')
        page1.creator = editor
        page2 = models.Page(name='BackPage', data='This is the back page')
        page2.creator = basic
        dbsession.add_all([basic, editor, page1, page2])

def test_root(testapp):
    res = testapp.get('/', status=303)
    assert res.location == 'http://example.com/FrontPage'

def test_FrontPage(testapp):
    res = testapp.get('/FrontPage', status=200)
    assert b'FrontPage' in res.body

def test_missing_page(testapp):
    res = testapp.get('/SomePage', status=404)
    assert b'404' in res.body

def test_successful_log_in(testapp):
    params = dict(
        **basic_login,
        csrf_token=testapp.get_csrf_token(),
    )
    res = testapp.post('/login', params, status=303)
    assert res.location == 'http://example.com/'

def test_successful_log_with_next(testapp):
    params = dict(
        **basic_login,
        next='WikiPage',
        csrf_token=testapp.get_csrf_token(),
    )
    res = testapp.post('/login', params, status=303)
    assert res.location == 'http://example.com/WikiPage'

def test_failed_log_in(testapp):
    params = dict(
        login='basic',
        password='incorrect',
        csrf_token=testapp.get_csrf_token(),
    )
    res = testapp.post('/login', params, status=400)
    assert b'login' in res.body

def test_logout_link_present_when_logged_in(testapp):
    testapp.login(basic_login)
    res = testapp.get('/FrontPage', status=200)
    assert b'Logout' in res.body

def test_logout_link_not_present_after_logged_out(testapp):
    testapp.login(basic_login)
    testapp.get('/FrontPage', status=200)
    params = dict(csrf_token=testapp.get_csrf_token())
    res = testapp.post('/logout', params, status=303)
    assert b'Logout' not in res.body

def test_anonymous_user_cannot_edit(testapp):
    res = testapp.get('/FrontPage/edit_page', status=303).follow()
    assert b'Login' in res.body

def test_anonymous_user_cannot_add(testapp):
    res = testapp.get('/add_page/NewPage', status=303).follow()
    assert b'Login' in res.body

def test_basic_user_cannot_edit_front(testapp):
    testapp.login(basic_login)
    res = testapp.get('/FrontPage/edit_page', status=403)
    assert b'403' in res.body

def test_basic_user_can_edit_back(testapp):
    testapp.login(basic_login)
    res = testapp.get('/BackPage/edit_page', status=200)
    assert b'Editing' in res.body

def test_basic_user_can_add(testapp):
    testapp.login(basic_login)
    res = testapp.get('/add_page/NewPage', status=200)
    assert b'Editing' in res.body

def test_editors_member_user_can_edit(testapp):
    testapp.login(editor_login)
    res = testapp.get('/FrontPage/edit_page', status=200)
    assert b'Editing' in res.body

def test_editors_member_user_can_add(testapp):
    testapp.login(editor_login)
    res = testapp.get('/add_page/NewPage', status=200)
    assert b'Editing' in res.body

def test_editors_member_user_can_view(testapp):
    testapp.login(editor_login)
    res = testapp.get('/FrontPage', status=200)
    assert b'FrontPage' in res.body

def test_redirect_to_edit_for_existing_page(testapp):
    testapp.login(editor_login)
    res = testapp.get('/add_page/FrontPage', status=303)
    assert b'FrontPage' in res.body

Running the tests

On Unix:

$VENV/bin/pytest -q

On Windows:

%VENV%\Scripts\pytest -q

The expected result should look like the following:

...........................                                         [100%]
27 passed in 6.91s