Adding Tests

We will now add tests for the models and views as well as a few functional tests in a new tests package. Tests ensure that an application works, and that it continues to work when changes are made in the future.

Test harness

The project came bootstrapped with some tests and a basic harness. These are located in the tests package at the top-level of the project. It is a common practice to put tests into a tests package alongside the application package, especially as projects grow in size and complexity. A useful convention is for each module in the application to contain a corresponding module in the tests package. The test module would have the same name with the prefix test_.

The harness consists of the following setup:

  • The project.optional-dependencies stanza of pyproject.toml - controls the dependencies installed when testing. When the list is changed, it is necessary to re-run $VENV/bin/pip install -e ".[testing]" to ensure the new dependencies are installed.

  • The tool.pytest.ini_options stanza of pyproject.toml controls basic pytest configuration, including where to find the tests. We have configured pytest to search for tests in the application package and in the tests package.

  • The tool.coverage.run stanza of pyproject.toml controls coverage config. In our setup, it works with the pytest-cov plugin that we use via the --cov options to the pytest command.

  • The testing.ini file is a mirror of development.ini and production.ini that contains settings used for executing the test suite. Most importantly, it contains the database connection information used by tests that require the database.

  • The tests/conftest.py file defines the core fixtures available throughout our tests. The fixtures are explained in more detail in the following sections. Open tests/conftest.py and follow along.

Session-scoped test fixtures

  • app_settings - the settings dict parsed from the testing.ini file that would normally be passed by pserve into your app's main function.

  • dbengine - initializes the database. It's important to start each run of the test suite from a known state, and this fixture is responsible for preparing the database appropriately. This includes deleting any existing tables, running migrations, and potentially even loading some fixture data into the tables for use within the tests.

  • app - the Pyramid WSGI application, implementing the pyramid.interfaces.IRouter interface. Most commonly this would be used for functional tests.

Per-test fixtures

  • tm - a transaction.TransactionManager object controlling a transaction lifecycle. Generally other fixtures would join to the tm fixture to control their lifecycle and ensure they are aborted at the end of the test.

  • dbsession - a sqlalchemy.orm.session.Session object connected to the database. The session is scoped to the tm fixture. Any changes made will be aborted at the end of the test.

  • testapp - a webtest.TestApp instance wrapping the app and is used to sending requests into the application and return full response objects that can be inspected. The testapp is able to mutate the request environ such that the dbsession and tm fixtures are injected and used by any code that's touching request.dbsession and request.tm. The testapp maintains a cookiejar, so it can be used to share state across requests, as well as the transaction database connection.

  • app_request - a pyramid.request.Request object that can be used for more lightweight tests versus the full testapp. The app_request can be passed to view functions and other code that need a fully functional request object.

  • dummy_request - a pyramid.testing.DummyRequest object that is very lightweight. This is a great object to pass to view functions that have minimal side-effects as it'll be fast and simple.

  • dummy_config — a pyramid.config.Configurator object used as configuration by dummy_request. Useful for mocking configuration like routes and security policies.

Modifying the fixtures

We're going to make a few application-specific changes to the test harness. It's always good to come up with patterns for things that are done often to avoid lots of boilerplate.

  • Initialize the cookiejar with a CSRF token. Remember our application is using pyramid.csrf.CookieCSRFStoragePolicy.

  • testapp.get_csrf_token() - every POST/PUT/DELETE/PATCH request must contain the current CSRF token to prove to our app that the client isn't a third-party. So we want an easy way to grab the current CSRF token and add it to the request.

  • testapp.login(params) - many pages are only accessible by logged in users so we want a simple way to login a user at the start of a test.

Update tests/conftest.py to look like the following, adding the highlighted lines:

  1import alembic
  2import alembic.config
  3import alembic.command
  4import os
  5from pyramid.paster import get_appsettings
  6from pyramid.scripting import prepare
  7from pyramid.testing import DummyRequest, testConfig
  8import pytest
  9import transaction
 10from webob.cookies import Cookie
 11import webtest
 12
 13from tutorial import main
 14from tutorial import models
 15from tutorial.models.meta import Base
 16
 17
 18def pytest_addoption(parser):
 19    parser.addoption('--ini', action='store', metavar='INI_FILE')
 20
 21@pytest.fixture(scope='session')
 22def ini_file(request):
 23    # potentially grab this path from a pytest option
 24    return os.path.abspath(request.config.option.ini or 'testing.ini')
 25
 26@pytest.fixture(scope='session')
 27def app_settings(ini_file):
 28    return get_appsettings(ini_file)
 29
 30@pytest.fixture(scope='session')
 31def dbengine(app_settings, ini_file):
 32    engine = models.get_engine(app_settings)
 33
 34    alembic_cfg = alembic.config.Config(ini_file)
 35    Base.metadata.drop_all(bind=engine)
 36    alembic.command.stamp(alembic_cfg, None, purge=True)
 37
 38    # run migrations to initialize the database
 39    # depending on how we want to initialize the database from scratch
 40    # we could alternatively call:
 41    # Base.metadata.create_all(bind=engine)
 42    # alembic.command.stamp(alembic_cfg, "head")
 43    alembic.command.upgrade(alembic_cfg, "head")
 44
 45    yield engine
 46
 47    Base.metadata.drop_all(bind=engine)
 48    alembic.command.stamp(alembic_cfg, None, purge=True)
 49
 50@pytest.fixture(scope='session')
 51def app(app_settings, dbengine):
 52    return main({}, dbengine=dbengine, **app_settings)
 53
 54@pytest.fixture
 55def tm():
 56    tm = transaction.TransactionManager(explicit=True)
 57    tm.begin()
 58    tm.doom()
 59
 60    yield tm
 61
 62    tm.abort()
 63
 64@pytest.fixture
 65def dbsession(app, tm):
 66    session_factory = app.registry['dbsession_factory']
 67    return models.get_tm_session(session_factory, tm)
 68
 69class TestApp(webtest.TestApp):
 70    def get_cookie(self, name, default=None):
 71        # webtest currently doesn't expose the unescaped cookie values
 72        # so we're using webob to parse them for us
 73        # see https://github.com/Pylons/webtest/issues/171
 74        cookie = Cookie(' '.join(
 75            '%s=%s' % (c.name, c.value)
 76            for c in self.cookiejar
 77            if c.name == name
 78        ))
 79        return next(
 80            (m.value.decode('latin-1') for m in cookie.values()),
 81            default,
 82        )
 83
 84    def get_csrf_token(self):
 85        """
 86        Convenience method to get the current CSRF token.
 87
 88        This value must be passed to POST/PUT/DELETE requests in either the
 89        "X-CSRF-Token" header or the "csrf_token" form value.
 90
 91        testapp.post(..., headers={'X-CSRF-Token': testapp.get_csrf_token()})
 92
 93        or
 94
 95        testapp.post(..., {'csrf_token': testapp.get_csrf_token()})
 96
 97        """
 98        return self.get_cookie('csrf_token')
 99
100    def login(self, params, status=303, **kw):
101        """ Convenience method to login the client."""
102        body = dict(csrf_token=self.get_csrf_token())
103        body.update(params)
104        return self.post('/login', body, **kw)
105
106@pytest.fixture
107def testapp(app, tm, dbsession):
108    # override request.dbsession and request.tm with our own
109    # externally-controlled values that are shared across requests but aborted
110    # at the end
111    testapp = TestApp(app, extra_environ={
112        'HTTP_HOST': 'example.com',
113        'tm.active': True,
114        'tm.manager': tm,
115        'app.dbsession': dbsession,
116    })
117
118    # initialize a csrf token instead of running an initial request to get one
119    # from the actual app - this only works using the CookieCSRFStoragePolicy
120    testapp.set_cookie('csrf_token', 'dummy_csrf_token')
121
122    return testapp
123
124@pytest.fixture
125def app_request(app, tm, dbsession):
126    """
127    A real request.
128
129    This request is almost identical to a real request but it has some
130    drawbacks in tests as it's harder to mock data and is heavier.
131
132    """
133    with prepare(registry=app.registry) as env:
134        request = env['request']
135        request.host = 'example.com'
136
137        # without this, request.dbsession will be joined to the same transaction
138        # manager but it will be using a different sqlalchemy.orm.Session using
139        # a separate database transaction
140        request.dbsession = dbsession
141        request.tm = tm
142
143        yield request
144
145@pytest.fixture
146def dummy_request(tm, dbsession):
147    """
148    A lightweight dummy request.
149
150    This request is ultra-lightweight and should be used only when the request
151    itself is not a large focus in the call-stack.  It is much easier to mock
152    and control side-effects using this object, however:
153
154    - It does not have request extensions applied.
155    - Threadlocals are not properly pushed.
156
157    """
158    request = DummyRequest()
159    request.host = 'example.com'
160    request.dbsession = dbsession
161    request.tm = tm
162
163    return request
164
165@pytest.fixture
166def dummy_config(dummy_request):
167    """
168    A dummy :class:`pyramid.config.Configurator` object.  This allows for
169    mock configuration, including configuration for ``dummy_request``, as well
170    as pushing the appropriate threadlocals.
171
172    """
173    with testConfig(request=dummy_request) as config:
174        yield config

Unit tests

We can test individual APIs within our codebase to ensure they fulfill the expected contract that the rest of the application expects. For example, we'll test the password hashing features we added to the tutorial.models.User object.

Create tests/test_user_model.py such that it appears as follows:

 1from tutorial import models
 2
 3
 4def test_password_hash_saved():
 5    user = models.User(name='foo', role='bar')
 6    assert user.password_hash is None
 7
 8    user.set_password('secret')
 9    assert user.password_hash is not None
10
11def test_password_hash_not_set():
12    user = models.User(name='foo', role='bar')
13    assert not user.check_password('secret')
14
15def test_correct_password():
16    user = models.User(name='foo', role='bar')
17    user.set_password('secret')
18    assert user.check_password('secret')
19
20def test_incorrect_password():
21    user = models.User(name='foo', role='bar')
22    user.set_password('secret')
23    assert not user.check_password('incorrect')

Integration tests

We can directly execute the view code, bypassing Pyramid and testing just the code that we've written. These tests use dummy requests that we'll prepare appropriately to set the conditions each view expects, such as adding dummy data to the session. We'll be using dummy_config to configure the necessary routes, as well as setting the security policy as pyramid.testing.DummySecurityPolicy to mock dummy_request.identity.

Update tests/test_views.py such that it appears as follows:

  1from pyramid.testing import DummySecurityPolicy
  2import sqlalchemy as sa
  3
  4from tutorial import models
  5
  6
  7def makeUser(name, role):
  8    return models.User(name=name, role=role)
  9
 10
 11def setUser(config, user):
 12    config.set_security_policy(
 13        DummySecurityPolicy(identity=user)
 14    )
 15
 16def makePage(name, data, creator):
 17    return models.Page(name=name, data=data, creator=creator)
 18
 19class Test_view_wiki:
 20    def _callFUT(self, request):
 21        from tutorial.views.default import view_wiki
 22        return view_wiki(request)
 23
 24    def _addRoutes(self, config):
 25        config.add_route('view_page', '/{pagename}')
 26
 27    def test_it(self, dummy_config, dummy_request):
 28        self._addRoutes(dummy_config)
 29        response = self._callFUT(dummy_request)
 30        assert response.location == 'http://example.com/FrontPage'
 31
 32class Test_view_page:
 33    def _callFUT(self, request):
 34        from tutorial.views.default import view_page
 35        return view_page(request)
 36
 37    def _makeContext(self, page):
 38        from tutorial.routes import PageResource
 39        return PageResource(page)
 40
 41    def _addRoutes(self, config):
 42        config.add_route('edit_page', '/{pagename}/edit_page')
 43        config.add_route('add_page', '/add_page/{pagename}')
 44        config.add_route('view_page', '/{pagename}')
 45
 46    def test_it(self, dummy_config, dummy_request, dbsession):
 47        # add a page to the db
 48        user = makeUser('foo', 'editor')
 49        page = makePage('IDoExist', 'Hello CruelWorld IDoExist', user)
 50        dbsession.add_all([page, user])
 51
 52        # create a request asking for the page we've created
 53        self._addRoutes(dummy_config)
 54        dummy_request.context = self._makeContext(page)
 55
 56        # call the view we're testing and check its behavior
 57        info = self._callFUT(dummy_request)
 58        assert info['page'] is page
 59        assert info['content'] == (
 60            '<div class="document">\n'
 61            '<p>Hello <a href="http://example.com/add_page/CruelWorld">'
 62            'CruelWorld</a> '
 63            '<a href="http://example.com/IDoExist">'
 64            'IDoExist</a>'
 65            '</p>\n</div>\n'
 66        )
 67        assert info['edit_url'] == 'http://example.com/IDoExist/edit_page'
 68
 69class Test_add_page:
 70    def _callFUT(self, request):
 71        from tutorial.views.default import add_page
 72        return add_page(request)
 73
 74    def _makeContext(self, pagename):
 75        from tutorial.routes import NewPage
 76        return NewPage(pagename)
 77
 78    def _addRoutes(self, config):
 79        config.add_route('add_page', '/add_page/{pagename}')
 80        config.add_route('view_page', '/{pagename}')
 81
 82    def test_get(self, dummy_config, dummy_request, dbsession):
 83        setUser(dummy_config, makeUser('foo', 'editor'))
 84        self._addRoutes(dummy_config)
 85        dummy_request.context = self._makeContext('AnotherPage')
 86        info = self._callFUT(dummy_request)
 87        assert info['pagedata'] == ''
 88        assert info['save_url'] == 'http://example.com/add_page/AnotherPage'
 89
 90    def test_submit_works(self, dummy_config, dummy_request, dbsession):
 91        dummy_request.method = 'POST'
 92        dummy_request.POST['body'] = 'Hello yo!'
 93        dummy_request.context = self._makeContext('AnotherPage')
 94        setUser(dummy_config, makeUser('foo', 'editor'))
 95        self._addRoutes(dummy_config)
 96        self._callFUT(dummy_request)
 97        page = dbsession.scalars(
 98            sa.select(models.Page).where(models.Page.name == 'AnotherPage')
 99        ).one()
100        assert page.data == 'Hello yo!'
101
102class Test_edit_page:
103    def _callFUT(self, request):
104        from tutorial.views.default import edit_page
105        return edit_page(request)
106
107    def _makeContext(self, page):
108        from tutorial.routes import PageResource
109        return PageResource(page)
110
111    def _addRoutes(self, config):
112        config.add_route('edit_page', '/{pagename}/edit_page')
113        config.add_route('view_page', '/{pagename}')
114
115    def test_get(self, dummy_config, dummy_request, dbsession):
116        user = makeUser('foo', 'editor')
117        page = makePage('abc', 'hello', user)
118        dbsession.add_all([page, user])
119
120        self._addRoutes(dummy_config)
121        dummy_request.context = self._makeContext(page)
122        info = self._callFUT(dummy_request)
123        assert info['pagename'] == 'abc'
124        assert info['save_url'] == 'http://example.com/abc/edit_page'
125
126    def test_submit_works(self, dummy_config, dummy_request, dbsession):
127        user = makeUser('foo', 'editor')
128        page = makePage('abc', 'hello', user)
129        dbsession.add_all([page, user])
130
131        self._addRoutes(dummy_config)
132        dummy_request.method = 'POST'
133        dummy_request.POST['body'] = 'Hello yo!'
134        setUser(dummy_config, user)
135        dummy_request.context = self._makeContext(page)
136        response = self._callFUT(dummy_request)
137        assert response.location == 'http://example.com/abc'
138        assert page.data == 'Hello yo!'

Functional tests

We'll test the whole application, covering security aspects that are not tested in the unit and integration tests, like logging in, logging out, checking that the basic user cannot edit pages that it didn't create but the editor user can, and so on.

Update tests/test_functional.py such that it appears as follows:

  1import pytest
  2import transaction
  3
  4from tutorial import models
  5
  6
  7basic_login = dict(login='basic', password='basic')
  8editor_login = dict(login='editor', password='editor')
  9
 10@pytest.fixture(scope='session', autouse=True)
 11def dummy_data(app):
 12    """
 13    Add some dummy data to the database.
 14
 15    Note that this is a session fixture that commits data to the database.
 16    Think about it similarly to running the ``initialize_db`` script at the
 17    start of the test suite.
 18
 19    This data should not conflict with any other data added throughout the
 20    test suite or there will be issues - so be careful with this pattern!
 21
 22    """
 23    tm = transaction.TransactionManager(explicit=True)
 24    with tm:
 25        dbsession = models.get_tm_session(app.registry['dbsession_factory'], tm)
 26        editor = models.User(name='editor', role='editor')
 27        editor.set_password('editor')
 28        basic = models.User(name='basic', role='basic')
 29        basic.set_password('basic')
 30        page1 = models.Page(name='FrontPage', data='This is the front page')
 31        page1.creator = editor
 32        page2 = models.Page(name='BackPage', data='This is the back page')
 33        page2.creator = basic
 34        dbsession.add_all([basic, editor, page1, page2])
 35
 36def test_root(testapp):
 37    res = testapp.get('/', status=303)
 38    assert res.location == 'http://example.com/FrontPage'
 39
 40def test_FrontPage(testapp):
 41    res = testapp.get('/FrontPage', status=200)
 42    assert b'FrontPage' in res.body
 43
 44def test_missing_page(testapp):
 45    res = testapp.get('/SomePage', status=404)
 46    assert b'404' in res.body
 47
 48def test_successful_log_in(testapp):
 49    params = dict(
 50        **basic_login,
 51        csrf_token=testapp.get_csrf_token(),
 52    )
 53    res = testapp.post('/login', params, status=303)
 54    assert res.location == 'http://example.com/'
 55
 56def test_successful_log_with_next(testapp):
 57    params = dict(
 58        **basic_login,
 59        next='WikiPage',
 60        csrf_token=testapp.get_csrf_token(),
 61    )
 62    res = testapp.post('/login', params, status=303)
 63    assert res.location == 'http://example.com/WikiPage'
 64
 65def test_failed_log_in(testapp):
 66    params = dict(
 67        login='basic',
 68        password='incorrect',
 69        csrf_token=testapp.get_csrf_token(),
 70    )
 71    res = testapp.post('/login', params, status=400)
 72    assert b'login' in res.body
 73
 74def test_logout_link_present_when_logged_in(testapp):
 75    testapp.login(basic_login)
 76    res = testapp.get('/FrontPage', status=200)
 77    assert b'Logout' in res.body
 78
 79def test_logout_link_not_present_after_logged_out(testapp):
 80    testapp.login(basic_login)
 81    testapp.get('/FrontPage', status=200)
 82    params = dict(csrf_token=testapp.get_csrf_token())
 83    res = testapp.post('/logout', params, status=303)
 84    assert b'Logout' not in res.body
 85
 86def test_anonymous_user_cannot_edit(testapp):
 87    res = testapp.get('/FrontPage/edit_page', status=303).follow()
 88    assert b'Login' in res.body
 89
 90def test_anonymous_user_cannot_add(testapp):
 91    res = testapp.get('/add_page/NewPage', status=303).follow()
 92    assert b'Login' in res.body
 93
 94def test_basic_user_cannot_edit_front(testapp):
 95    testapp.login(basic_login)
 96    res = testapp.get('/FrontPage/edit_page', status=403)
 97    assert b'403' in res.body
 98
 99def test_basic_user_can_edit_back(testapp):
100    testapp.login(basic_login)
101    res = testapp.get('/BackPage/edit_page', status=200)
102    assert b'Editing' in res.body
103
104def test_basic_user_can_add(testapp):
105    testapp.login(basic_login)
106    res = testapp.get('/add_page/NewPage', status=200)
107    assert b'Editing' in res.body
108
109def test_editors_member_user_can_edit(testapp):
110    testapp.login(editor_login)
111    res = testapp.get('/FrontPage/edit_page', status=200)
112    assert b'Editing' in res.body
113
114def test_editors_member_user_can_add(testapp):
115    testapp.login(editor_login)
116    res = testapp.get('/add_page/NewPage', status=200)
117    assert b'Editing' in res.body
118
119def test_editors_member_user_can_view(testapp):
120    testapp.login(editor_login)
121    res = testapp.get('/FrontPage', status=200)
122    assert b'FrontPage' in res.body
123
124def test_redirect_to_edit_for_existing_page(testapp):
125    testapp.login(editor_login)
126    res = testapp.get('/add_page/FrontPage', status=303)
127    assert b'FrontPage' in res.body

Running the tests

On Unix:

$VENV/bin/pytest -q

On Windows:

%VENV%\Scripts\pytest -q

The expected result should look like the following:

...........................                                         [100%]
27 passed in 6.91s