Substance D¶

Demonstration application¶

See the application running at http://demo.substanced.net for a demonstration of the Substance D management interface.

You can deploy the demonstration application locally by performing the following steps.

1. Create a new directory somewhere and cd to it:

   $ virtualenv -p python2.7 hack-on-substanced
   $ cd hack-on-substanced
   

2. Install Substance D either from PyPI or from a git checkout:

   $ bin/pip install substanced
   

   OR:

   $ bin/pip install -e git+https://github.com/Pylons/substanced#egg=substanced
   

   Alternatively create a writeable fork on GitHub and check that out.

3. Check that the python-magic library has been installed:

   $ bin/python -c "from substanced.file import magic; assert magic is not None, 'python-magic not installed'"
   

   If you then see “python-magic not installed” then you will need to take additional steps to install the python-magic library. See Installing python-magic.

4. Move back to the parent directory:

   $ cd ..
   

5. Now you should be able to create new Substance D projects by using pcreate. The following pcreate command uses the scaffold substanced to create a new project named myproj:

   $ hack-on-substanced/bin/pcreate -s substanced myproj
   

6. Now you can make a virtual environment for your project and move into it:

   $ virtualenv -p python2.7 myproj
   $ cd myproj
   

7. Install that project using pip install -e into the virtual environment:

   $ bin/pip install -e .
   

8. Run the resulting project via bin/pserve development.ini. The development server listens to requests sent to http://0.0.0.0:6543 by default. Open this URL in a web browser.

9. The initial Administrator password is randomly generated automatically. Use the following command to find the login information:

   $ grep initial_password *.ini
   development.ini:substanced.initial_password = hNyrGI5nnl
   production.ini:substanced.initial_password = hNyrGI5nnl
   

Create a project from a scaffold in Substance D¶

After creating a development checkout, you can create a new project from the default substanced scaffold by using pcreate.

$ cd ../env
$ bin/pcreate -s substanced myproj

Then install that project using pip install -e . into the virtual environment.

$ cd myproj
$ ../bin/pip install -e .

Run the resulting project.

$ ../bin/pserve development.ini

Then start hacking on your new project.

Hacking on Substance D¶

See Hacking on Substance D, or look in your checked out local git repository for HACKING.txt, for information and guidelines to develop your application, including testing and internationalization.

Benefits and features¶

• Create, Read, Update, Delete (CRUD) operations on content resources
• Extensible actions for each content type via management views
• Built-in support for hierarchies with security
• Already-done UIs for all supported features (e.g., references, principals)
• Undo facility to back out of the last transaction
• Copy and paste

Background and motivation¶

In prehistoric times there was a Python-based application server, derived from a commercial predecessor released in 1996. Zope and its predecessor had a unique “through-the-web” (TTW) UI for interacting with the system. This UI, called the “Zope Management Interface” (ZMI), had a number of capabilities for a number of audiences. Plone, built on Zope, extended this idea. Other systems, such as Django, have found that providing an out-of-the-box (OOTB) starting point with attractive pixels on the screen can be a key selling point.

Substance D taps into this. In particular, lessons learned from our long experience in this area are applied to the SDI:

• Attractive, official, supported OOTB management UI
• Be successful by being very clear what the SDI isn’t

What is and isn’t the SDI¶

The SDI is for:

• Developers to use while building their application
• Administrators to use after deployment, to manage certain Substance D or application settings provided by the developer
• Certain power users to use as a behind-the-scenes UI

The SDI is not for:

• The retail UI for your actual application. Unlike Plone, we don’t expect developers to squeeze their UX expectations into an existing UX
• Overridable, customizable, replaceable, frameworky new expectations

The SDI does have a short list of clearly-defined places for developers to plug in parts of their application. As a prime example, you can define a Management View that gets added as a new tab on a resource.

The SDI is extensible and allows you to plug your own views into it, so you can present nontechnical users with a way to manage arbitrary kinds of custom content.

Once again, for clarity: the SDI is not a framework, it is an application. It is not your retail UI.

Layout¶

The SDI has a mostly-familiar layout:

• A header that shows the username as a dropdown menu containing a link to the principal screen as well as a logout link
• Breadcrumbs with a path from the root
• A series of tabs for the management views of the current resource
• Optionally, a flash message showing results of the previous operation, a warning, or some other notice
• A footer

Folder contents¶

Folders show a listing of items they contain using a powerful data grid based on SlickGrid:

This dynamic grid features:

• Only loading the items needed for display, to speed up operations on large folders
• “Infinite scrolling” via the scrollbar to go directly to a batch at any point in the folder
• Column resizing and re-ordering
• Sorting on sort-supported columns
• Filtering based on search string
• Selection of one or more items and performing an operation by clicking on a button
• Styling integrated with Twitter Bootstrap
• Detection and re-layout on responsive design operations

The Configuring Folder Contents chapter covers how Substance D developers can plug their custom content types into folder contents.

Undo¶

In Substance D, many transactions can be undone and redone after a commit. This “Undo” ability is one of the key features that people notice immediately and it has real, deep value to a developer’s customers.

Many of the built-in operations display an undo button. For example, if we delete an item from a folder, we get a “flash” message telling us the deletion was performed, but with a button allowing us to undo it if that was a mistake:

Clicking “undo” restores the deleted item, with a flash message offering to redo the undo:

Undo button support is enabled by the developer in their management views that commit data. It isn’t available on every kind of change. Instead developers need to wrap their commit with certain information used by the SDI’s undo features.

All actions that change data (even ones without undo button support) can be undone. These screenshots show an Undo tab on the site’s root folder. This provides a global way to see recent transactions and perform an undo:

Sometimes a particular transaction cannot be undone without undoing an earlier transaction. For example, if you make three changes to a resource, the first two can’t be undone without first undoing the last, as the resource will have been changed by a later transaction.

Catalog¶

With cataloging developers have a powerful indexing and searching facility that can be added to their application. Like other first-class parts of Substance D’s machinery, it includes an SDI UI for interacting with the catalog:

Catalogues are content, meaning they show up as folder items in the SDI. You can visit a catalog and update its indexes, or see some statistics for those indexes. You can also use the SDI to reindex the contents of an index, if you suspect it has gotten out of sync with the content.

The catalog also registers a management view on content resources, which gain an Indexing tab:

This shows some statistics and allows an SDI user to reindex an individual resource.

Principals¶

Managing users and groups, a.k.a., principals, is more interesting in a system like Substance D with rich hierarchies. You can add a folder of principals to any folder or other kind of container that allows adding principals:

A principals folder allows you to manage (e.g., add, edit, delete, or rename) users and groups via the SDI, as well as password resets. Since users and groups are content, you gain some of the other SDI tabs for managing them (e.g., Security, References):

Users and groups can also grow extra attributes and behavior because they’re just content, so you can customize your user model out of the box.

Workflows¶

The workflows service provides a powerful system for managing states and transitions. This service shows up in the SDI as a tab on content types that have workflows registered for them:

This provides a way, via the SDI, to transition the workflow state of a resource.

References¶

With the built-in support for references, Substance D helps manage relationships between resources. The SDI provides a UI into the reference service.

If the resource you are viewing has any references, a References tab will appear:

In this example, mydoc1 is a target of an ACL reference from the admin1 user.

An integrity error can occur if you try to delete a source or target of a reference that claims to be “integral”. The SDI will then show this with an explanation:

Manage Database¶

The object database inside Substance D has some management knobs that can be adjusted via the SDI:

This tab appears on the root object of the site and lets you:

• Pack the old revisions of objects in the database.
• Inspect and run evolution steps.
• Flush the object cache.
• See details and statistics about the database, the connection, and activity.

Implementation Notes¶

While it doesn’t matter for developers of Substance D applications, some notes are below, regarding how the SDI is implemented:

• We use a high-performance, modern, responsive UI based on Twitter Bootstrap
• We use the upstream LESS variables from Bootstrap in a LESS file for parts of the SDI.
• Our grid is based on SlickGrid.

Registering Content¶

In order to add new content to the system, you need to associate a resource factory with a content type. A resource factory that generates content must have these properties:

• It must be a class, or a factory function that returns an instance of a resource class.
• Instances of the resource class must be persistent (it must derive from the persistent.Persistent class or a class that derives from Persistent such as substanced.folder.Folder).
• The resource class or factory must be decorated with the @content decorator, or must be added at configuration time via config.add_content_type.
• It must have a type. A type acts as a globally unique categorization marker, and allows the content to be constructed, enumerated, and introspected by various Substance D UI elements such as “add forms”, and queries by the management interface for the icon class of a resource. A type can be any hashable Python object, but it’s most often a string.

Here’s an example which defines a content resource factory as a class:

# in a module named blog.resources

from persistent import Persistent
from substanced.content import content

@content('Blog Entry')
class BlogEntry(Persistent):
    def __init__(self, title='', body=''):
        self.title = title
        self.body = body

Here’s an example of defining a content resource factory using a function instead:

# in a module named blog.resources

from persistent import Persistent
from substanced.content import content

class BlogEntry(Persistent):
    def __init__(self, title, body):
        self.title = title
        self.body = body

@content('Blog Entry')
def make_blog_entry(title='', body=''):
    return BlogEntry(title, body)

When a resource factory is not a class, Substance D will wrap the resource factory in something that changes the resource object returned from the factory. In the above case, the BlogEntry instance returned from make_blog_entry will be changed; its __factory_type__ attribute will be mutated.

Notice that when we decorate a resource factory class with @content, and the class’ __init__ function takes arguments, we provide those arguments with default values. This is mandatory if you’d like your content objects to participate in a “dump”. Dumping a resource requires that the resource be creatable without any mandatory arguments. It’s a similar story if our factory is a function; the function decorated by the @content decorator should provide defaults to any argument. In general, a resource factory can take arguments, but each parameter of the factory’s callable should be given a default value. This also means that all arguments to a resource factory should be keyword arguments, and not positional arguments.

In order to activate a @content decorator, it must be scanned using the Pyramid config.scan() machinery:

# in a module named blog.__init__

from pyramid.config import Configurator

def main(global_config, **settings):
    config = Configurator()
    config.include('substanced')
    config.scan('blog.resources')
    # .. and so on ...

Instead of using the @content decorator, you can alternately add a content resource imperatively at configuration time using the add_content_type method of the Configurator:

# in a module named blog.__init__

from pyramid.config import Configurator
from .resources import BlogEntry

def main(global_config, **settings):
    config = Configurator()
    config.include('substanced')
    config.add_content_type('Blog Entry', BlogEntry)

This does the same thing as using the @content decorator, but you don’t need to scan() your resources if you use add_content_type instead of the @content decorator.

Once a content type has been defined (and scanned, if it’s been defined using a decorator), an instance of the resource can be constructed from within a view that lives in your application:

# in a module named blog.views

from pyramid.httpexceptions import HTTPFound
from pyramid.view import (
    view_config,
    view_defaults,
    )

@view_config(name='add_blog_entry', request_method='POST')
def add_blogentry(context, request):
    title = request.POST['title']
    body = request.POST['body']
    entry = request.registry.content.create('Blog Entry', title, body)
    context[title] = entry
    return HTTPFound(request.resource_url(entry))

The arguments passed to request.registry.content.create must start with the content type, and must be followed with whatever arguments are required by the resource factory.

Creating an instance of content this way isn’t particularly more useful than creating an instance of the resource object by calling its class __init__ directly unless you’re building a highly abstract system. But even if you’re not building a very abstract system, types can be very useful. For instance, types can be enumerated:

# in a module named blog.views

@view_config(name='show_types', renderer='show_types.pt')
def show_types(request):
    all_types = request.registry.content.all()
    return {'all_types':all_types}

request.registry.content.all() will return all the types you’ve defined and scanned.

Metadata¶

A content’s type can be associated with metadata about that type, including the content type’s name, its icon in the SDI management interface, an add view name, and other things. Pass arbitrary keyword arguments to the @content decorator or config.add_content_type to specify metadata.

# in a module named blog.resources

from persistent import Persistent
from substanced.content import content

@content('Blog Entry', name='Cool Blog Entry')
class BlogEntry(Persistent):
    def __init__(self, title='', body=''):
        self.title = title
        self.body = body

# in a module named blog.resources

from persistent import Persistent
from substanced.content import content

@content('Blog Entry', icon='glyphicon glyphicon-file')
class BlogEntry(Persistent):
    def __init__(self, title='', body=''):
        self.title = title
        self.body = body

from persistent import Persistent
from substanced.content import content

def blogentry_icon(context, request):
    if context.body:
        return 'glyphicon glyphicon-file'
    else:
        return 'glyphicon glyphicon-gift'

@content('Blog Entry', icon=blogentry_icon)
class BlogEntry(Persistent):
    def __init__(self, title='', body=''):
        self.title = title
        self.body = body

# in a module named blog.resources

from persistent import Persistent
from substanced.content import content

@content('Blog Entry', add_view='add_blog_entry')
class BlogEntry(Persistent):
    def __init__(self, title='', body=''):
        self.title = title
        self.body = body

from persistent import Persistent
from substanced.content import content
from substanced.folder import Folder

def add_blog_entry(context, request):
    if request.registry.content.istype(context, 'Blog'):
        return 'add_blog_entry'

@content('Blog')
class Blog(Folder):
    pass

@content('Blog Entry', add_view=add_blog_entry)
class BlogEntry(Persistent):
    def __init__(self, title='', body=''):
        self.title = title
        self.body = body

request.registry.content.metadata(blogentry, 'icon')

request.registry.content.metadata(blogentry, 'icon',
                                  'glyphicon glyphicon-file')

Affecting Content Creation¶

In some cases you might want your resource to perform some actions that can only take place after it has been seated in its container, but before the creation events have fired. The @content decorator and add_content_type method both support an after_create argument, pointed at a callable.

For example:

@content(
    'Document',
    icon='glyphicon glyphicon-align-left',
    add_view='add_document',
    propertysheets = (
        ('Basic', DocumentPropertySheet),
        ),
    after_create='after_creation'
    )
class Document(Persistent):

    name = renamer()

    def __init__(self, title, body):
        self.title = title
        self.body = body

    def after_creation(self, inst, registry):
        pass

If the value provided for after_create is a string, it’s assumed to be a method of the created object. If it’s a sequence, each value should be a string or a callable, which will be called in turn. The callable(s) are passed the instance being created and the registry. Afterwards, substanced.event.ContentCreatedEvent is emitted.

Construction of the root folder in Substance D is a special case. Most Substance D applications will start with:

from substanced.db import root_factory
def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application.
    """
    config = Configurator(settings=settings, root_factory=root_factory)

The substanced.db.root_factory() callable contains the following line:

app_root = registry.content.create('Root')

In many cases you want to perform some extra work on the Root. For example, you might want to create a catalog with indexes. Substance D emits an event when the root is created, so you can subscribe to that event and perform some actions:

from substanced.root import Root
from substanced.event import subscribe_created
from substanced.catalog import Catalog

@subscribe_created(Root)
def root_created(event):
    root = event.object
    catalog = Catalog()
    catalogs = root['catalogs']
    catalogs.add_service('catalog', catalog)
    catalog.update_indexes('system', reindex=True)
    catalog.update_indexes('sdidemo', reindex=True)

Names and Renaming¶

A resource’s “name” (__name__) is important to the system in Substance D. For example, traversal uses the value in URLs and paths to walk through hierarchy. Containers need to know when a resource’s __name__ changes.

To help support this, Substance D provides substanced.util.renamer(). You use it as a class attribute wrapper on resources that want “managed” names. These resources then gain a name attribute with a getter/setter from renamer. Getting the name returns the __name__. Setting name grabs the container and calls the rename method on the folder.

For example:

class Document(Persistent):
    name = renamer()

Special Colander Support¶

Forms and schemas for resources become pretty easy in Substance D. To make it easier for forms to interact with the Substance D machinery, it includes some special Colander schema nodes you can use on your forms.

class BlogEntrySchema(Schema):
    name = NameSchemaNode()

class BlogEntrySchema(Schema):
    name = NameSchemaNode(max_len=20)

class BlogEntrySchema(Schema):
    name = NameSchemaNode(
        editing=lambda c, r: r.registry.content.istype(c, 'BlogEntry')
        )

class UserSchema(Schema):
    """ The property schema for :class:`substanced.principal.User`
    objects."""
    groupids = MultireferenceIdSchemaNode(
        choices_getter=groups_choices,
        title='Groups',
        )

Overriding Existing Content Types¶

Perhaps you would like to slightly adjust an existing content type, such as Folder, without re-implementing it. For exampler, perhaps you would like to override just the add_view and provide your own view, such as:

@mgmt_view(
    context=IFolder,
    name='my_add_folder',
    tab_condition=False,
    permission='sdi.add-content',
    renderer='substanced.sdi:templates/form.pt'
)
class MyAddFolderView(AddFolderView):

    def before(self, form):
        # Perform some custom work before validation
        pass

With this you can override any of the view predicates (such as permission) and override any part of the form handling (such as adding a before that performs some custom processing.)

To make this happen, you can re-register, so to speak, the content type during startup:

from substanced.folder import Folder
from .views import MyAddFolderView
config.add_content_type('Folder', Folder,
                        add_view='my_add_folder',
                        icon='glyphicon glyphicon-folder-close')

This, however, keeps the same content type class. You can also go further by overriding the content type definition itself:

@content(
    'Folder',
    icon='glyphicon glyphicon-folder-close',
    add_view='my_add_folder',
)
@implementer(IFolder)
class MyFolder(Folder):

    def send_email(self):
        pass

The class for the Folder content type has now been replaced. Instead of substanced.folder.Folder it is MyFolder.

Adding Automatic Naming for Content¶

On some sites you don’t want to set the name for every piece of content you create. Substance D provides support for this with a special kind of folder. You can configure your site to use the autonaming folder by overriding the standard folder:

from substanced.folder import SequentialAutoNamingFolder
from substanced.interaces import IFolder
from zope.interface import implementer

@content(
   'Folder',
    icon='glyphicon glyphicon-folder-close',
    add_view='add_folder',
)
@implementer(IFolder)
class  MyFolder(SequentialAutoNamingFolder):
    """ Override Folder content type """

The add view for Documents can then be edited to no longer require a name:

def add_success(self, appstruct):
    registry = self.request.registry
    document = registry.content.create('Document', **appstruct)
    self.context.add_next(document)
    return HTTPFound(
        self.request.sdiapi.mgmt_path(self.context, '@@contents')
    )

Affecting the Tab Order for Management Views¶

The tab_order parameter overrides the mgmt_view tab settings for a content type. Its value should be a sequence of view names, each corresponding to a tab that will appear in the management interface. Any registered view names that are omitted from this sequence will be placed after the other tabs.

Handling Content Events¶

Adding and modifying data related to content is, thanks to the framework, easy to do. Sometimes, though, you want to intervene and, for example, perform some extra work when content resources are added. Substance D has several framework events you can subscribe to using Pyramid events.

The substanced.events module imports these events as interfaces from substanced.interfaces and then provides decorator subscribers as convenience for each:

• substanced.interfaces.IObjectAdded as subscriber @subscriber_added
• substanced.interfaces.IObjectWillBeAdded as subscriber @subscriber_will_be_added
• substanced.interfaces.IObjectRemoved as subscriber @subscriber_removed
• substanced.interfaces.IObjectWillBeRemoved as subscriber @subscriber_will_be_removed
• substanced.interfaces.IObjectModified as subscriber @subscriber_modified
• substanced.interfaces.IACLModified as subscriber @subscriber_acl_modified
• substanced.interfaces.IContentCreated as subscriber @subscriber_created

As an example, the substanced.principal.subscribers.user_added() function is a subscriber to the IObjectAdded event:

@subscribe_added(IUser)
def user_added(event):
    """ Give each user permission to change their own password."""
    if event.loading: # fbo dump/load
        return
    user = event.object
    registry = event.registry
    set_acl(
        user,
        [(Allow, get_oid(user), ('sdi.view', 'sdi.change-password'))],
        registry=registry,
        )

As with the rest of Pyramid, you can do imperative configuration if you don’t like decorator-based configuration, using config.add_content_subscriber Both the declarative and imperative forms result in substanced.event.add_content_subscriber().

The IACLModified event (and @subscriber_acl_modified subscriber) is used internally by Substance D to re-index information in the system catalog’s ACL index. Substance D also uses this event to maintain references between resources and principals. Substance D applications can use this in different ways, for example recording a security audit trail on security changes.

Sometimes when you perform operations on objects you don’t want to perform the standard events. For example, in folder contents you can select a number of resources and move them to another folder. Normally this would fire content change events that re-index the files. This is fairly pointless: the content of the file hasn’t changed.

If you looked at the interface for one of the content events, you would see some extra information supported. For example, in substanced.interfaces.IObjectWillBeAdded:

class IObjectWillBeAdded(IObjectEvent):
    """ An event type sent when an before an object is added """
    object = Attribute('The object being added')
    parent = Attribute('The folder to which the object is being added')
    name = Attribute('The name which the object is being added to the folder '
                     'with')
    moving = Attribute('None or the folder from which the object being added '
                       'was moved')
    loading = Attribute('Boolean indicating that this add is part of a load '
                        '(during a dump load process)')
    duplicating = Attribute('The object being duplicated or ``None``')

moving, loading, and duplicating are flags that can be set on the event when certain actions are triggered. These help in cases such as the one above: certain subscribers might want “flavors” of standard events and, in some cases, handle the event in a different way. This helps avoid lots of special-case events or the need for a hierarchy of events.

Thus in the case above, the catalog subscriber can see that the changes triggered by the event where in the special case of “moving”. This can be seen in substanced.catalog.subscribers.object_added.

mgmt_view View Predicates¶

Since mgmt_view is an extension of Pyramid’s view_config, it re-uses the same concept of view predicates as well as some of the same actual predicates:

• request_type, request_method, request_param, containment, attr, renderer, wrapper, xhr, accept, header, path_info, context, name, custom_predicates, decorator, mapper, and http_cache are supported and behave the same.
• permission is the same but defaults to sdi.view.

The following are new view predicates introduced for mgmt_view:

• tab_title takes a string for the label placed on the tab.

• tab_condition takes a callable that returns True or False, or True or False. If you state a callable, this callable is passed context and request. The boolean determines whether the tab is listed in a certain situation.

• tab_before takes the view name of a mgmt_view that this mgmt_view should appear after (covered in detail in the next section.)

• tab_after takes the view name of a mgmt_view that this mgmt_view should appear after. Also covered below.

• tab_near takes a “sentinel” from substanced.sdi (or None) that makes a best effort at placement independent of another particular mgmt_view. Also covered below. The possible sentinel values are:

  substanced.sdi.LEFT
  substanced.sdi.MIDDLE
  substanced.sdi.RIGHT
  

Tab Ordering¶

If you register a management view, a tab will be added in the list of tabs. If no mgmt view specifies otherwise via its tab data, the tab order will use a default sorting: alphabetical order by the tab_title parameter of each tab (or the view name if no tab_title is provided.) The first tab in this tab listing acts as the “default” that is open when you visit a resource. Substance D does, though, give you some options to control tab ordering in larger systems with different software registering management views.

Perhaps a developer wants to ensure that one of her tabs appears first in the list and another appears last, no matter what other management views have been registered by Substance D or any add-on packages. @mgmt_view (or the imperative call) allow a keyword of tab_before or tab_after. Each take the string tab name of the management view to place before or after. If you don’t care (or don’t know) which view name to use as a tab_before or tab_after value, use tab_near, which can be any of the sentinel values MIDDLE, LEFT, or RIGHT, each of which specifies a target “zone” in the tab order. Substance D will make a best effort to do something sane with tab_near.

As in many cases, an illustration is helpful:

from substanced.sdi import LEFT, RIGHT

@mgmt_view(
    name='tab_1',
    tab_title='Tab 1',
    renderer='templates/tab.pt'
    )
def tab_1(context, request):
    return {}


@mgmt_view(
    name='tab_2',
    tab_title='Tab 2',
    renderer='templates/tab.pt',
    tab_before='tab_1'
    )
def tab_2(context, request):
    return {}


@mgmt_view(
    name='tab_3',
    tab_title='Tab 3',
    renderer='templates/tab.pt',
    tab_near=RIGHT
    )
def tab_3(context, request):
    return {}


@mgmt_view(
    name='tab_4',
    tab_title='Tab 4',
    renderer='templates/tab.pt',
    tab_near=LEFT
    )
def tab_4(context, request):
    return {}


@mgmt_view(
    name='tab_5',
    tab_title='Tab 5',
    renderer='templates/tab.pt',
    tab_near=LEFT
    )
def tab_5(context, request):
    return {}

This set of management views (combined with the built-in Substance D management views for Contents and Security) results in:

Tab 4 | Tab 5 | Contents | Security | Tab 2 | Tab 1 | Tab 3

These management view arguments apply to any content type that the view is registered for. What if you want to allow a content type to influence the tab ordering? As mentioned in the content type docs, the tab_order parameter overrides the mgmt_view tab settings, for a content type, with a sequence of view names that should be ordered (and everything not in the sequence, after.)

Filling Slots¶

Each management view that you write plugs into various parts of the SDI UI. This is done using normal ZPT fill-slot semantics:

• page-title is the <title> in the <head>
• head-more is a place to inject CSS and JS in the <head> after all the SDI elements
• tail-more does the same, just before the </body>
• main is the main content area

SDI API¶

All templates in the SDI share a common “layout”. This layout needs information from the environment to render markup that is common to every screen, as well as the template used as the “main template.”

This “template API” is known as the SDI API. It is an instance of the sdiapi class in substanced.sdi.__init__.py and is made available as request.sdiapi.

The template for your management view should start with a call to requests.sdiapi:

<div metal:use-macro="request.sdiapi.main_template">

The request.sdiapi object has other convenience features as well. See the Substance D interfaces documentation for more information.

Flash Messages¶

Often you perform an action on one view that needs a message displayed by another view on the next request. For example, if you delete a resource, the next request might confirm to the user “Deleted 1 resource.” Pyramid supports this with “flash messages.”

In Substance D, your applications can make a call to the sdiapi such as:

request.sdiapi.flash('ACE moved up')

…and the next request will process this flash message:

• The message will be removed from the stack of messages
• It will then be displayed in the appropriate styling based on the “queue”

The sdiapi provides another helper:

request.sdiapi.flash_with_undo(‘ACE moved up’)

This displays a flash message as before, but also provides an Undo button to remove the previous transaction.

• title, content, flash messages, head, tail

FormView¶

Form handling is ground that is frequently covered, usually in different ways. Substance D provides a class to help implement common patterns in form handling.

Imagine this example:

@mgmt_view(
    context=IFolder,
    name='add_document',
    tab_title='Add Document',
    permission='sdi.add-content',
    renderer='substanced.sdi:templates/form.pt',
    tab_condition=False,
)
class AddDocumentView(FormView):
    title = 'Add Document'
    schema = DocumentSchema()
    buttons = ('add',)

    def add_success(self, appstruct):
        registry = self.request.registry
        name = appstruct.pop('name')
        document = registry.content.create('Document', **appstruct)
        self.context[name] = document
        return HTTPFound(
            self.request.mgmt_path(self.context, '@@contents'))

This mgmt_view adds a view add_document to resources with the IFolder interface. The form gets a title, a Colander schema, and asks for just one button.

Since the mgmt_view is associated with a renderer, we have an SDI template form.pt which does the basics of laying out the rendering before handing the work over to Deform.

The @action of the form is the mgmt_view itself, making it a self-posting form. The button that was clicked causes the FormView to, upon validation success, route processing to a handler for that button. By convention, FormView looks for a method starting with the name of the button (e.g. add) and finishing with _success (e.g. add_success.) The class also supports a similar protocol for _failure.

FormView also supports the following methods that can be overridden:

• before(self, form) is called before validation and processing of any _success or _failure methods
• failure(self, e) is called with the exception, if the there is no button-specific _failure method
• show(self, form) returns {'form':form.render()} and thus can be a place to affect form rendering

The Default Catalog¶

A default catalog named system is installed into the root folder’s catalogs subfolder when you start Substance D. This system catalog contains a default set of indexes:

• path (a path index)

  Represents the path of the content object.

• name (a field index), uses content.__name__ exclusively

  Represents the local name of the content object.

• interfaces (a keyword index)

  Represents the set of interfaces possessed by the content object.

• content_type (a field index)

  Represents the Substance D content-type of an object.

• allowed (an allowed index)

  An index which can be used to filter resultsets using principals and permissions.

• text (a text index)

  Represents the text searched for when you use the filter box within the folder contents view of the SDI.

Adding a Catalog¶

The system catalog won’t have enough information to form all the queries you need. You’ll have to add a catalog via code related to your application. The first step is adding a catalog factory.

A catalog factory is a collection of index descriptions. Creating a catalog factory doesn’t actually add a catalog to your database, but it makes it possible to add one later.

Here’s an example catalog factory named mycatalog:

from substanced.catalog import (
    catalog_factory,
    Text,
    Field,
    )

@catalog_factory('mycatalog')
class MyCatalogFactory(object):
    freaky = Text()
    funky = Field()

In order to activate a @catalog_factory decorator, it must be scanned using the Pyramid config.scan() machinery. This will allow you to use substanced.catalog.CatalogsService.add_catalog() to add a catalog with that factory’s name:

# in a module named blog.__init__

from pyramid.config import Configurator

def main(global_config, **settings):
    config = Configurator()
    config.include('substanced')
    config.scan('blog.catalogs')
    # .. and so on ...

Once you’ve done this, you can then add the catalog to the database in any bit of code that has access to the database. For example, in an event handler when the root object is created for the first time.

from substanced.root import Root
from substanced.event import subscribe_created

@subscribe_created(Root)
def created(event):
    root = event.object
    service = root['catalogs']
    service.add_catalog('mycatalog', update_indexes=True)

Object Indexing¶

Once a new catalog has been added to the database, each time a new catalogable object is added to the site, its attributes will be indexed by each catalog in its lineage that “cares about” the object. The object will always be indexed in the “system” catalog. To make sure it’s cataloged in custom catalogs, you’ll need to do some work. To index the object in a custom application index, you will need to create an index view for your content using substanced.catalog.indexview, and scan the resulting index view using pyramid.config.Configurator.scan():

For example:

from substanced.catalog import indexview

class MyCatalogViews(object):
    def __init__(self, resource):
        self.resource = resource

    @indexview(catalog_name='mycatalog')
    def freaky(self, default):
        return getattr(self.resource, 'freaky', default)

An index view class should be a class that accepts a single argument, (conventionally named resource), in its constructor, and which has one or more methods named after potential index names. When it comes time for the system to index your content, Substance D will create an instance of your indexview class, and it will then call one or more of its methods; it will call methods on the indexview object matching the attr passed in to add_indexview. The default value passed in should be returned if the method is unable to compute a value for the content object.

Once this is done, whenever an object is added to the system, a value (the result of the freaky(default) method of the catalog view) will be indexed in the freaky field index.

You can attach multiple index views to the same index view class:

from substanced.catalog import indexview

class MyCatalogViews(object):
    def __init__(self, resource):
        self.resource = resource

    @indexview(catalog_name='mycatalog')
    def freaky(self, default):
        return getattr(self.resource, 'freaky', default)

    @indexview(catalog_name='mycatalog')
    def funky(self, default):
        return getattr(self.resource, 'funky', default)

You can use the “index_name” parameter to indexview to tell the system that the index name is not the same as the method name in the index view:

from substanced.catalog import indexview

class MyCatalogViews(object):
    def __init__(self, resource):
        self.resource = resource

    @indexview(catalog_name='mycatalog')
    def freaky(self, default):
        return getattr(self.resource, 'freaky', default)

    @indexview(catalog_name='mycatalog', index_name='funky')
    def notfunky(self, default):
        return getattr(self.resource, 'funky', default)

You can use the context parameter to indexview to tell the system that this particular index view should only be executed when the class of the resource (or any of its interfaces) matches the value of the context:

from substanced.catalog import indexview

class MyCatalogViews(object):
    def __init__(self, resource):
        self.resource = resource

    @indexview(catalog_name='mycatalog', context=FreakyContent)
    def freaky(self, default):
        return getattr(self.resource, 'freaky', default)

    @indexview(catalog_name='mycatalog', index_name='funky')
    def notfunky(self, default):
        return getattr(self.resource, 'funky', default)

You can use the indexview_defaults class decorator to save typing in each indexview declaration. Keyword argument names supplied to indexview_defaults will be used if the indexview does not supply the same keyword:

from substanced.catalog import (
    indexview,
    indexview_defaults,
    )

@indexview_defaults(catalog_name='mycatalog')
class MyCatalogViews(object):
    def __init__(self, resource):
        self.resource = resource

    @indexview()
    def freaky(self, default):
        return getattr(self.resource, 'freaky', default)

    @indexview()
    def notfunky(self, default):
        return getattr(self.resource, 'funky', default)

The above configuration is the same as:

from substanced.catalog import indexview

class MyCatalogViews(object):
    def __init__(self, resource):
        self.resource = resource

    @indexview(catalog_name='mycatalog')
    def freaky(self, default):
        return getattr(self.resource, 'freaky', default)

    @indexview(catalog_name='mycatalog')
    def notfunky(self, default):
        return getattr(self.resource, 'funky', default)

You can also use the substanced.catalog.add_indexview() directive to add index views imperatively, instead of using the @indexview decorator.

Querying the Catalog¶

You execute a catalog query using APIs of the catalog’s indexes.

from substanced.util import find_catalog

catalog = find_catalog(resource, 'system')
name = catalog['name']
path = catalog['path']
# find me all the objects that exist under /somepath with the name 'somename'
q = name.eq('somename') & path.eq('/somepath')
resultset = q.execute()
for contentob in resultset:
    print contentob

The calls to name.eq() and path.eq() above each return a query object. Those two queries are ANDed together into a single query via the & operator between them (there’s also the | character to OR the queries together, but we don’t use it above). Parentheses can be used to group query expressions together for the purpose of priority.

Different indexes have different query methods, but most support the eq method. Other methods that are often supported by indexes: noteq, ge, le, gt, any, notany, all, notall, inrange, notinrange. The AllowedIndex supports an additional allows() method.

Query objects support an execute method. This method returns a hypatia.util.ResultSet. A hypatia.util.ResultSet can be iterated over; each iteration returns a content object. hypatia.util.ResultSet also has methods like one and first, which return a single content object instead of a set of content objects. A hypatia.util.ResultSet also has a sort method which accepts an index object (the sort index) and returns another (sorted) hypatia.util.ResultSet.

catalog = find_catalog(resource, 'system')
name = catalog['name']
path = catalog['path']
# find me all the objects that exist under /somepath with the name 'somename'
q = name.eq('somename') & path.eq('/somepath')
resultset = q.execute()
newresultset = resultset.sort(name)

Querying Across Catalogs¶

In many cases, you might only have one custom attribute that you need indexed, while the system catalog has everything else you need. You thus need an efficient way to combine results from two catalogs, before executing the query:

system_catalog = find_catalog(resource, 'system')
my_catalog = find_catalog(resource, 'mycatalog')
path = system_catalog['path']
funky = my_catalog['funky']
# find me all funky objects that exist under /somepath
q = funky.eq(True) & path.eq('/somepath')
resultset = q.execute()
newresultset = resultset.sort(system_catalog['name'])

Filtering Catalog Results Using the Allowed Index¶

The Substance D system catalog at substanced.catalog.system.SystemCatalogFactory contains a number of default indexes, including an allowed index. Its job is to index security information to allow security-aware results in queries. This index allows us to filter queries to the system catalog based on whether the principal issuing the request has a permission on the matching resource.

For example, the below query will find:

• all of the subresources inside a folder
• which is of content type News Item
• which the current user also possesses the view permission against

system_catalog = find_catalog(resource, 'system')
path = system_catalog['path']
content_type = system_catalog['content_type']
allowed = system_catalog['allowed']
q = ( path.eq(resource, depth=1, include_origin=False) &
      content_type.eq('News Item') &
      allowed.allows(request, 'view')
    )
return q

Filtering Catalog Results Using The Objectmap¶

It is possible to postfilter catalog results using the substanced.objectmap.ObjectMap.allowed() API. For example:

def get_allowed_to_view(context, request):

    catalog = find_catalog(context, 'system')
    q = catalog['content_type'].eq('News Item')
    resultset = q.execute()

    objectmap = find_objectmap(context)
    return objectmap.allowed(
              resultset.oids, request.effective_principals, 'view')

The result of allowed() is a generator which returns oids, so the result must be listified if you intend to index into it, or slice it, or what-have-you.

Setting ACLs¶

The objectmap keeps track of ACLs in a cache to make catalog security functionality work. Note that for the object map’s cached version of ACLs to be correct, you will need to set ACLs in a way that helps keep track of all the contracts. For this, the helper function substanced.util.set_acl() can be used. For example, the site root at substanced.root.Root finishes with:

set_acl(
    self,
    [(Allow, get_oid(admins), ALL_PERMISSIONS)],
    registry=registry,
    )

Using set_acl this way will generate an event that will keep the objectmap’s cache updated. This will allow the allowed index to work and the substanced.objectmap.ObjectMap.allowed() method to work.

Deferred Indexing and Mode Parameters¶

As a lesson learned from previous cataloging experience, Substance D natively supports deferred indexing. As an example, in many systems the text indexing can be done after the change to the object is committed in the web request’s transaction. Doing so has a number of performance benefits: the user’s request processes more quickly, the work to extract text from a Word file can be performed later, less chance to have a conflict error, etc.

As such, the substanced.catalog.system.SystemCatalogFactory, by default, has indexes that aren’t updated immediately when a resource is changed. For example:

# name is MODE_ATCOMMIT for next-request folder contents consistency
name = Field()

text = Text(action_mode=MODE_DEFERRED)
content_type = Field()

The Field indexes use the default of MODE_ATCOMMIT. The Text overrides the default and set action_mode to MODE_DEFERRED.

There are three such catalog “modes” for indexing:

• substanced.interfaces.MODE_IMMEDIATE means indexing action should take place as immediately as possible.
• substanced.interfaces.MODE_ATCOMMIT means indexing action should take place at the successful end of the current transaction.
• substanced.interfaces.MODE_DEFERRED means indexing action should be performed by an external indexing processor (e.g. drain_catalog_indexing) if one is active at the successful end of the current transaction. If an indexing processor is unavailable at the successful end of the current transaction, this mode will be taken to imply the same thing as MODE_ATCOMMIT.

Running an Indexer Process¶

Great, we’ve now deferred indexing to a later time. What exactly do we do at that later time?

Indexer processes are easy to write and schedule with supervisor. Here is an example of a configuration for supervisor.conf that will run in indexer process every five seconds:

[program:indexer]
command = %(here)s/../bin/sd_drain_indexing %(here)s/production.ini
redirect_stderr = true
stdout_logfile = %(here)s/../var/indexing.log
autostart = true
startsecs = 5

This calls sd_drain_indexing which is a console script that Substance D automatically creates in your bin directory. Indexing messages are logged with standard Python logging to the file that you name. You can view these messages with the supervisorctl command tail indexer. For example, here is the output from sd_drain_indexing when changing a simple Document content type:

2013-01-07 11:07:38,306 INFO  [substanced.catalog.deferred][MainThread] no actions to execute
2013-01-07 11:08:38,329 INFO  [substanced.catalog.deferred][MainThread] executing <substanced.catalog.deferred.IndexAction object oid 5886459017869105529 for index u'text' at 0x106e52910>
2013-01-07 11:08:38,332 INFO  [substanced.catalog.deferred][MainThread] executing <substanced.catalog.deferred.IndexAction object oid 5886459017869105529 for index u'interfaces' at 0x106e52dd0>
2013-01-07 11:08:38,333 INFO  [substanced.catalog.deferred][MainThread] executing <substanced.catalog.deferred.IndexAction object oid 5886459017869105529 for index u'content_type' at 0x1076e2ed0>
2013-01-07 11:08:38,334 INFO  [substanced.catalog.deferred][MainThread] committing
2013-01-07 11:08:38,351 INFO  [substanced.catalog.deferred][MainThread] committed

Overriding Default Modes Manually¶

Above we set the default mode used by an index when Substance D indexes a resource automatically. Perhaps in an evolve script, you’d like to override the default mode for that index and reindex immediately.

The index_resource on an index can be passed an action_mode flag that overrides the configured mode for that index, and instead, does exactly what you want for only that call. It does not permanently change the configured default for indexing mode. This applies also to reindex_resource and unindex_resource. You can also grab the catalog itself and reindex with a mode that overrides all default modes on each index.

Autosync and Autoreindex¶

If you add substanced.catalogs.autosync = true within your application’s .ini file, all catalog indexes will be resynchronized with their catalog factory definitions at application startup time. Indices which were added to the catalog factory since the last startup time will be added to each catalog which uses the index factory. Likewise, indices which were removed will be removed from each catalog, and indices which were modified will be modified according to the catalog factory. Having this setting in your .ini file is like pressing the Update indexes button on the Manage tab of each of your catalogs. The SUBSTANCED_CATALOGS_AUTOSYNC environment variable can also be used to turn this behavior on. For example export SUBSTANCED_CATALOGS_AUTOSYNC=true.

If you add substanced.catalogs.autoreindex = true within your application’s .ini file, all catalogs that were changed as the result of an auto-sync will automatically be reindexed. Having this setting in your .ini file is like pressing the Reindex catalog button on the Manage tab of each catalog which was changed as the result of hitting Update indexes. The SUBSTANCED_CATALOGS_AUTOREINDEX environment variable can also be used to turn this behavior on. For example export SUBSTANCED_CATALOGS_AUTOREINDEX=true.

Forcing Deferral of Indexing¶

There may be times when you’d like to defer all catalog indexing operations, such as during a bulk load of data from a script. Normally, only indexes marked with MODE_DEFERRED use deferred indexing, and actions associated with those indexes are even then only actually deferred if an index processor is active.

You can force Substance D to defer all catalog indexing using the substanced.catalogs.force_deferred flag in your application’s .ini file. When this flag is used, all catalog indexing operations will be added to the indexer’s queue, even those indexes marked as MODE_IMMEDIATE or MODE_ATCOMMIT. Deferral will also happen whether or not the indexer is running, unlike during normal operations.

When you use this flag, you can stop the indexer process, do your bulk load, and start the indexer again when it’s convenient to have all the content indexing done in the background.

The SUBSTANCED_CATALOGS_FORCE_DEFERRED environment variable can also be used to turn this behavior on. For example export SUBSTANCED_CATALOGS_FORCE_DEFERRED=true.

ACLs and Principal References¶

When an ACL is modified on a resource, a statement is being made about a relationship between that resource and a principal or group of principals. Wouldn’t it be great if a reference was established, allowing you to then see such connections in the SDI?

This is indeed exactly how Substance D behaves: a source-integral PrincipalToACLBearing reference is set up between an ACL-bearing resource and the principals referred to within the ACL.

Features¶

• Site-wide workflows
• Multiple workflows per object
• Content type specific workflows
• Restrict transitions by permission
• Configurable callbacks when entering state
• Configurable callbacks when executing transition
• Reset workflow to initial state

Adding a workflow¶

Suppose we want to add a simple workflow:

/-----\   <-- to_draft -----   /---------\
|draft|                        |published|
\-----/   --- to_publish -->   \---------/

Using add_workflow() Pyramid configuration directive:

>>> workflow = Workflow(initial_state="draft", type="article")
>>> workflow.add_state("draft")
>>> workflow.add_state("published")
>>> workflow.add_transition('to_publish', from_state='draft', to_state='published')
>>> workflow.add_transition('to_draft', from_state='published', to_state='draft')

...

>>> config.add_workflow(workflow, ('News',))

Interaction with the workflow¶

Retrieve a Workflow instance using the substanced.workflow.get_workflow():

>>> from substanced.workflow import get_workflow

>>> workflow = get_workflow(request, type='article', content_type='News')

Suppose there is a context object at hand, you can reset() its workflow to initial state:

>>> workflow.reset(context, request)

You could check it has_state() and assert state_of() context is initial state name of the workflow:

>>> assert workflow.has_state(context) == True
>>> assert workflow.state_of(context) == workflow.initial_state

List possible transitions from the current state of the workflow with get_transitions():

>>> workflow.get_transitions(context, request)
[{'from_state': 'draft',
  'callback': None,
  'permission': None,
  'name': 'to_publish',
  'to_state': 'published'}]

Execute a transition():

>>> workflow.transition(context, request, 'to_publish')

List all states of the workflow with get_states():

>>> workflow.get_states(context, request)
[{'name': 'draft',
  'title': 'draft',
  'initial': True,
  'current': False,
  'transitions': [{'from_state': 'draft',
                   'callback': None,
                   'permission': None,
                   'name': 'to_publish',
                   'to_state': 'published'}],
  'data': {'callback': None}},
 {'name': 'published',
  'title': 'published',
  'initial': False,
  'current': True,
  'transitions': [{'from_state': 'published',
                   'callback': None,
                   'permission': None,
                   'name': 'to_draft',
                   'to_state': 'draft'}],
  'data': {'callback': None}}]

Execute a transition_to_state():

>>> workflow.transition_to_state(context, request, 'draft')

Using callbacks¶

Typically you will want to define custom actions when transition is executed or when content enters a specific state. Let’s define a transition with a callback:

>>> def cb(context, **kw):
...     print "keywords: ", kw

>>> workflow.add_transition('to_publish_with_callback',
...                         from_state='draft',
...                         to_state='published',
...                         callback=cb)

When you execute the transition, callback is called:

>>> workflow.transition(context, request, 'to_publish_with_callback')
keywords: {'workflow': <Workflow ...>, 'transition': {'to_state': 'published', 'from_state': 'draft', ...}, request=<Request ...>}

To know more about callback parameters, read add_transition() signature.

Dumping Resources Using sd_dump¶

The sd_dump console script loads your Substance D application, connects to your object database, and writes serialized representations of resources to disk in a directory hierarchy:

$ ../bin/sd_dump --help
Usage: sd_dump [options]

 Dump an object (and its subobjects) to the filesystem:  sd_dump [--source
=ZODB-PATH] [--dest=FILESYSTEM-PATH] config_uri   Dumps the object at ZODB-
PATH and all of its subobjects to a   filesystem path.  Such a dump can be
loaded (programmatically)   by using the substanced.dump.load function  e.g.
sd_dump --source=/ --dest=/my/dump etc/development.ini

Options:
  -h, --help            show this help message and exit
  -s ZODB-PATH, --source=ZODB-PATH
                        The ZODB source path to dump (e.g. /foo/bar or /)
  -d FILESYSTEM-PATH, --dest=FILESYSTEM-PATH
                        The destination filesystem path to dump to.

For example:

$ ../bin/sd_dump ../etc/development.ini
2013-01-07 13:27:03,939 INFO  [ZEO.ClientStorage][MainThread] ('localhost', 9963) ClientStorage (pid=93148) created RW/normal for storage: 'main'
2013-01-07 13:27:03,941 INFO  [ZEO.cache][MainThread] created temporary cache file '<fdopen>'
2013-01-07 13:27:03,981 WARNI [ZEO.zrpc][Connect([(2, ('localhost', 9963))])] (93148) CW: error connecting to ('fe80::1%lo0', 9963): EHOSTUNREACH
2013-01-07 13:27:03,982 WARNI [ZEO.zrpc][Connect([(2, ('localhost', 9963))])] (93148) CW: error connecting to ('fe80::1%lo0', 9963): EHOSTUNREACH
2013-01-07 13:27:04,002 WARNI [ZEO.zrpc][Connect([(2, ('localhost', 9963))])] (93148) CW: error connecting to ('::1', 9963): EINVAL
2013-01-07 13:27:04,003 INFO  [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) Testing connection <ManagedClientConnection ('127.0.0.1', 9963)>
2013-01-07 13:27:04,004 INFO  [ZEO.zrpc.Connection(C)][('localhost', 9963) zeo client networking thread] (127.0.0.1:9963) received handshake 'Z3101'
2013-01-07 13:27:04,105 INFO  [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) Server authentication protocol None
2013-01-07 13:27:04,106 INFO  [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) Connected to storage: ('localhost', 9963)
2013-01-07 13:27:04,108 INFO  [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) No verification necessary -- empty cache
2013-01-07 13:27:04,727 INFO  [substanced.catalog][MainThread] system update_indexes: no indexes added or removed
2013-01-07 13:27:04,730 INFO  [substanced.catalog][MainThread] sdidemo update_indexes: no indexes added or removed
2013-01-07 13:27:04,732 INFO  [substanced.dump][MainThread] Dumping /
2013-01-07 13:27:04,749 INFO  [substanced.dump][MainThread] Dumping /principals
2013-01-07 13:27:04,754 INFO  [substanced.dump][MainThread] Dumping /principals/users
2013-01-07 13:27:04,760 INFO  [substanced.dump][MainThread] Dumping /principals/users/admin
2013-01-07 13:27:04,779 INFO  [substanced.dump][MainThread] Dumping /principals/resets
2013-01-07 13:27:04,783 INFO  [substanced.dump][MainThread] Dumping /principals/groups

…with logging messages being emitted until all known content is dumped. A dump subdirectory in the current directory is created (if no argument is provided) containing:

$ ls
acl.yaml    propsheets      references.yaml resource.yaml   resources

- !!python/tuple [Allow, 1644064392535565429, !all_permissions '']

!interface 'substanced.interfaces.PrincipalToACLBearing':
  sources: [1644064392535565429]

!!python/object:persistent.mapping.PersistentMapping
data: {document: draft}

{body: !!python/unicode 'The quick brown fox jumps over the lazy dog. The quick brown
    fox jumps over the lazy dog.  The quick brown fox jumps over the lazy dog.
    The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the
    lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps
    over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown
    fox jumps over the lazy dog. ', name: !!python/unicode 'document_0', title: !!python/unicode 'Document
    0 Binder 0'}

{content_type: Root, created: !!timestamp '2013-01-07 14:23:23.133436', is_service: false,
  name: null, oid: 1644064392535565415}

Custom Dumping with __dump__¶

The built-in facilities allow automatic dumping of most information for your content, including information in your property sheets, the content type, security settings, references, workflows, etc.

If you do need extra information dumped to YAML about your content type, Substance D has a Python protocol using an __dump__ on your @content class. As an example, :py:meth:substanced.principal.User.dump is a callable which returns a mapping of simple Python objects. The dumper checks to see if a resource has a __dump__ method. If so, it calls the method, encodes the result to YAML, and writes it to an adhoc.yaml file in the dumped-resource’s directory.

The inverse is also true. If a content type has a __load__ method, information from that method is added to the state that is loaded.

Adding New Dumpers¶

The adhoc.yaml file that we just saw is an example of the AdhocAttrDumper. There are seven other dumpers built-in: acl, workflow, references, sdiproperties, interfaces, order, and propsheets.

If you would like a custom dumper, you can register it with config.add_dumper. For example, substanced.dump.includeme() registers the existing dumpers and their dumper factories:

def includeme(config):
    DEFAULT_DUMPERS = [
        ('acl', ACLDumper),
        ('workflow', WorkflowDumper),
        ('references', ReferencesDumper),
        ('sdiproperties', SDIPropertiesDumper),
        ('interfaces', DirectlyProvidedInterfacesDumper),
        ('order', FolderOrderDumper),
        ('propsheets', PropertySheetDumper),
        ('adhoc', AdhocAttrDumper),
        ]
    config.add_directive('add_dumper', add_dumper)
    for dumper_name, dumper_factory in DEFAULT_DUMPERS:
        config.add_dumper(dumper_name, dumper_factory)

Running an Evolution from the Command Line¶

Substance D applications generate a console script at bin/sdi_evolve. Running this without arguments displays some help:

$ bin/sd_evolve
Requires a config_uri as an argument

    sd_evolve [--latest] [--dry-run] [--mark-finished=stepname] [--mark-unfinished=stepname] config_uri
      Evolves new database with changes from scripts in evolve packages
         - with no arguments, evolve displays finished and unfinished steps
         - with the --latest argument, evolve runs scripts as necessary
         - with the --dry-run argument, evolve runs scripts but does not issue any commits
         - with the --mark-finished argument, marks the stepname as finished
         - with the --mark-unfinished argument, marks the stepname as unfinished

    e.g. sd_evolve --latest etc/development.ini

Running with your INI file, as explained in the help, shows information about the version numbers of various packages:

$ bin/sd_evolve etc/development.ini

Finished steps:

    2013-06-14 13:01:28 substanced.evolution.legacy_to_new

Unfinished steps:

This shows that one evolution step has already been run and that there are no unfinished evolution steps.

Running an Evolution from the SDI¶

The Evolution section of the Database tab of the Substance D root object allows you to do what you might have otherwise done using the sd_evolve console script described above.

In some circumstances when Substance D itself needs to be upgraded, you may need to use the sd_evolve script rather than the GUI. For example, if the way that Substance D Folder objects work is changed and folder objects need to be evolved, it may be impossible to view the evolution GUI, and you may need to use the console script.

Autoevolve¶

If you add substanced.autoevolve = true within your application .ini file, all pending evolution upgrade steps will be run when your application starts. Alternately you can use the SUBSTANCED_AUTOEVOLVE evnironment variable (e.g. export SUBSTANCED_AUTOEVOLVE=true) to do the same thing.

Adding Evolution Support To a Package¶

Let’s say we have been developing an sdidemo package and, with content already in the database, we want to add evolution support. Our sdidemo package is designed to be included into a site, so we have the traditional Pyramid includeme support. In there we add the following:

import logging

logger = logging.getLogger('evolution')

def evolve_stuff(root, registry):
    logger.info('Stuff evolved.')

def includeme(config):
    config.add_evolution_step(evolve_stuff)

We’ve used the substanced.evolution.add_evolution_step() API to add an evolution step in this package’s includeme function.

Running sd_evolve without --latest (meaning, without performing an evolution) shows that Substance D’s evolution now knows about our package:

$ bin/sd_evolve etc/development.ini

Finished steps:

    2013-06-14 13:01:28 substanced.evolution.legacy_to_new

Unfinished steps:

                        sdidemo.evolve_stuff

Let’s now run sd_evolve “for real”. This will cause the evolution step to be executed and marked as finished.

$ bin/sd_evolve --latest etc/development.ini

2013-06-14 13:22:51,475 INFO  [evolution][MainThread] Stuff evolved.
Evolution steps executed:
   substanced.evolution.evolve_stuff

This examples shows a number of points:

• Each package can easily add evolution support via the config.add_evolution_step() directive. You can learn more about this directive by reading its API documentation at substanced.evolution.add_evolution_step().
• Substance D’s evolution service looks at the database to see which steps haven’t been run, then runs all the needed evolve scripts, sequentially, to bring the database up to date.
• All changes within an evolve script are in the scope of a transaction. If all the evolve scripts run to completion without exception, the transaction is committed.

Manually Marking a Step As Evolved¶

In some cases you might have performed the work in an evolve step by hand and you know there is no need to re-perform that work. You’d like to mark the step as finished for one or more evolve scripts, so these steps don’t get run. The --mark-step-finished argument to sd_evolve accomplishes this. The “Mark finished” button in the SDI evolution GUI does the same.

Baselining¶

Evolution is baselined at first startup. When there’s no initial list of finished steps in the database. Substance D, in the root factory, says: “I know all the steps participating in evolution, so when I first create the root object, I will set all of those steps to finished.”

If you wish to perform something after Root was created, see Affecting Content Creation.

Adding Columns¶

Perhaps your system has content types with extra attributes that are meaningful and you’d like your contents listings to show that column. You can change the columns available on folder contents listings by passing in a columns argument to the @content directive. The value of this argument is a callable which returns a sequence of mappings conforming to the datagrid’s contract. For example:

def binder_columns(folder, subobject, request, default_columnspec):
    subobject_name = getattr(subobject, '__name__', str(subobject))
    objectmap = find_objectmap(folder)
    user_oid = getattr(subobject, 'creator', None)
    created = getattr(subobject, 'created', None)
    modified = getattr(subobject, 'modified', None)
    if user_oid is not None:
        user = objectmap.object_for(user_oid)
        user_name = getattr(user, '__name__', 'anonymous')
    else:
        user_name = 'anonymous'
    if created is not None:
        created = created.isoformat()
    if modified is not None:
        modified = modified.isoformat()
    return default_columnspec + [
        {'name': 'Title',
        'value': getattr(subobject, 'title', subobject_name),
        },
        {'name': 'Created',
        'value': created,
        'formatter': 'date',
        },
        {'name': 'Last edited',
        'value': modified,
        'formatter': 'date',
        },
        {'name': 'Creator',
        'value': user_name,
        }
        ]

@content(
    'Binder',
    icon='glyphicon glyphicon-book',
    add_view='add_binder',
    propertysheets = (
        ('Basic', BinderPropertySheet),
        ),
    columns=binder_columns,
    )

The callable is passed the folder, a subobject, the request, and a set of default column specifications. To display the datagrid column headers, your callable is invoked on the first resource. Later, this callable is used to get the value for the fields of each column for each resource in a request’s batch.

The mappings returned can indicate whether a particular column should be sorted. If you want your column to be sortable, you must provide a sorter key in the mapping. If supplied, the sorter value must either be None if the column is not sortable, or a function which accepts a resource (the folder), a “resultset”, a limit keyword argument, and a reverse keyword argument and which must return a sorted result set. Here’s an example sorter:

from substanced.util import find_index

def sorter(folder, resultset, reverse=False, limit=None):
    index = find_index(folder, 'mycatalog', 'date')
    if index is not None:
        resultset = resultset.sort(index, reverse=reverse, limit=limit)
    return resultset

def my_columns(folder, subobject, request, default_columnspec):
    return default_columnspec + [
        {'name': 'Date',
        'value': getattr(subobject, 'title', subobject_name),
        'sorter': 'sorter',
        },

Most often, sorting is done by passing a catalog index into the resultset.sort method as above (resultset.sort returns another resultset), but sorting can be performed manually, as long as the sorter returns a resultset.

Buttons¶

As we just showed, you can extend the folder contents with extra columns to display and possibly sort on. You can also add new buttons that will trigger operations on selected resources.

As with columns, we pass a new argument to the @content directive. For example, the folder contents view for the catalogs folder allows you to reindex multiple indexes at once:

The Reindex button illustrates a useful facility for performing many custom operations at once.

The substanced.catalog module’s @content directive has a buttons argument:

@content(
    'Catalog',
    icon='glyphicon glyphicon-search',
    service_name='catalog',
    buttons=catalog_buttons,
    )

This points at a callable:

def catalog_buttons(context, request, default_buttons):
    """ Show a reindex button before default buttons in the folder contents
    view of a catalog"""
    buttons = [
        {'type':'single',
         'buttons':
         [
             {'id':'reindex',
              'name':'form.reindex',
              'class':'btn-primary btn-sdi-sel',
              'value':'reindex',
              'text':'Reindex'}
             ]
         }
        ] + default_buttons
    return buttons

In this case, the Reindex button was inserted before the other buttons, in the place where an add button would normally appear.

The class on your buttons affect behavior in the datagrid:

• btn-primary gives this button the styling for the primary button of a form, using Twitter Bootstrap form styling
• btn-sdi-act makes the button always enabled
• btn-sdi-sel disables the button until one or more items are selected
• btn-sdi-one disables the button until exactly one item is selected
• btn-sdi-del disables the button if any of the selected resources is marked as “non-deletable” (discussed below)

When clicked, this button will do a form POST of the selected docids to a view that you have implemented. Which view? The 'name': 'form.reindex' item sets the parameter on the POST. You can then register a view against this. substanced.catalog.views.catalog shows this:

@mgmt_view(
    context=IFolder,
    content_type='Catalog',
    name='contents',
    request_param='form.reindex',
    request_method='POST',
    renderer='substanced.folder:templates/contents.pt',
    permission='sdi.manage-contents',
    tab_condition=False,
    )
def reindex_indexes(context, request):
    toreindex = request.POST.getall('item-modify')
    if toreindex:
        context.reindex(indexes=toreindex, registry=request.registry)
        request.sdiapi.flash(
            'Reindex of selected indexes succeeded',
            'success'
            )
    else:
        request.sdiapi.flash(
            'No indexes selected to reindex',
            'danger'
            )

    return HTTPFound(request.sdiapi.mgmt_path(context, '@@contents'))

Selection and Button Enabling¶

As mentioned above, some buttons are driven by the selection. If nothing is selected, the button is disabled.

Buttons can also be disabled if any selected item is “non-deletable”. How does that get signified? An item is ‘deletable’ if the user has the sdi.manage-contents permission on folder and if the subobject has a __sdi_deletable__ attribute which resolves to a boolean True value.

It is also possible to make button enabling and disabling depend on some application-specific condition. To do this, assign a callable to the enabled_for key in the button spec. For example:

def catalog_buttons(context, request, default_buttons):
    def is_indexable(folder, subobject, request):
        """ only enable the button if subobject is indexable """
        return subobject.is_indexable()

    buttons = [
        {'type':'single',
         'buttons':
         [
             {'id':'reindex',
              'name':'form.reindex',
              'class':'btn-primary btn-sdi-sel',
              'value':'reindex',
              'enabled_for': is_indexable,
              'text':'Reindex'}
             ]
         }
        ] + default_buttons
    return buttons

In the example above, we define a button similar to our previous reindex button, except this time we have an enabled_for key that is assigned the is_indexable function. When the buttons are rendered, each element is passed to this function, along with the folder and request. If any one of the folder subobjects returns False for this call, the button will not be enabled.

Filtering What Can Be Added¶

Not all kinds of resources make sense to be added inside a certain kind of container. For example, substanced.catalog.Catalog is a content type that can hold only indexes. That is,it isn’t meant to hold any arbitrary kind of thing.

To tell the SDI what can be added inside a container content type, add a __sdi_addable__ method to your content type. This method is passed the folder object representing the place the object might be added, and a Substance D introspectable for a content type. When Substance D tries to figure out whether an object is addable to a particular folder, it will call the __sdi_addable__ method of your folderish type once for each content type.

The introspectable is a dictionary-like object which contains information about the content type. The introspectable contains the following keys:

meta

A dictionary representing “meta” values passed to add_content_type(). For example, if you pass add_view='foo' to add_content_type(), the meta of the content type will be {'add_view':'foo'}.

content_type

The content type value passed to add_content_type().

factory_type

The factory_type value passed to add_content_type().

original_factory

The original content factory (without any wrapping) passed to add_content_type().

factory

The potentially wrapped content factory derived from the original factory in add_content_type().

See Registering Content for more information about content type registration and what the above introspectable values mean.

Your __sdi_addable__ method can perform some logic using the values it is passed, and then it must return a filtered sequence.

As an example, the __sdi_addable__ method on the Catalog filters out the kinds of things that can be added in a catalog.

Extending Which Columns Are Displayed¶

The folder contents grid displays a number of columns by default. If you are managing content with custom properties, in some cases you want to list those properties in the columns the grid can display. You can do so on custom folder content types by adding a columns argument to your @content decorator.

As an example, imagine a Binder kind of container. It has a content type declaration:

@content(
    'Binder',
    icon='glyphicon glyphicon-book',
    add_view='add_binder',
    propertysheets = (
        ('Basic', BinderPropertySheet),
        ),
    columns=binder_columns,
    )

The binder_columns points to a callable where we perform the work to both add the column to the list of columns, but also specify how to get the row data for that column:

def binder_columns(folder, subobject, request, default_columnspec):
    subobject_name = getattr(subobject, '__name__', str(subobject))
    objectmap = find_objectmap(folder)
    user_oid = getattr(subobject, 'creator', None)
    created = getattr(subobject, 'created', None)
    modified = getattr(subobject, 'modified', None)
    if user_oid is not None:
        user = objectmap.object_for(user_oid)
        user_name = getattr(user, '__name__', 'anonymous')
    else:
        user_name = 'anonymous'
    if created is not None:
        created = created.isoformat()
    if modified is not None:
        modified = modified.isoformat()
    return default_columnspec + [
        {'name': 'Title',
        'value': getattr(subobject, 'title', subobject_name),
        },
        {'name': 'Created',
        'value': created,
        'formatter': 'date',
        },
        {'name': 'Last edited',
        'value': modified,
        'formatter': 'date',
        },
        {'name': 'Creator',
        'value': user_name,
        }
        ]

Here we add four columns to the standard set of grid columns, whenever we are in a Binder folder.

Adding New Folder Contents Buttons¶

The grid in folder contents makes it easy to select multiple resources then click a button to perform an action. Wouldn’t it be great, though, if we could add a new button to all or certain folders, to perform custom actions?

In the previous section we saw how to pass another argument to the @content decorator. We do the same for new buttons. A content type can pass in buttons=callable to modify the list of buttons on a particular kind of folder.

For example, the substanced.catalog.catalog_buttons() callable adds a new Reindex button in front of the standard set of buttons:

def catalog_buttons(context, request, default_buttons):
    """ Show a reindex button before default buttons in the folder contents
    view of a catalog"""
    buttons = [
        {'type':'single',
         'buttons':
         [
             {'id':'reindex',
              'name':'form.reindex',
              'class':'btn-primary btn-sdi-sel',
              'value':'reindex',
              'text':'Reindex'}
             ]
         }
        ] + default_buttons
    return buttons

The button is disabled until one or more resources are selected which have the correct permission (discussed above.) If our new button is clicked, the form is posted with the form.reindex value in post data. You can then make a @mgmt_view with request_param='form.reindex' in the declaration to handle the form post when that button is clicked.

Broken Objects and Class Aliases¶

Let’s assume that there’s an object in your database that is an instance of the class myapplication.resources.MyCoolResource. If that class is subsequently renamed to myapplication.resources.MySuperVeryCoolResource, the MyCoolResource object that exists in the database will become broken. This is because the ZODB database used by Substance D uses the Python pickle persistence format, and pickle writes the literal class name into the record associated with an object instance. Therefore, if a class is renamed or moved, when you come along later and try to deserialize a pickle with the old name, it will not work as it used to.

Persistent objects that exist in the database but which have a class that cannot be resolved are called “broken objects”. If you ask a Substance D folder (or the object map) for an object that turns out to be broken in this way, it will hand you back an instance of the pyramid.util.BrokenWrapper class. This class tries to behave as much as possible like the original object for data that exists in the original objects’ __dict__ (it defines a custom __getattr__ that looks in the broken object’s state). However, you won’t able to call methods of the original class against a broken object.

You can usually delete broken objects using the SDI folder contents view if necessary.

If you must rename or move a class, you can leave a class alias behind for backwards compatibility to avoid seeing broken objects in your database. For example:

class MySuperVeryCoolResource(Persistent):
    pass

MyCoolResource = MySuperVeryCoolResource # bw compat alias

Configuring the Audit Database¶

In order to enable auditing, you have to add an audit database to your Substance D configuration. This means adding a key to your application’s section in the .ini file associated with the app:

zodbconn.uri.audit = <some ZODB uri>

An example of “some ZODB URI” above might be (for a FileStorage database, if your application doesn’t use multiple processes):

zodbconn.uri.audit = file://%(here)s/auditlog.fs

Or if your application uses multiple processes, use a ZEO URL.

The database cannot be your main database. The reason that the audit database must live in a separate ZODB database is that we don’t want undo operations to undo the audit log data.

Note that if you do not configure an audit database, real-time SDI features such as your folder contents views updating without a manual refresh will not work.

Once you’ve configured the audit database, you need to add an audit log object to the new database. You can do this using pshell:

[chrism@thinko sdnet]$ bin/pshell etc/development.ini
Python 3.3.2 (default, Jun  1 2013, 04:46:52)
[GCC 4.6.3] on linux
Type "help" for more information.

Environment:
  app          The WSGI application.
  registry     Active Pyramid registry.
  request      Active request object.
  root         Root of the default resource tree.
  root_factory Default root factory used to create `root`.

>>> from substanced.audit import set_auditlog
>>> set_auditlog(root)
>>> import transaction; transaction.commit()

Once you’ve done this, the “Auditing” tab of the root object in the SDI should no longer indicate that auditing is not configured.

Viewing the Audit Log¶

The root object will have a tab named “Auditing”. You can view the currently active audit log entries from this page. Accessing this tab requires the sdi.view-auditlog permission.

Adding an Audit Log Entry¶

Here’s an example of adding an audit log entry of type NailsFiled to the audit log:

from substanced.util import get_oid, get_auditlog

def myview(context, request):
    auditlog = get_auditlog(context)
    auditlog.add('NailsFiled', get_oid(context), type='fingernails')
    ...

This will add a``NailsFiled`` event with the payload {'type':'fingernails'} to the audit log. The payload will also automatically include a UNIX timestamp as the key time. The first argument is the audit log typename. Audit entries of the same kind should share the same type name. It should be a string. The second argument is the oid of the content object which this event is related to. It may be None indicating that the event is global, and unrelated to any particular piece of content. You can pass any number of keyword arguments to substanced.audit.AuditLog.add(), each will be added to the payload. Each value supplied as a keyword argument must be JSON-serializable. If one is not, you will receive an error when you attempt to add the event.

Using The auditstream-sse View¶

If you have auditing enabled, you can use a view named auditstream-sse against any resource in your resource tree using JavaScript. It will return an event stream suitable for driving an HTML5 EventSource (an HTML 5 feature, see http://www.html5rocks.com/en/tutorials/eventsource/basics/ for more information). The event stream will contain auditing events. This can be used for progressive enhancement of your application’s UI. Substance D’s SDI uses it for that purpose. For example, when an object’s ACL is changed, a user looking at the “Security” tab of that object in the SDI will see the change immediately, rather than upon the next page refresh.

Obtain events for the context of the view only:

var source = new EventSource(
   "${request.sdiapi.mgmt_path(context, 'auditstream-sse')}");

Obtain events for a single OID unrelated to the context:

var source = new EventSource(
   "${request.sdiapi.mgmt_path(context, 'auditstream-sse', query={'oid':'12345'})}");

Obtain events for a set of OIDs:

var source = new EventSource(
   "${request.sdiapi.mgmt_path(context, 'auditstream-sse', query={'oid':['12345', '56789']})}");

Obtain all events for all oids:

var source = new EventSource(
   "${request.sdiapi.mgmt_path(context, 'auditstream-sse', query={'all':'1'})}");

The executing user will need to possess the sdi.view-auditstream permission against the context on which the view is invoked. Each event payload will contain detailed information about the audit event as a string which represents a JSON dictionary.

See the acl.pt template in the substanced/sdi/views/templates directory of Substance D to see a “real-world” usage of this feature.

Locking a Resource¶

To lock a resource:

from substanced.locking import lock_resource
from pyramid.security import has_permission

if has_permission('sdi.lock', someresource, request):
    lock_resource(someresource, request.user, timeout=3600)

If the resource is already locked by the owner supplied as owner_or_ownerid (the parameter filled by request.user above), calling this function will refresh the lock. If the resource is not already locked by another user, calling this function will create a new lock. If the resource is already locked by a different user, a substanced.locking.LockError will be raised.

Using the substanced.locking.lock_resource() function has the side effect of creating a “Lock Service” (named locks) in the Substance D root if one does not already exist.

Unlocking a Resource¶

To unlock a resource:

from substanced.locking import unlock_resource
from pyramid.security import has_permission

if has_permission('sdi.lock', someresource, request):
    unlock_resource(someresource, request.user)

If the resource is already locked by a user other than the owner supplied as owner_or_ownerid (the parameter filled by request.user above) or the resource isn’t already locked with this lock type, calling this function will raise a substanced.locking.UnlockError exception. Otherwise the lock will be removed.

Using the substanced.locking.unlock_resource() function has the side effect of creating a “Lock Service” (named locks) in the Substance D root if one does not already exist.

To unlock a resource using an explicit lock token:

from substanced.locking import unlock_token
from pyramid.security import has_permission

if has_permission('sdi.lock', someresource, request):
    unlock_token(someresource, token, request.user)

If the lock identified by token belongs to a user other than the owner supplied as owner_or_ownerid (the parameter filled by request.user above) or if no lock exists under token , calling this function will raise a substanced.locking.LockError exception. Otherwise the lock will be removed.

Using the substanced.locking.unlock_token() function has the side effect of creating a “Lock Service” (named locks) in the Substance D root if one does not already exist.

Discovering Existing Locks¶

To discover any existing locks for a resource:

from substanced.locking import discover_resource_locks

locks = discover_resource_locks(someresource)
# "locks" will be a sequence

The substanced.locking.discover_resource_locks() function will return a sequence of substanced.locking.Lock objects related to the resource for the lock type provided to the function. By default, only valid locks are returned. Invalid locks for the resource may exist, but they are not returned unless the include_invalid argument passed to :discover_resource_locks() is True.

Under normal circumstances, the length of the sequence returned will be either 0 (if there are no locks) or 1 (if there is any lock). In some special circumstances, however, when the substanced.locking.lock_resource() API is not used to create locks, there may be more than one lock related to a resource of the same type.

By default, the discover_resource_locks API returns locks for the provided object, plus locks on any object in its lineage. To suppress this default, pass include_lineage=False, e.g.:

locks = discover_resource_locks(someresource)
# "locks" will be only those set on 'someresource'

In some applications, the important thing is to ensure that a particular user could lock a resource before updating it (e.g., from a browser view on a property sheet). The :could_lock_resource() API is designed for these cases: if the supplied userid could not lock the resource, it raises a substanced.locking.LockError exception:

from substanced.locking import could_lock_resource, LockError

try:
    could_lock_resource(someresource, request.user)
except LockError as e:
    raise FormError('locked by "%s"' % e.lock.owner.__name__)

Viewing The Lock Service¶

Once some locks have been created, a lock service will have been created. The lock service is an object named locks in the Substance D root.

You can use the SDI UI of this locks service to delete and edit existing locks. It’s a good idea to periodically use the “Delete Expired” button in this UI to clear out any existing expired locks that were orphaned by buggy or interrupted clients.

Scan and Include¶

When writing Pyramid applications, the Configurator supports config.include and config.scan Because of ordering effects, do all your config.include calls before any of your config.scan calls.

Using RelStorage¶

Content in Substance D is stored in a Python object database called the ZODB. The ZODB has deep integration with Pyramid. When developing Python applications that use ZODB, you have a number of storage options:

• FileStorage is the simplest and is used in the development scaffolds for Substance D. That is, development.ini is configured use FileStorage. Just a file on disk, no long-running server process.
• ZEO keeps a file on disk but runs a server process that manages transactions over a socket. This allows multiple app servers on multiple boxes, or background processes such as deferred indexing, to access the database.
• RelStorage stores and retrieves the Python objects from a relational database. This is the preferred deployment option for applications that need trusted reliability and scalability.

Switching between storages is mostly a matter of editing your configuration file and choosing a different storage.

Although RelStorage supports a number of RDBMS packages, we’ll focus on PostgreSQL in these docs.

zodbconn.uri = zconfig://%(here)s/relstorage.conf

%import relstorage
<zodb main>
  <relstorage>
    blob-dir ../var/blobs
    <postgresql>
      dsn dbname='zodb' user='zodbuser' host='localhost' password='zodbuser'
    </postgresql>
  </relstorage>
  cache-size 100000
</zodb>

Setting Up¶

To enable statistics gathering in your site, edit your .ini configuration file and add the following lines to your [app:main] section:

substanced.statsd.enabled = true
substanced.statsd.host = localhost
substanced.statsd.port = 8125
substanced.statsd.prefix = substanced

Using DataDog with SubstanceD statistics¶

Substance D supports DataDog, a Software-as-a-Service (SaaS) provider for monitoring and visualizing performance data. DataDog installs an dogstatsd agent for sending custom metrics on your local system. The agent is based on StatsD.

Using DataDog is an an easy way to get started with Substance D statistics. Sign up for an account with DataDog. This will provide you with the instructions for downloading and running the local agent. You’ll need to get the agent installed before proceeding.

Once you’ve got the agent installed, and the proper settings in your Substance D ini file, you will be able to see statistics in the DataDog user interface. Once you log into your DataDog dashboard, click on Infrastructure and you’ll see any hosts configured as part of your account:

The substanced entry in Apps table column is from the substanced.statsd.prefix configured in Settings up section. Clicking on that brings up Substance D specific monitoring in DataDog:

Clicking settings symbol on a graph will lead you to graph editor, where you can change how DataDog interprets and renders your graphs. A good resource how the editor works is Graphing Primer.

DataDog also supports Metric Alerts allowing you to send alerts when your statistics reach certain state.

Logging Custom Statistics¶

Over time, Substance D itself will include more framework points where statistics are collected. Most likely, though, you’ll want some statistics that are very meaningful to your application’s specific functionality.

If you look at the docs for the Python statsd module you will see three main types:

• Counters for simply incrementing a value,
• Timers for logging elapsed time in a code block, and
• Gauges for tracking a constant at a particular point in time

Each of these map to methods in substanced.stats.StatsdHelper. This class is available as an instance available via import:

from substanced.stats import statsd_gauge

Your application code can then make calls to these stats-gathering methods. For example, substanced.principal.User does the following to note that check password was used:

statsd_gauge('check_password', 1)

Here is an example in substanced.catalog.Catalog.index_resource() that measures elapsed indexing time inside a Python with block:

with statsd_timer('catalog.index_resource'):
    if oid is None:
        oid = oid_from_resource(resource)
    for index in self.values():
        index.index_resource(resource, oid=oid, action_mode=action_mode)
    self.objectids.insert(oid)

sd_adduser¶

Add a new user, making them part of the ‘admins’ group. Useful when recovering from a forgotten password for the default ‘admin’ user. E.g.:

$ /path/to/virtualenv/bin/sd_adduser /path/to/virtualenv/etc/production.ini phred password

sd_drain_indexing¶

Process deferred indexing actions. E.g., run this from a cron job to drain the queue every two minutes:

0-59/2 * * * * /path/to/virtualenv/bin/sd_drain_indexing /path/to/virtualenv/etc/production.ini

sd_dump¶

Dump an object (and its subobjects) to the filesystem:

sd_dump [--source=ZODB-PATH] [--dest=FILESYSTEM-PATH] config_uri
Dumps the object at ZODB-PATH and all of its subobjects to a
filesystem path.  Such a dump can be loaded (programmatically)
by using the substanced.dump.load function

E.g.:

$ /path/to/virtualenv/bin/sd_dump --source=/ --dest=/tmp/dump /path/to/virtualenv/etc/development.ini

sd_evolve¶

Query for pending evolution steps, or run them to get the database up-to-date. See Running an Evolution from the Command Line.

sd_reindex¶

Reindex the catalog. E.g.:

$ /path/to/virtualenv/bin/sd_reindex /path/to/virtualenv/etc/development.ini