Serialization and Deserialization

Serialization is the act of converting application data into a form rendering. Deserialization is the act of converting data resulting from a form submission into application data.

Serialization

Serialization is what happens when you ask Deform to render a form given a schema. Here's a high-level overview of what happens when you ask Deform to do this:

  • For each schema node in the schema provided by the application developer, Deform creates a field. This happens recursively for each node in the schema. As a result, a tree of fields is created, mirroring the nodes in the schema.
  • Each field object created as a result of the prior step knows about its associated schema node (it has a field.schema attribute); each field also knows about an associated widget object (it has a field.widget attribute). This widget object may be a default widget based on the schema node type or it might be overridden by the application developer for a particular rendering.
  • Deform passes an appstruct to the root schema node's serialize method to obtain a cstruct. The root schema node is responsible for consulting its children nodes during this process to serialize the entirety of the data into a single cstruct.
  • Deform passes the resulting cstruct to the root widget object's serialize method to generate an HTML form rendering. The root widget object is responsible for consulting its children nodes during this process to serialize the entirety of the data into an HTML form.

If you were to attempt to produce a high-level overview diagram this process, it might look like this:

appstruct -> cstruct -> form
           |          |
           v          v
        schema      widget

Peppercorn Structure Markers

You'll see the default deform widget "serializations" (form renderings) make use of Peppercorn structure markers.

Peppercorn is a library that is used by Deform. It allows Deform to treat the form controls in an HTML form submission as a stream instead of a flat mapping of name to value. To do so, it uses hidden form elements to denote structure.

Peppercorn structure markers come in pairs which have a begin token and an end token. For example, a given form rendering might have a part that looks like so:

1
2
3
4
5
6
7
8
9
  <html>
   ...
    <input type="hidden" name="__start__" value="date:mapping"/>
    <input name="day"/>
    <input name="month"/>
    <input name="year"/>
    <input type="hidden" name="__end__"/>
   ...
  </html>

The above example shows an example of a pair of Peppercorn structure markers which begin and end a mapping. The example uses this pair to mean that the widget related to the date node in the schema will be be passed a pstruct that is a dictionary with multiple values during deserialization. The dictionary will include the keys day , month, and year, and the values will be the values provided by the person interacting with the related form controls.

Other uses of Peppercorn structure markers include a "confirm password" widget that can render a Peppercorn mapping with two text inputs in it, or a "mapping widget" that can serve as a substructure for a fieldset. Basically Peppercorn makes it more pleasant to deal with form submission data by pre-converting the data from a flat mapping into a set of mappings, sequences, and strings during deserialization.

However, if a widget doesn't want to do anything fancy and a particular widget is completely equivalent to one form control, it doesn't need to use any Peppercorn structure markers in its rendering.

Note

See the Peppercorn documentation for more information about using Peppercorn structure markers in HTML.

Deserialization

The following is a high-level overview of how "deserialization" (converting form control data resulting from a form submission to application data) works:

  • For each schema node in the schema provided by the application developer, Deform creates a field. This happens recursively for each node in the schema. As a result, a tree of fields is created, mirroring the nodes in the schema.
  • Each field object created as a result of the prior step knows about its associated schema node (it has a field.schema attribute). Each field also knows about an associated widget object (it has a field.widget attribute). This widget object may be a default widget based on the schema node type, or it might be overridden by the application developer for a particular rendering.
  • Deform passes a set of form controls to the parse method of Peppercorn in order to obtain a pstruct.
  • Deform passes the resulting pstruct to the root widget node's deserialize method in order to generate a cstruct.
  • Deform passes the resulting cstruct to the root schema node's deserialize method to generate an appstruct. This may result in a validation error. If a validation error occurs, the form may be re-rendered with error markers in place.

If you were to attempt to produce a high-level overview diagram of this process, it might look like this:

formcontrols -> pstruct -> cstruct -> appstruct
             |          |          |
             v          v          v
         Peppercorn   widget    schema

When a user presses the submit button on any Deform form, Deform itself runs the resulting form controls through the peppercorn.parse method. This converts the form data into a mapping. The structure markers in the form data indicate the internal structure of the mapping.

For example, if the form submitted had the following data:

1
2
3
4
5
6
7
8
9
  <html>
   ...
    <input type="hidden" name="__start__" value="date:mapping"/>
    <input name="day"/>
    <input name="month"/>
    <input name="year"/>
    <input type="hidden" name="__end__"/>
   ...
  </html>

There would be a date key in the root of the pstruct mapping which held three keys: day, month, and year.

Note

See the Peppercorn documentation for more information about the result of the peppercorn.parse method and how it relates to form control data.

The bits of code that are "closest" to the browser are called "widgets". A chapter about creating widgets exists in this documentation at Writing Your Own Widget.

A widget has a deserialize method. The deserialize method is passed a structure (a pstruct) which is shorthand for "Peppercorn structure". A pstruct might be a string, it might be a mapping, or it might be a sequence, depending on the output of peppercorn.parse related to its schema node against the form control data.

The job of the deserialize method of a widget is to convert the pstruct it receives into a cstruct. A cstruct is a shorthand for "Colander structure". It is often a string, a mapping, or a sequence.

An application eventually wants to deal in types less primitive than strings, such as a model instance or a datetime object. An appstruct is the data that an application using Deform eventually wants to deal in. Therefore once a widget has turned a pstruct into a cstruct, the schema node related to that widget is responsible for converting that cstruct to an appstruct. A schema node possesses its very own deserialize method, which is responsible for accepting a cstruct and returning an appstruct.

Raising Errors During Deserialization

If a widget determines that a pstruct value cannot be converted successfully to a cstruct value during deserialization, it may raise an colander.Invalid exception.

When it raises this exception, it can use the field object as a "scratchpad" to hold on to other data, but it must pass a value attribute to the exception constructor. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import colander

def serialize(self, field, cstruct, readonly=False):
    if cstruct is colander.null:
        cstruct = ''
    confirm = getattr(field, 'confirm', '')
    template = readonly and self.readonly_template or self.template
    return field.renderer(template, field=field, cstruct=cstruct,
                          confirm=confirm, subject=self.subject,
                          confirm_subject=self.confirm_subject,
                          )

def deserialize(self, field, pstruct):
    if pstruct is colander.null:
        return colander.null
    value = pstruct.get('value') or ''
    confirm = pstruct.get('confirm') or ''
    field.confirm = confirm
    if value != confirm:
        raise Invalid(field.schema, self.mismatch_message, value)
    return value

The schema type associated with this widget is expecting a single string as its cstruct. The value passed to the exception constructor raised during the deserialize when value != confirm is used as that cstruct value when the form is re-rendered with error markers. The confirm value is picked off the field value when the form is re-rendered at this time.

Say What?

Q: "So Deform, Colander, and Peppercorn are pretty intertwingled?"

A: "Colander and Peppercorn are unrelated; Deform is effectively
something that integrates Colander and Peppercorn together."