Default

Web Application Development Tools

Home

Overview

Abstract

Writing Common Gateway Interface (CGI) or server side programs in Python either involves using the cgi module or an application framework like Zope. The tools described in this document permit higher level programming than the cgi module without needing to "buy in" to a completely new environment like Zope.

Author: Paul Boddie (paul@boddie.net)

Overview

Currently, the toolkit consists of the following components:

The cgi_wrapper program.
The planning module.
The Storyboard package.
Examples, demonstrations and miscellaneous modules.

Finding Your Level

You might not need to use all the features of the toolkit. This table summarises the applications of certain component configurations:

cgi_wrapper	planning	Storyboard	Forms
Simple, discrete server-side functionality.
Simple, discrete functionality needing to process structured data.
Connected, multiple state functionality with possible structured data requirements.
Connected, multiple state functionality needing output templates.

The cgi_wrapper program is used to invoke functions in appropriately structured Python packages. Follow the instructions in the program's docstring (found at the top of the program text) to configure the program.

The planning module provides support for the transformation of "flattened" HTML form inputs into hierarchies of values. Read the module's docstring for more details; more documentation will be forthcoming.

The Storyboard package provides support for the definition of forms and the transitions between them. Such definitions are then used to determine which action handlers are required, the data types they can accept, and the outcomes that should follow.

The Forms package provides support for the definition of forms, types and transitions within pseudo-XML documents. These definitions can be used to build packages of action handlers which then only require customisation before deployment.

The examples and demonstrations illustrate the workings of the programs and modules. Read the README file to find out how to set your Web server up to run the demonstrations.

Tutorial

The easiest way to try out the toolkit is to embrace all the different modules and write some psuedo-XML form definitions. They look like this:

<xxx-form name='name_example'>

<html>
<head>
<title>Name Example</title>
</head>

<body>

<h1>Name Example</h1>

<xxx-data type='family' location='root' decoration='form'>

  <p>
  Let's have a family name <xxx-var name='family_name' required='1' decoration='input'/>
  and the first names of everyone in the family:
  </p>

  <xxx-collection name='members' class='Member' marker='member_index'>

    <p>
    First name of this person is <xxx-var name='first_name' decoration='input'/> and their
    middle names are:
    </p>

    <ul>

    <xxx-collection name='middle_names' class='Names' marker='name_index'>

      <li><xxx-var name='middle_name' decoration='input'/>
          (Select it: <xxx-selector name='name_select' decoration='checkbox'/> )</li>

    </xxx-collection>

    </ul>

    <p>
    (Select this person: <xxx-selector name='member_select' decoration='checkbox'/> )
    </p>

  </xxx-collection>

  <p>
  What do you want to do now?

  <xxx-function name='add_someone' label='Add another member'>
    <xxx-target-form name='name_example'/>
  </xxx-function>

  <xxx-function name='remove_someone' label='Remove a member'>
    <xxx-target-form name='name_example'/>
  </xxx-function>

  <xxx-function name='add_name' label='Add a name'>
    <xxx-target-form name='name_example'/>
  </xxx-function>

  <xxx-function name='remove_name' label='Remove a name'>
    <xxx-target-form name='name_example'/>
  </xxx-function>

  <xxx-function name='finish_off' label='Finish off'>
  </xxx-function>

  </p>

</xxx-data>

</body>
</html>

</xxx-form>

Examining this document, it is clear that two different types of tags are in use: "normal" HTML tags and "special" <xxx-...> tags. This form definition would not validate as a proper XML document, but if we were to extract the <xxx-...> tags, we would get something very close to a valid XML document, with the remaining HTML also being valid, or very close to it.

Form definitions such as this are actually two merged documents and, if we only consider the <xxx-...> tags as being real, we can consider the HTML document it divides up as "just text". This enables us to manipulate the HTML document, substituting and repeating sections where appropriate, before presenting it as output in an application.

Such documents define a number of different types, each containing a hierarchy of information, along with operations that a particular "application" of a type is involved in. The above definition can be summarised as follows:

Family name: `family_name`
Family members: `members`	First name: `first_name`
Family members: `members`	Middle names: `middle_names`	Middle name: `middle_name`

Forms and Data Types

We always define a xxx-form section in a form definition, so that the software knows the name of the form being defined.

<xxx-form name='name_example'>
...
</xxx-form>

We must also define xxx-data sections where data is being transformed in the form. Inside these sections we find different elements and sections which define what primitive elements a data type is composed from.

<xxx-data type='family' location='root' decoration='form'>
...
</xxx-data>

The location affects the resulting HTML document, indicating where an HTML form is going to direct a request resulting from the submission of the form in the browser. Currently, only the special value 'root' is supported.

The decoration is used to specify that an HTML form is to be produced. Eventually, lots of decorations will be implemented to deal with different kinds of outputs.

Variables

Variables or simple attributes at a particular level in a data type are defined using the xxx-var element.

<xxx-var name='family_name' required='1' decoration='input'/>

The name is used to identify the variable in the level at which the variable is defined.

The decoration attribute of this element is only important for template display purposes, indicating in this case that an HTML input element is to be substituted when the template is prepared for output.

The required attribute dictates whether the software should expect values of a variable at the variable's position in the data type.

Collections

Collections in a data type are defined using the xxx-collection section, which may contain variables or other collections.

<xxx-collection name='members' class='Member' marker='member_index'>
...
</xxx-collection>

This construct indicates that the data type contains, at a particular level, a collection of other things. The collection is identified by the given name at that level.

The class attribute refers to the structure of the "other things" as defined by a Python class which will be built from any xxx-var elements and xxx-collection sections found inside this particular section.

The marker attribute refers to a secret element embedded in template output which is used to reconstruct the structure after an HTML form submission of "flattened" data.

Functions

Although defined in the xxx-data section, xxx-function sections are used to record the action functions that this particular application of the given type will lead to.

<xxx-function name='add_name' label='Add a name'>
  <xxx-target-form name='name_example'/>
</xxx-function>

The name attribute refers to an action function written in Python. The label is for template output preparation purposes.

The xxx-target-form elements indicate the forms which an action will in turn lead to, with their name attributes.

Generating this Application

It is sufficient to generate most of this application using the Forms/make.py program:

python Forms/make.py test_names name_example directory,classes,types,templates,actions \
  Examples/XML/name_example.xml

This specifies that the package test_names will be produced, with the name_example form (given above) starting the "story".

All aspects of the package are to be generated:

The directory used to store the package within.
The classes used to represent the form data type(s) in Python.
The type definitions themselves - used to transform the data in an HTML form to the real data type's hierarchical structure.
The templates used to produce the application output.
The actions which comprise the application itself.

You cannot run this application yet, though, as the actions consist of pre-defined "guesses" made by the Forms/make.py program.

Customising the Application

Looking at the test_names/root.py file, it becomes apparent that the action functions need some work before they will do what they have been advertised to do.

The root Function

The root function is the starting point for the application. Let's just make sure that an empty family definition is initialised.

 	import family

 	family_name = ""
 	members = []

 	return test_names.store.template_repository.name_example(family_name=family_name, members=members)

We do not pass the *_select values to the template (the thing used in the return statement) as they really only contain information of use when an action function receives the family details. We shall see this when considering the other action functions.

The add_someone Function

The add_someone function adds a family member to the list. We only need to add something to the members list in order to achieve this.

 	import family

 	members.append(family.Member(""))

The add_name Function

In the add_name function, we should consider which member to add the name to. In this case, we check the member_select variable - a special "selector" variable used to indicate that parts of the data type's structure have been "selected".

 	import family

 	for i in range(0, len(members)):
 		if str(i) in member_select:
 			members[i].middle_names.append(family.Names(""))
 
 	return test_names.store.template_repository.name_example(family_name=family_name, members=members)

We need to be careful with member_select as it consists of a list of character strings. Therefore, we need to compare the string version of any list index in order to detect whether that index is mentioned in member_select.

The remove_member Function

Here we need to build a new list of members because traversing a list and deleting elements as we go usually results in problems with the resulting list.

 	members2 = []
 	for i in range(0, len(members)):
 		if str(i) in member_select:
 			continue
 
 		members2.append(members[i])
 
 	return test_names.store.template_repository.name_example(family_name=family_name, members=members2)

The remove_name Function

Here we behave similarly to the remove_member function, but descend a level deeper to delete names. We do not need to create a new members list as we are not modifying that list itself, but instead just the contents of its elements.

 	for i in range(0, len(members)):

 		middle_names2 = []
 		for j in range(0, len(members[i].middle_names)):
 			if "%s,%s" % (i, j) in name_select:
 				continue
 
 			middle_names2.append(members[i].middle_names[j])
 
 		members[i].middle_names = middle_names2
 
 	return test_names.store.template_repository.name_example(family_name=family_name, members=members)

Running the Application

Assuming that your Web server is set up to use cgi_wrapper packages at http://localhost/forms/..., try and visit http://localhost/forms/test_names/root to start editing the family names.

Finishing the editing will return a non-existent form. We could invent another form to be displayed in this case, and then add it to the form definition:

<xxx-function name='finish_off' label='Finish off'>
  <xxx-target-form name='end_of_editing'/>
</xxx-function>

Writing a New Storyboard

To write a useful storyboard, first plan the following aspects of your application:

The forms involved, along with the data types they transform.
The transitions from form to form.
The actions which take place in each transition.
How the form will look.

Example

In the Examples/XML directory, three form definitions exist:

a0.xml - the starting screen of a simple application.
b0.xml - the address editing screen.
c0.xml - the finishing screen.

So, taking the information found inside each of these files:

Three forms exist: the first (a) manipulating type x0, the second (b) manipulating type y0, the third (c) manipulating type z0.
Form a leads to form b or itself only through the finish_definition action; otherwise, it leads to itself. Form b leads to form a if the back action is followed; otherwise it leads to form c or form a if the finish_address action is followed. Form c leads to nothing else. (Note that some actions may lead to a choice of forms, so we assume that the action decides which form is chosen when invoked.)
The appearance of the form. Here, HTML tables have been used to give an orderly arrangement of the labels and fields, and care had to be taken to make sure that when collections are "instantiated" and the contents of the collection block are repeated, the resulting output is valid HTML.

The data types would be defined in detail as follows:

Type x0 is a structure containing customer name, site, machine and module details.
Type y0 is a structure containing customer name, site name and site address details.
Type z0 is a structure containing a customer name.

Generating a Package

The Forms/make.py program should be invoked and will attempt to generate a Python package containing the type, template and action information. This will almost produce a working application, but it is rather likely that some specialisation will be required to make the actions process the data given to them and transform that data into other types (structures) in order to produce different outputs.

The test_template package uses the above form definitions and specialises the actions generated from them to produce a working application. Although the HTML elements employed by these forms are quite simple, when they are displayed, they demonstrate the principles behind the editing of data. Moreover, this application manages to provide editing capabilities which many Web applications avoid, due to the supposed complexity of managing the many possible interactions. The code which transforms the structures, for actions like "add" and "delete", is found in Examples/test_library.py and consists of a set of fairly straightforward "tree" manipulation functions. One possibility is to generalise these to work on any structure.

Rationale

When writing applications which deal with HTML forms, it becomes apparent that a number of activities are repeated over and over again:

Acquisition, detection and decoding of form inputs.
Acquisition, detection and decoding of cookie inputs.
Explicit invocation of code depending on the nature of the inputs, specifically the detection of form "actions" and the invocation of different pieces of code which handle those actions.

It seemed that these activities could be handled automatically:

Form and cookie inputs could be acquired automatically.
Activities could be stated in such a way that the inputs they require could be explicitly stated in a function signature, and the code associated with an activity could be encapsulated inside that function.
The form action could be detected and the appropriate function invoked automatically. Thus no explicit effort need be made by the developer to recognise such action invocations, and the developer's code can be organised around distinct activities rather than be a mess of if...elif...else statements.

Rather than writing programs which manually inspect the values of parameters, possibly originating from fields on HTML forms or stated in a URL, which are passed through a Web server directly to such programs as a result of an HTTP request, the cgi_wrapper framework acts as an intermediary between the Web server and the real application software and arranges the necessary input decoding, permitting the Web developer to concentrate on defining the actions in the application concerned with such a request.

Thus, a developer can write a set of functions, defined in a Python package structure, corresponding to the possible actions. Then, the cgi_wrapper framework organises the invocation of such functions according to the nature of any given request.

Function Selection

The way the cgi_wrapper framework decides on the functions to call, and the passing of the parameters in the URL to those functions, is a result of some exposure to Zope and its predecessor, Bobo. However, the selection of a function based on an input (itself representing a form action), as opposed to a path component, is really a result of experiences in form handling. As noted above, it is nice to be able to have the form action influence the called function, rather than be obliged to deal with many actions inside a common function associated solely with a path component.

Forms and Actions

When an HTML form consists of only a few fields and a single "Submit" button, the task of developing a program to interpret these fields is straightforward. There is only one action, and the same behaviour is associated with that action every time it is invoked. However, consider forms where more than one action is available. How is it possible to distinguish between them?

The way to identify a particular form action is to examine the parameters given in a request and look at the values of parameters having the names given to "submit fields" in the form's HTML source. For example:

<input name="confirm" type=submit value="Confirm">
<input name="delete" type=submit value="Delete">

When only one submit field appears in a form, the name and value attributes of the action's field, and the resulting input associated with the parameter of the same name in the request, can be ignored - the CGI (or server-side) program is activated and there is no need to work out what the action means, given that only one thing can possibly happen in that form.

When many actions appear in a form, however, the name and value of a given field become important - the corresponding input in the request will have the value stated in the value attribute if that field caused the request to occur. All other parameters associated with submit fields will be undefined, since they were not involved in causing the request, or action, to occur.

Now, it is obviously possible to write code which explicitly checks for non-empty values in parameters associated with the submit fields, but this makes it difficult or inelegant to structure code around the different actions in a form:

if confirm != None:
	do_confirm()
elif delete != None:
	# Do some delete things.

	another_function()
else:
	# Other things.

	yet_another_function()

However, if functions are invoked according to the presence of an input whose name matches that of a function "published" in a package, the structuring of the code becomes somewhat easier and more elegant:

def confirm(...):
	# Only confirm type things happen in here.
	...

def delete(...):
	# Only delete type things happen in here.
	...

Storyboards

Storyboards are an attempt at raising the level of abstraction when developing systems which lead users through sequences of forms, perhaps using different routes and involving different data types. It is arguably of minimal interest to be concerned initially with the actions required to process form inputs, and in focusing on such details many developers may find that the "big picture" - the flow of data through a user's dialogue with the software - becomes obscured.

Storyboards consist of a number of form definitions, each of which contains the following pieces of information:

The structure of data being passed through a form.
The means of processing the data to be placed into the form.
The means of extracting the data from the form.
The forms which should follow this form in a dialogue with system users, and the names of the actions which should be used to take the user to those forms.

Data Types and Structures

Fortunately, due to the development of the planning module, we already have a means of defining the structure of data to be extracted from a form. We can use this information to define the structure of data to be placed in a form as well.

Placing Data in a Form

Converting the hierarchical data possibly implied by the processing module's data type definitions into a suitable HTML document typically requires some kind of template system. We can use the DocumentTemplate package from Digital Creations or, more specifically, Zope. This has the ability to take hierarchical inputs and, with a suitably written template, produce an "instantiated" form with the data embedded within it.

Extracting Data from a Form

The first stage of extracting data from a form is fortunately done for us by the Web browser, and thanks to the planning module and its usage within the cgi_wrapper program, we already have the ability to arrange the retrieved data into the necessary hierarchical structure expected by action handlers.

Following Forms and Transitions

It is natural to consider a state diagram showing how one form might lead to a number of others in a dialogue with a user. On the transitions between forms, we might make annotations concerning the conditions for following such transitions after leaving a given form, or any kind of activity which might be associated with a particular transition. In the Storyboard implementation, we make a compromise between the purity of such a diagram and the necessity of defining action handlers by requiring that transitions between forms pass through particular action handlers, and these handlers may then direct the user to different forms.

Making Actions from Forms

A most important aspect of the storyboard approach is the generation of action handler function signatures from form definitions. Given the data type processed by a particular form, and the actions which follow it, it is possible to bind particular input types to actions. If, in a different form, the data type processed by that form is different yet supposedly passed to the same action, an error is signalled to indicate that the action cannot be asked to deal with more than one data type.

Another restrictive, but useful, check made when forms are processed is the registration of the forms which actions are stated to lead to. If one form defines a transition to a collection of forms through a particular action, then another form defines a transition to a different collection of forms through the same action, then an error is signalled to indicate that the action cannot be asked to produce different sets of outcomes in different situations. Although one could indicate how an action was invoked, using various form field values (for example), this would complicate the action code unnecessarily and arguably make the system harder to understand.

Details of the Storyboard Implementation

To make a storyboard yourself, take a look at the Storyboard/test.py file, which contains example data type and form definitions. Note that this file contains a main program which will generate the appropriate test directory containing action handler function templates ready for specialisation.

To get an idea of how the Storyboard package processes forms, take a look at the Examples/storyboard.py program. It will take some form definitions and process them, with various results.

Prerequisites

The following things must be available on your system:

A Python interpreter.
A Web server which can execute CGI programs.

Installing the Tools

Download the Web.tar.gz file, saving it as opposed to viewing it in your Web browser.
Decide upon a directory where your CGI executable programs will reside. Unpack the file so that the contents are located in this CGI directory. For example, you might unpack the file in the /home/httpd directory, so that a directory /home/httpd/Web exists, and then you might make this directory a CGI directory by configuring your Web server appropriately.
Make sure that the contents of the directory are really accessible by the Web server user.
Change the first line of cgi_wrapper to suit your system. For example:
```
#!/usr/local/bin/python
```
Note that the usual /usr/bin/env construct doesn't always work with programs run by Web servers, since the environment may not be set up to include the python executable in the PATH.
Read the README file for some hints about configuring your Web server.

Security

Several security aspects need investigating:

Module importing - is it safe enough?
Package access - are package contents protected from accidental Web server publishing?

Status

(August 9th, 2000)