Python notes 6/28/07

Summary

This month I've written a simple component programming system similar to what I did for Scientist's Workbench and Scientist's Component Toolkit, although both of those systems were written in C++ rather than Python. If it survives, this new system can be the basis for Python Workbench. At present, there is no graphical interface for creating components or hooking them together as there was in SWB and SCT, although you can create component GUIs that both draw in windows and take their input from graphical controls. The component programming interface is a purely textual one, which has both disadvantages from and advantages over SWB and SCT, as will become clear in the examples below. Nevertheless, the use of the current API does not preclude the addition of a graphical layout system sometime in the future.

Required software

In all graphics examples, I have decided to use PyQt 4.2, as that seems to work well and I seem to be able to understand it. After using so many different but similar event-driven graphics systems, I find this kind of programming to be fairly intuitive. Therefore, the current software is built using the following required pieces:

Python 2.5.1
Numpy 1.0.3
Pyfits 1.1
Qt 4.2 Open Source, SIP 4.6, and PyQt 4.2

Component architecture

In the following, a 'component program' is defined as a system possessing the following characteristics:

A set of software components (defined below).
A set of connections between those components.
An initial state for all components.
A way for each component to update its state from step s to step s+1. This updating may involve significant 'side effects' such as reading and writing files, getting user input from graphical controls, or sending graphics output to windows and plots. In fact, the bulk of what the program is designed to do may be in the form of these side effects.
A way to termintate the step iterations.

This paradigm encompasses a vast variety of potential programs, including initial value problems, simulations, and data processing pipelines. However, as is demonstrated below, this paradigm can also be used to build most of the interactive GUI programs that we might want to use.

A 'component' is defined as a piece of software (data and code) possessing the following characteristics:

An internal state.
A way to initialize and reset that state.
A way to describe its state in a human-readable format.
A way to transform the state at step s to a state at step s+1, possibly performing the action of one or more 'side effects' such as reading and writing files, getting user input from graphical controls, or sending graphics output to windows and plots.
A set of connections from and to other components.
A way to send data to other components, and to receive data from other components.

A prototype implementation of these ideas is in the file pwb.py. In this case, a component is a Python class having some instance variables including inputs and outputs. These are lists of other components, and, in the case of inputs, a data queue and intermediate buffer for each input. As an aside, I would like to point out that code editors like XCode (and others) provide syntax coloring that makes Python code much easier to read:

A simple component program

The bulk of the component functionality exists in just a few class functions:

__init__:  Create and initialize component variables.

     run:  Perform component operation.  Get input data, process, send outputs.  Read/write files, 
           interact with graphics system, etc...

describe:  Print an informative message about the component and its variables.

    done:  Decide if the component is done doing its thing.

   reset:  Reset component variables to initial state.

The basic Component class provides defaults for these and other class functions, so it is up to the developer of a derived component to provide overloads only for those functions which will perform non-default behavior. Most of the work will be done in the 'run' function, as this is where input data will be obtained, processing performed, and output data sent. In the default case, the 'run' function just passes any input data along to its outputs. This behavior can be used to demonstrate a simple component program, which can be entered from the interactive Python command line:

(Start Python)

23: ~/Projects/Python/pwb > python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) 
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

(Import pwb module.  Use 'from pwb import *' so that names in module
can be used without prepending 'pwb.')

>>> from pwb import *

(Create some components)

>>> a = Component('a')
>>> b = Component('b')
>>> c = Component('c')

(Connect components in loop)

>>> connect(a, b)
>>> connect(b, c)
>>> connect(c, a)

(Prime loop with data sent from c to a)

>>> c.send(0, 'foo')
>>> a.update()

(Print out current state of components)

>>> describe()
---
name = a
inputs:
c ['foo'] []
outputs:
b
---
name = b
inputs:
a [] []
outputs:
c
---
name = c
inputs:
b [] []
outputs:
a

(Run 10 steps)

>>> run(10)
---
Component.run(): a
a got 'foo' from c
Component.run(): b
Component.run(): c
---
Component.run(): a
Component.run(): b
b got 'foo' from a
Component.run(): c
---
Component.run(): a
Component.run(): b
Component.run(): c
c got 'foo' from b
---
Component.run(): a
a got 'foo' from c
Component.run(): b
Component.run(): c
---
Component.run(): a
Component.run(): b
b got 'foo' from a
Component.run(): c
---
Component.run(): a
Component.run(): b
Component.run(): c
c got 'foo' from b
---
Component.run(): a
a got 'foo' from c
Component.run(): b
Component.run(): c
---
Component.run(): a
Component.run(): b
b got 'foo' from a
Component.run(): c
---
Component.run(): a
Component.run(): b
Component.run(): c
c got 'foo' from b
---
Component.run(): a
a got 'foo' from c
Component.run(): b
Component.run(): c
>>>

This example simply passes data around from component to component in a loop. There is no stopping condition. The data could be anything: strings, numbers, arrays, images, or more complicated types. A component wishing to make use of polymorphic input data must perform checks to determine the actual type of the data, and call the appropriate functions to process it.

More component programs

For a finite program, try the following:

(Import components module (described below))

>>> from components import *

(Clear component list)

>>> pwb.clist = []

(Create a producer and consumer component)

>>> signal = Signal('signal', range(10))
>>> display = Display('display')
>>> connect(signal, display)
>>> describe()
---
name = signal
inputs:
outputs:
display
data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
data2 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
---
name = display
inputs:
signal [] []
outputs:
data = []

(Run to completion)

>>> run(-1)
---
---
display got 0 from signal
---
display got 1 from signal
---
display got 2 from signal
---
display got 3 from signal
---
display got 4 from signal
---
display got 5 from signal
---
display got 6 from signal
---
display got 7 from signal
---
display got 8 from signal
---
display got 9 from signal
>>> describe()
---
name = signal
inputs:
outputs:
display
data = []
data2 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
---
name = display
inputs:
signal [] []
outputs:
data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

(Reset components)

>>> reset()
>>> describe()
---
name = signal
inputs:
outputs:
display
data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
data2 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
---
name = display
inputs:
signal [] []
outputs:
data = []
>>>

The components.py file contains several component classes which are derived from Component, but which perform more specialized functions by overloading some of the Component class functions. The Signal component is initialized with a list of data values, and sends one of those values to all of its outputs at each step until are all gone. The Display component accumulates the values it receives from its inputs into a list. It runs until there is no more data in any of its input queues.

Here is an example of adding two sets of numbers and saving the result:

>>> pwb.clist = []
>>> c1 = Signal('signal 1', range(10))
>>> c2 = Signal('signal 2', range(10, 20))
>>> c3 = Add('add')
>>> c4 = Display('display')
>>> connect(c1, c3)
>>> connect(c2, c3)
>>> connect(c3, c4)
>>> run(-1)
---
---
add got 0 from signal 1
add got 10 from signal 2
---
add got 1 from signal 1
add got 11 from signal 2
display got 10 from add
---
add got 2 from signal 1
add got 12 from signal 2
display got 12 from add
---
add got 3 from signal 1
add got 13 from signal 2
display got 14 from add
---
add got 4 from signal 1
add got 14 from signal 2
display got 16 from add
---
add got 5 from signal 1
add got 15 from signal 2
display got 18 from add
---
add got 6 from signal 1
add got 16 from signal 2
display got 20 from add
---
add got 7 from signal 1
add got 17 from signal 2
display got 22 from add
---
add got 8 from signal 1
add got 18 from signal 2
display got 24 from add
---
add got 9 from signal 1
add got 19 from signal 2
display got 26 from add
---
display got 28 from add
>>>

This program works even if the connections from the signals to the adder are of different lengths, although then there is a corresponding delay in the final output to the display:

>>> pwb.clist = []
>>> c1 = Signal('signal 1', range(10))
>>> c2 = Signal('signal 2', range(10, 20))
>>> c3 = Add('add 1')
>>> c4 = Add('add 2')
>>> c5 = Add('add 3')
>>> c6 = Display('display')
>>> connect(c1, c5)
>>> connect(c2, c3)
>>> connect(c3, c4)
>>> connect(c4, c5)
>>> connect(c5, c6)
>>> run(-1)
---
---
add 1 got 10 from signal 2
---
add 1 got 11 from signal 2
add 2 got 10 from add 1
---
add 1 got 12 from signal 2
add 2 got 11 from add 1
add 3 got 0 from signal 1
add 3 got 10 from add 2
---
add 1 got 13 from signal 2
add 2 got 12 from add 1
add 3 got 1 from signal 1
add 3 got 11 from add 2
display got 10 from add 3
---
add 1 got 14 from signal 2
add 2 got 13 from add 1
add 3 got 2 from signal 1
add 3 got 12 from add 2
display got 12 from add 3
---
add 1 got 15 from signal 2
add 2 got 14 from add 1
add 3 got 3 from signal 1
add 3 got 13 from add 2
display got 14 from add 3
---
add 1 got 16 from signal 2
add 2 got 15 from add 1
add 3 got 4 from signal 1
add 3 got 14 from add 2
display got 16 from add 3
---
add 1 got 17 from signal 2
add 2 got 16 from add 1
add 3 got 5 from signal 1
add 3 got 15 from add 2
display got 18 from add 3
---
add 1 got 18 from signal 2
add 2 got 17 from add 1
add 3 got 6 from signal 1
add 3 got 16 from add 2
display got 20 from add 3
---
add 1 got 19 from signal 2
add 2 got 18 from add 1
add 3 got 7 from signal 1
add 3 got 17 from add 2
display got 22 from add 3
---
add 2 got 19 from add 1
add 3 got 8 from signal 1
add 3 got 18 from add 2
display got 24 from add 3
---
add 3 got 9 from signal 1
add 3 got 19 from add 2
display got 26 from add 3
---
display got 28 from add 3
>>>

Here is a textual description of these connections:

>>> describe()
---
name = signal 1
inputs:
outputs:
add 3
data = []
data2 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
---
name = signal 2
inputs:
outputs:
add 1
data = []
data2 = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
---
name = add 1
inputs:
signal 2 [] []
outputs:
add 2
---
name = add 2
inputs:
add 1 [] []
outputs:
add 3
---
name = add 3
inputs:
signal 1 [] []
add 2 [] []
outputs:
display
---
name = display
inputs:
add 3 [] []
outputs:
data = [10, 12, 14, 16, 18, 20, 22, 24, 26, 28]
>>>

Here is a visual description of the same connections:

This demonstrates that even simple component connections are difficult to visualize from text alone.

Feedback loops

Components can be connected to themselves, either directly or via other intermediate components. However, in these cases care must be taken to ensure that (a) component input requirements are satisfied sufficiently for the program to do anything at all, and (b) stopping conditions are also satisfied to prevent infinite loops (although this in itself is not an error condition, and is not prohibited). For example, consider the following simple connections:

>>> pwb.clist = []
>>> c1 = Signal('signal', range(10))
>>> c2 = Add('add')
>>> c3 = Display('display')
>>> connect(c1, c2)
>>> connect(c2, c2)
>>> connect(c2, c3)
>>> run(10)
---
---
---
---
---
---
---
---
---
---
>>> describe()
---
name = signal
inputs:
outputs:
add
data = []
data2 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
---
name = add
inputs:
signal [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] []
add [] []
outputs:
add
display
---
name = display
inputs:
add [] []
outputs:
data = []
>>>

In this case, nothing happens, because the adder is waiting for data on all of its inputs before it performs any function and sends out any results. So, the signal data just queues up in one of the adder's inputs, while it is waiting for data from itself on the other input. One way around this is to use the Add2 component from components.py, which does not require data on all inputs before it does its thing. It will add all available inputs and send them out, as long as there is data on at least one input:

>>> pwb.clist = []
>>> c1 = Signal('signal', range(10))
>>> c2 = Add2('add2')
>>> c3 = Display('display')
>>> connect(c1, c2)
>>> connect(c2, c2)
>>> connect(c2, c3)
>>> run(15)
---
---
add2 got 0 from signal
---
add2 got 1 from signal
add2 got 0 from add2
display got 0 from add2
---
add2 got 2 from signal
add2 got 1 from add2
display got 1 from add2
---
add2 got 3 from signal
add2 got 3 from add2
display got 3 from add2
---
add2 got 4 from signal
add2 got 6 from add2
display got 6 from add2
---
add2 got 5 from signal
add2 got 10 from add2
display got 10 from add2
---
add2 got 6 from signal
add2 got 15 from add2
display got 15 from add2
---
add2 got 7 from signal
add2 got 21 from add2
display got 21 from add2
---
add2 got 8 from signal
add2 got 28 from add2
display got 28 from add2
---
add2 got 9 from signal
add2 got 36 from add2
display got 36 from add2
---
add2 got 45 from add2
display got 45 from add2
---
add2 got 45 from add2
display got 45 from add2
---
add2 got 45 from add2
display got 45 from add2
---
add2 got 45 from add2
display got 45 from add2
>>> describe()
---
name = signal
inputs:
outputs:
add2
data = []
data2 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
---
name = add2
inputs:
signal [] []
add2 [45] []
outputs:
add2
display
---
name = display
inputs:
add2 [45] []
outputs:
data = [0, 1, 3, 6, 10, 15, 21, 28, 36, 45, 45, 45, 45]
>>>

Now the adder continues to add and send its feedback result even after the signal component has run out of data. In this case, the program must have some other means of stopping, since the adder will always have data to process. Some possible stopping methods will be demonstrated below, when infinite looping programs are considered.

Plotting data

PWB components can send data to Qt graphics objects such as plot windows. Qt can be run from an interactive Python session without blocking the interpreter by (a) periodically calling Qt to process events (like drawing), but (b) never calling the QApplication.exec_ main event loop (which never exits, and blocks further execution of the interpreter). The performance is not nearly as good as running a stand-alone PyQt program, but it does make it possible to create plots and GUI controls on-the-fly from Python commands entered by hand. For example:

29: ~/Projects/Python/pwb > python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) 
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pwb import *
>>> from components import *
>>> from numpy import *
>>> c1 = Signal('signal', list(sin(arange(100) / 99.0 * 2.0 * pi)))
>>> c2 = Display2('display2')
>>>

At this point, a blank plot window will appear:

>>> connect(c1, c2)
>>> run(-1)
---
---
display2 got 0.0 from signal
---
display2 got 0.0634665182543 from signal
---
display2 got 0.126933036509 from signal
---
etc...
---
display2 got -0.0634239196566 from signal
---
display2 got -2.44921270764e-16 from signal
>>>

At this point you can reset the components and then step through the entire run, one point at a time:

>>> reset()
>>> run(1)
---
>>> run(1)
---
display2 got 0.0 from signal
>>> run(1)
---
display2 got 0.0634239196566 from signal
>>> etc...

Or you can construct a loop in which each step is followed by a short delay so that the plot appears more slowly:

>>> reset()
>>> import time
>>> while not done():
...     run(1)
...     time.sleep(0.1)
... 
---
---
display2 got 0.0 from signal
---
display2 got 0.0634239196566 from signal
etc...

The PlotWindow class, which is used by the Display2 component, is defined in the file plots.py. You can use this plot type directly from Python as well:

>>> from PyQt4 import QtCore, QtGui
>>> from numpy import *
>>> from plots import *
>>> app = QtGui.QApplication([])
>>> data = sin(arange(1000) / 999.0 * 8.0 * pi)
>>> window = PlotWindow('A plot', data)
>>> window.show()
>>>

Page 2 -->

�Sky Coyote 2007