Europython 2004 Day 1

testing track


Software Testing with Python

Software Testing With Python - session chair

magnus lyckå

Software Testing With Python - Magnus Lyckå

started as an electronics tester

{ SQA { Software Verification { Software Testing } } }

"V model" with more levels in practice:

system specification        ->      acceptance test
        |                             ^
        v                             |
 API Specification           ->    integration test
        |                             ^
        v                             |
 detailed design           ->     unit test

XP version:

system specified through | customer tests
           v                   ^
api validated through | continuous integration
           v                   ^
design by writing | programmer tests

Consequences of the XP approach: automated test -> more software to maintain?

unit testing: unit tests are about programmer intent, not specs

test fixture/case/suite/runner

trick in setUp: avoid "global" by doing RESULTS[:] = []

sys.exit(0) problem?

doctest - just mentioned, what about scaling?

Integration testing

http://staf.sourceforge/net/ - IBM framework with python API and other language support

TextTest - tables in web pages?

wxPython testing stub?

ask later - doesn't doctest put pressure against lots of tests?

Unit testing with mock code

Stefan Schwarzer

Unit testing with mock code - Stefan Schwarzer and initial slide

"mock code" is substitute code for production code, which is used by the code under test.

"mock object" is mock code in form of an object

"difficult tests" have hairy test setups like

  • filesystem full
  • net connection dropped
  • mail server can't be reached

typical usage:

  • test code prepares the mock object
  • calls the code under test, passing in the mock object (ie. "connection object")
  • checks the mock object for evidence, signs of usage

example: Logger object, which just appends args to history (and then, just dump the trace for comparison)

StringIO are natural mock objects

Keep mock code simple - "universal" mock objects get difficult to too complicated to maintain (they need their own unittests!)

Try not to depend on the implementation, only the interface (ie. the mock object might pick up operations from lower layers)

Use this when testing is needed, but convention tests are difficult, and a mock object works well; if it does require modifications to production code to support it, make sure they aren't bad for maintainability.

Magnus Lyckå comment: this shows why you should write tests first - so the code is inherently testable...

Towards a Better Test Runner

Marius Gedminas - Programmers of Vilnius

Towards a Better Test Runner - Marius Gedminas and initial slide

Yay - green-on-black slides!


SchoolTool test runner, inspired by the one in Zope 3; test runners get complicated as the tree gets complicated


  • find all modules automatically
  • allow you to run a subset
  • show progress indication

Also, automate various kinds of analysis to make life easier.

Conventions: or tests/test*.py in every package.

test_suite function, returns a TestSuite.

Helper - warn about TestFoo classes that aren't listed

Filtering - you want to be able to run them quickly to check if a failure was fixed.

  • regex on filenames (avoids some slow imports)
  • regex on class names (more granular but doesn't avoid the imports)
  • test levels?


  • can do code coverage analysis (using builtin python trace)
  • detect common mistakes (leftover prints, global changes)
  • find slowest tests (zope3, not SchoolTool)
  • find memory leaks (zope3, not SchoolTool)


std: a complementary second standard library

Holger Krekel

Much of the standard lib, in particular unittest, are java-like, not pythonic

Standard lib exposes too much in terms of implementation details, leading to things like urllib vs. urllib2

"The best API is one that doesn't exist" (riffing on Raskin's "best UI" comments)

Innumerable incompatible extensions around to do the bookkeeping of testing, many of which are larger than the module itself.

No standard way to run the tests for an arbitrary app.

std - Holger Krekel and output


  • use existing assert statement
  • but magically reinterpret it to get more data than just "Failed"
  • shows the source of the test, exactly the line of failure, and a reinterpretation of the statement! (std.magic)
  • raises(WhatToCatch, "expr")

Does the last (.pop()) case in actually show the wrong thing, namely the context after the test?

Looks for test_*.py, test_* functions, in the module.

"restartable iterators", called "collectors" - for walking file/directory trees... collect functions and classes within them... (Do I want to use this for tree walking for stufflog?)

Collector/Runner/Reporter model; tests just start.

utest.conf simply in the namespace for options

std has a uniform access method for subversion repositories (svn-url and svn-working, not just filesystems.

Also has more explicit namespace control (std.initpkg)

(Armin Rigo is the other developer)

1.0 release real soon

svn co

TextTest - a test framework

Johan Andersson, Geoff Bache - Carmen Systems AB -

TextTest - a test framework - Johan Andersson

Acceptance Testing

Carmen does "development planning systems" for airline and railway industries, 190 employees; TextTest regression tests a number of their products.

Tests drive development, absolutely.

Focussed on "customer approval" tests (vs. developer intent)

Started out as a highly product-specific shell script using grep; became generic python, and eventually went on sourceforge.

Has both interactive and batch reporting tools

output, errors, and performance files - explicitly measures expected CPU time consumed!

PyGTK GUIs -- GUI selftest being presented at XP2004

"diagnostic debugging" practice (from XP?) is handled by additional tools, that go beyond the "it failed" defaults.

The actual end users don't deal with the text themselves, they have viewer apps and such.

Testing Panel Discussion

Magnus Lyckå Stefan Schwarzer Marius Gedminas Johan Andersson Holger Krekel plus Stefan Holek - zope2 testing tool author

Testing Panel Discussion - panel members

Testing Panel Discussion - panel members- 1

Johan Andersson: "Not writing unit test code saves lots of time" - because they could automate the acceptance tests externally. It loses some of the benefits of unit tests, but they get back some of the benefit by additional practices: 'usage first', you write the caller first (sort of like test-first), and 'diagnostic debugging' - write more detailed trace statements for suspicious areas. (40 hour tests are performance tests from real customer demands.)

Holger Krekel: we have lots of different demands for testing, PyPy sees it very differently. Maybe there's a way to come up with a common ground so we can share tools?

Mock objects: sqlite "memory", zodb (zope2) - DemoStorage, cancel transactions; avoids having to duplicate all of Zope2 as mock objects, which need their own unit tests...

Audience question: how do you test the Infinite Filing cabinet...

Why is there so much more testing interest? More systems are around, have aged more, become complex, and need self test...

Quixote - web applications


Quixote - web applications - Harald Armin Masson and initial slide

Harald Armin Masson (did the "why not to tell people about python" humour talk last year)

  • training and process development - EU600/day + expenses

"program based"

  • 3 of them "CherryPy" style (ie. embed python in HTML text)

Quixote - done for Mems Exchange

  • logic, not templates
  • no magic
  • only 7000 lines of code, implemented in a week or two
  • PTL, but mostly URLs == module paths
  • can do XML-RPC too
  • Triggered IE bugs by being to fast (but a service pack fixed them)
  • subsumed medusa/asyncore; works with twisted
  • preferred use: mod_scgi + apache
  • mod_python works, but doesn't help (is a pain to configure)
  • is slow, but it reloads everything always, which is a win for development

Quixote - web applications - Harald Armin Masson and sample code

audience: suggest cheetah, or at worst ZPT for designers

response: they wanted to use Word (not even DreamWeaver) so he fed Word's HTML output to mxTidy and extracted it from there.

Real World Email


Real World Email - Anders Hamerquist

Anders Hamerquist

"CAPS" system that Alex Martelli talks about. Handles 100k messages (over what timescale/)

Problems all boil down to MIME - charset, but other things too.

Anders coped well with not having a working projector, writing mime header encoding examples on the chalkboard...

Real World Email - Anders Hamerquist- 3

RFC2231 - encoding in structured fields, like mime-content-filename (outlook botches this, but the hack of setting the obsolete Content-type name= field, in addition to content-disposition, seems to work.)

PyKI - Public Key Infrstructure Development Framework


PyKI - Public Key Infrstructure Development Framework - Richard Zoni and initial slide

Richard Zoni Piero Ribichini

PyKI - Public Key Infrstructure Development Framework - Piero Ribichini

Context: X.509 certs, sigs, PKI, ASN.1, PKCS, LDAP

PyKI - Public Key Infrstructure Development Framework - Richard Zoni- 1

Goals: framework for PKI and PKI-enabled apps

Python: for rapid prototyping, but ended up in the final product, since performance from libs was sufficient (and the slow part is the smartcard)

Bignum Ext on GMP, crypto EXT, CryptoKI pkcs#11 drivers, PyQT

ASN.1 engine - Maps ASN.1 objs to Python Classes

"Crypto Pipes" - Buffer/Feeder/Eater, [rsa.rsa, pkcs1_5.pad]

timestamping, TLS, then their apps on top

ASN1 modules - manually generated from spec, since they need to be enriched with live methods too:

class Cert(ASN1.Sequence):
   def model(self):
      return [ foo("fooName"), ... ]

Signo Professional - 20,000 users, mostly because it was adopted by Italian CA, bundled with a USB token.

Servers, Dev; also SIGNO SecureLogOn - smartcard logon for Linux.

Open Source release of the core tech in Q3-2004

SIGNO-ETSI support ("better" signatures?)


Steve Alexander

SchoolTool - Steve Alexander and slide- 1

Funded by The Shuttleworth Foundation (in South Africa; founded to do risky things, that benefit SA and countries like it, not first world.)

Focussed very narrowly on school administration (Mark Shuttleworth, fownder of Thawte; keynote is tomorrow) not on delivering courseware etc.

Surveys indicated that a GUI was important, but then schools have gone off and decided that web based apps were the big thing...

Real concrete project, test driven development - not theoretical or abstract, involves school people.

A couple of schools have tried it and are going further, but no major adoptions yet. (There is funding for this.) As an open source platform, it is intended to force interopration, in the first world.

Third world countries don't need software per se; they need management tools to let them know how better to use the resources they have.

Development has bounties for funding (google "here be bounties" :-)

Models classes, departments, groups; students, teachers; attendance; bookable resources (rooms, equipment); allows for changing roles.

Milestone 5 released; fairly raw demo.

(Think about using this for IAP scheduling?)

Needs usability testing, or rather, tight development with actual educators in the loop.

There are debian packages and an apt-source.

A Case Study from Bristol UK


Chris Withers (Noted UK Python/zope consultant)

A Case Study from Bristol, UK - Chris Withers

X2Y Document Conversion Server (not ready yet)

University of Bristol print services department (thesis printing, etc.)

Lots of request forms - paper, incorrectly filled in forms, no audit trail, phone in status checked. Too much word-of-mouth; more effort on tracking than working.

Goal: paperless job management. ePrint Direct (client/server commercial system) looked good at first. It's actually a windows printer driver, for submission. Problems:

  • windows only
  • license per client!
  • still end up with job-sheets inside the facility

Proposal: user - web/email, print services - web (zope), x2y - talks http to zope. uses reportlab tools. Standardizing on PDF helps this...

Live run against the real server!

(Common problem - file attachment gets reset on reload with errors.)

X2Y itself - currently a script that just polls a URL. Moving to Twisted, as an actual server. XML-RPC, not because it is any good, but it is popular for integration. Email-in as well.

Swapped in "EasyPDFCreator" COM object, instead of openoffice. (the workflow framework may be adaptable)

dLCMS - dynamic Learning Content Management System

Samuel Schluep
(speaker sounds like a young Peter Lorre :-) Institute for Hygiene and Applied Physiology.

dynamic Learning Content Management System - Samuel Schluep and initial slide- 1

Very little reuse in classic "e-learning" content.

"Learning Objects" - small, modular, reusable, durable (10 years or longer lifetime)

Standardization - metadata, packaging

This project uses XML directly.

No existing LCMS was accessible (or affordable, even for ETHZ) so based one on Silva/Zope.

Example: "Gesundheitsgefahren bei Lärm"

Working towards standardized packaging techniques (XML in ZIP seems to work?)

Track Lead Comments

Looking for a track chair for an education track...

Keynote - Guido van Rossum

293 people, not just 245, wow.

Keynote - Guido van Rossum

"Go have a beer with your friends, that's one of the more important
parts of conferences"

Rumenon(?) - far end of student union - for Mark Shuttleworth's keynote.

Anna - the redhead :-) - lightning talks

Python 2.2 is "resting" ("pining for the fjords"), probably not getting bug fixes.

Python 2.3.4 just came out, 2.3.5 due later this year (Anthony Baxter, Guido is out of the loop)

Python 2.4 alpha 1 released soon, final release Q4-2004.

Keynote - Guido van Rossum- 6

Generator Expressions: sum(x**2 for x in range(100))

  • makes analogy to unix pipes

  • intended to encourage people to use sum() because you don't have the expense of memory you'd have in a list comprehension

  • warning that they have values, and are late-bound:

    for f in functions: a.append(f(x) for x in range(10))

    yields 3 identical generators - "don't do that", it is convoluted, and not what they're for. genexps are intended to be immediate and not delayed.

Decorators: replace staticmethod() syntax with a def syntax...

  • decorators could add "doctest"-like things, but Tim Peters doesn't think so...
  • very useful for cross-language bindings
  • abuse it for adding things to the function dict...

One of:

def func(args) [decorators]: body

def func(args): body

Either way, it's syntactically a list, but not a list comprehension, and never actually instantiates the list.

Guido likes the "but I can just grep for ..." argument in syntax judgement, in particular def foo etc.

Guido quote: "I would say... don't do that." (in regard to excess blank space between the decorator list and the def.)

Voting: yes we want it, we want it right not now...

Before def, after arg list, anything else: the latter got the most hands, the first two about the same size but clearly less than the third.

Post-keynote language ranting - Guido van Rossum- 1

Future: there will not be a python 2.10 (2.5..2.9 may parallel 2.0)

"3.0 won't have radically different syntax - I'll leave that to people with more linguistic background." --GvR (subtle dig at Larry Wall?)

Python 3.0 will make many things return iterators instead of lists.


The lecture hall doesn't have wireless, doesn't have power at the seats, but does have an air-conditioning vent at every seat!