Europython 2004 Day 3

"social skills track"

Database Programming with Python

Database Programming with Python - Magnus Lyckå (using chalkboard instead)

Magnus Lyckå

(Started late due to projector problems) [eichin:20040609T0923+02]

Axes: standalone database server vs. embedded in process

There are still a lot of quirks in the python db apis (we knew this.)

import mx.ODBC.Windows as dbi is a reasonable use of aliasing.

No embedding, explicit-strings in an execute() method; no consistent substitution-syntax.

dbi-style interfaces don't default to autocommit.

There are standardized exceptions.

A Graph-Based Data Store

A Graph-Based Data Store - Dr. Duncan Grisby

[eichin:20040609T0934+02] Dr. Duncan Grisby

Tideway Systems: software to help large orgs undestand and manage their IT application infrastructure; venture funded, hiring in London.

Many node types ("apache", "workstation", "Fred"), complex data in each node, highly interconnected, cross-platform access to data; also needs to support distributed transactions.

SQL sucks for graph traversal - complex to write, slow to execute (there are higher level tools to build it, but the results are hard to maintain.)

Berkeley DB - needs commercial license for what they're doing (which they're ok with.)

Can store multiple tables in a file; can use it in a mode where multiple stores lead to multiple values per key; can do "find keys beginning with this substring"... also supports some kind of table cross-linking. Alternative storage modes (btree, for example) are very useful for performance (locality) tuning.

Each "node" is a CORBA object - gives universal access, can pass object references around directly. Each node state is also a set of key-value (string/CORBA-ANY) pairs which also lets other languages look at it. They can be manipulated in sets; full text search, and other APIs, are implemented on top.

Node is pretty obvious, but also has get_edges. Their edge-storage model is proprietary, though.

(My question:) Locking is done at the Berkeley DB layer, though they also have a higher-level transaction layer.

CORBA "default servant" python object representing all nodes. (Effectively a cursor?)

A Graph-Based Data Store - Implementation slide (and arm)

Keys are id + "kind" (single char tag for the kind of value); values are CORBA self-describing syntax. For example:

NNNt  Host
NNNk  Hostname,ip,os
NNN_hostname  fred

etc. Since they're near each other lexicographically, they have locality in caching too, because of the use of btree (ie. intentionally.)

Tim Bray's page was useful for figuring out how to index.

BerkDB transactions - 2PC, pessimistic locking, may raise deadlock exception so you have to have retry handlers.

DB cache size is important - default is 256K, 512M works better :-)

Conclusions: relational isn't the right solution for some things; Berk DB and CORBA are good building blocks; Python is a great language for this kind of thing.

Ldaptor - pure python LDAP servers and clients

Ldaptor - pure python LDAP servers and clients - Tommi Virtanen

Tommi Virtanen

(Had trouble with the mac - drove the display but only gave an aqua screen, possible resolution problem... ended up ethernet-sharing with an RH9 laptop.)

17K lines of python, including docs and examples - probably the largest Twisted app ever.

Has simple interface for Distinguished Name objects.

Ldaptor - pure python LDAP servers and clients - ldaptor overview slide

Pure python - "no segfaults, unlike OpenLDAP" :-) Also stylistically pythonic, not wrapped-C-API.

Stringized result objects are LDIF.

Uses Nevow to build a web app to do LDAP lookups from a form (see slides on the Wiki)

See LDAP lightning talk later "grok LDAP fast".

"If you're doing networking with python at all, You must learn Twisted. It took about 5 hours to learn the essentials."


PyFx - Lennart Ohlsson and initial slide

Lennart Ohlsson - Lund University

Interface for exploiting the kind of things modern graphics cards (Programmable GPUs in particular) can do, shading and reflectivity tricks, volumetric 3d.

Shaders and such are simple, applications integration is hard and messy.

Existing frameworks (CgFX, DirectX Effects) built around the "Effect" abstraction, but still an insufficient mini-language.

PyFX - use python to actually specify the effects, instead of an adhoc minilanguage. Framework as well. Currently on PyOpenGL and SWIG of Cg for the runtime. Simple effects look a lot like CgFX.

Generic Application Interface - a simple rendering loop.

GPGPU - using the GPU as a number cruncher. (Hypnotic swirl effects, but it isn't clear how much it really needs the GPU)

(see Wiki for references, later)

The MEMOPS Programming Framework

The MEMOPS Programming Framework - Wayne Boucher, being introduced

[eichin:20040609T1146+02] (previous talk ran late)

Wayne Boucher, Cambridge

CCPN: Collaborate Computing Project for NMR MEMOPS: MEtaMOdelling Programming System

10% of the audience of X-ray crystallography, so there are fewer tools and more of them are adhoc and proprietary.

Standardized on a data model rather than format because it describes the domain, and is more accessible to other languages and tools. "Model Driven Architecture", using UML and hardcore tools (Object Domain, in particular, because it uses Python for scripting, after starting with Rational Rose - which uses VB - "I lasted about three hours before giving up".)

The MEMOPS Programming Framework - UML example (horror) slide

The UML screenshot really doesn't look very pictorial...

UML usage further constrainted by XML ("parent/child") and SQL (by having keys - though apparently their SQL users said they wanted integers instead...)

Initial scripts are directly pulled down from UML, but most people work directly with the model. (Couldn't use XMI, because it doesn't do placement, just structure.)

Over 300 classes, 2300 metaobjects

Currently around 370k lines of python (600k in java!) in 22 packages.

They do UML models of new applications too, sometimes. Primary existing applications are conversion scripts.

Licensing for the generated code is LGPL, scripts will probably be GPL.

Pythonic Alternatives for declarative XML-based specifications of robot behaviour

Pythonic Alternatives for declarative XML-based specifications of robot behaviour - intro slide

[eichin:20040609T1210+02] Theo de Ridder

SE-RTES - Software Engineering for RealTime Embedded Systems

Pythonic Alternatives for declarative XML-based specifications of robot behaviour - Theo de Ridder

His experience: Lisp in the 60's, Simula in the 70's, etc.

Hates UML - because it just shows attributes, which are "silly things", and you don't see Responsibilities or Performance. "If a child of 6 years gave you a drawing like that, you'd think he had a handicap..."

Tensions/axes: generic/specific, textual/graphical, physical/virtual

AIBO - 576mhz MIPS R7000, 64M, 802.11b, etc. "Very rich" for programming. - Dutch Aibo robocup league

XABSL - XML Agent Behavior Specification Language, used by the German Team to win in 2003: 250kloc, lots of tooling, modular architecture. (huge system.) agents, options, symbols, basic behaviors.

Basic behaviours in C++, strategic behaviour declared in XML, XSLT generated IC from that. 16kloc of XML, low signal/noise ratio, clumsy and error prone - so, reconsider using python, since it's language after all.

Pythonic Alternatives for declarative XML-based specifications of robot behaviour - Tagged Hierarchies example slide

Tagged Hierarchy syntax. Map ":" to "__", xi__include(href=...) for example. Since you pass in an object for the "tree" base, you can do arbitrary things with it on output...

Basically, hairy support structure but clear python resulting operation code. Complete recursive state machine implemented with changing-base-classes?

X3D syntax examples - overloaded ~ and using it as a sort of macro lead-in marker.

He wants a very minimal RTOS that "only runs python", along with a minimal interpreter for embedded use.

Also wants a 3d blender/world editor that is live, no attribute editors, so you can interact on either side (python code or images.)

questions: (? university 3d programming-teaching project in python?)

Lightning Talks

Anna to-be-Martelli track lead. 5 minutes to talk, she even has a hooked pole...

Lightning Talks - ftputil: a high level FTP client library - Stefan Schwarzer

Stefan Schwarzer - ftputil: a high level FTP client library

Side track: Pet Peeves survey:

  1. Spelling
  2. Projectors don't work
  3. Talking to the Wall
  4. Reading Slides
  5. Mumbling
  6. Small text
  7. Speaking too quietly
  8. Broken Python Code
  9. Typing code in really slowly in front of the audience
  10. Deleting source code before running the demo
  11. Launching *.py with JAVA
  12. Showing URLs for 2 seconds

Lightning Talks - Eunuchs - Missing manly parts of UNIX API for Python - Tommi Virtanen and slide

Tommi Virtanen Eunuchs - Missing manly parts of UNIX API for Python

  • C/Pyrex bits, cover stdlib missing things
  • have them now, not wait for new release: tuntab, fchdir, sendmsg
  • low level, dirty, absolutely necessary, "real systems programming"
  • Guido: "There's a 2 year old PEP on how to add new libraries, I don't know what's up with it either"
  • send me other small ones as patches.
  1. Not knowing where the lights are
  2. Poor timekeeping
  3. Contrast
  4. Transitions
  5. Repetition
  6. No audience participation
  7. Excess audience participation
  8. Avoiding questions
  9. Not Knowing subject
  10. Door slamming
  11. Relying on connectivity

Lightning Talks - Astrazenica and CLAB - the team lead from Astrazenica (a sponsor?) talking about their python efforts

? Astrazenica(?) (one of the sponsors): CLAB

  • pharmaceutical company, 60k employees, R&D is 11k, Global R&D IS is 600...
  • science teams have university culture which is python friendly
  • CLAB is 20k lines of python, 200 unit tests, 175 acceptance tests
  • (general success story)

24. Track Chairs who don't know how to work the equipment 24. Missing Track Chairs (moshe?)

Lightning Talks - Grok LDAP fast - Tommi pointing out parts of the LDAP namespace on a slide

Tommi Virtanen Grok LDAP fast

  • using DNS domain as a root is "modern"
  • LDIF - lines broken as long as they're indented ("stupid idea" - GvR)

Lightning Talks - lxml - a sane Python wrapper for libxml - Martijn Faassen, slide

Martijn Faassen, Infrae: lxml - a sane Python wrapper for libxml

  • libxml is fast and good, xpath, relaxng, etc.
  • current python bindings not pythonic - c-ish, utf-8 not python-unicode, manual memory management, only documented in C
  • currently only has refcounting and ElementTree API
  • moving into svn on All in pyrex

Lightning Talks - more geeks at laptops - Holger Krekel prepping for SHPY talk- 1

Holger Krekel: SHPY

  • pygame-based cell-based editor, gives you a python-interactive "pad"...
  • multiple cursors for multiple clients... "latency-free"?
  • sending code over the wire.
  • "SHARE" server, "execution" server (roughly, python -c "exec input()"
  • rough hack done in 4 days by Holger Krekel and Armin Rigo
  • can use ssh, other authentication
  • in codespeak svn -- but talk to them, it's rough.
  • not quite a subethaedit replacement, because it uses a server, but...
  • finally, the got it working, and scribble a lot...

Lightning Talks - yagni, whui & don - Steve Alexander and initial slide

Steve Alexander: yagni, whui & don

  • You aren't gonna need it - get moving faster
  • We haven't used it - means you should have yagni'd more
  • Don: Knuth's quote of Hoare about "premature optimisation is the root of all evil" -- or, Don't Optimize Now
  • question: "How to tactfully handle your open source community" (doesn't have an acronym yet)

Lightning Talks - Pyano - Benjamin ?

Benjamin ?: Pyano

  • Python Archive NetwOrk
  • CPAN-like for Python
  • what about PyPI? It doesn't do dependencies (audience comment, I think from GvR, was to "work with PyPI instead")

(Jacob distributes prizes from feedback forms - bottle of wine, and Apress-donated "Practical Python")

Lightning Talks - Sonnets from Pythia - Aroldo Souza-Leite and slide of Steve Alexander's painful "A common gateway" sonnet- 1

Aroldo Souza-Leite: Sonnets from Pythia

  • a collection of poems, "by, around and about" the Python Community.
  • experiment in (post) modern literature
  • not compilable (that's perl)
  • not about bad jokes in bad poems
  • sonnets
  • recited Steve Alexander's painful "A common gateway" sonnet

Lightning Talks - "The Bub's Brothers" - Michael Hudson, Armin Rigo, on-screen shot of multiplayer pygame based game

Michael Hudson: "The Bub's Brothers"

Lightning Talks - Insecticide: the ultimate Python debugger - Michael Salib (faintly) and initial slide

Michael Salib: Insecticide: the ultimate Python debugger

  • interviewed with Green Hills, looked at embedded chip trace port idea
  • time travel support - 1G on traceboard == 1 minute of execution
  • so, just an idea: so add a traceport to the python VM...
  • or integrate cleverness in the VM directly
  • record complete state
  • show me all state-snapshots when a condition is true
  • perform execution-diffs!
  • object-level history

Lightning Talks - Amaretto Venture project - Michel Marquez and initial slide ("Think Python, run Series 60")

Michel Marquez:

Lightning Talks - Amaretto Venture project - Michel Marquez with phone in hand

  • actually works on Amaretto Venture project (ooooh)
  • massive overhead to write normal programs, even short ones (.pkg, .SIS)
  • python packaging could be much simpler
  • Hello World is 46 lines of code - but with OK/Cancel buttons, and a wizard, about 200 lines of code...
  • Python 2.2.2, last stable when project started
  • 60% of GUI wrapped
  • "Not released until Friday" - customer edition available then?
  • "under 800k" target - which was easy.
  • currently needs the SDK, for the emulator, but it should just work on the phone

Lightning Talks - Amaretto Venture project - "Write Series 60 Python Extensions"

Lightning Talks - Amaretto Venture project - "Few highlights" - Python 2.2.2 interpreter

-- break --

Talked to Marquez; talked to Michael Salib

Side track: Good things

  1. Repeating the question
  2. Not using the projector
  3. Speaking to the audience
  4. Involving the audience
  5. Using pygame for demos -- "making it fun"
  6. Maintaining eye contact with audience
  7. Smiling
  8. Talking about interesting stuff
  9. Staying on topic
  10. AV assistance

Lightning Talks - Highlighting python in slides - greg? and slide

greg? - Highlighting python in slides

  • tokenize in cookbook
  • idle
  • SilverCity - built on top of Scintilla
  • doesn't need valid code
  • has a cgi script, but doesn't need to be

Lightning Talks - ZPackage (Zope Packaging System) - Jim Fulton- 1

Jim Fulton: ZPackage (Zope Packaging System)

  • Zope is an application, not a library, different needs from distutils
  • Zope applications themselves are more like modules, but expect to be installed into a Zope software area, not site-packages
  • Want to reuse distutils
  • Wants to be good, like CPAN or apt-get or yum - dependency modeling, install and download, update
  • needed for Zope 3 release
  • written by packaging tool
  • this uses packman/pimp
  1. Modulating your voice
  2. Making slides readable
  3. Using graphics on slides, not text
  4. Showing live demos
  5. Making people laugh (Harald was giving out chocolate?)
  6. Explain the target audience
  7. Give context, clear structure
  8. Putting stuff in the Wiki

bad side:

  1. Using proportional fonts to show python interactive sessions

Lightning Talks - Kamaelia's Component System - Michael Sparks (BBC R&D) getting his laptop working

Michael Sparks (BBC R&D): Kamaelia's Component System

  • streaming server infrastructure replacement
  • open standards: quicktime/real/windows are all proprietary
  • scale to millions of concurrent viewers, not thousands (entire UK population)
  • Concurrency is the big hit: process/threads too expensive, event based systems are "obfuscating goto" systems :-)
  • but real-world things work concurrently, so it can't be that hard
  • Generators with IPC - modeled after pipelines, but multi-dimensional
  • things don't know where their inputs/outputs are, just that they have dataflow from/to them.
  • postman, "coordination and tracking"
  • service is (component, inbox) tuple
  • basic DCT transform component done
  • trivial inherit-from-Component and add yield mechanism
  • Not released yet, working on open source
  • makes maintenance simpler, because it is much more readable
  • try/except to warn about threads dying, but keep it running
  • started a couple of years ago, stackless not really ready then.
  • plan to use it on the front end for streaming, it is the most expensive part of the system. (questioner mentions a Norwegian equivalent)

Lightning Talks - open source work at Divmod - amir (blurry)

Amir Bakhtiar: open source work at Divmod (

  • atop, lupy, etc.
  • demo of reverend, lupy, quotient
  • [later correction: he's at divmod, which employs key members of the Twisted team, but he's not himself on it]
  • quotient does spam filtering, integrates pictures into galleries, fully searchable, auto-files lists, does an address book...
  • $9.95/mo for the service, or just install it
  • and it's his (and the team's) primary email server...


Finally - multiplayer run of Bub's Brothers!


"Something's wrong with your keyboard -- Yeah, it's US layout -- OK, find me the colon key, then"

"I'm missing out some of the details..." - meaning "leaving out", UK British...

"beamer" - slang for video projector.

EuroHaskell - post-conference - ads for the EuroHaskell conference the next day