Persistence (17 Apr 2003)

Over the past few weeks I've been knocking about with ideas (which generally go under the name of Landscape) for holding data between code. Generally a byte stream (be it a pipe or a file) is the highest we get when sharing data. Too much data is far too difficult to get to and too little data is well linked.

One key part of fixing this could be a persistent object store and I've been messing with Python versions of this. Firstly, ZODB is far too slow and Prevayler keeps everything in memory.

Prototype 1

This kept data in XFS extended file attributes and each object was a file.



Prototype 2/3

This used SQL to store the objects.



Prototype 4

This uses proxy object that act as far pointers and pickle/unpickle objects on demand




These are, of course, only prototypes to play about with these ideas. The true way to do it would be to use EROS. But EROS isn't seeing much take-up (I've not even got it to boot) and it might be better to put the neat bits of EROS into Linux, even if they don't all fit.

Prototypes 1 and 2 only support dictionary type data and Prototype 3 has slightly bodged support for other objects (they can have a .type link, pointing to the python code to be imported). Supporting other types of objects is very important for objects like (for example) the mixer which has to get and get its values from/to somewhere else.

The interface is also very important. Finding the right set of ideas to simply abstract data is a very hard problem. The interface to the same is possibly just as hard.

At the moment the interface is shell like but different types of objects will need more than that. It remains to be seen if some objects will need to handle the interface themselves to the level that they do today (e.g. GTK/QT level APIs) or if a system akin to XHTML/CSS will do.