For compatibility with the Object Pascal memory model, which ameliorated the lack of physical memory management, C++ on the Mac generally used handle-based heap objects. This had serious structural issues because they just weren't firmly located in memory.

Ultimately regular pointer-based heap objects won out even on Mac and reference counting became usable. I encapsulated the necessary activities in the smart pointer ZRef and the object base class ZCounted, which stabilized in the late 1990s after some struggles with the advent of preemptive multithreading.

All of which is to justify why I haven't simply embraced the C++11 standard library's smart_ptr — by implementing reference counting for myself I think I stumbled over every possible mistake, but also found a really useful mechanism I haven't encountered elsewhere. In the literature reference counting is considered to have two fundamental problems — reference cycles and object resurrection. In my experience a cycle (ultimately) indicates a broken data model.

However, inadvertent resurrection is a significant problem.

The issue is that we may want to take additional action when an entity's count transitions from one to zero, but if that action itself needs to treat the entity as counted, then the count will necessarily increment from zero, must ultimately transition down again, and we're into recursive execution of finalization.

Another situation is when we don't want simply to release the entity when it's no longer in general circulation, but want to change its treatment, perhaps by placing it in a cache. So we actually still have a physical reference to the entity, but managed with some different mechanism. This kind of changeover is very susceptible to thread issues. We could acquire a lock every time we manipulate any reference count, but that utterly kills performance and creates a pervasive and non-obvious source of deadlocks.

The solution is in the statement — we know to take action when the reference count changes from one to zero. What if we don't change the count in that case? Remember we're tracking the number of extant physical references. Seeing the one to zero transition tells us there's no other reference to the object than the one in our hands at that moment. So any further reference originates from the one we're considering finalizing. Rather than the typical atomic-decrement-and-test, releasing a reference uses a get, test and CAS sequence. The finalization code is now free to pass the entity to other code and no (legal) operation can again take the count from one to zero. Ultimately execution returns to the finalization. If the count has returned to (or remained) one, then we release the entity, otherwise do a simple atomic decrement and exit.

Transfers (by subscribers) use perfect forward secrecy, and can be entirely asynchronous, so sender and receiver need never be online at the same time. Of course if they are, then transmission goes as fast as the physical network allows.

We built the backend and client mainly in Python, supporting Mac and Windows back to 10.5 and XP respectively. For Linux we support 32 and 64 bit Ubuntu 12 and later.

Much of my focus has been on keeping OS variations from impacting the main body of the code. To that end, we're using wxWidgets to abstract the GUI, and I've implemented such native code as needed to interface with unwrapped aspects of the OS and third-party libraries. In some cases the fluent Python bindings to OS facilities just aren't fast enough, and native code is used there also.

In ZooLib I made several attempts at creating a clean suite of cartesian types and operations, with conversion operators and constructors, pseudo constructors and miscellaneous helper functions. But the approach just never scaled well.

Recently I've taken a very different tack. In ZCartesian I've defined a suite of template accessor functions -- passed a rectangle/point the edges, corners and centers can be extracted. Offsetting and aligning operations build on those accessors, as do sensible algebraic operators.

The accessor functions themselves forward to functions defined in traits templates. There's a suite of base classes for rectangles that are defined by variations of origin/extent or left/top/right/bottom. Supporting a new suite of cartesian types can be as simple as providing the definition of a traits class inheriting from one of the provided bases.

In this way we can work with NSRect, CGRect, XRectangle, QuickDraw Rect, Windows Rect, SDL_Rect, and their associated point (and size) structures using identical notation. And when we need to switch representations there's just a single pseudo-constructor to do the job.

Zorap ran as a browser extension, so could be instantiated and destroyed many times within the life of a process. Any code it used had to be robust, and could not depend on operating system facilities to dispose of resources like network sockets.

ZBlackBerry can also make a BlackBerry connected to a PC accessible via the same API as is used when it's connected directly to a Mac. This lets Mac application developers run their BlackBerry application under the Windows-based debugger and still have it communicate with their Mac application. This is a crucial capability when writing anything more than the most trivial application.

ZBlackBerry is part of ZooLib, our open source C++ library.

Zen itself is only about 15,000 lines of C++, using ZooLib to run on Mac, Windows and Linux. It uses a tuplebase to store configuration and state, and uses the same tuplebase as the communication medium between its components.

To earn its stripes as the foundation for a smart home its higher-level behavior needs to be customizable. Some of that customizability can be captured as data-driven C++ code, but there are always edge cases to deal with, and C++ is not as malleable as I'd like when experimenting. So for a long time I'd been looking for an embeddable programming language that could talk with the C++. I've considered SIOD, but I'm just not a Scheme person. And although Javascript is a much nicer language than you might believe from only having seen its use in web pages, the various open source implementations are quite hard to incorporate into an application that's not the web browser that spawned them.

Lua is written in ANSI C, not requiring much even of the standard C library. It took only a couple of hours from downloading Lua to having it compiled using GCC on Linux and CodeWarrior on Mac. Actually connecting it to C++ and to Zen took a bit longer, but only because its scheme for sharing data between C++ and Lua is unique in my experience and took some thought to fully grok.

There is an excellent book, "Programming in Lua" whose first edition is available online and the second edition in printed form.

Lua supports a small range of data types, inluding the usual number, string and booleans. More complex structures are represented as what Lua calls tables, which are very like a blend of LISP s-lists and a-lists, and thus map nicely to ZooLib's tuples.

Lua's controlling mechanisms themselves use tables, so overriding library functions and global and function scopes use the same techniques as working with a program's own data structures.

Zen compiles Lua code that's kept in its tuplebase as that code is modified, and when the code is invoked provides access to state, which can be inspected and modified by Lua code. Reads from state variables are trapped and the code's dependency on those variables is noted so that external changes to those variables can cause the code to be re-run. This is much simpler than the scheme I initially implemented where each Lua function had to be declared in its own tuple, and its dependencies noted in that tuple.

Frankly, WebDAV is an inelegant protocol. It puts some information in the URL, some in header lines, and some is encoded as XML in the body of requests and responses. But it has three sterling characteristics:

• It exists
• It works
• It's usable out of the box on Mac, Windows and many flavors of UNIX

There are two problems with connecting a UI to a transaction-based store like the tuplebase. First, how do you efficiently identify the minimal set of interesting changes that have been made in the tuplebase. Second, in reading or writing data, what do you do when a transaction fails to commit?

ZTSoup addresses these problems by maintaining objects locally that record a snapshot of the subset of the tuplebase in which the software is interested. Local changes are recorded in those objects, and they are sent to the tuplebase as a single atomic update operation. It's simply a matter of hooking a call to the soup's Update method into the normal event loop -- all additions and removals of interest in croutons and sieves are sent to the tuplebase, and remote changes to the croutons and sieves are returned.

Because the soup may be talking to a tuplebase on the end of a comms link the update work is split into a synchronous piece, which executes in a small amount of time with no blocking, and an async piece that accumulates work done by the synchronous piece and communicates with the server as fast as latency and server CPU will allow.

Language agnosticism is a key feature of tuplebase. Another is its runtime flexibility, which contrasts with most relational database implementations.

The power of a relational database is that the data it stores conforms to a rigorously defined structure. For a single piece of software it's always possible to find a single structure that supports all the work to be performed. As the nature of that work changes the database schema is evolved along with it. But this evolution must be coordinated so that all software that depends on the schema is updated to accommodate the changes, or virtual views are created to preserve a logical interface to the data.

Tuplebase takes a very different approach. Rather than enforcing a structure it allows any structure at all to be stored. Different pieces of software with overlapping concerns obviously must coordinate amongst themselves, but there's no requirement that there be any single authority managing everything. And because there's no mandated structure it's possible at runtime to treat multiple physical tuplebases as a single logical tuplebase, even if they contain data of different provenances. If the code using the data is written to be accommodating to the gradual migration of meaning then newer code can correctly interpret older data, and older code will ignore (but preserve) newer data.

So, a ZBlockStore takes a single file (or file-like entity) and allows an application to work with resizable blocks allocated within it, easily, efficiently and (with ZBlockStore_PhaseTree) safely.

• Easy to use — blocks are managed with the same API as real files.
• Efficient — all the data lives in a wide fan-out B+ Tree.
• Safe — modifications to nodes in the tree cause the entire ancestor branch to be replicated into existing free space, and the new root is atomically updated only when descendant nodes are known to be in stable storage.

ZooLib abstracts threads, files, networking, graphics and the user interface. And there are escape hatches whereby OS-specific features can be accessed without compromising the platform-agnostic nature of the rest of the code.

ZooLib has been under continual development since 1992, and incorporates fixes for every OS bug or anomaly I've encountered, and generally provides cleaner mechanisms for most tasks than OSes traditionally provide. It's been the basis of Measurement in Motion, NetPhone, and every version of the Knowledge Forum server and client applications.

A key design feature is that it never touches its hosting environment directly. Instead it simply posts update notifications, and so the hosting environment is free to clip or draw on top of a painting as it is being worked on. This makes it easier to host in environments with tricky constraints, and to achieve effects like multiple views onto different parts of the same painting.

This was my first serious use of B-Trees, the essential basis of most high-performance disk-based storage systems. The searching of B-Tree content and the insertion of new data is well documented in numerous sources, but the deletion of data is invariably left as an exercise for the reader, which was a bit frustrating but ultimately empowering.