I've been enamoured of reference counting since devouring Smalltalk-80: The Language and its Implementation in 1987, and actively using it since reading Coplien's Advanced C++ in 1993.
Continue reading "Reference Counting, Finalization and Resurrection" »
Continue reading "WireOver - Fast, Free, Secure File Sending" »
Continue reading "Unwind_SjLj the Top Items in Your Profile?" »
So what happens when bodies of code collide? Well, we end up with ad hoc transference of information from one representation to another, and chunks of code using a rich representation just look different from chunks using something more primitive.
This is not my first full HTTP proxy; that was the desktop component of iMobimac Modem. In this case there were a couple of requirements that made things interesting.
Continue reading "Smalltalk in Byte Magazine, August 1981" »
YadTranscode is a useful little command line tool. There's a bunch of data formats out there that might look different at first glance but on reflection can be seen to be isomorphic.
Electric Magic implemented the Mac-specific portions of the Zorap web plugin, supporting Safari, Firefox and other modern web browsers.
SiteGrinder’s UI is beautifully implemented in Flash, and Electric Magic helped MediaLab get that UI working in older and newer versions of PhotoShop.
Our client’s application is a scriptable form-filling engine, used by their customers to automate computer-based form submission. Their engine is written in Java, and uses the Abbot GUI testing framework to drive third-party UIs. For Mac OS X support we implemented a JNI shim that made the AX API usable from Java.
ZBlackBerry is a suite of code that implements the BlackBerry USB communications protocol in a generic fashion. A few hundred lines of code let Macs use that protocol. A few hundred more allow multiple Mac applications to talk to a single BlackBerry simultaneously, something that has not been possible till now.
Under the hood KF 4.6 has moved from using ZDBase for its backing store, instead using a tuplebase. This makes it possible to split HTML page generation into separate processes, potentially running on multiple front end machines. It also restores support for a rich client application, now written in Java, using ZTSoup to efficiently communicate changes in the tuplebase, whether made by other clients or by the web interface.
WebDAV is an extension to the HTTP protocol. It is the basis for Apple's iDisk and Windows' Web Folders, standard features of Mac OS X and Windows XP. It is thus the easiest way for a server to make data available to a client machine without requiring that client software be installed first.
ZooLib provides a generic WebDAV server. Your application need only implement subclasses of ZNodeRep to represent nodes in your desired hierarchy. Standard subclasses of ZNodeRep let you lay one tree over another, or expose part of the server's file system to clients.
UNIX does not make an explicit distinction between programs that are servers vs programs invoked by a user. Each flavor of UNIX provides a mechanism for the management of these programs, generally by using naming and placement conventions of descriptor files in the file-system.
In contrast, Windows NT/XP supports servers with a specialized API and management tools. Windows services are packaged as regular applications, but when they are invoked by the service framework they register a callback with that framework to which status and control messages can be delivered.
The tuplebase API is well-suited to data processing needs. But it's clumsy as the mechanism by which data in a tuplebase is to be presented and maintained in a live UI, rather than web pages or generated reports.
ZTSoup gets its name and inspiration from the Apple Newton concept. A ZTSoup is backed by the same data as a tuplebase, but that data is accessed by instantiating a ZTCrouton for each tuple that code is interested in, and a ZTSieve for the result set of any query of interest. Why the funny names? A soup has croutons floating in it, interesting ones are sieved out of it.
Initially I provided Java access to a tuplebase instance by implementing Java classes whose most interesting methods were marked as native, and thus invoked via JNI. This was very powerful because Java could use any tuplebase implementation simply by calling the appropriate factory function and I could expose any existing C++ functionality simply by implementing the appropriate JNI glue.
ZDCPixmapBlit uses templates to generate the code for source/destination and source/matte/destination blitting, matte/destination filling, and destination-only munging, using the four Porter/Duff composition operators Copy, Over, In and Plus.
Electric Fence is an immensely useful library written by Bruce Perens. It uses the standard UNIX memory mapping API to put guard pages on either side of memory allocations, thus causing a segment violation as soon as your code writes where it shouldn't. I made a few small changes to make it possible to use from a preemptively multi-threaded application.
It was thus relatively straightforward to implement ZFakeWindow_NSPlugin, which translates between the Netscape browser plugin and the ZFakeWindow APIs. What was actually more difficult was finding a decent implementation of the plugin glue code and header files. In 2002 there wasn't anything that would work with current compilers and with current browsers, so most of the effort was in putting together ZNSPlugin, which is a usable implementation of the glue.
ZooLib's tuplebase is derived from the tuplespace concept initially explored in the Linda coordination language, another well known derivation of which is Sun's JavaSpaces system. Whereas JavaSpaces is Java-only and relies on many of that languages's features, ZooLib's tuplebase works today with C++ and Java, and is well suited to work with other languages.
ZooLib defines and implements a graphics API that produces identical results across all supported platforms. It is geared towards creating user interface elements and so is pixel-based rather than geometric so that a programmer can know precisely which pixels will be touched by a drawing operation. It supports pixel-plotting, lines of any thickness, text, rectangles, rounded rectangles, ovals, arbitrary regions and the drawing of masked-pixmaps. ZDC_ZooLib is an implementation of this API using no OS facilities at all, and thus can be used in server applications without the difficulties that normally poses (gaining access to a window or graphics server from a low-privilege process).
The implementation uses operations on ZBigRegion instances to decide which pixels to touch, decomposes the regions into rectangles and then calls ZDCPixmapBlit to actually do the work.
IFF is a venerable data meta-format, introduced in 1985 by Electronic Arts as a standard framing mechanism for multimedia data. In short it defines a nested chunked format, where the file as a whole is considered to be a chunk. A chunk has as its first four bytes a tag that indicates the type of data in the payload, a 32 bit (four bytes again) count of the number of bytes in the payload, and then that number bytes being the payload itself. Some chunk types are known to contain zero or more other chunks in their payload, so an arbitrarily complex hiearchy can be established. There are other details, but chunk types and sizes are IFF's essentials. QuickTime's file format is almost the same, except that the size comes first, and includes the eight bytes of the chunk header.
IFF/QT are simpler than most file formats, but can still be fiddly to work with. To make it easier for me to parse and generate QT files manually (on platforms which don't have QT libraries) I put together a suite of ZooLib streams that take care of all the bookkeeping.
For most of my career I've been very suspicious of dynamic data representations. After all, what's the point of having a compiler if it isn't provided with enough information about the shape of the data being manipulated to tell you when your code is going wrong. However, that really only works for data created and consumed within a single application. In the mid-90s, every C++ framework worth its salt had a huge bunch of code dedicated to turning arbitrary C++ objects into something that could be serialized and regenerated as objects later; and in fact that's about as far as most people got with CORBA before giving up.
But most objects in a C++ program, if they're objects at all, simply don't need to be serialized. The ones that do have disparate needs, and special cases abound. The approach I've found most flexible and least intrusive is to provide ZTuple. In this context a tuple is isomorphic to a LISP a-list, Python or Cocoa dictionary, Perl or Ruby Hash or to a Java Map. It's simply a list of name/value pairs, where values can be primitives (strings, numbers, raw bytes etc), tuples or lists of values.
ZooLib provides a suite of facilities that read and write tuples to binary streams, generate and parse a well-defined and easy to read text format, and that can read and write appropriate data formats as tuples.
Most of my work till this point had not required 'interesting' operations with file systems, being restricted to creating and opening files in externally determined locations, then reading and writing the files' contents. When it became necessary to ennumerate the contents of directories, and to deal with permissions and locking I took the opportunity to define an API that would be consistent across Windows, Mac and UNIX whilst cleanly accomodating their differences.
There's a maxim, a citation for which I can't locate now, which pours scorn on the idea of multiplexing streams over TCP. It kinda made sense when I first saw it, but in these days of ornery system adminstrators and their firewalls that sometimes allow connections seemingly on the roll of a die, there's a lot to be said for the all or nothing of getting a connection to a server, and then sharing it locally.
I wrote ZStreamMUX after coming across an early draft of WebMUX. ZStreamMUX takes a read stream and a write stream, and runs a lightweight protocol over them to support multiple independent sessions. The overall interface is similar to that provided by sockets. The protocol is asynchronous, and uses buffer credits to prevent deadlock at the protocol level.
Applications often have to satisfy two conflicting requirements. On the one hand the data created by a user has a structure whose parts should be managed independently of one another, ideally with each piece placed in its own file. On the other hand users like to think of their data as a single entity which can be copied, emailed and backed up in its entirety. Although Mac OS X has the notion of a bundle, a specially marked directory which behaves like a single file when manipulated by the Finder, other operating systems do not.
I picked the name 'asset' as an alternative to 'resource', a term that already has too many disparate meanings on different platforms. That said, assets are used in the same situations that MacOS/Win32/BeOS resources would be, although the mechanism is more flexible.
The data making up an asset tree is directly usable by any processor, big or little-endian. It can be kept in a file, loaded into RAM, memory-mapped from disk or accessed from a stream.
ZStreamR_Boundary is a ZooLib stream derivative that uses Boyer-Moore-Horspool to take a source stream and efficiently provide to a caller only that data preceding a boundary. This is very useful when parsing MIME multipart streams, which can be nested arbitrarily and otherwise get very fiddly to deal with.
NPainter is a suite of classes that provides a MacPaint-like interface, but supporting indexed and true color, across all platforms supported by ZooLib.
Measurement in Motion 1.0 had an interesting interface. A document contained a set of measurements, and various objects that could display and modify those measurements. The objects were represented as 'windoids', windows within the document window. MiM 1.0 was a Mac application, so the windoids could depend only on ZooLib's graphics API, with no help from the OS. When demand for a Windows version became overwhelming I implemented a full windowing system that conformed to ZooLib's ZOSWindow API, so that any code that worked within a regular OS window could instead be hosted in a windoid. QuickTime by this point was writing to the screen buffer asynchronously, so any change in the visible region of a windoid is notified before it happens and after its completed, somewhat like the BeOS BDirectWindow mechanism.
In porting ZooLib to BeOS I found that BeOS's region API was missing key features. Exclusive or and equivalence testing could be handled by composing more primitive operations, but insetting was trickier. So I dug through the X source and found that their implementation of insetting was totally generic, basically shifting the source region for each set bit in the inset distance and accumulating a union (for expanding) or intersection (for contracting).
But BRegion was also quite slow. The internal representation was pretty close to that used by the X code, being a list of non-overlapping rectangles organized into bands. BRegion did not always maintain the same tight constraints as the X code, but it still proved possible to significantly improve performance by manipulating BRegion's internal data using the X code.
By this point I was intimately familiar with X regions and it was a small step to take the X source and rework it, the result being ZBigRegion - a portable region implementation using 32 bit coordinates. Combined with ZDCPixmapBlit it forms the basis of ZDC_ZooLib.
Fibers, at least in this context, are threads implemented using setjmp/longjmp to transfer control in a manner reminiscent of co-routines. They are a generalization of the stack-swapping used by NetPhone to run in the constrained environment it experienced when its deferred tasks were scheduled when an application with a tiny stack was current (yes Print Monitor, I'm talking about you).
ZooLib had always maintained the ZFakeWindow abstract interface between user interface elements and the hosting environment. This allowed ZooLib UI widgets to be used within MacApp applications, standalone applications, Control Panels, XCMD windows and XObject windows. It worked very well.
However, when the portability axis was across platforms rather than hosting environments there was the potential for a lot of replication and ugly code in ZWindow, the standalone application derivative of ZFakeWindow.
So, I abstracted the interface to OS-windows, placing it in ZOSWindow, and modified ZWindow to use a ZOSWindow, rather than be conditionally compiled for different platforms.
ZDBase is a relatively simple database storage engine. It's more of an alternative to Berkeley DB (as it was) than a replacement for MySQL. It supports an arbitrary number of tables, records and typed/named fields within those records. Each table maintains as many indices as desired, and the schema can be updated at any time. All data is kept in a blockstore, so everything lives in a single file, and the code and data are portable across platforms.
FinderHider simply put a border round the stage that would dynamically size to fill the entire screen.
Marrakech took the hypermedia concepts I explored in WorkSpace and applied them to the problem of managing workflow and assets for multimedia development.
