"Everything is a file"

One of the defining factors about UNIX operating systems is it's use of files for everything^[1]. Documents are files, images are files, but also processes and hardware such as the running session of Maya and your keyboard.

Universal Namespace

Accessing an image on Linux can be done by traversing to the directory in which that image resides and then read or write to it. Accessing a process can also be done by traversing to the directory in which the process resides and then reading or writing to it.

/home/image.png

/proc/photoshop

Now, clearly your brand new floppy disk drive is not a file. It's a collection of components physically soldered together in some factory in Utah. What Unix provides however is a unified mechanism for accessing the hardware in your PC and the documents on file-system. Lets look at an example.

Reading from a text document involves launching Notepad and giving it the file you wish to read. Writing to it involves much the same process, only this time Notepad is giving the filesystem something. Accessing the contents of your Shadows of Darkness Disk 6 involves in much the same way handing the file over to the appropriate reading mechanism and overwriting in turn means giving your Full Throttle Disk 54 to the writing mechanism of your original diskette. Makes sense? Of course it does. Lets move on.

What about Pipi?

Pipi likes this way of handling data. In addition to every asset that you create being a file, so is its users and running processes. So is its metadata and so are its events. This ensures that we can completely bundle components together without worrying about handles to external entities such as database entries or memory addresses. Moving an asset to another project? No problem, anything related to that asset is safely stored within its own hierarchy of files and folders.

The Unix "Everything is a File" isn't great with metadata

This is true. In Linux, the floppy disk is a mere file and a file-system only allows so much metadata to be appended to it. The date at which it was modified, its author and perhaps even a tag if you're lucky. This forces developers to use an alternative approach for its data about data, such as XMP information in photographs, which strays from the universality of everything in fact being files.

In Pipi "If it isn't a File, it's a Folder"

Pipi solves this by adding another dimension to files; parents. By storing something abstract, such as a user or process, as a folder instead of a file, we can append files, or even more folders, within. These files may represent XMP information, images or settings for that particular piece of data.

Folders are treated as files. They have a name, and extension and basic metadata such as dates and original author. In a file, content is a stream of bytes whereas in folders it is the files that it contains and Pipi accesses these folders as though they were files.

Advantages

Using folders for files has several advantages, the main one being uniformity and ease of use allowing less of a learning curve for developers and a lower rate of error. Another benefit is the modularity it provides. Logical entities can easily be compartmentalized and transferred without any loss of related information.

Disadvantages

While folders provide a common logical interface, they do come at a price. Traditionally, the metadata of files are stored in separate database structures, such as MongoDB. This separates the binary and metadata information of entities to where they can both perform at their best. Accessing hundreds of thousands of properties via the database model is many times faster than that of accessing it via individual files on disk.

Conclusion

While performance is an important factor in any activity, the encapsulation and uniformity provided by this approach is hard to overlook. When one thinks performance, we usually think about one of three things; it is either responsive, sluggish or a long-running process. It is therefore important to consider the environment in which your application runs and whether or not the plausible numbers in a high-activity situation require you to jump through hoops or whether you can direct your focus towards simplicity and stability instead.

In the case of Pipi, the domain for which within creative studios of a 1-100/ppl crew, activity peaks at three times during the day; in the morning (when artists all load up their scenes), during lunch (when long-running processes such as renders or pointcaching are set off) and at 6 pm (when everyone either saves their work or sets of an even longer running process for execution during the night). Since digital content is quite heavy on the file-size front, the master bottleneck in all three of these scenarios is the read/write performance of the server and in the case of a 100 people crew, a low amount of file accesses might hover around 1 read/sec (giving the crew a gentle amount of 2 min from sitting down to start working), a vanilla amount around 3 reads/sec and in busy situations around 50 reads/sec.

Mind you these numbers are mere expectations at this point and via the power given by the ease and usability of this approach, unexpected ways of accessing this data is bound to be discovered and in cases where file access does reach into the hundreds of thousands per second, the user will simply have to wait a few milliseconds before he or she can continue working.

References

1. "http://ph7spot.com/musings/in-unix-everything-is-a-file"

2. "http://en.wikipedia.org/wiki/Everything_is_a_file"
3. "http://www.faqs.org/docs/artu/"

Finding a Common Language of Design pt. 1

It dawned upon me quite early that to design several interconnected parts of software, they would have to share a common language in terms of how they looked and behaved. Luckily, the default look and feel of an application developed under any operating system using Qt is identical to other applications built under the same operating system. It mimics the window frame, the drag and resize facilities, closing and minimizing behaviours and also, colors and sound. In fact, developing with Qt provides promise of "write once, run everywhere", or WORE, meaning that whatever you write will not only work on any platform, but also mimic the look and feel of that platform, while still performing as you designed.

"But then.." I hear you say, "..why not use that to your advantage when developing Pipi?". It's a fair question.

Pipi does not adhere to this flexibility of mimicking its host. This is due to the fact that Pipi is a cross-platform framework. It will be run on a variety of platforms (Windows, Linux and Mac) and thus the behaviour of its host platform will vary but its users, however, won't. Users will come and go and their environments may change, the very platform they work on may change for each project they enroll which is why it is so important to maintain a "beacon of light" and provide a consistent and streamlined UI that any artist can access and use without the slightest doubt of anything being different. Regardless of which platform the artist is working under.

This does not mean, however, that the practices put in place by these hosts are ignored. Throughout the evolution of software, users have come to expect many things from their interaction with computers and it is your responsibility to maintain these expectations if you decide to stray from the path. There is a saying which says that the greatest design is that which behaves just as users expect. This applies to any interaction design, not least so webpages. If you have ever found yourself entering an unfamiliar website with a goal in mind, only to come out of that site with goal achieved 5 minutes later not remembering exactly how you did it but that in fact you did, you have experienced a rare and well-designed site and should take note of its whereabouts and learn from its design. This has happened to me only a few times and sadly, their successfull design makes them the most easily forgettable websites out there. They provide no friction, no moment of wonder or scratching of the head and thus no time to commit this flawless design to memory.

As you might suspect, if you are designing for a fashion site or any other market promoting content rather than functionality, this may not be the best route. For design that evolves around ease-of-use however, focus and simplicity (to borrow from the mantra of Steve Jobs) is key.