Alpha

Since heading off to StartupWeekend #London on the 12th of July 2013, Pipi has been featured in Campaign Magazine and PRWeek, gotten a website, a Facebook and Twitter account but most importantly -  I now have a team.

We're a hacker, hustler and designer fulfilling the three D's of responsibility - Design, Data and Distribution - and we've been hard at work trying to figure out how to deliver the first pass of Pipi to you guys.

 Alpha

For the Alpha version of Pipi, I'm envisioning two separate tools for File Handling and Metadata respectively.

Perfect File Handling

As we started to collect feedback from various studios (the deep pockets), technical artists (the integrators) and artists (our end-users), it became clear very early that the first step towards establishing a foundation within a production would be to properly manage their file-system.

Files are super important in any sort of work, especially so in an environment with potentially thousands of files generated, created and modified on a daily basis by the many people throughout a production.

There needs to versioning, categorisation, metadata and above all - consistency. And then there's the one law that such a tool abides to above all - it must do no harm.

For the alpha, we'll introduce the Dashboard how it solves these matters.

Meta.. what?

A file cannot tell you why it exists. It cannot tell you about where it came from or what it's purpose is. For these matters, there is the concept of Metadata, or "data about data".

One of the first barriers I faced when envisioning Pipi was how to store metadata together with the content I was looking to enrich. As it turns out, this has been done to death in the past and the go-to solution for doing so is to use a database, such as MongoDB.

But to us (you and me), metadata is so much more than just text. Metadata means screenshots per version, reference videos per shot, audio for animators as well as key/value pairs for automation throughout the pipeline. Short text, long text, tables and hierarchies of text, all of that is part of the data about our content.

For the Alpha version of Pipi, I'll introduce Augment and how it solves this in a way which is both easy to learn (no coding required) and infinitely scalable and adaptable (any OS, any content, any data-format, any size)

What is Pipi.. to the technical artist?

originally posted on tech-artists.org

Beating the bush

Ok, I think it's safe to say that what we're developing is already being developed in several places on this planet and is something that many of you may already familiar with. What we have done is recognize that not everyone has this luxury, especially the little guy. Our goal then is to make the most elegant and beautifully designed solution we possibly can and then make it available to all.

$$$

We love cg. We work with it every day. And whether Pipi is to be distributed for free, as open-source software or in some other way sold will depend greatly on the feedback given here. Our primary objective is for Pipi to grow to be as beautiful as possible so that it may be useful enough for one, ten, or thousands of users. If the way to get there is giving it away for free or selling it on a per-user basis, that's what we're here to figure out. The way we do get there is up to you, me, and all of us together. Starting now.

Our mantra is..

..enable artists. This means building a foundation upon simple ideas. Simple enough so that when a tool fails, you know what to do. Too many times have I experience situations in which the tool, blindly trusted upon, fails - leaving the user helpless and in need of technical support. To truly enable, the user must be given a choice.

Our design is..

..beautiful. We believe frustration degrades productivity. And one of the ways frustration is born is through the way in which we interact with our tools. That is why user experience and its technical implementation bear equal importance to us. Suffice to say, we build [B]from[/B] the user experience, not towards it.

Where we are

We would like to think of Pipi as Process Management, rather than Task Management. This is where Pipi fits in compared to existing Task Management Suites, such as Shotgun and FTrack.

Task Management vs. Process Management

Development

We love agile development methodologies and the lean startup method and recognize that no process is a linear as this. Thats one of the reasons why we're here. Our initial goal is for Pipi to become useful to 1 studio, reasonably small yet know what they want and how to go about doing it manually. We are on the lookout for one that is interested and fits the role of the earlyvangelist.

From there, development will orbit around what is most urgent to them. We will then branch outwards into a closed-beta. A cycle which we'll repeat for each of the main development areas - foundation, usability, tools - until we then move on to the next area of application - Pre-production, Animation, Post-production - covering aspects from Animation, VFX and Games as separate beasts.


I've tried to illustrate this process

 (click to enlarge)

Benefits, not features

We've broken down the most basic usage of a pipeline to a set of 6 set of tools. Those are Dashboard, Editor, Library, Publisher, Inspector and Manager.

• Dashboard (previously "Launcher")
• This is what each user of Pipi will be most familiar with. It serves as an entry point for each artist at the start of each new day. It is where the user specifies what to work with and in what application. The Dashboard then sets up the environment and runs the application.
• Editor (previously "Creator")
• When creating active (aka working-state) assets or shots, the Editor is where metadata is entered.
• Library
• For browsing available assets
• Publisher 
• For creating an inactive (aka published) asset
• Inspector
• For exploring metadata and events
• Manager
• For loading and modifying instances of assets from within a DCC application, such as Maya.

From here, we will move on to more complex benefits, catering to extensibility and debugging followed by in-the-dirt tools used by artists in specific disciplines, such as utilities for the animator or setup artist.

Illustration here

 (click to enlarge)

I'll hold off showing you actual designs and prototypes until next time. TMI! TMI my friend! :)

Let me know what you think!

Best,
Marcus




FAQ

Wait, who the hell are you?
My name is Marcus Ottosson. I've been making beautiful cg since 2008, time spent at over 10 vfx studios across Europe ranging from commercials to feature film to idents to triple-A game cinematics.

The video on your website doesn't contain all of the information I need
It doesn't. But we hope it provides you with enough of an introduction to understand our initial goals.

Isn't this basically Shotgun?
No. Shotgun connects people. Pipi connects the software that people uses.

Python, python, python
Yes. We develop with Python and Qt.

I'm using software [X]. Will it be supported?
Where there's a will there's a way. We've been successfully prototyping with Maya and Nuke and aim to cover all of the packages that you use. Let us know what is important to you and we'll move it higher on our checklist.

What about a database?
Yes, who needs 'em. Pipi runs under the Unix philosophy "Everything is a file". That, coupled with a technique similar to the localised metadata storage used by OS X (.DS_Store), help keep the guts of Pipi simple and logical with very small learning curve.

How about version control?
I'll have to rely on you to light the way in this regard as I have no experience with Perforce or other any off-the-shelf solution to binary media revision control, nor have I heard or seen of any DCC house (besides in games) that make use of it.

Will we have to reformat our folder structure to work with Pipi?
No. Our mantra is "Enable Artists", rather than "Enforce Artists". Pipi wraps any existing layout you might have (given it is logical) and provides a unified way of accessing content through code rather than static filepaths.

I need more information
Just ask.

New Market

When developing a product, the choice of market boils down to these three alternatives^[1].

Existing

You've got the existing market in which shops such as restaurants are established. The rules of the game are known, customers know what to expect and plenty of resources are when looking to establish one.

Re-segmented

There is the re-segmented market. A niche market in which a specialized product within an already existing market aims to change the rules of the game, such as the iPad. The customer knew what they had before and can be told about how why they should get this instead of that.

New

Lastly there is the new market. Here, the solution to the problem is unique and in some cases people aren't aware of ever having the problem in the first place, like with cloud computing. The main issue with this type of market is that it takes a considerable amount of time to get customer adoption. Since they went along just fine without you in the first place, you'll have to come up with something appealing enough for them want to replace it with one or more of their currently used products, like the iPhone did with phones and the PDA.

What about Pipi?

It doesn't fit in either one of those. Instead, the product does exist, the population knows how to use it and how beneficial it is only no one is taking the time to develop a product that is available for everyone. 

Remember the Sports Almanac from Back to the Future II? Marty went to the future and got that almanac, which contained sports statistics between the 1950s and 2000. Back in the 80s, he could then use those statistics to win safe bets on any sporting event up until the new millenia. Pipi is similar to that. I've been to the future and not only seen the many benefits available but also applied them to my own work and now I'm back, in the real world, forwarding my experiences to you via my own design. Doing so within a market without competition and plenty of demand. For what more could I ask?

References
1. Four Steps to the Epiphany by Steven Blank

"Everything is a file"

One of the defining factors about UNIX operating systems is it's use of files for everything^[1]. Documents are files, images are files, but also processes and hardware such as the running session of Maya and your keyboard.

Universal Namespace

Accessing an image on Linux can be done by traversing to the directory in which that image resides and then read or write to it. Accessing a process can also be done by traversing to the directory in which the process resides and then reading or writing to it.

/home/image.png

/proc/photoshop

Now, clearly your brand new floppy disk drive is not a file. It's a collection of components physically soldered together in some factory in Utah. What Unix provides however is a unified mechanism for accessing the hardware in your PC and the documents on file-system. Lets look at an example.

Reading from a text document involves launching Notepad and giving it the file you wish to read. Writing to it involves much the same process, only this time Notepad is giving the filesystem something. Accessing the contents of your Shadows of Darkness Disk 6 involves in much the same way handing the file over to the appropriate reading mechanism and overwriting in turn means giving your Full Throttle Disk 54 to the writing mechanism of your original diskette. Makes sense? Of course it does. Lets move on.

What about Pipi?

Pipi likes this way of handling data. In addition to every asset that you create being a file, so is its users and running processes. So is its metadata and so are its events. This ensures that we can completely bundle components together without worrying about handles to external entities such as database entries or memory addresses. Moving an asset to another project? No problem, anything related to that asset is safely stored within its own hierarchy of files and folders.

The Unix "Everything is a File" isn't great with metadata

This is true. In Linux, the floppy disk is a mere file and a file-system only allows so much metadata to be appended to it. The date at which it was modified, its author and perhaps even a tag if you're lucky. This forces developers to use an alternative approach for its data about data, such as XMP information in photographs, which strays from the universality of everything in fact being files.

In Pipi "If it isn't a File, it's a Folder"

Pipi solves this by adding another dimension to files; parents. By storing something abstract, such as a user or process, as a folder instead of a file, we can append files, or even more folders, within. These files may represent XMP information, images or settings for that particular piece of data.

Folders are treated as files. They have a name, and extension and basic metadata such as dates and original author. In a file, content is a stream of bytes whereas in folders it is the files that it contains and Pipi accesses these folders as though they were files.

Advantages

Using folders for files has several advantages, the main one being uniformity and ease of use allowing less of a learning curve for developers and a lower rate of error. Another benefit is the modularity it provides. Logical entities can easily be compartmentalized and transferred without any loss of related information.

Disadvantages

While folders provide a common logical interface, they do come at a price. Traditionally, the metadata of files are stored in separate database structures, such as MongoDB. This separates the binary and metadata information of entities to where they can both perform at their best. Accessing hundreds of thousands of properties via the database model is many times faster than that of accessing it via individual files on disk.

Conclusion

While performance is an important factor in any activity, the encapsulation and uniformity provided by this approach is hard to overlook. When one thinks performance, we usually think about one of three things; it is either responsive, sluggish or a long-running process. It is therefore important to consider the environment in which your application runs and whether or not the plausible numbers in a high-activity situation require you to jump through hoops or whether you can direct your focus towards simplicity and stability instead.

In the case of Pipi, the domain for which within creative studios of a 1-100/ppl crew, activity peaks at three times during the day; in the morning (when artists all load up their scenes), during lunch (when long-running processes such as renders or pointcaching are set off) and at 6 pm (when everyone either saves their work or sets of an even longer running process for execution during the night). Since digital content is quite heavy on the file-size front, the master bottleneck in all three of these scenarios is the read/write performance of the server and in the case of a 100 people crew, a low amount of file accesses might hover around 1 read/sec (giving the crew a gentle amount of 2 min from sitting down to start working), a vanilla amount around 3 reads/sec and in busy situations around 50 reads/sec.

Mind you these numbers are mere expectations at this point and via the power given by the ease and usability of this approach, unexpected ways of accessing this data is bound to be discovered and in cases where file access does reach into the hundreds of thousands per second, the user will simply have to wait a few milliseconds before he or she can continue working.

References

1. "http://ph7spot.com/musings/in-unix-everything-is-a-file"

2. "http://en.wikipedia.org/wiki/Everything_is_a_file"
3. "http://www.faqs.org/docs/artu/"

Finding a Common Language of Design pt. 1

It dawned upon me quite early that to design several interconnected parts of software, they would have to share a common language in terms of how they looked and behaved. Luckily, the default look and feel of an application developed under any operating system using Qt is identical to other applications built under the same operating system. It mimics the window frame, the drag and resize facilities, closing and minimizing behaviours and also, colors and sound. In fact, developing with Qt provides promise of "write once, run everywhere", or WORE, meaning that whatever you write will not only work on any platform, but also mimic the look and feel of that platform, while still performing as you designed.

"But then.." I hear you say, "..why not use that to your advantage when developing Pipi?". It's a fair question.

Pipi does not adhere to this flexibility of mimicking its host. This is due to the fact that Pipi is a cross-platform framework. It will be run on a variety of platforms (Windows, Linux and Mac) and thus the behaviour of its host platform will vary but its users, however, won't. Users will come and go and their environments may change, the very platform they work on may change for each project they enroll which is why it is so important to maintain a "beacon of light" and provide a consistent and streamlined UI that any artist can access and use without the slightest doubt of anything being different. Regardless of which platform the artist is working under.

This does not mean, however, that the practices put in place by these hosts are ignored. Throughout the evolution of software, users have come to expect many things from their interaction with computers and it is your responsibility to maintain these expectations if you decide to stray from the path. There is a saying which says that the greatest design is that which behaves just as users expect. This applies to any interaction design, not least so webpages. If you have ever found yourself entering an unfamiliar website with a goal in mind, only to come out of that site with goal achieved 5 minutes later not remembering exactly how you did it but that in fact you did, you have experienced a rare and well-designed site and should take note of its whereabouts and learn from its design. This has happened to me only a few times and sadly, their successfull design makes them the most easily forgettable websites out there. They provide no friction, no moment of wonder or scratching of the head and thus no time to commit this flawless design to memory.

As you might suspect, if you are designing for a fashion site or any other market promoting content rather than functionality, this may not be the best route. For design that evolves around ease-of-use however, focus and simplicity (to borrow from the mantra of Steve Jobs) is key.

Product Development as a Film Process

As I was working the other day, it struck me that the approach I had adopted in developing the graphical context of Pipi is very similar to how I would develop the graphical content for Film. In fact, the more similar I made it, the more comfortable and predictable it became. So, if you're familiar film production, I'd like to share this approach with you as it has helped me in materialising my ideas into concrete matter in a rapid fashion.

Firstly, lets have a brief look at what the process of making film looks like today. A director has a vision. The director refines this vision, discusses it amongst his peers and starts materialising his thoughts onto paper in the form of text (the script) and art (storyboards). Then, one or more studios are hired to perform the bulk of work required in turning the director's vision into reality. Starting with more concept art, he then progresses into pre-visualisation in which the film is blocked out in time and space to represent his vision as a kid would when trying to convey his idea using crayons and the surface of your kitchen table. Its crude, its fast and inaccurate. This is when the vision is tested against its target audience, this is where the vision is iterated upon until the foundation is strong. Once a foundation has been built and once the vision is shared amongst everyone involved in its inception, production begins.

Lets stop here for a bit and look how this correlates to product development. An entrepreneur has a vision. The entrepreneur refines his vision, discusses it amongst his peers and starts materialising his thoughts onto paper in the form of text (requirements) and art (wireframes). Then, one or more companies are erected to perform the bulk of work required in turning the entrepreneur's vision into reality. Starting with more wireframes, he then progresses into pre-visualisation.

This is where I at first got stopped in my tracks and gazed out into the horizon, looking for ways to achieve the same fluidity as what has become natural to me when working in film. To "sketch" your vision and to then iterate on this sketch. But how can one sketch in product development? How can a product be tested against an audience without having it actually built? In User Interface and User Experience (UI/UX) Design, there is this concept of wireframing. Wireframing shares many similarities with that of storyboarding. It is a crude and simple, yet clear and fast version that facilitates iteration. UI/UX designers have been using it for years apparently, and it makes perfect sense for interface design just as storyboarding does to film.

The interesting part, however, is what comes next.

In film, once the script has been written, storyboards laid out, it is time to put the vision against the clock and make a so called animatic. The animatic serves as a useful guide to timing and continuity. Do shots fit together in time? Does this sequence make sense when played out like this? How do we go from this shot to this other one? The animatic answers all of these questions and more.

In developing Pipi, the most natural thing for me to do was to apply the same concept of an animatic, but to User Interface Design. So I took my storyboards drawn hastily on paper, refined them in photoshop to look like the product I had in mind and simply animated the UI with a mocked up version of a cursor so as to give the appearance of a user interacting with my future software.

Click to play

This is great! I can now easily send this off so anyone and, without further explanation, the recipient would understand not only what it looks like and what it can do, but also how those things are done.

Taking this one step further, we can insert the sequence of images into a demo context, such as an operating system, Windows in this case, and host application, Maya.

Click to play

As you can see, there are actually two separate user interfaces being demonstrated simultaneously, one spawning the other, while the fictional user clicks his way through the menu items to achieve his goal. Additionally, the video contains some backing explanation to the side of the window, accentuating things not clear from simply looking at the screen.

Using these techniques, I can iterate on an idea I just finished plotting on paper to the level of what you just saw in these videos in less than a day's worth of work each. And at such high level of clarity, I can collect feedback on features, user experience and visual design more accurately and quickly from my target audience, having the next version out the following morning. If not the very same day.

To finish off, I'd like to provide you with one final demonstration. A user experience and feature demonstration of another application that was hastily developed in order to gain the same level of understanding that one of the previous "wirematics" could have provided, except the execution time was closer to two weeks of Python coding.

Click to play

What is Pipi

It has taken me a long time to define Pipi. As I made my way from studio to studio over the course of the past five years, each one had this "something" about it that made work a much more pleasant experience. Turning these, usually repetitive and cumbersome tasks, into welcome distractions.

Of the 10+ studios I've worked at, each and every one displayed patterns of the "earlyvangelist". The characteristics are as follows; the earlyvangelist:

1. Has a problem
2. Is aware of having a problem
3. Has been actively looking for a solution
4. Has put together a solution out of piece parts
5. Has or can acquire a budget

I remember this one artist of a mid-sized house here in the UK who told me he had suggested to management that he wanted to build better tools that could help reduce the rate of mistakes and boost productivity of their crew, but the benefits was unseen by management and so never allowed him to proceed.

Management is not to blame, however. It is difficult to visualise the abstract benefits of what you do not know. It reminds me of a famous quote by Henry Ford. "If I had asked people what they wanted, they would have said faster horses."  Although fake, its message holds true. It is a trait of human nature to try and improve on what we have by mere optimisation rather than stepping outside of that box and into unfamiliar territory. The territory of innovation.

Developing tools is expensive. Both in terms of money and time. There is an inevitable period of trial-and-error when breaking new ground. But what you get out of this process is invaluable. Not only will it accelerate your rate of delivery, it will enhance it, make it stronger and more accurate. The rate of mistakes will drop significantly and the happiness-factor of your workforce will increase.

_____________________________________________________________________________________________

As I have spent the past year refining this idea and the solutions that came with it, the smoke has cleared and my vision is slightly less blurry. I would now like to give you an overview of what some of my conclusions thus far.

What is Pipi?

Pipi is a Digital Content Creation (DCC) framework for the film, commercial and games industries.

Why Pipi?

Digital content creation is rapidly getting more and more divided into areas of special expertise and along the way we have forgotten to keep the pieces in sync. Pipi aims to solve exactly that.

How does it work?

Pipi is a framework. It provides you with an essential set of tools along with a foundation upon which to build your own tools that fit into your studio's unique way of working. It integrates with the major existing applications, such as Maya and Nuke, and frameworks delivered by third-party developers, such as Shotgun and FTrack, to help take the hassle out of fast-paced production development.

Tools?

Pipi ships with six essential tools that serve as pivot-points for your work and own custom tools development. These tools, in order of relevance, are Launcher, Inspector, Creator, Library, Publisher and Handler

I will now go over the supplied set of tools in more detail.

_____________________________________________________________________________________________

Launcher

Main entry-point for artists. This is what is booted up at the dawn of each day as artists begin their work. It supplies artists with an overview of each project and its content along with the ability to manage projects from a higher-level point of view;  creating sequences, editing assets, removing shots is all performed via the Launcher.

_____________________________________________________________________________________________

Inspector

Each job, each asset and each shot come bundled with data. This "data about data" is referred to as metadata, and shots et. al. are in turn known as entities.

As the number of entities grow, metadata helps to keep things organized. To give an example, it can be used to keep track of frame-ranges in a shot, comments in playblasts or artist-supplied descriptions for assets. The Inspector then provides the means to visualise this data.

In addition to visualising data, the Inspector also provides the means to monitor what is referred to as "events". Events are recorded throughout the use of Pipi. Any time an artist comments on a shot, publishes his/her work or playblasts a shot in preparation for dailies, the recorded event can be monitored by other artists. This helps artists working on the same asset or shot stay in sync with each other and makes it easy to stay up to date on the overall progress for each chunk of work.

_____________________________________________________________________________________________

Creator

As you start a new job, the Creator provides the means of adding the necessary sequences, shots, assets and variants into that job. It is in the Creator where you append or remove metadata such as camera information, storyboards, references, audio or video et al. The metadata can then be accessed by artists, other Pipi tools or your custom tools.

Creator is most commonly accessed via the Launcher. Whenever an entity is edited, Creator will appear.


_____________________________________________________________________________________________

Publisher

I mentioned that in a studio, work is divided into chunks and that each chunk is delegated to specialists. These specialists communicate with each other by "publishing" their finished material onto a central database. Other specialists can then access this material knowing that it conforms to a fixed set of studio rules. Background props, for instance, may have to conform to a certain polygon-count, character setups may have to provide a certain set of controls familiar to animators and so on.

The Publisher then acts as the funnel through which each shared chunk of work is passed before it reaches the public space. The publisher will perform sanity-checks (e.g. "is the geometry visible?") on your work to ensure it conforms to the already published work, in addition to help artists meet the required guidelines.

_____________________________________________________________________________________________

Library

Once an artist has published his/her work, it can be found in the Library. The Library provides a birds-eye view of each published entity of each job within a studio. It allows artists not only to load animations, point-caches and assets, but also to perform an at-a-glance inspection of metadata, relations between it and other entities, their history and preview the data directly in the Library. This makes the process of finding what you are looking effortless for any artist.

_____________________________________________________________________________________________

Handler

The Handler is the only application-specific tool so far. It deals with the specifics of how an application deals with assets, such as rigs or point-caches. To aid in explaining I'll use Maya as case-in-point. In Maya, once an asset such as a rig has been loaded, it provides the user with a few options. Such as whether the asset should be imported or referenced, which version to load and the ability to up- or downgrade assets. Things which makes a difference only within the specific domain in which the artist is working at the time.

I mentioned that it is application-specific. This means that for each application to make use of it, it has to be provided with an implementation. This is due to the alternating ways applications deal with assets. In Nuke, an asset is an image-sequence loaded via its internal nodes known as "Read". Maya can also load image-sequences, but does so via different means. The Handler provides a high-level abstraction from these technicalities that can be manipulated via it's user interface so that the artist only has to focus on the "what" and not the "how".

Initially, the Handler will be provided for all major DCC applications, such as Maya, Softimage and Houdini, along with Nuke and Mari.

With these tools at hand, an additional set of tools are being developed. Such as a relational viewer to display relationships between the entities within a studio, a rigging framework including "weightshift" and an animation library utilities for storing poses and reusable segments of animation.

Hope you enjoyed and thanks for your attention.