New home

Head over to pipi.io/blog for the latest and best blog posts! :D

Pipi Ideas

As a test, I've launched a site where you can now submit your ideas for what you think should be in a perfect pipeline.

Have a look, vote on what you like, submit what you miss and get it done before I take it down.

pipi.uservoice.com

Super Generic Folder Layout (SGFL)

Had a vision yesterday evening. A vision about how file handling could get more generic and less hard-coded.

Solution

Previously, I've been thinking that each type of entity, like an asset or a sequence, could only have one type of child - assets have variants and sequences shots - so that there would be some structure in how the users worked.

This is very constrained however and will not apply to how everyone works and might be rejected by those who don't.

I'm thinking of generalising file management enough so that it basically acts like other file management app -Explorer on Windows versus Finder on OSX - where you can create any folder under any entity - like a shot under a job directly, skipping the sequence-parent. Or any random folder wherever.

Where's the structure then?

Presets

Once the user has created a project shell, like 1 sequence and 1 shot, they have essentially specified how they've chosen to structure their work.

We could store that in a preset. So that for their next similar job, they could hit "Add by preset" which makes an empty shell identical to the one they saved.

Templates 

To expand on the pre-built types, like shot, asset or sequence, by letting users enter any type as the name, setting properties on their new type, and then saving it as a new type.

Next time, they could enter that type and it would use their previous template.

Bottom Line

Making Dashboard more into the explorer enforces the one benefit of using the Dashboard over the explorer - Name conventions and placing of files and folders. That's what they get and that's what the Dashboard is supposed to take care of.

Benefit

More generic means more flexible. Advanced users will like it more and our development would get drastically simplified.

Disadvantage

More generic means less rigidity which may be counter-productive for new users who may need more hand-holding. But, this is where presets and templates would step in.

Alpha

Since heading off to StartupWeekend #London on the 12th of July 2013, Pipi has been featured in Campaign Magazine and PRWeek, gotten a website, a Facebook and Twitter account but most importantly -  I now have a team.

We're a hacker, hustler and designer fulfilling the three D's of responsibility - Design, Data and Distribution - and we've been hard at work trying to figure out how to deliver the first pass of Pipi to you guys.

 Alpha

For the Alpha version of Pipi, I'm envisioning two separate tools for File Handling and Metadata respectively.

Perfect File Handling

As we started to collect feedback from various studios (the deep pockets), technical artists (the integrators) and artists (our end-users), it became clear very early that the first step towards establishing a foundation within a production would be to properly manage their file-system.

Files are super important in any sort of work, especially so in an environment with potentially thousands of files generated, created and modified on a daily basis by the many people throughout a production.

There needs to versioning, categorisation, metadata and above all - consistency. And then there's the one law that such a tool abides to above all - it must do no harm.

For the alpha, we'll introduce the Dashboard how it solves these matters.

Meta.. what?

A file cannot tell you why it exists. It cannot tell you about where it came from or what it's purpose is. For these matters, there is the concept of Metadata, or "data about data".

One of the first barriers I faced when envisioning Pipi was how to store metadata together with the content I was looking to enrich. As it turns out, this has been done to death in the past and the go-to solution for doing so is to use a database, such as MongoDB.

But to us (you and me), metadata is so much more than just text. Metadata means screenshots per version, reference videos per shot, audio for animators as well as key/value pairs for automation throughout the pipeline. Short text, long text, tables and hierarchies of text, all of that is part of the data about our content.

For the Alpha version of Pipi, I'll introduce Augment and how it solves this in a way which is both easy to learn (no coding required) and infinitely scalable and adaptable (any OS, any content, any data-format, any size)

What is Pipi.. to the technical artist?

originally posted on tech-artists.org

Beating the bush

Ok, I think it's safe to say that what we're developing is already being developed in several places on this planet and is something that many of you may already familiar with. What we have done is recognize that not everyone has this luxury, especially the little guy. Our goal then is to make the most elegant and beautifully designed solution we possibly can and then make it available to all.

$$$

We love cg. We work with it every day. And whether Pipi is to be distributed for free, as open-source software or in some other way sold will depend greatly on the feedback given here. Our primary objective is for Pipi to grow to be as beautiful as possible so that it may be useful enough for one, ten, or thousands of users. If the way to get there is giving it away for free or selling it on a per-user basis, that's what we're here to figure out. The way we do get there is up to you, me, and all of us together. Starting now.

Our mantra is..

..enable artists. This means building a foundation upon simple ideas. Simple enough so that when a tool fails, you know what to do. Too many times have I experience situations in which the tool, blindly trusted upon, fails - leaving the user helpless and in need of technical support. To truly enable, the user must be given a choice.

Our design is..

..beautiful. We believe frustration degrades productivity. And one of the ways frustration is born is through the way in which we interact with our tools. That is why user experience and its technical implementation bear equal importance to us. Suffice to say, we build [B]from[/B] the user experience, not towards it.

Where we are

We would like to think of Pipi as Process Management, rather than Task Management. This is where Pipi fits in compared to existing Task Management Suites, such as Shotgun and FTrack.

Task Management vs. Process Management

Development

We love agile development methodologies and the lean startup method and recognize that no process is a linear as this. Thats one of the reasons why we're here. Our initial goal is for Pipi to become useful to 1 studio, reasonably small yet know what they want and how to go about doing it manually. We are on the lookout for one that is interested and fits the role of the earlyvangelist.

From there, development will orbit around what is most urgent to them. We will then branch outwards into a closed-beta. A cycle which we'll repeat for each of the main development areas - foundation, usability, tools - until we then move on to the next area of application - Pre-production, Animation, Post-production - covering aspects from Animation, VFX and Games as separate beasts.


I've tried to illustrate this process

 (click to enlarge)

Benefits, not features

We've broken down the most basic usage of a pipeline to a set of 6 set of tools. Those are Dashboard, Editor, Library, Publisher, Inspector and Manager.

• Dashboard (previously "Launcher")
• This is what each user of Pipi will be most familiar with. It serves as an entry point for each artist at the start of each new day. It is where the user specifies what to work with and in what application. The Dashboard then sets up the environment and runs the application.
• Editor (previously "Creator")
• When creating active (aka working-state) assets or shots, the Editor is where metadata is entered.
• Library
• For browsing available assets
• Publisher 
• For creating an inactive (aka published) asset
• Inspector
• For exploring metadata and events
• Manager
• For loading and modifying instances of assets from within a DCC application, such as Maya.

From here, we will move on to more complex benefits, catering to extensibility and debugging followed by in-the-dirt tools used by artists in specific disciplines, such as utilities for the animator or setup artist.

Illustration here

 (click to enlarge)

I'll hold off showing you actual designs and prototypes until next time. TMI! TMI my friend! :)

Let me know what you think!

Best,
Marcus




FAQ

Wait, who the hell are you?
My name is Marcus Ottosson. I've been making beautiful cg since 2008, time spent at over 10 vfx studios across Europe ranging from commercials to feature film to idents to triple-A game cinematics.

The video on your website doesn't contain all of the information I need
It doesn't. But we hope it provides you with enough of an introduction to understand our initial goals.

Isn't this basically Shotgun?
No. Shotgun connects people. Pipi connects the software that people uses.

Python, python, python
Yes. We develop with Python and Qt.

I'm using software [X]. Will it be supported?
Where there's a will there's a way. We've been successfully prototyping with Maya and Nuke and aim to cover all of the packages that you use. Let us know what is important to you and we'll move it higher on our checklist.

What about a database?
Yes, who needs 'em. Pipi runs under the Unix philosophy "Everything is a file". That, coupled with a technique similar to the localised metadata storage used by OS X (.DS_Store), help keep the guts of Pipi simple and logical with very small learning curve.

How about version control?
I'll have to rely on you to light the way in this regard as I have no experience with Perforce or other any off-the-shelf solution to binary media revision control, nor have I heard or seen of any DCC house (besides in games) that make use of it.

Will we have to reformat our folder structure to work with Pipi?
No. Our mantra is "Enable Artists", rather than "Enforce Artists". Pipi wraps any existing layout you might have (given it is logical) and provides a unified way of accessing content through code rather than static filepaths.

I need more information
Just ask.

New Market

When developing a product, the choice of market boils down to these three alternatives^[1].

Existing

You've got the existing market in which shops such as restaurants are established. The rules of the game are known, customers know what to expect and plenty of resources are when looking to establish one.

Re-segmented

There is the re-segmented market. A niche market in which a specialized product within an already existing market aims to change the rules of the game, such as the iPad. The customer knew what they had before and can be told about how why they should get this instead of that.

New

Lastly there is the new market. Here, the solution to the problem is unique and in some cases people aren't aware of ever having the problem in the first place, like with cloud computing. The main issue with this type of market is that it takes a considerable amount of time to get customer adoption. Since they went along just fine without you in the first place, you'll have to come up with something appealing enough for them want to replace it with one or more of their currently used products, like the iPhone did with phones and the PDA.

What about Pipi?

It doesn't fit in either one of those. Instead, the product does exist, the population knows how to use it and how beneficial it is only no one is taking the time to develop a product that is available for everyone. 

Remember the Sports Almanac from Back to the Future II? Marty went to the future and got that almanac, which contained sports statistics between the 1950s and 2000. Back in the 80s, he could then use those statistics to win safe bets on any sporting event up until the new millenia. Pipi is similar to that. I've been to the future and not only seen the many benefits available but also applied them to my own work and now I'm back, in the real world, forwarding my experiences to you via my own design. Doing so within a market without competition and plenty of demand. For what more could I ask?

References
1. Four Steps to the Epiphany by Steven Blank

"Everything is a file"

One of the defining factors about UNIX operating systems is it's use of files for everything^[1]. Documents are files, images are files, but also processes and hardware such as the running session of Maya and your keyboard.

Universal Namespace

Accessing an image on Linux can be done by traversing to the directory in which that image resides and then read or write to it. Accessing a process can also be done by traversing to the directory in which the process resides and then reading or writing to it.

/home/image.png

/proc/photoshop

Now, clearly your brand new floppy disk drive is not a file. It's a collection of components physically soldered together in some factory in Utah. What Unix provides however is a unified mechanism for accessing the hardware in your PC and the documents on file-system. Lets look at an example.

Reading from a text document involves launching Notepad and giving it the file you wish to read. Writing to it involves much the same process, only this time Notepad is giving the filesystem something. Accessing the contents of your Shadows of Darkness Disk 6 involves in much the same way handing the file over to the appropriate reading mechanism and overwriting in turn means giving your Full Throttle Disk 54 to the writing mechanism of your original diskette. Makes sense? Of course it does. Lets move on.

What about Pipi?

Pipi likes this way of handling data. In addition to every asset that you create being a file, so is its users and running processes. So is its metadata and so are its events. This ensures that we can completely bundle components together without worrying about handles to external entities such as database entries or memory addresses. Moving an asset to another project? No problem, anything related to that asset is safely stored within its own hierarchy of files and folders.

The Unix "Everything is a File" isn't great with metadata

This is true. In Linux, the floppy disk is a mere file and a file-system only allows so much metadata to be appended to it. The date at which it was modified, its author and perhaps even a tag if you're lucky. This forces developers to use an alternative approach for its data about data, such as XMP information in photographs, which strays from the universality of everything in fact being files.

In Pipi "If it isn't a File, it's a Folder"

Pipi solves this by adding another dimension to files; parents. By storing something abstract, such as a user or process, as a folder instead of a file, we can append files, or even more folders, within. These files may represent XMP information, images or settings for that particular piece of data.

Folders are treated as files. They have a name, and extension and basic metadata such as dates and original author. In a file, content is a stream of bytes whereas in folders it is the files that it contains and Pipi accesses these folders as though they were files.

Advantages

Using folders for files has several advantages, the main one being uniformity and ease of use allowing less of a learning curve for developers and a lower rate of error. Another benefit is the modularity it provides. Logical entities can easily be compartmentalized and transferred without any loss of related information.

Disadvantages

While folders provide a common logical interface, they do come at a price. Traditionally, the metadata of files are stored in separate database structures, such as MongoDB. This separates the binary and metadata information of entities to where they can both perform at their best. Accessing hundreds of thousands of properties via the database model is many times faster than that of accessing it via individual files on disk.

Conclusion

While performance is an important factor in any activity, the encapsulation and uniformity provided by this approach is hard to overlook. When one thinks performance, we usually think about one of three things; it is either responsive, sluggish or a long-running process. It is therefore important to consider the environment in which your application runs and whether or not the plausible numbers in a high-activity situation require you to jump through hoops or whether you can direct your focus towards simplicity and stability instead.

In the case of Pipi, the domain for which within creative studios of a 1-100/ppl crew, activity peaks at three times during the day; in the morning (when artists all load up their scenes), during lunch (when long-running processes such as renders or pointcaching are set off) and at 6 pm (when everyone either saves their work or sets of an even longer running process for execution during the night). Since digital content is quite heavy on the file-size front, the master bottleneck in all three of these scenarios is the read/write performance of the server and in the case of a 100 people crew, a low amount of file accesses might hover around 1 read/sec (giving the crew a gentle amount of 2 min from sitting down to start working), a vanilla amount around 3 reads/sec and in busy situations around 50 reads/sec.

Mind you these numbers are mere expectations at this point and via the power given by the ease and usability of this approach, unexpected ways of accessing this data is bound to be discovered and in cases where file access does reach into the hundreds of thousands per second, the user will simply have to wait a few milliseconds before he or she can continue working.

References

1. "http://ph7spot.com/musings/in-unix-everything-is-a-file"

2. "http://en.wikipedia.org/wiki/Everything_is_a_file"
3. "http://www.faqs.org/docs/artu/"