Showing posts with label PiAssetLibrary. Show all posts
Showing posts with label PiAssetLibrary. Show all posts

Friday, January 4, 2013

Separating Data From the Information About It

In any collection that includes a binary chunk of data and its related metadata, it can be convenient to store each in separate locations, such as having files in a hierarchical format on a server and metadata in a database table.

This way, you have two independent sources of information that relate to one anther.  You could store information in the table that you wouldn't as easily be able to store in the file itself. Especially when dealing with multiple file-types that share a common usage.

The problem is how to know which piece of the database table refers to which file. This is called referential integrity. Either each cell keeps a record of which data it represents, or each data contains reference to which cell its data is stored. "integrity" thus refers to how strong this link is.

This link is rather important. Lets take an example.

On your server, you have a file. A maya scene file representing a Character Rig. This rig has properties, such as who made it, when and how it was made. In addition, this rig exists in more than a vacuum. As a buildingblock, it has been built out of other buildingblocks such as models and perhaps even other rigs. The rig contains links to other rigs and thus maintains it's own referential integrity. More on that later.

Then, in your database table, at position A5-C5, you have stored its creator and date at which it was created and a reference to its location on disk.

 __________A_____________B_________________C________________
|5 |  Marcus      | 2012-04   | c:\path\charactersetup.ma   |
|--|--------------|-----------|-----------------------------|

Pretty cool huh? This way, I could look up the path to my binary data in this table and retrieve information about it not commonly stored by default file systems. Anything could be put in this table!

Synchronization

Now imagine what would happen if we were to move charactersetup.ma to c:\anotherpath\charactersetup.ma. The link would be broken and the metadata would no longer know what binary file it relates to.

We've got a couple of options in this case. We could:
1. Prevent files from being moved
2. Adjust cell when moving file
3. Automate cell-modification when moving the file

Prevention
This is the easiest approach. Simply ensure that files don't need to be moved and enforce that fact either by setting files and directories to be read-only. In many cases, this is sufficient.

Manual Intervention
Whenever you move a file, simply update the cell. You could have either a web-based software for managing your database, making the editing not so bad. When moving more than one file on a regular basis however can make this process rather tedious. That's when automation can help.

Automation
Whenever you move a file, simply signal the database to follow. This requires some work. If you are on windows, you know how to move files via Windows Explorer. You could set up a callback from Explorer to your database, letting it know whenever a relevant file is being moved and act accordingly.

A more common approach however is to only allow files to be modified by your own tools. That is, instead of teaching Windows Explorer to your Database, let your Database and your Modification Tools speak the same language to begin with.

Having all of your tools under the same roof has it's advantages. For instance, a callback from Windows Explorer could not send information about why the change was made. Something that could be critical in a collaborative environment. On the other hand, it could help prevent files from being moved by artists who don't have write-access to certain files. Something you would get for free due to the tight integration with the current user and his permissions.

Synchronize utilizing existing tools or roll your own?

If a user is allowed to use common tools such as Windows Explorer to move files, should they also be allowed to rename files this way? How about modifying the properties for a file from executable to non-executable? Integrating an existing tool has the drawback that you might end up muddling the interface to your toolchain. The user needs to know what is supposed to be possible and what is not, he needs to know the constraints of the system and only work within them.

By integrating a common, yet comprehensive and existing toolset into your application, you need to ensure that all bases are covered by either clearly stating what is allowed up front or informing the user whenever an unsupported action is taking place. As you might suspect, this requires you to have a comprehensive understanding of what the Explorer is able to do in the first place and ensure that whatever it is able to do is handled appropriately.

Windows Explorer Feature List
Create
Read (incl. their properties)
Update (move, rename)
Delete

Another way of ensuring that your data and metadata are synchronized is to only allow file modification through the use of custom tools. This means that for whatever feature you wish to enable, you must explicitly build it. This makes the support for it rather fool-proof in that you give yourself the ability to only enable features which are easily supported. It does however mean that the more features you require, the more code you must write and the more code you write, the more room there is for bugs to creep in.

Is there another way?

This brings me to the climax of this post. Keeping your data synchronized with your metadata is always a headache and there will always be times where there is a mismatch, either due to errors in your program or human-error.

Let me show you the way I chose to approach it in the Asset Library.

What do we have? Our assets all have additonal information about them, such as change-logs and relations to other assets in the project, but most importantly, it contains information about how it should get treated. An Asset has no files, it is merely a container for various types of character setups, textures, shaders etc. Which means we could not rely on file format to tell us how to handle it, we must give it an identity through other means. Thus, we have binary data inside of groups along with its corresponding metadata and we would like to keep them synchronized.

Extrapolation
In windows, you can attach additional information to a file such as dimension, frame-rate, comments and even thumbnails. This is used by a few file formats to make browsing media a more immersive experience in apps such as Explorer. It is however to the best of my knowledge limited to only a few types of metadata on only a few types of file formats. Unless, as far as I know, you go more deeply in which case things seems less stable. In addition, this would require reading and writing entire files when modifying metadata which is a big deal in most scenarios but an especially big deal when modifying large pointcaches of several hundreds of gigabytes. Not sure how this works on folders.

Integration
What would be nice is if we could somehow store the metadata directly in the binary file. Maya files for example may be stored in an ASCII format, thus allowing us to extrapolate it with additional information inside the file in the form of comments. Nuke also allows for this. However, some formats do not and thus doing it this way would limit us to only extrapolate certain file formats and not others. Additionally, the same fate is suffered with that of Extrapolation in that large files need to be read and written whenever a change occurs. This also wouldn't be that easy with folders.

Bundling
Another way is to keep them separate in terms of files, but store them together somehow. This would allow us to benefit from a database approach whilst not having to worry too much about synchronizing them since they will always be together. 

How can we bundle them? Well, we want them to always be together and thus separating them should involve an obvious and difficult process of an either manual or tool-based intervention. How about storing metadata in a .txt file along with charactersetup.ma binary file in an uncompressed archive of sorts. Such as .zip. This would allow assets to be passed around easily without running the risk of it loosing its metadata and suffer the performance issues whenever reading or writing to compressed archives. Reading and writing to them is quite easy as most operating systems allow for zip archives to be treated as regular folders, with the addition of them having a different icon. Which in our case is a good thing as it enlightens the fact that you are within a bundle that should not be dissolved.

In addition, it would allow us to give the archives custom file extensions, such as .asset for archives of type Asset.

This allows us to store Any metadata independently from the type of binary data or group in addition to the tight coupling between the two. It does however require us to read and write potentially large files whenever a change occurs. It also requires us to deal with that of nested archives.

o PuffAdder
    o texture
    o character_setup
    o simulation_setup

PuffAdder is a group with metadata, and character_setup is binary data with additional metadata, which means we end up with a bundle of bundles. Ultimately, we would end up with the entire data-set being a large bundle of bundles of bundles..

o MyProject.project
    o MyDatabase.database
        o MyAsset.asset
            o MyVariant1.variant
            o MyVariant2.variant
How about we turn the situation around. Instead of relying on file telling us what type of data we are dealing with, how about letting the folder have a type? If the folder is a type, that means it's content is the data just as the ones and zeros are the data of a binary file (read "container"). It would mean metadata and binary data could live as equals under the same roof, being accessed in a common fashion, without causing performance penalties when accessing either of them separately.

A folder could have dots in its name, essentially enabling them to have an extension just like files. However, as a matter of preference, adding dots to folders clutters their representation in an explorer as they would be more difficult to distinguish from files other than by looking at their icon. Of course, the library could strip the name during their display if one wanted to go that route without the visual clutter.

Thus, the asset library expects its library of assets to be formatted as follows:
- Each folder has a metadata file
- Each folder may specify a type other than the default via its metadata
- Each folder may have additional folders

o MyProject
    - metadata
    o MyDatabase
        - metadata
        o MyAsset
            - metadata
            o MyVariant1
                - metadata
                - binary data
            o MyVariant2
                - metadata
                - binary data

Having an additional file per folder clutters their interface, and thus we set the metadata to be "hidden"
 
o MyProject
    o MyDatabase
        o MyAsset
            o MyVariant1
                - binary data
            o MyVariant2
                - binary data

 Now we have a hierarchy of files, each folder knows what type it is without cluttering its interface.

How do I deal with keeping binary data and metadata toghether? The only accesspoint to the file system is through custom tools which only allow certain features to be performed. Artists use the library and the library is only allowed to read files. Supervisors use an Editor which may edit files and may expose more information about the system.

Some topics were left out of this post. Such as reading and writing metadata, type of metadata that may be stored and how to store it.

Tuesday, January 1, 2013

Composition for Grouping Attributes

In my previous post, I briefly mentioned that I stored instance attributes together with composed instances whose attributes I could fetch via instance.composited_instance.attr

Is composition good for this sort of thing?

In books, you might find composition to be used for grouping functionality by letting smaller objects form larger objects via nested encapsulation. E.g. instead of storing name, home_number and work_number under Person, you could instead choose to store home_number and work_number under a Contact object that is then stored under Person.

o Person
    o .name
    o .contact
        o .home_number
        o .work_number

So whats the benefit of doing this? Well, besides the benefits of composition in general, you get to stow away accessors that might not be directly related to the overall operation of the object into less obvious paths, thus encouraging the use of parts of the functionality over others. The directly available accessors is intuitively more comfortable to use than to go 6 level deep for a simple operation.
>>> you.name() # This
"marcus" 
>>> you.personaldetails.contactdetails.publicinformation.name() # vs. this
"marcus"
>>> # Which do you prefer?
The neat thing about the latter is once you start gathering functionality that relates to the bigger picture of the parent object, but aren't really necessary for everyday use. Remember, the idea is to make the interface as simple, yet as complete, as possible.





>>> you.countries_visited()
['france', 'russia', 'usa', 'finland', 'sweden']
>>> you.history.travel.countries_visited()
['france', 'russia', 'usa', 'finland', 'sweden']
Storing that level of detailed functionality directly into the parent object can easily obfuscate it's interface. That's why it can be beneficial to stow them away somewhere where, when accessed, the user knows he's dealing with less common functionality.

An alternative to composing accessors is to collect external sets of functionality. Rather than storing all functionality inside an object and it's composed objects, you can choose to store it in separated functions, either in the same file or in different modules or packages.





>>> you.nearby_cinemas()
['canary wharf cineon 1.5km', 'north greenwich odeon 0.4km']
>>> externalmodule.position.nearby_cinemas(you)
['canary wharf cineon 1.5km', 'north greenwich odeon 0.4km']


As you can see, you is no longer responsible for processing this command. Rather, an exernal module takes care of the necessary steps in order to produce the result. This can lead to deep and thorough functionality whilst still providing a simple interface for you. The user now has to think about what tool is right for the job as opposed to figuring out how to use one tool for everything.

Sunday, December 30, 2012

Common Data Access vs 1-for-each

Benefits of storing all data access in a .data member method rather than one method per type of data.

Initially, I chose to store data in member variables and access them through specific means such as
>>> self.name
'marcus'
>>> self.icon
<QPixmap instance at 0x00000..&gt
 I recently found myself in a situation where the number of data access routes I have in one of my primary classes grew into quite a mess. Inbetween state-getters, such as self.is_hidden() and essential getters such as self.name() were composited classes, each with their own getters, and god knows how many accessors I've inherited from the underlying Qt class.

Qt has a rather neat way of solving this problem. Instead of accessing variables classifiable as being "data" through specific methods or properties of a class, the class holds a much broader method called simply "data". The data method takes two inputs, column and role. The column parameter is only applicable with the Model/View architecture of Qt, such as when using Tree- or ItemView. It passes the column of the TreeView that data is requested for. Lets assume our tree only has one column for now.

o root
|--o child1
|--o child2

That means the column parameter will always be 0. The role is the more interesting one. It specifies an enum for which type of data is requested. Say I wish to know your name. I could simply do
>>> you.name()
'nicholas'
With the .data() access however, I can instead do
>>> you.data(role='name')
>>> 'nicholas'
>>> me.data(role='icon')
<QPixmap instance at 0x00000..&gt
Cool huh? Actually, it might seem rather dirty at first since you.name() is clearly much more to the point.

This really starts to shine when you've got plenty of things you might want to ask your object for. It might have a name, an adress, a car and perhaps an additional long list of various types of data, each with it's own accessor method. This can muddy down the interface for a class. A QAbstractItemModel for instance has 14 different roles associated, such as DisplayRole, DecorationRole and EditRole. Each of the roles may provide unique types of data. DisplayRole for instance values for use in Tree- and Item-Models, whilst EditRole returns an editor of depending on which type of data is to be edited.

Using strings as role keys probably isn't a great of an idea though, which is why Qt uses identifiers instead.
>>> qtitem.data(role=Qt.DisplayRole)
>>> 'nicholas'
>>> me.data(role=Qt.DecorationRole)
<QPixmap instance at 0x00000..>
Each identifier returns a integer corresponding to the role.

Now, how can you know when the use of a data accessor is the better alternative to direct access? How many accessors are too many? The approach seems rather un-OOP in that you both lack the dot-syntax to access data and you lose the ability to encapsulate common data, name and adress might go under personal_information for instance.