Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Wednesday, January 9, 2013

Working in Context pt. 2

So how would you let your application know about the context? Well, there are a few ways. The most convenient way might be to try and find a common ground of communication between you and the application. A place where both you and the application could read and write to

Environment Variables

In most operating systems there is the concept of environment variables. A common-ground for all applications on your computer, most of which will only read from it but some who also writes to it. You can think of it as a file on disk that everything accesses whenever the user stores a setting or an application requests one. In programming terms, it can be thought of as a singleton or global variable.

So how does one access the environment? Well, it depends on your point of entry.

From the command line in Windows 7, you can go
>>> set // Print all environment variables
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=3a09
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
...

>>> set windir // Print a single variable
windir=C:\Windows

>>> set myVariable=MyValue // Create a new variable and print it
>>> set myVariable
myVariable=MyValue

And on Both Mac and Linux\Ubuntu using bash, you can go
>>> env // Print all environment variables
XDG_CURRENT_DESKTOP=Unity
LESSCLOSE=/usr/bin/lesspipe %s %s
LC_TIME=en_GB.UTF-8
COLORTERM=gnome-terminal
XAUTHORITY=/home/marcus/.Xauthority
LC_NAME=en_GB.UTF-8
_=/usr/bin/env
...

>>> printenv XDG_CURRENT_DESKTOP // Print a single variable
Unity

>>> export myVariable=MyValue // Create a new variable and print it
>>> printenv myVariable
MyValue

We, however, are mainly interested in accessing it via Python
>>> import os
>>> for key, value in os.environ.iteritems():
>>>     print "%s=%s" % (key, value) // Print all environment variables
XDG_CURRENT_DESKTOP=Unity
LESSCLOSE=/usr/bin/lesspipe %s %s
LC_TIME=en_GB.UTF-8
COLORTERM=gnome-terminal
XAUTHORITY=/home/marcus/.Xauthority
LC_NAME=en_GB.UTF-8
_=/usr/bin/env
...

>>> os.environ['MyVariable'] = MyValue // Create a new variable and print it
>>> print os.environ['MyVariable']
MyValue

One variable you might already be familiar with is PATH. Which is, according to wiki.. "..a list of directory paths. When the user types a command without providing the full path, this list is checked if it contains a path that leads to the command." You usually store executables that you would like to run from a command line of sorts.

Another useful and common one is the PYTHONPATH. This is a variable which python looks for when determining which paths to use when searching for modules during import.

So, cmd.exe uses PATH to determine what executables are available and python uses PYTHONPATH. How about we make our own dependency? Couldn't we specify a CurrentProject variable that our library could use to determine which project we are in, in addition to Maya, Nuke, or any other program that might like to know about it? In this sense, CurrentProject is a global variable, accessible by everything that is run as a child of the OS.

Global is usually not the answer, however, but there is one thing that helps us with compartmentalization. Any time you open an application that needs the environment in any way, the environment is copied in to the running process. This means that modifying the environment from inside a child process is merely modifying it's own copy of the environment. It's own duplicate. You can think of this as a child process inheriting from its parent process.

This can be both a blessing and a curse. In some cases, it would be great to modify a variable and have all applications know about it so that they can update accordingly. But there are better ways around that. For instance, whenever a change occurs, a signal could be emitted to dependent processes to update accordingly. That way, not only can you control the flow of updates, but it also helps in keeping things tidy.

What's next

So, we'd like to modify our environment to store information about the context and then have Maya et al. make use of it. How do we go about doing this?

The terminal

Via the terminal, you can read the file system, create and edit files, folders. You can also modify the environment. This, in addition to the fact that the terminal contains a full copy of all environment variables, means that we can edit the environment inside the terminal and then start an application from it. This would make the terminal a child of the OS, and the application a child of the terminal. The OS would maintain it's own original of the environment, the terminal would contain a modified version and then this modified version would get passed along to Maya (which, in turn, would then make another duplicate of this environment). In fact, we could launch several processes via the terminal and each one would get the same modified copy of the environment from the parent, our terminal.

o Windows
    o Terminal
        o Maya
        o Nuke
        o Mari

Each level of this hierarchy is an opportunity to modify the environment, which means that we could separate modifications into groups related to the context in which it is getting modified. For example, the operating system could assign variables related to the overall operation of every application, independent of the user. The terminal then lets the user add to that via custom commands, such as setting the current project. Maya would then have a final copy with all of the information that it needs to operate.

But wait, there's more! In addition to these three levels of updating our environment, it can in fact happen at a few more levels.

o Boot
    o Windows
        o Login (per user)
            o Launch Terminal (per terminal)
                o Custom Commands
                    o Per Project
                        o Per Asset
                            o Maya
                            o Nuke
                            o Mari

As you can see, we can assign each of these level a responsibility and have a very dynamic way of interacting with our applications. Letting them know exactly what is going on with very little effort.

In the next part, I'll go through implementation and some of the problems I encountered along this path. Such as how to work around the fact that a process cannot edit it's parent environment.

Tuesday, January 8, 2013

Working in Context pt. 1

"Hey Maya, how are you?", said the Publisher Tool
"Oh, hey. It's allright. Whats up?", said Maya

"Nothing, just wanted to check some stuff with you if that's okay"
"Ok"

"What asset is your user working with?"
"PuffAdder"

"Cool, do you also know which variant of that asset?"
"It's called animationSetup"

"Would you mind making me a burger?"
"Sorry, not in my job description"

"Oh. Okay, thanks!"


Having your application know under which context it is working under can be a real treat sometimes. It allows you to compartmentalize your work and keep things clean while also giving you that feeling that you are within a larger whole, that youre in a team, not only physically and emotionally, but also digitally. The feeling that whatever you do under this "digital umbrella" is somehow recorded and monitored and thus progresses the global status of the project.

In the example conversation above, between a Publisher and Maya, the Publisher asks Maya for information regarding what is about to get published. This relieves the user from having to specify it once for each publish and enables them to instead specify it per-application or perhaps once per several applications. With the application informed, the user may simply hit "Publish", enter a comment about what has changed, and click "Ok". Incrementing versions and determining where the physical file eventually ends up on disk is left as implementation details where it belongs.

Friday, January 4, 2013

Working in Context pt. 0

Why use Cygwin when windows already has a built-in terminal?

Background

Having an application know under which context to operate is essential to providing a truly immersive experience for the user. Whenever an action is being performed the action should happen within the context that the user is working under. Take publishing for instance. The user may be publishing an variant multiple times under the same session. It makes little sense for him to specify which asset is being published each time, the surrounding should be able to provide such information. The only information relevant for the user to specify is what has been changed since the last publish. Saving is another example. Whenever the user saves a working file, they should only have one place to go and there should be no mistake as to where that place is. This helps the user with encapsulating parts of his work that relate to a certain building-block of the film. It can also be extended to providing sandboxed areas under each shot or variant for whenever the user feels the need to experiment.

Approaches

So how can we inform the application of the context? Two ways, you can either start the application and then tell it about the context, or you can tell the parent of the application, the operating system, about the context and boot up the application under that context. The application can then ask the os for information it does not already know. It also allows us to have multiple applications share the same context. Say for instance you wish to work on shot 23. You set it once, and boot up Maya, Mari and Nuke to work on various aspects of the shot simultaneously. Since the context exists outside the scope of each application, each application may ask the higher source for information and may modify that source collectively, effectively staying synchronized with each other and may even allow a level of communication between them.

I chose to support the latter of the two approaches.

Implementation

How do we inform the os of the context? One word, "environment variables". The os has a common area for storing both temporary and persistent metadata. The metadata may be altered either globally to affect everything always, or per instance of say a terminal.

A terminal, such as the built in Cmd.exe is booted up within this context (read "set of environment variables") and stores a copy of it internally. Whenever we make any change to the context, only this instance knows about it. That means that once we boot up Maya from within this terminal, Maya will also know about it, but the next terminal you boot up will not. (unless you boot it up from within this terminal). This lets us encapsulate modifications to the higher source (the os) whilst still allowing us to keep multiple applications living under the same encapsulation (namespace).

Reasoning

Cmd.exe has it's drawbacks however, which is why I chose Cygwin. The main reasons being:

Cross-platform
Considering the pipeline should work across any platform, it makes sense to conform windows to an otherwise working standard, rather than the other way around. Additionally, knowledge gained from using cygwin translates transparently to other operating systems such as Linux and OSX as they both have access to bash, making any future transition effortless. Additionally, many studios have already adopted Linux for their productions. Using Cygwin thus would allow us to benefit from an existing knowledge pool whilst also facilitating the transition between studios for artists.

Built-in apps
Cygwin, like linux, allow for the use of editors within the terminal, mainly Text Editors. That allows you to quickly modify files that don't need the feature-richness of an external app.

In addition to:

Command line scripting
Custom commands can be made by both user and administrator to allow for easy access to common data such as project and current shot being worked on.

Per-project environment variables
Users can run a bash-script, setting an environment variable PROJECT to correspond to under which project is currently being worked on. PROJECT can then delegate paths relevant to this project, such as which bash-script is being used to launch the correct version of applications such as Maya.

The latter ones are possible using Cmd.exe as well and preferring one over the other however is a matter of style.

Headaches

One of the thing about Cygwin that separates it from being native is that it uses its own directory scheme. C:\ does not exist and is instead kept under a directory like /cygdrive/c/. This may seem silly at first but I'm guessing their reasoning is the same as mine. Facing the choice between either conforming all of linux functionality to work under windows paths, or altering windows to work with all of linux, I'm guessing it was the right thing to do.

Tuesday, January 1, 2013

Composition for Grouping Attributes

In my previous post, I briefly mentioned that I stored instance attributes together with composed instances whose attributes I could fetch via instance.composited_instance.attr

Is composition good for this sort of thing?

In books, you might find composition to be used for grouping functionality by letting smaller objects form larger objects via nested encapsulation. E.g. instead of storing name, home_number and work_number under Person, you could instead choose to store home_number and work_number under a Contact object that is then stored under Person.

o Person
    o .name
    o .contact
        o .home_number
        o .work_number

So whats the benefit of doing this? Well, besides the benefits of composition in general, you get to stow away accessors that might not be directly related to the overall operation of the object into less obvious paths, thus encouraging the use of parts of the functionality over others. The directly available accessors is intuitively more comfortable to use than to go 6 level deep for a simple operation.
>>> you.name() # This
"marcus" 
>>> you.personaldetails.contactdetails.publicinformation.name() # vs. this
"marcus"
>>> # Which do you prefer?
The neat thing about the latter is once you start gathering functionality that relates to the bigger picture of the parent object, but aren't really necessary for everyday use. Remember, the idea is to make the interface as simple, yet as complete, as possible.





>>> you.countries_visited()
['france', 'russia', 'usa', 'finland', 'sweden']
>>> you.history.travel.countries_visited()
['france', 'russia', 'usa', 'finland', 'sweden']
Storing that level of detailed functionality directly into the parent object can easily obfuscate it's interface. That's why it can be beneficial to stow them away somewhere where, when accessed, the user knows he's dealing with less common functionality.

An alternative to composing accessors is to collect external sets of functionality. Rather than storing all functionality inside an object and it's composed objects, you can choose to store it in separated functions, either in the same file or in different modules or packages.





>>> you.nearby_cinemas()
['canary wharf cineon 1.5km', 'north greenwich odeon 0.4km']
>>> externalmodule.position.nearby_cinemas(you)
['canary wharf cineon 1.5km', 'north greenwich odeon 0.4km']


As you can see, you is no longer responsible for processing this command. Rather, an exernal module takes care of the necessary steps in order to produce the result. This can lead to deep and thorough functionality whilst still providing a simple interface for you. The user now has to think about what tool is right for the job as opposed to figuring out how to use one tool for everything.

Sunday, December 30, 2012

Common Data Access vs 1-for-each

Benefits of storing all data access in a .data member method rather than one method per type of data.

Initially, I chose to store data in member variables and access them through specific means such as
>>> self.name
'marcus'
>>> self.icon
<QPixmap instance at 0x00000..&gt
 I recently found myself in a situation where the number of data access routes I have in one of my primary classes grew into quite a mess. Inbetween state-getters, such as self.is_hidden() and essential getters such as self.name() were composited classes, each with their own getters, and god knows how many accessors I've inherited from the underlying Qt class.

Qt has a rather neat way of solving this problem. Instead of accessing variables classifiable as being "data" through specific methods or properties of a class, the class holds a much broader method called simply "data". The data method takes two inputs, column and role. The column parameter is only applicable with the Model/View architecture of Qt, such as when using Tree- or ItemView. It passes the column of the TreeView that data is requested for. Lets assume our tree only has one column for now.

o root
|--o child1
|--o child2

That means the column parameter will always be 0. The role is the more interesting one. It specifies an enum for which type of data is requested. Say I wish to know your name. I could simply do
>>> you.name()
'nicholas'
With the .data() access however, I can instead do
>>> you.data(role='name')
>>> 'nicholas'
>>> me.data(role='icon')
<QPixmap instance at 0x00000..&gt
Cool huh? Actually, it might seem rather dirty at first since you.name() is clearly much more to the point.

This really starts to shine when you've got plenty of things you might want to ask your object for. It might have a name, an adress, a car and perhaps an additional long list of various types of data, each with it's own accessor method. This can muddy down the interface for a class. A QAbstractItemModel for instance has 14 different roles associated, such as DisplayRole, DecorationRole and EditRole. Each of the roles may provide unique types of data. DisplayRole for instance values for use in Tree- and Item-Models, whilst EditRole returns an editor of depending on which type of data is to be edited.

Using strings as role keys probably isn't a great of an idea though, which is why Qt uses identifiers instead.
>>> qtitem.data(role=Qt.DisplayRole)
>>> 'nicholas'
>>> me.data(role=Qt.DecorationRole)
<QPixmap instance at 0x00000..>
Each identifier returns a integer corresponding to the role.

Now, how can you know when the use of a data accessor is the better alternative to direct access? How many accessors are too many? The approach seems rather un-OOP in that you both lack the dot-syntax to access data and you lose the ability to encapsulate common data, name and adress might go under personal_information for instance.