Scripting: Alfred Language Summary

Synopsis

Alfred input scripts (a.k.a. worklists) are composed of expressions using the following operators:

##AlfredToDo 3.0

Job [options]

Task {title} [options]

Cmd {launch_expr} [options]

RemoteCmd {launch_expr} [options]

Instance {task_title_to_reference}

Iterate varname -from n -to m -by i -template {script} [options]

Assign  varname {value_string}

Introduction

A worklist is a structured script which describes the work to be dispatched and monitored by alfred. Worklists are typically created by application programs as they spool new rendering jobs to alfred. That is: the following syntax discussion will probably be useful only to people developing alfred-compliant applications.

Worklists support a simple hierarchical structuring scheme which can encode dependencies that in turn determine the order of task execution; worklists are somewhat similar to makefiles in this regard. The predefined worklist operators are: Task, Cmd, and Instance. There are also some special conventions for embedding certain run-time values into command expressions. As the worklist script is parsed during job initialization, alfred constructs both the internal execution hierarchy for the dispatcher as well as the widget hierarchy for the monitor. The dispatcher then continuously traverses this internal representation looking for opportunities to launch new work.

The alfred expression syntax consists of some simple list-formation rules. Specifically, each operator accepts a list of argument strings, some of which may be optional. Strings are delimited by either double-quotes or curly-braces; they may contain newlines, they are also allowed to be empty. Case is usually important, while whitespace is generally not (see the description of Instances for an exception). Newlines and semicolons, when not in strings, are end-of-expression markers. Backslash is used (as in C) to produce literal versions of syntactically significant characters and other special character codes (and to disable newline processing). Leading sharp-signs (#) introduce one-line comments. See the special characters discussion for more information.

Hierarchical nesting is accomplished by embedding child Task descriptions within the subtask field of the parent Task operator. It is only possible to create nested strings using the curly-brace delimiters.

See the dispatching overview for a general discussion of job concepts such as task hierarchies, launching sequences, and basic Alfred terminology.

A simple example

Here is a simple script which defines an example task hierarchy. Assume we have two shadow maps to compute, and a final frame which references them; in this case the shadow maps must be complete before rendering can begin on the final frame. The alfred "reserved words" are Job, Task, and Cmd, and these operators can take options, such as "-subtasks", which begin with a leading hyphen:

	##AlfredToDo 3.0

	Job -title {A Simple Job} -subtasks {

		Task {Scene One} -subtasks {

			Task {Shadow A} -cmds {
				Cmd {render light.1.rib}
			}

			Task {Shadow B} -cmds {
				Cmd {render light.2.rib}
			}

		} -cmds {
			Cmd {render scene01.rib}
		}
	}

This worklist describes a two-level execution tree in which the task named Scene One will launch its command (render scene01.rib) after the two tasks upon which it depends have completed successfully (the Shadows). The nested task descriptions define the (depth-first) order of execution; hence, the render command for light.1.rib is launched first, then the one for light.2.rib follows. Then, when they are both complete, scene01.rib is rendered. Commands are launched as separate sub-processes, in the order that they appear in the script.

Note that the two shadow commands are not dependent on each other, i.e. they aren't nested with respect to each other; hence, they represent a parallel processing opportunity. Given sufficient resources, alfred will typically launch both commands and allow them to execute concurrently. Hence, the script defines a launching order, some commands may still be running when another is launched, and these concurrent commands may complete in any order. In no case, however, will a Task launch its own commands until all of its sub-tasks have completed their commands successfully.

Note also that the first line of worklists start with the identifying comment
##AlfredToDo 3.0
by which alfred distinguishes scripts from other input (i.e. RIB files).
The "3.0" indicates the worklist language-version in use.

The assumptions made in this example are:

• that there is an executable named render somewhere in the current search path.
• that the two shadow map images are created where scene01.rib expects to find them (i.e. file system references in the RIB files are consistent).

Alfred will not attempt to confirm these sorts of assumptions, that's the domain of the script creator. If an error does occur, the task becomes blocked, the corresponding UI widget converts to the error state, and all tasks which depend on the broken one will remain unexecuted (others may proceed however).

Things to note about Tasks:

• the title string will appear in the alfred UI, so it's handy to choose a brief but descriptive title.
• child tasks are listed in the -subtasks field; if there are no dependencies it can be empty or missing.
• the Cmd list in the -cmds field describes the actual actions to take when the dependencies have been met; it can be empty or missing; it can also contain multiple (newline separated) Cmds, which will be processed in sequence.

Process Launch and Tracking

Remote Servers, Launch Expressions, and Runtime Substitution

However, alfred also provides some powerful features beyond hierarchical ordering of launches, most notably the binding of remote servers. Some commands, such as netrender, are executed on the local processor but communicate with remote servers. For example, consider a typical (non-alfred) invocation:

	netrender -h antigone -h percival  a.rib

	Cmd {netrender %H a.rib} -service {pixarNRM} -atleast 2

Note: the schedule file also allows the site administrator or project coordinator to restrict server access to certain users at certain times of day, and it defines the relative priority of groups of users on subsets of the servers. The Alfred user-interface provides schedule editors which modify these settings.

When the dispatcher is processing a job and it encounters a Cmd like the one above which specifies a service key, it requests an available slot from the maitre-d, which in turn searches for slots that match the service keys. So, these service key expressions are used to filter through the list of remote server slots which are available to the requesting user, and find one which matches the Cmd requirements.

The schedule file entry for each remote server contains a field called Selection Keys, which is just a blank-delimited list of words. The keywords chosen are arbitrary, but obviously they must be coordinated between the script writers and the schedule administrators. A typical use for service keys is to indicate the type of software (server) running on the associated remote host. Pixar applications which create Alfred scripts, such as MTOR, assume the use of a few descriptive service key conventions.

Sometimes it is also useful to include keys which describe the hardware or operating system of the remote system as well, since some applications only run on certain types of machines. Keys are also a handy way to indicate project "ownership" of particular servers. System attributes which vary over time, such as resource utilization, are typically treated using dynamic server metrics rather than static keys.

Service keywords can also be combined using a simple logical-expression syntax. For example:

	Cmd {ralf %h render a.rib} -service {pixarNRM|pixarRender}

For example, consider a schedule file which has the following Services defined (among others):

    Name    host    UDI  Selection Keys
   [Gumby] [iris42] [5] [pixarRAT pixarRender SGI big fast FieldTrip]
   [Pokey] [sun007] [1] [pixarRender pixarNRM Sun fast NightWalrus]

   Cmd {someApp %h args} -service {NightWalrus,(big|fast)}

Note: in addition to the keywords specified in the schedule, each server slot always has two additional implicit keywords: the name of the slot, and the name of the host. This feature is seldom used, but by specifying a slot or host name as the the service key, it allows a script to restrict execution of a particular command to a predefined host (or slot on a host), rather than allowing alfred to pick the best slot from all available servers.

A special scoping directive can be added to the -service specification to limit the selections to Services defined on the dispatcher's host (i.e. the local host, from the dispatcher's point of view).

   Cmd {netrender -f -Progress some.rib} -service {local:pixarNRM}

Note: It is important to remember that the Cmd directive does not actually send work to remote hosts. It launches local applications, such as netrender or rsh, which then manage interactions with a remote server themselves. If there is a copy of Pixar's alfserver running on the remote server machines, then RemoteCmd can be used to cause direct remote execution to occur with no local client.
 

Substitutions
The following symbols are expanded at launch time when they occur within launch or message expressions:

Sharing One Server Check-Out Among Several Commands
Sometimes it can be useful to acquire a remote server and run several commands in sequence on that server. This is called a shared server scenario. The mechanism for accomplishing this goal is to add a server check-out specification to the enclosing Task and then reference the task-id on each Cmd or RemoteCmd that needs the server slot. For example:

  Task {SharedServer example} -id {bob} -service {pixarRender} -cmds {
	Cmd {rsh %h mkdir /tmp/somedir} -refersto {bob}
	Cmd {rsh %h render -Progress some.rib} -refersto {bob}
  } -cleanup {
	# Clean-up commands go here.
	# This example uses RemoteCmd just to illustrate its use as an
	# alternative to rsh above.  Note that there is an implicit '%h'
	# used to determine where the command should be run.
	#
	RemoteCmd {/bin/rm -rf /tmp/somedir} -refersto bob
  } -subtasks {
	# the description of any nested dependent tasks go here
	[...]
  }

Syntax note: the characters allowed in the idref names are letters, numbers, period, and underbar. For all of the % substitutions above, braces may be used delimit names, as in csh or Tcl; for example, if the current schedule has a service of type "renderserver" defined for the host named "darwin", then a Task containing this command:

Cmd {renderer -use %{h}.profile} -service {renderserver}

renderer -use darwin.profile

Special Characters and Escapes in Alfred Scripts
It is important to remember that alfred makes two, very different, passes through the scripts submitted to it. The first spool-time parsing pass is used to construct the "shape" of the job; it sets up the hierarchy of tasks. The second pass (and subsequent identical passes) is the dispatching, run-time, pass during which commands are launched and run-time substitutions are made, etc.

The initial parsing of the alfred script is done using a system built around John Ousterhaut's TCL interpreter. As a result, the TCL syntax rules apply to command arguments and string formation. Alfred scripts consist of calls to the Job, Task, Cmd, and Instance commands (operators), which in turn take strings as arguments. In the case of Task, the -subtasks option can contain additional script which is parsed recursively. In general the TCL rules, as they apply to alfred operator arguments, are much simpler than those for the Bourne shell (sh) or csh(1). Shell programmers should be aware of several things:

• launch expressions are tokenized and passed directly to execvp(2); hence semi-colon separated command sequences, command pipelines, environment variable references, and filename meta-characters (e.g. *,?,[]) are not directly supported (instead use an appropriate sub-shell).
• alfred will do tilde-expansion, as in csh
• single-quotes have no special meaning (i.e. they don't group words together), use braces {} for grouping.
• only one tokenizing pass is made through each argument string

   find ~bob -type f -name 'preview.*' -exec /bin/rm {} \;

   Cmd {find ~bob -type f -name preview.* -exec /bin/rm \{\} ;}

Shell Pipelines and other Expressions
Given the restrictions just described above on Cmd launch expressions there are nonetheless occasions when shell constructs, such as command pipelines or run-time filename expansion, can be very useful. In these situations, a simple solution is to launch an appropriate shell as the Cmd, passing the pipeline expression to the shell via command-line arguments

   Cmd {/bin/sh -e "cd /tmp/frames; ls -1 | xargs -I+ cp + /DDR"}

Another approach is to use the Cmd -msg option to pass arbitrary expressions to a launched shell. This is essentially equivalent to the above approach, but not as compact, however it does allow for persistent reuse of the shell, if that's desired. Consider the following examples which send mail, which can sometimes be handy at the end of job. Recall that the -s option to Mail looks for the next "word" as the subject, so spaces need to be escaped (using csh(1) syntax this time):

   Cmd {/bin/csh -fc "/usr/sbin/Mail -s 'job done' jean < ~/errlog"}
   Cmd {/bin/csh -fet} -msg {/usr/sbin/Mail -s 'job done' jean < ~/errlog}

   Cmd {myscript}

   Cmd {myscript %h} -service {someServerType}

See the notes on writing alfred-compliant application programs for details on the behavior requirements for apps launched by alfred.  

Assign: Initializing Global Job Variables

The Job -init {initializations} block can be used to define variables which have global scope with respect to the Tasks of the job. Typically these variables are just used to make the job script more compact by using them instead of frequently occurring (long) filenames. The init-block can contain multiple Assign statements, for example:

   ##AlfredToDo 3.0

   Job -title {A Job} -init {

      Assign frmdir {/usr/people/buzz/projects/toys/ribfiles}

   } -subtasks {

      Task {Frame 0} -cmds {
         Cmd {render $frmdir/frame.0.rib}
      }

      Task {Frame 1} -cmds {
         Cmd {render $frmdir/frame.1.rib}
      }

      [... etc ...]
   }

Using Iterate to Simplify Repetitive Scripts

Many scripts have a repetitive structure, especially those used to render a sequence of frames from the same shot. The Iterate operator can often be used to simplify the construction of these scripts: basically a template of the repetitive structure is described just once, and it is replicated by Alfred while the job is executing. Here's a simple example:

   ##AlfredToDo 3.0

   Job -title {A Repetitive Job} -subtasks {

      Task {Environment.0} -cmds {
         Cmd {render environment.0.rib}
      }

      Iterate frame -from 1 -to 100 -by 1 -subtasks {
         Instance {Environment.0}
      } -template {
         Task {Frame $frame} -cmds {
            Cmd {render Frm.$frame.rib}
         }
      }

   }

On-Demand Processing, Walk-Ahead Limits, and Thwarting

This determination is based on the walk-ahead metric which is a measure, at a particular task node, of all the ready-to-execute leaf nodes which precede it. Recall that alfred traverses a job tree in depth-first order looking for tasks which have no outstanding subtasks (dependencies). At any given moment, all such leaf-nodes can potentially be launched in parallel, although typically there are scarce resources such as remote servers or tag limits which restrict the number of active tasks.

An Iterate node will perform one substitute-and-copy cycle when the number of unlaunched leaf nodes preceding it, in the top-to-bottom, depth-first sense, is less than the user-defined walk-ahead limit, which is set in the preferences dialog. Since an Iterate cycle usually results in additional unlaunched nodes, the next cycle is delayed by the walk-ahead test until the job can catch up. When a node is delayed in this way it is said to have been thwarted.

This gating behavior is particularly useful when the template script contains tasks which generate, render, and then clean up RIB input files on a per-frame basis. The disk space footprint will be approximately (RIB_file_size) x (walk_ahead_limit).

Formatting During Substition and Leading Zero Padding

    Iterate fn -from 9 -to 15 -by 1 -template {
        Task {Frame $fn} -cmds {
            Cmd {render Frm.[format "%0.5d" $fn].rib}
        }
    }

Iterate Nodes and Job Percent-Done

In the short job example above the default level of zero will track the iterated nodes properly.

Note that when a job has only one level-zero node   - a sort of master task with frame tasks below it -   then you should track the subtasks, e.g. level 1, since progress estimates based on the single level-zero node are unlikely to be accurate.  

Conditional Cmd Launches using if Expressions

The Cmd  -if {expression} conditional test is useful in situations where certain individual commands may not need to be executed. For example:

    Cmd {render backdrop.rib} -if {[file exists backdrop.rib]}

The Tcl file command, used in the example above, can also be used to construct other kinds of tests related to files.

Alfred adds a utility command "fnewer" which can be used in if-expressions to compare file modification times. For example, if an artist is running many test renders of foreground elements over a fixed background, it might be useful to only render the background geometry once, during the first job of the day, or if the RIB-file is updated:

    Cmd {render backdrop.rib} -if {[fnewer backdrop.rib backdrop.tif]}

Note that both the curly and square brackets are typically required in these expressions, as described in expr(n) and more generally in Tcl(n).  

Referencing results from previous commands

Alfred provides a limited mechanism by which commands can pass information to each other. For example, one command may generate RIB and choose a temporary filename for its results, and another command needs to know the filename to render it. This scheme has two components; first, a launched app writes a single line to its standard-out of the form:

	ALF_RET text text text

  Task {command result example} -subtasks {
	# the description of any nested dependent tasks go here
	[...]
  } -cmds {
	# these are the commands for this task, executed in sequence
	Cmd {someApp -genrib 2} -id Frm2
	Cmd {netrender -f -Progress %H %Frm2(result)} \
		-service {nrmserver} -refersto Frm2
  } -cleanup {
	# clean-up commands go here
	Cmd {/bin/rm -f %Frm2(result)} -refersto Frm2
  }

Progress or percent-done indication

	ALF_PROGRESS nnn%

Exit status

	ALF_EXIT_STATUS nnn

RALF
The program called ralf ships with alfred; it functions like the remote shell rsh(1), except that it also provides a means of retrieving the exit status of the remotely launched app, and it will transmit SIGTERM signals to the remote side.