New Features in 3.0

General

 

• Protocol changes, the message protocols used by the various alfred system components has been changed to support the new per-job contexts, to improve performance, and to support the new status messages. These changes make the 3.0 dispatcher/nimby/maitre-d components incompatible with those from previous releases.

   

• New priority model, the mechanism for assigning servers to competing dispatchers now applies a dynamic "fairness" criteria, which improves on the previous "strict priority queuing" scheme.

   

• User selectable dispatching scheme, each user can now select which dispatching scheme is in force for their local dispatcher. This setting determines how commands from their current local queue of jobs are selected for launching. Choices are:
  • Task Sequential - typically for debugging, no parallel execution
  • Job Sequential - only one active job at a time
  • Job Spill-Over - secondary jobs are active if resources are available
  • Job Parallel - all jobs are active and compete equally
  See the dispatching discussion for details.

Master Schedule Management

 

• Adding servers to the schedule is easier. There is a new Server Editor which simplifies the task of adding new remote servers to the Alfred schedule. This is now the default Master Schedule interface. It focuses solely on server definitions and leaves all of the other more advanced schedule attributes such as Crews, User-Groups, and Server-Groups, to the secondary schedule editors. This should be a useful simplification since many sites use only the most general Crew definitions (all servers available to all users, all the time), and so their schedule editing is confined to straightforward changes in the basic server pool.

   

• Schedule updates are now performed by the maitre-d. As part of the improved schedule editing scheme, the "Write Schedule" mechanism has been changed so that all schedule file updates are now performed directly by the maitre-d process, on behalf of users around the network.

  In previous releases, the user's editor itself wrote the new file, which sometimes failed to cause the desired global scheduling changes. This problem arose when the editor and the maitre-d were not sharing the same schedule file.

  Under this new system, the user's changes are sent to the current maitre-d which then writes the new schedule. The file is written from the maitre-d host, using the maitre-d process owner's permissions.

  As a result of this change, it is important for the schedule file ($RATTREE/lib/alfred/alfred.schedule) to be writable by the owner of the maitre-d process.

  The remote editing user's permissions are also checked before editing begins. Also, the identity of the user making the change is recorded in the first line of the new file.

   

• Schedule changes are now archived. As schedule changes are made (via the new interface to the maitre-d), old versions of the schedule file are now archived to provide a multi-level back-up as well as an audit trail. Old versions are stored in $RATTREE/lib/alfred/oldSchedules with filenames that correspond to their creation dates.

   

• Temporarily offline servers. The schedule now supports the ability to flag alfred servers as being temporarily unavailable, without having to edit or disable Crew definitions. In the new top-level server editor, the right-mouse button can be used to enable or disable the server(s) on a particular host, across all Crews.

   

• The limit and ping definitions have been moved to the schedule file to simplify run-time changes; and there are dialogs for making changes. Syntax has changed for added functionality.

   

• New MTOR service key and ping. An additional default service key "pixarMTOR" has been added to identify servers which can properly execute the stand-alone version of the MTOR product.

User Interface

 

• User Control of Per-Job Scheduling Parameters.
  Many of the important settings which control job execution and remote server selection can now be changed on a per-job basis in addition to the previous per-dispatcher and schedule defaults:
  • per-job pause
  • per-job priority control
  • per-job server huntgroups
  • per-job Crew selection  (to restrict server groups by Crew)
  See the priority discussion for details.

   

• Improved watch-servers information, the new watch-servers status display contains new types of information, such as remote server load average, and it is formatted in a more informative way.

   

• The new auto-discard preference now allows users to specify a fixed number of done jobs to "keep" on the job queue. Also, a new option allows done jobs to be automatically discarded when the alfred UI closes.

   

• Delayed job start times, jobs can be spooled but inactive until after a specified time, when processing begins. The start time can be given during spooling as a command line option or as an internal parameter in the job script itself; or the value can be changed after a job has been spooled using the Priority settings huntgroup dialog.

   

• Job respool and complete restart, jobs can now be restarted from the beginning, even if they have already completed. Note: the restart may not always work successfully due to the previous actions of clean-up tasks, such as crucial input files being removed, and other changes outside of alfred's control.

   

• Faster UI start-up for very large jobs, the overhead of DAG-view widget construction only occurs on demand, when the DAG window is first opened.

   

• Confirm active job deletion, Alfred now prompts for user confirmation prior to deleting active jobs. This prompting is optional, and can be disabled by the user in the Preferences dialog.

   

• Editable server At-Least / At-Most counts, the spooled min/max-server values, which control the number of servers bound to each command, can now be redefined while the job is running through the the huntgroup (job properties) dialog.

   

• The limit and ping definitions can be editted via the advanced schedule editor.

   

• Better time estimates, the job time-to-completion estimates have been improved.

Scripting

 

• top-level "Job" keyword, a new scripting operator
  Job -options -subtasks {...}
  should be used to enclose all of the Tasks in a job. It has several options for specifying important job parameters.

   

• Iterate operator, a new looping construct has been added for replicating common sub-DAGs with variable substitutions:  Iterate i -from 1 -to 10 ...
  

   

• Conditional launch/skip, individual commands can now perform simple tests to determine if they should be run or skipped:
  Cmd -if {[file exists ...]}

   

• Semaphore / lock to ensure sequential execution, commands can be forced to acquire a token before being allowed to execute
  Cmd -grab token ...

   

• New Cmd variables, there are several new runtime values for Cmd expressions (in addition to %H, etc). These are especially useful in clean-up commands which need extra details of the execution context:

   

  %s - is converted to the bound Service name (whereas %h is the hostname on which the service resides).

   

  %b - is converted to the list of all hostnames ever bound to the Job (useful in Cmds in the new Job -cleanup section).

   

  %B - the list of all Service (slot) names ever bound to the current Job.

   

  %x - the execution state of the current Task; useful for clean-up commands that need a command-line argument that distinguishes between normal Task/Job termination and clean-ups which are run during the deletion of still-active jobs.

  See Scripting: Alfred Language Summary for more details.

Maitre-d

 

• Performance and robustness, there have been overall improvements due to query caching and state management changes.

   

• More status data, there is more detail in data sent to the "watch-servers" status reports.

   

• Servers with multiple network interfaces, host queries and assignments now interoperate better with servers that have more than one network address/interface.

   

• Case is preserved in keywords, service type keys, such as "pixarNRM", are no longer forced into lower-case during schedule processing. Case is now preserved when a new schedule file is written. However, the keyword search for matching servers still ignores case.

Dispatching

 

• Dispatching schemes, there are now several user-selectable dispatching schemes. Job "spill-over" mode is now the default; the old serial mode is still available via preferences.

   

• On-Done / On-Error commands, two new preferences allow users to define shell commands which are executed when a job finishes or encounters an error. These can be used to send mail, or play a sound, etc (see the built-in help on the preferences dialog for more details).

   

• Walk-Ahead Thwarting, Iterate and expand commands are automatically "thwarted" (temporarily restricted from launching) using a new user-defined walk-ahead limit. This test replaces, and improves upon, the previous available-server-count test.

   

• Elapsed time counters changed, elapsed time in spill over mode only counts time spent spent in Cmd execution for secondary jobs (in spill-over mode); primary job still is considered active until complete, paused, or error-waiting.

   

• Dispatcher keep-alive test for socket connections to maitre-d. A new active "heartbeat" protocol allows the maitre-d to determine when a dispatcher is unable to continue using its checked-out resources, and they can then be freed for use by other waiting dispatchers. The maitre-d has always been able to detect most types of dispatcher disconnects (when the program exits or is killed, or when the dispatcher host is properly shutdown). The new test allows the maitre-d to also detect the situation in which a dispatcher has stalled but not properly closed the TCP/IP sockets. This situation arises most frequently when the dispatching host is unplugged from the network or crashes due to a kernel panic.

   

• Pause-on-first-error behavior modified to free resources. The resource usage behavior of jobs with errors has been improved for dispatchers which have the "pause-on-first-error" preference set. This preference controls whether or not a job continues to dispatch new work after an error occurs (in jobs with lots of parallelism it is possible for there to be many tasks which are unaffected by a single error). If the preference is set, new dispatching for the affected job is stopped, and any already running commands are allowed to complete, or "drain", then the job is placed into the "Paused" state. In the previous release, if the preference was set, the active commands were suspended when an error occurred; and so the resources associated with these commands, such as remote servers, remained unavailable to other dispatchers until the error was addressed by the user.

   

• Improved job checkpointing, the job checkpoint mechanism has been changed to reduce file i/o overhead and to ensure that complete restart files exist even if the system crashes during checkpointing.

The Web/HTTP Interface

 

• Status synopsis - the top-level HTML status report, from the maitre-d, which lists all active dispatchers on the network, has been modified (in default, non-wrangler mode) to include a count of waiting, active, and done jobs on each dispatcher; it also indicates if there are any jobs with errors. These additions provide a quick status snapshot to regular users.

   

• Multi-job restart - a new control has been added to address the situation where there are many jobs on the same dispatcher which have errors, due to network errors, for example. The per-dispatcher HTML status page, which lists all of the jobs on a particular dispatcher, has a new control menu which can be used to restart all of the jobs which have errors. There is a choice between simply retrying all of the tasks with errors (across all jobs), or, more drastically, of interrupting and respooling all the jobs containing tasks with errors so that the jobs completely restart from the beginning.

   

• Fast wrangler status, the layout of (HTML) wrangler-mode information provides better status tracking. Also, the information itself is generated very quickly from cached maitre-d state, rather than from round-trip queries to each dispatcher.

   

• Miscellaneous
  • compact per-job pause, priority, reordering, discard
  • wrangler-mode sessions have a configurable time-out

Bug fixes in 3.2

 

• MTOR generated scripts provided a scenario whereby -atmost was less on a single Cmd, than the Job defined -atleast value. This could cause multiple rsh render processes to be started on a single image. This bug has been fixed. If ( -atleast > -atmost ), an implicit ( atleast = atmost ) is performed.

   

• Internal Job -atleast, -atmost tags Ignored. Alfred was ignoring Job level -atmost and -atleast tags. This bug has been corrected.

   

• Commandline option -tags not functional. A problem causing commandline specified tags to be ignored has been fixed.

   

• Maitre_d Deadlock Fixed. The maitre_d process would occasionally deadlock when attempting to assign servers to a multiproc request, where minimum value was greater than zero.

   

• Maitre-d sometimes unavailable under heavy load. In some circumstances, particularly when heavy network traffic caused DNS lookups to be slow, the maitre-d would sometimes become unavailable. This was due to problem in the way that license checks were being handled for the 'Alfping nrm' case. This has been fixed.

   

• Temporarily disabling servers from the new "simplified" Master Schedule interface had no effect in some cases. This has been fixed. Use mouse button-3 (usually the right-most button) to click on a server name, the pop-up menu will allow you to temporarily take a server offline (or re-enable it later), without having to alter any other crew or schedule settings.

   

• Bound server names were sometimes not initially visible in freshly opened job task-tree windows; this has been fixed.

   

• Some paused jobs hang after restart. Under some circumstances, when the pause-on-first-error preference was enabled, jobs with errors appeared to become permanently paused when the errored-out task was restarted. This was due to problems draining task clean-up commands, and has been fixed.

   

• Server "spillage" between similarly named crews. When a job script specified a particular crew subset, it was sometimes possible for the job to be bound to servers from other normally-accessible crews not specified in the subset. This occurred when the whole specified crew name was the first part of another crew name, e.g. "art" and "artists"; the problem has been fixed.

   

• Web browser version-related problems. Occasional problems related to handling forms and current state have been addressed.