Matlab on the Arrow Cluster Tutorial

To start, read Daniel Eaton's tutorial on Matlab with the ICICS beta cluster. Follow this tutorial to get started, noting the following differences between the beta and arrow cluster for some of the steps:
  • Get yourself access: For the Arrow cluster, you need to talk to Kevin LB or Kevin M, while Frank or Lin can set up the appropriate permissions.
  • Get network storage: An alternate location for network storage is /lci/SCRATCH. You can also use your home directory if this has enough space
  • Start on a linux command-line: The submit hosts are
    samos, stoker, arrow01, and arrow50
  • Setup: To set environment variables from the TCSH shell, use
    source /cs/beta/lib/pkg/sge-6.0u7_1/default/common/settings.csh
  • Submit a job: When submitting a job, you may need to specify a priority class using the -P option. For students of Kevin Murphy, this changes the command to:
    qsub -P kpm myscript.sh
  • Monitor your job: To monitor the jobs of a particular user, use the following:
    qstat -u user-name Another option for monitoring the status of the cluster is to use the Ganglia Cluster Report
For more information on the priority classes, and more extensive information on the Arrow cluster, see the Wiki.

Using Compiled Matlab Code

In many cases, you probably want to run compiled code rather than starting Matlab for each job, below is a guide to getting compiled code running:

Making Matlab Code Deployable:
  • Functions Only Your job must consist of a .m file containing a function (ie. myFunction.m), scripts can not be compiled. This main function can call other .m files.
  • Convert Numeric Arguments: Once compiled, the function's arguments will be passed as strings. For numeric arguments, you will want to convert these strings into numerical representations:
    if isdeployed
          myNumericArg1 = str2double(myNumericArg1);
          myNumericArg2 = str2double(myNumericArg2);
          etc.
    end
  • Include Annonymous Functions: For functions called by 'feval' or passed as arguments, you can get the Matlab compiler to include them by adding the following to the m-file (for a function 'myAnnonymousFunction.m'):
    %#function myAnnonymousFunction
  • No Changing the Path: Your function can not modify the Matlab path (all functions must be either on the path at compile time, or included as above).
  • Turn off Verbose Output: When running jobs on the Arrow cluster, output is logged. It might be a good idea to turn off warnings (and other verbose options) to avoid going over your disk-quota with billions of warning messages:
    warning off all
Compiling Matlab Code:

Once the function that you want to compile is ready, compiling it is straightforward:
  1. Start Matlab: Start the version of Matlab that you want the compiled code to use:
    /cs/local/generic/lib/pkg/matlab-7.3/bin/matlab
  2. Update Path: Add all needed functions to the Matlab path (note, the MCR library should NOT be put on the Matlab path). If you have a `root' code directory (where all relevant code is in a sub-directory), you can simply switch to this directory and use:
    addpath(genpath(pwd))
  3. Compile Code: Call the Matlab compiler to make a stand-alone binary (makes an executable file 'myFunction'):
    mcc -m myFunction.m
Preparing the MCR Library

The binaries generated with the Matlab compiler need to link with the MCR library in order to run. The MCR library only needs to built once, and it can subsequently be used with any compiled function. To prepare the MCR library:
  1. Start Matlab: Start the version of Matlab that you want the compiled code to use:
    /cs/local/generic/lib/pkg/matlab-7.3/bin/matlab
  2. Build the MCR library: To build the library into './MCR.zip', use:
    buildmcr('.','MCR.zip')
  3. Unzip the MCR library: Exit Matlab, find a suitable location on disk (needs to have sufficient space and should be separate from your Matlab code), and unzip the library:
    unzip MCR.zip
Linking the compiled executable with the MCR library requires setting some environment variables. This is described here, but I found that you also to need to add the bug-fix described here before it will work (an example is given below).

Running Compiled Code

The final step before submitting compiled jobs is to write a script that sets environment variables appropriately and calls the compiled binary (updating the environment variables in your log-in script is another option, but I have found that adding the MCR library can make other software not work). In addition to the LD_LIBRARY_PATH variable, the DISPLAY variable should be cleared (on some arrow machines, compiled code will not run when the DISPLAY variable is set). Below is an example script. In this script, the 'code directory' and 'MCR directory' must be set appropriately, and the 'export LD_LIBRARY_PATH' command should be a single line.

#!/bin/bash

hostname

echo $1

unset DISPLAY

cd {code directory}

export MCR={MCR directory}/v75

export LD_LIBRARY_PATH=
$MCR/bin/glnx86:$MCR/runtime/glnx86:
$MCR/sys/os/glnx86:$MCR/sys/java/jre/glnx86/jre1.5.0/lib/i386/native_threads:
$MCR/sys/java/jre/glnx86/jre1.5.0/lib/i386/server:
$MCR/sys/java/jre/glnx86/jre1.5.0/lib/i386/

export XAPPLRESDIR=$MCR/X11/app-defaults

case $1 in
1)
{code directory}/myFunction 1 2 3
2)
{code directory}/myFunction 1 3 2
3)
{code directory}/myFunction 2 3 3
esac

In the above file (call it myscript.sh), myFunction will be called with arguments {1,2,3} under the invocation: myscript.sh 1. Alternately, it will be called with arguments {1,3,2} under the invocation myscript.sh 2 and {2,3,3} if you call it as myscript.sh 3.

For submitting this script as an array job (ie. running it with arguments 1 through 3), replace $1 in the above script with $SGE_TASK_ID, and submit the job as follows:
qsub -P kpm -t 1-3 myscript.sh

You can turn off the error and output files by using the following command
qsub -e /dev/null -o /dev/null -P kpm -t 1-3 myscript.sh

Here is a matlab file which generates a script file similar to example described above.

You may also check if the script is working or not on your local machine by typing ./myscript.sh, However you will need to add export LD_LIBRARY PATH = ... as described in the example script above.


Mark's page