WESTGRID < LCI

WESTGRID ([[http://www.westgrid.ca/]]) provides high performance computing, networking, and collaboration tools hosted by Western universities (Simon Fraser University, The Banff Centre, TRIUMF, University of Alberta, University of British Columbia, University of Calgary, University of Lethbridge, University of Victoria, University of Northern British Columbia, Athabasca University, University of Saskatchewan, University of Regina, University of Manitoba, University of Winnipeg, Brandon University).  Since WESTGRID is CFI funded, WESTGRID also welcomes external users from any other institutions in Canada (ie: any student enrolled with any Canadian university).


Here's quick list of general facilities available:

1) storage: storage facility

2) shared memory: nexus group, cortex group

3) cluster: glacier, robson

4) cluster with fast interconnect: lattice, matrix, tantalus

5) visualization: hydra

Note #1: Here is their hardware in detail: http://www.westgrid.ca/support/Facilities

Note #2: Here is their software in detail: http://www.westgrid.ca/support/software



Get an account here: https://rsg.nic.ualberta.ca/new_account.php

Note: To get an account, one must be supervised by a project leader.  Any professor with any Canadian university may become a project supervisor and approve accounts under his project.  Use the same website above to get a project leader account and a project ID number.



Here are some things found out about WESTGRID that might help someone who is new to WESTGRID (after reading the [[http://www.westgrid.ca/support/support][WESTGRID FAQ]]).

1) Opening an account is more of a formality than anything. Everyone gets accepted.

2) Matlab is available but only in single processor format on WESTGRID GLACIER.

3) I have made a Matlab script that will take an M-file (whose primary function is to run many identical test cases with different parameters after some setup code at the top of the M-file) and create a queue to submit to the WESTGRID queue. See a graphics explanation [[https://bugs.cs.ubc.ca/twiki/pub/LCI/WESTGRID/clumeqm-explanation.jpg][here]]. See the code here [[https://bugs.cs.ubc.ca/twiki/pub/LCI/WESTGRID/qsuball.m][qsuball.m]].

Note: The picture was intended for [[CLUMEQ]], but it serves to explain WESTGRID as well.

4) Resource allocation is a complete mystery despite the assurance of [[http://www.westgrid.ca/support/Resource_Allocations][these documents]].

5) The admin for GLACIER, Roman Baranowski, if a very useful and responsive. When troubles arise, do contact him ( roman@chem.ubc.ca ).

6) [[CLUMEQ]] is the same thing as GLACIER WESTGRID.  It is woefully under used most of the time. It might be worth it to race your queues on both systems.

7) WESTGRID GLACIER makes 2 files, stdout and errout, after running your job queue file (ie: PBS file). However, the error file sometimes cuts off if Matlab is writing to that error file (unknown why). This makes debugging Matlab sometimes difficult. Beware....

8) I have made a matlab script that will kill all the user queues. Check it out here [[https://bugs.cs.ubc.ca/twiki/pub/LCI/WESTGRID/qdelall.m][qdelall.m]]. One uses it like this under matlab: qdelall('your_user_name'); 

9) Sometimes there aren't enough licenses free to execute your program.  This is a pain.  However, someone has solved this [[https://bugs.cs.ubc.ca/twiki/pub/LCI/WESTGRID/license-juggling.pdf][Stupid Cluster Tricks]].  Good luck with this.

10) To solve the point above for matlab licenses, one may include the following line of execution into your matlab code:
<pre>
    % halt execution for a minute if we can't get the image toolbox
    while (~license('checkout','image_toolbox'))
        pause(60); 
    end;
</pre>
Note: This can be done for every single toolbox required.  However, to prevent possible license-required [[http://www.google.ca/search?q=deadlock][deadlocks]], I highly suggest one manually (within your matlab script, and most probably during your initilisation procedure) control the checking out of required matlab toolbox licenses as the matlab [[http://www.mathworks.com/access/helpdesk/help/techdoc/ref/license.html][license()]] function does not allow one to release a toolbox license. But, of course, this only works if one can initiate a maltab session.

11) To solve the above problem (not being able to start a matlab session due to lack of licenses), one may have a PBS call that doesn't execute your script until a license is available. Just add "#PBS -l software=MATLAB" to your PBS file (this is done automatically in [[https://bugs.cs.ubc.ca/twiki/pub/LCI/WESTGRID/qsuball.m][qsuball.m]].

Note: Some PBS systems don't accept this option.

12) If the system does have matlab but lacks licenses for a particular toolbox (ie: it only has 1 license for a required toolbox-- sorta meaningless in a parallel system) or (yet worst) has the wrong version of matlab, then one can take the ultimate step and compile the M-file into a system executable.  This is done via [[http://www.mathworks.com.au/access/helpdesk/help/toolbox/compiler/index.html][matlab's compiler]].  However, one must be aware of the [[http://www.mathworks.com.au/products/compiler/compiler_support.html][licensing limitations]] associated with a stand-alone system executable from the matlab compiler.  Of course, if one uses a call in the "Can not be compiled" column, then one must revert back to the solution above.

13) If all else fails, then the ultimate matlab licensing solution would involved a system executable which attempts to launch a [[http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_external/f38569.html][matlab engine]].  Whilst monitoring the standard error output stream (ie: stderr), if it detects an error "-4" (the error number associated with insufficient matlab licenses), then it can relaunch the matlab engine until it doesn't fail.  The problem with this solution is that it is racing against all the other processes in a polling manner-- quite inelegant.  So really this isn't the ultimate solution.

14) Here ([[lmstat]]) is the details of the matlab licenses available to WESTGRID's GLACIER.  Additionally, GLACIER was asked by UBC IT Services (the maintainers of the MATLAB license manager) to limit GLACIER to 20 concurrent matlab jobs.


15) For a complete PBS (Portable Batch System) reference on, please see [[https://bugs.cs.ubc.ca/twiki/pub/LCI/CLUMEQ/PBSProUG_7.0.pdf][PBS Professional User Guide 7.0]].


Note: Please feel free to edit this page.  If something isn't clear, then feel free to email [[mailto:bleuant@cim.mcgill.ca][Albert Law]].
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
jpg	clumeqm-explanation.jpg	r1	manage	168.4 K	2006-05-21 - 20:52	UnknownUser
pdf	license-juggling.pdf	r1	manage	144.4 K	2006-05-21 - 20:56	UnknownUser	http://bioteam.net/dag/license-juggling.pdf
m	qdelall.m	r1	manage	0.4 K	2006-05-21 - 22:31	UnknownUser
m	qsuball.m	r2 r1	manage	6.8 K	2006-06-05 - 03:28	WeiLwunLu
This topic: LCI > WebHome > WESTGRID
Topic revision: r6 - 2006-06-05 - WeiLwunLu