Batch server

As of 16-Aug-2007, the following paragraph is obsolete. The machine hermes is being repaired. The replacement condor batch manager is riverside.nevis.columbia.edu. However, as documented below, you don't need to login to riverside to use condor.

The system responsible for administering batches services is hermes.nevis.columbia.edu. Users typically cannot log in to this machine; you submit and monitor jobs from your local box on the Linux cluster. As far as job submission and execution are concerned, the existence of hermes may be completely transparent to you.

In addition to any RAID drives attached to your workgroup's servers, there are additional "common" RAID drives that are intended to be shared among the users of Nevis batch system. They are initially to be used by the ATLAS and D0 groups, as noted below, but may be made available to other groups as the need arises. These disks are available via automount on the Linux cluster; each has a capacity of about 1.5TB. The names of these drives are:

• /a/data/condor/array1/ - This is to be used for Nevis ATLAS FEB data, and by the Nevis D0 group.
• /a/data/condor/array2/ - This is to be used by the Nevis ATLAS group. Loosely speaking, this disk is intended to hold large files of data for individual use.
• /a/data/condor/array3/ - This is to be used by the Nevis ATLAS group. Loosely speaking, this disk is intended to hold large files for common use.

For example, the permissions on the drives have been set so that you can do the following from any machine on the Linux cluster (if you're a member of the ATLAS group):

Important! If you're skimming this page, stop and read the following paragraph!

The files on these /data partitions, like those on the /data partitions of any other systems on the Nevis cluster, are not backed up. They are stored on RAID5 arrays, which are a reliable form of storage; there is monitoring software that warns if any individual drives have failed. However, RAID arrays have been known to fail (and we've had at least one such failure at Nevis). If you have any critical data stored on these drives, make sure you backup the files yourself.

One more time: the disks on these partitions are not backed up!

Submitting batch jobs

The batch job submission system we're using at Nevis is Condor, developed at the University of Wisconsin. You can learn more about Condor from the User's Manual.

To use Condor at Nevis, the simplest way is to use the setup command:

Condor tips and tricks

• If you're just starting to learn Condor (as I am), a good start is to copy the Condor examples: cp -arv ~condor/condor-6.8.2/examples <your-own-work-area> cd <your-own-work-area>/examples

  Read the README file; type make to compile the programs; type sh submit to submit a few test jobs.

  You may notice that the sh_loop script will not execute; it will sit in the "Idle" state indefinitely. It won't execute unless you submit it in the vanilla universe as described below.

• The following commands will show you the machines available to run your jobs, their status, and their resources: condor_status condor_status -server

  Obviously, some machines are more powerful than others. Before you arbitrarily decide that only the most powerful machines are good enough for your jobs, consider:

  • It's true, a machine that's 1/4 as fast will take 4 times as long to execute your jobs. However, the demand for the faster machine may be more than four times as much; it's possible that your job will sit waiting in the queue longer than it would have taken to run on the slower box.
  • The CPU cycles on the slower machines are presently being wasted. You might be able to put them to some use.
  • If you have a large number of jobs to submit, the slower machine can chug away at a couple of them while the rest are waiting to execute on the faster processors.

  The best way to tell Condor that you'd prefer your job to execute on the faster machines is to use the Rank attribute in your submit file:

  Rank = Mips

  With all that said, if you want to restrict your job to the faster machines, you can try a statement like the following in your submit file:

  Requirements = (Mips > 2000)

  This would restrict your job to the fastest processors on the cluster.

• We have both 32-bit and 64-bit machines at Nevis. As a general rule, the 64-bit systems can execute programs compiled for 32-bit architectures, though not the other way around. However, condor appears to restrict job execution to machines with the same architecture as the machine that submitted the job.

  If you wish to enable your job to execute on both the 32-bit and 64-bit machines at Nevis, include the following statement in your submit file:

  Requirements = Arch=="INTEL" || Arch=="x86_64"

  This can be combined with other job requirements; e.g.:

  Requirements = (Mips > 2000) && ( Arch=="INTEL" || Arch=="x86_64" )

• As you read through the User's Manual chapter on job submission, note that we use a shared file system at Nevis.

• We have discovered that the vanilla environment described in the Condor manual does not behave exactly as documented at Nevis. The following advice may be helpful:
  • Unless you've specifically used the condor_compile command to compile your programs, you must submit your jobs in the "vanilla" universe. In particular, the Athena and D0 distribution kits do not use condor_compile, and must have the following line in a command script that makes use of those kits: universe = vanilla

  • Because we use a shared file system at Nevis that's based on automount, it's a good idea to include the initialdir attribute in your command scripts. For example, when I submit a script that makes use of files in my directory /a/data/tanya/seligman/kit/TestArea/, I include the following line in my command script to make sure the executing machine has correctly mounted the directory: initialdir = /a/data/tanya/seligman/kit/TestArea/

  • There appears to be a difference in the way the sh and csh shells handle files in Condor. In sh, bash, or zsh (the default at Nevis) the examples in the Condor manual basically work as they are. In csh or tcsh, the scripts will fail due to "file not found" errors unless you do one of the following:
    • Transfer all working files to the executing machine with the lines: should_transfer_files = YES when_to_transfer_output = ON_EXIT

    • Include the full path name when you reference any file in your command script. For example, in my scripts, the following line fails: executable = athena.csh but the following succeeds: executable = /a/data/tanya/seligman/kit/TestArea/athena.csh

• If you want to transfer files as part of your Condor job (e.g., for I/O efficiency), bear in mind that the temporary working directory for your job is in the ~condor home directory. This means that all Condor jobs from all users and all hosts share the same partition for their temporary files. If you fill up this disk, everyone's jobs may crash. Please be prudent, and only transfer files up to a few MB.

  If I/O efficiency is important, then consider manually transferring your files to a particular server, and requiring your job to execute on that machine in your submit file; for example:

  Requirements = (Machine == "sasha.nevis.columbia.edu")

  If you don't mind if the files are transferred, but would prefer it if they were not, instead of the above command you can use the Rank attribute (in this example, we also include a preference for a faster machine):

  Rank = (Machine == "sasha.nevis.columbia.edu") * 2000. + Mips

  The 2000 is a scaling factor. The value of Machine == "sasha.nevis.columbia.edu" will be 1.0 or 0.0, so we must scale it in order for the term to have roughly the same weight as the number of Mips.

• The batch farm is a heterogenous collection of machines; that is, they're not all running the same version of Fedora Core, nor do they all have the same version of gcc installed. If you're having problems with programs crashing on some systems but not on others, please read this page on compiler tools that can help solve this problem.

Many of the above tips, and others, have been combined into a set of example scripts. They are in ~seligman/condor/; start with the README file, which will point you to the other relevant files in the directory.

Availability of batch services

Use of Condor is not available to all systems at Nevis. If you would like access to the batch services (or feel that your system was omitted in error), please contact both Gustaaf Brooijmans and Bill Seligman.

	to the Nevis Home Page.
	to the previous Page.
	Send comments and suggestions to the webmaster