Condor

Now that you have some idea of what a Batch Farm is, let’s consider a particular batch farm.

HTCondor is a suite of batch-related software tools. It’s maintained by the Computer Science Department of the University of Wisconsin-Madison. It’s probably the most popular batch software in use by scientific institutions.1\(^,\)2

Here’s how the abstract “batch farm” in Figure 113 looks in condor.

condor farm — Figure 115: This figure defines a couple of terms: the “condor master” is the central computer that manages all the others; the “condor pool” are the groups of computers dedicated to running the jobs scheduled by the condor master.

The uniform color and shapes of the “Batch nodes” in the above figure may give you the impression that all the nodes have to be identical: the same amount of memory, disk space, CPU, etc. However, one of the nice features of condor is that it can manage a heterogenous collection of nodes. Consider the following:

multiple configs — Figure 116: In this figure, the different colors of the batch nodes represent different computer configurations. Those configurations might represent different amounts of memory or disk storage; they can also represent different CPU or GPU architectures; they can even represent different operating systems. (Yes: UNIX, Windows, and Mac OS can all exist on a single condor farm!)

This means that when you submit a job to condor (I’ll describe how do this soon, I promise!) there needs to be some mechanism by which condor can match what you need to run your program with what a batch node offers. This mechanism is called ClassAds.

classads — Figure 117: Here’s the ClassAd mechanism in action. Your program (“job”) has some set of ClassAds, even if you don’t specify any explicitly. Each batch node has its own set of ClassAds, such as the amount of memory it offers per job. Condor matches the job to run on an appropriate computer.

In the next section we’ll go over how to actually use HTCondor.

unreachable_state — Figure 118: https://xkcd.com/2200/ by Randall Munroe

1

I base this statement on the fact that I know of no other large-scale batch software manager used by any of the particle-physics institutions associated with Nevis.

The probable reason for this is that HTCondor is free. You can’t beat that price!

2

You may have noticed that the title of this section is Condor, but the name of the software suite is HTCondor. The maintainers of HTCondor had to change the name from Condor a few years ago due to legal conflicts.

This is the sort of thing that can happen in the legal world when you name your software after a species of vulture.

Since the names of all the HTCondor-related commands begin with the string condor_, I’m probably going to lapse into calling the suite condor. Please forgive the inaccuracy.