The following sections describe how to set up Condor for use in
special environments or configurations.
See section 3.4 on
page ![[*]](crossref.png)
for installation instructions on
the various Contrib modules that can be optionally downloaded and
installed.
If you are using AFS at your site, be sure to read section 3.3.5 on ``Shared Filesystem Config Files Entries'' for details on configuring your machines to interact with and use shared filesystems, AFS in particular.
Condor does not currently have a way to authenticate itself to AFS. This is true of the Condor daemons that would like to authenticate as AFS user Condor, and the condor_ shadow, which would like to authenticate as the user who submitted the job it is serving. Since neither of these things can happen yet, there are a number of special things people who use AFS with Condor must do. Some of this must be done by the administrator(s) installing Condor. Some of this must be done by Condor users who submit jobs.
The most important thing is that since the Condor daemons can't authenticate to AFS, the LOCAL_DIR (and it's subdirectories like ``log'' and ``spool'') for each machine must be either writable to unauthenticated users, or must not be on AFS. The first option is a VERY bad security hole so you should NOT have your local directory on AFS. If you've got NFS installed as well and want to have your LOCAL_DIR for each machine on a shared file system, use NFS. Otherwise, you should put the LOCAL_DIR on a local partition on each machine in your pool. This means that you should run condor_ install to install your release directory and configure your pool, setting the LOCAL_DIR parameter to some local partition. When that's complete, log into each machine in your pool and run condor_ init to set up the local Condor directory.
The RELEASE_DIR, which holds all the Condor binaries, libraries and scripts can and probably should be on AFS. None of the Condor daemons need to write to these files, they just need to read them. So, you just have to make your RELEASE_DIR world readable and Condor will work just fine. This makes it easier to upgrade your binaries at a later date, which means that your users can find the Condor tools in a consistent location on all the machines in your pool, and that you can have the Condor config files in a centralized location. This is what we do at UW-Madison's CS department Condor pool and it works quite well.
Finally, you might want to setup some special AFS groups to help your users deal with Condor and AFS better (you'll want to read the section below anyway, since you're probably going to have to explain this stuff to your users). Basically, if you can, create an AFS group that contains all unauthenticated users but that is restricted to a given host or subnet. You're supposed to be able to make these host-based ACLs with AFS, but we've had some trouble getting that working here at UW-Madison. What we have instead is a special group for all machines in our department. So, the users here just have to make their output directories on AFS writable to any process running on any of our machines, instead of any process on any machine with AFS on the Internet.
The condor_ shadow process runs on the machine where you submitted your Condor jobs and performs all file system access for your jobs. Because this process isn't authenticated to AFS as the user who submitted the job, it will not normally be able to write any output. So, when you submit jobs, any directories where your job will be creating output files will need to be world writable (to non-authenticated AFS users). In addition, if your program writes to stdout or stderr, or you're using a user log for your jobs, those files will need to be in a directory that's world-writable.
Any input for your job, either the file you specify as input in your submit file, or any files your program opens explicitly, needs to be world-readable.
Some sites may have special AFS groups set up that can make this unauthenticated access to your files less scary. For example, there's supposed to be a way with AFS to grant access to any unauthenticated process on a given host. That way, you only have to grant write access to unauthenticated processes on your submit machine, instead of any unauthenticated process on the Internet. Similarly, unauthenticated read access could be granted only to processes running on your submit machine. Ask your AFS administrators about the existence of such AFS groups and details of how to use them.
The other solution to this problem is to just not use AFS at all. If you have disk space on your submit machine in a partition that is not on AFS, you can submit your jobs from there. While the condor_ shadow is not authenticated to AFS, it does run with the effective UID of the user who submitted the jobs. So, on a local (or NFS) file system, the condor_ shadow will be able to access your files normally, and you won't have to grant any special permissions to anyone other than yourself. If the Condor daemons are not started as root however, the shadow will not be able to run with your effective UID, and you'll have a similar problem as you would with files on AFS. See the section on ``Running Condor as Non-Root'' for details.
Beginning with Condor version 6.0.1, you can use a single, global config file for all platforms in your Condor pool, with only platform-specific settings placed in separate files. This greatly simplifies administration of a heterogeneous pool by allowing you to change platform-independent, global settings in one place, instead of separately for each platform. This is made possible by the LOCAL_CONFIG_FILE parameter being treated by Condor as a list of files, instead of a single file. Of course, this will only help you if you are using a shared filesystem for the machines in your pool, so that multiple machines can actually share a single set of configuration files.
If you have multiple platforms, you should put all platform-independent settings (the vast majority) into your regular condor_config file, which would be shared by all platforms. This global file would be the one that is found with the CONDOR_CONFIG environment variable, user condor's home directory, or /etc/condor/condor_config.
You would then set the LOCAL_CONFIG_FILE parameter from that global config file to specify both a platform-specific config file and optionally, a local, machine-specific config file (this parameter is described in section 3.3.2 on ``Condor-wide Config File Entries'').
The order in which you specify files in the LOCAL_CONFIG_FILE parameter is important, because settings in files at the beginning of the list are overridden if the same settings occur in files later in the list. So, if you specify the platform-specific file and then the machine-specific file, settings in the machine-specific file would override those in the platform-specific file (which is probably what you want).
To specify the platform-specific file, you could simply use the ARCH and OPSYS parameters which are defined automatically by Condor. For example, if you had Intel Linux machines, Sparc Solaris 2.6 machines, and SGIs running IRIX 6.x, you might have files named:
condor_config.INTEL.LINUX
condor_config.SUN4x.SOLARIS26
condor_config.SGI.IRIX6
Then, assuming these three files were in the directory held in the ETC macro, and you were using machine-specific config files in the same directory, named by each machine's hostname, your LOCAL_CONFIG_FILE parameter would be set to:
LOCAL_CONFIG_FILE = $(ETC)/condor_config.$(ARCH).$(OPSYS), \
$(ETC)/$(HOSTNAME).local
Alternatively, if you are using AFS, you can use an ``@sys link'' to specify the platform-specific config file and let AFS resolve this link differently on different systems. For example, perhaps you have a soft linked named ``condor_config.platform'' that points to ``condor_config.@sys''. In this case, your files might be named:
condor_config.i386_linux2
condor_config.sun4x_56
condor_config.sgi_64
condor_config.platform -> condor_config.@sys
and your LOCAL_CONFIG_FILE parameter would be set to:
LOCAL_CONFIG_FILE = $(ETC)/condor_config.platform, \
$(ETC)/$(HOSTNAME).local
The only settings that are truly platform-specific are:
Reasonable defaults for all of these settings will be found in the default config files inside a given platform's binary distribution (except the RELEASE_DIR, since it is up to you where you want to install your Condor binaries and libraries). If you have multiple platforms, simply take one of the condor_config files you get from either running condor_ install or from the <release_dir>/etc/examples/condor_config.generic file, take these settings out and save them into a platform-specific file, and install the resulting platform-independent file as your global config file. Then, find the same settings from the config files for any other platforms you are setting up and put them in their own platform specific files. Finally, set your LOCAL_CONFIG_FILE parameter to point to the appropriate platform-specific file, as described above.
Not even all of these settings are necessarily going to be different. For example, if you have installed a mail program that understands the ``-s'' option in /usr/local/bin/mail on all your platforms, you could just set MAIL to that in your global file and not define it anywhere else. If you've only got Digital Unix and IRIX machines, the DAEMON_LIST will be the same for each, so there's no reason not to put that in the global config file (or, if you have no IRIX or Digital Unix machines, DAEMON_LIST won't have to be platform-specific either).
It is certainly possible that you might want other settings to be platform-specific as well. Perhaps you want a different startd policy for one of your platforms. Maybe different people should get the email about problems with different platforms. There's nothing hard-coded about any of this. What you decide should be shared and what should not is entirely up to you and how you lay out your config files.
Since the LOCAL_CONFIG_FILE parameter can be an arbitrary list of files, you can even break up your global, platform-independent settings into separate files. In fact, your global config file might only contain a definition for LOCAL_CONFIG_FILE, and all other settings would be handled in separate files.
You might want to give different people permission to change different Condor settings. For example, if you wanted some user to be able to change certain settings, but nothing else, you could specify those settings in a file which was early in the LOCAL_CONFIG_FILE list, give that user write permission on that file, then include all the other files after that one. That way, if the user was trying to change settings she/he shouldn't, they would simply be overridden.
As you can see, this mechanism is quite flexible and powerful. If you have very specific configuration needs, they can probably be met by using file permissions, the LOCAL_CONFIG_FILE setting, and your imagination.
In order to take advantage of two major Condor features: checkpointing and remote system calls, users of the Condor system need to relink their binaries. Programs that are not relinked for Condor can run in Condor's ``vanilla'' universe just fine, however, they cannot checkpoint and migrate, or run on machines without a shared filesystem.
To relink your programs with Condor, we provide a special tool, condor_ compile. As installed by default, condor_ compile works with the following commands: gcc, g++, g77, cc, acc, c89, CC, f77, fort77, ld. On Solaris and Digital Unix, f90 is also supported. See the condor_ compile(1) man page for details on using condor_ compile.
However, you can make condor_ compile work transparently with all commands on your system whatsoever, including make.
The basic idea here is to replace the system linker (ld) with the Condor linker. Then, when a program is to be linked, the condor linker figures out whether this binary will be for Condor, or for a normal binary. If it is to be a normal compile, the old ld is called. If this binary is to be linked for condor, the script performs the necessary operations in order to prepare a binary that can be used with condor. In order to differentiate between normal builds and condor builds, the user simply places condor_ compile before their build command, which sets the appropriate environment variable that lets the condor linker script know it needs to do its magic.
In order to perform this full installation of condor_ compile, the following steps need to be taken:
The actual commands that you must execute depend upon the system that you are on. The location of the system linker (ld), is as follows:
Operating System Location of ld (ld-path) Linux /usr/bin Solaris 2.X /usr/ccs/bin OSF/1 (Digital Unix) /usr/lib/cmplrs/cc
On these platforms, issue the following commands (as root), where ld-path is replaced by the path to your system's ld.
mv /[ld-path]/ld /[ld-path]/ld.real
cp /usr/local/condor/lib/ld /[ld-path]/ld
chown root /[ld-path]/ld
chmod 755 /[ld-path]/ld
On IRIX, things are more complicated in that there are multiple ld binaries that need to be moved, and symbolic links need to be made in order to convince the linker to work, since it looks at the name of it's own binary in order to figure out what to do.
mv /usr/lib/ld /usr/lib/ld.real
mv /usr/lib/uld /usr/lib/uld.real
cp /usr/local/condor/lib/ld /usr/lib/ld
ln /usr/lib/ld /usr/lib/uld
chown root /usr/lib/ld /usr/lib/uld
chmod 755 /usr/lib/ld /usr/lib/uld
mkdir /usr/lib/condor
chown root /usr/lib/condor
chmod 755 /usr/lib/condor
ln -s /usr/lib/uld.real /usr/lib/condor/uld
ln -s /usr/lib/uld.real /usr/lib/condor/old_ld
If you remove Condor from your system latter on, linking will continue to work, since the condor linker will always default to compiling normal binaries and simply call the real ld. In the interest of simplicity, it is recommended that you reverse the above changes by moving your ld.real linker back to it's former position as ld, overwriting the condor linker. On IRIX, you need to do this for both linkers, and you will probably want to remove the symbolic links as well.
NOTE: If you ever upgrade your operating system after performing a full installation of condor_ compile, you will probably have to re-do all the steps outlined above. Generally speaking, new versions or patches of an operating system might replace the system ld binary, which would undo the full installation of condor_ compile.
The condor keyboard daemon (condor_ kbdd) monitors X events on machines where the operating system does not provide a way of monitoring the idle time of the keyboard or mouse. In particular, this is necessary on Digital Unix machines and IRIX machines.
NOTE: If you are running on Solaris, Linux, or HP/UX, you do not need to use the keyboard daemon.
Although great measures have been taken to make this daemon as robust as possible, the X window system was not designed to facilitate such a need, and thus is less then optimal on machines where many users log in and out on the console frequently.
In order to work with X authority, the system by which X authorizes processes to connect to X servers, the condor keyboard daemon needs to run with super user privileges. Currently, the daemon assumes that X uses the HOME environment variable in order to locate a file named .Xauthority, which contains keys necessary to connect to an X server. The keyboard daemon attempts to set this environment variable to various users home directories in order to gain a connection to the X server and monitor events. This may fail to work on your system, if you are using a non-standard approach. If the keyboard daemon is not allowed to attach to the X server, the state of a machine may be incorrectly set to idle when a user is, in fact, using the machine.
In some environments, the keyboard daemon will not be able to connect to the X server because the user currently logged into the system keeps their authentication token for using the X server in a place that no local user on the current machine can get to. This may be the case if you are running AFS and have the user's X authority file in an AFS home directory. There may also be cases where you cannot run the daemon with super user privileges because of political reasons, but you would still like to be able to monitor X activity. In these cases, you will need to change your XDM configuration in order to start up the keyboard daemon with the permissions of the currently logging in user. Although your situation may differ, if you are running X11R6.3, you will probably want to edit the files in /usr/X11R6/lib/X11/xdm. The Xsession file should have the keyboard daemon startup at the end, and the Xreset file should have the keyboard daemon shutdown. As of patch level 4 of Condor version 6.0, the keyboard daemon has some additional command line options to facilitate this. The -l option can be used to write the daemons log file to a place where the user running the daemon has permission to write a file. We recommend something akin to $HOME/.kbdd.log since this is a place where every user can write and won't get in the way. The -pidfile and -k options allow for easy shutdown of the daemon by storing the process id in a file. You will need to add lines to your XDM config that look something like this:
condor_kbdd -l $HOME/.kbdd.log -pidfile $HOME/.kbdd.pid
This will start the keyboard daemon as the user who is currently logging in and write the log to a file in the directory $HOME/.kbdd.log/. Also, this will save the process id of the daemon to /.kbdd.pid, so that when the user logs out, XDM can simply do a:
condor_kbdd -k $HOME/.kbdd.pid
This will shutdown the process recorded in /.kbdd.pid and exit.
To see how well the keyboard daemon is working on your system, review the log for the daemon and look for successful connections to the X server. If you see none, you may have a situation where the keyboard daemon is unable to connect to your machines X server. If this happens, please send mail to condor-admin@cs.wisc.edu and let us know about your situation.
The CondorView server is an alternate use of the condor_ collector that logs information on disk, providing a persistent, historical database of pool state. This includes machine state, as well as the state of jobs submitted by users. Historical information logging can be turned on or off, so you can install the CondorView collector without using up disk space for historical information if you don't want it.
The CondorView collector is a condor_ collector that has been specially configured and running on a different machine from the main condor_ collector. Unfortunately, installing the CondorView collector on a separate host generates more network traffic (from all the duplicate updates that are sent from each machine in your pool to both collectors).
The following sections describe how to configure a machine to run a CondorView server and to configure your pool to send updates to it.
To configure the CondorView collector, you have to add a few settings
to the local configuration file of the chosen machine (a
seperate machine from the main condor_ collector) to enable
historical data collection.
These settings are described in detail in the Condor Version 6.6.0
Administrator's Manual, in
section 3.3.15 on
page .
A short explanation of the entries you must customize is provided
below.
NOTE: This directory should be separate and different from the spool or log directories already set up for Condor. There are a few problems putting these files into either of those directories.
Once these settings are in place in the local configuration file for your CondorView server host, you must to create the directory you specified in POOL_HISTORY_DIR and make it writable by the user your CondorView collector is running as. This is the same user that owns the CollectorLog file in your log directory. The user is usually condor.
After you've configured the CondorView attributes, you must configure Condor to automatically start the CondorView server. You do this by adding VIEW_SERVER to the DAEMON_LIST on this machine and defining what VIEW_SERVER means. For example:
VIEW_SERVER = $(SBIN)/condor_collector
DAEMON_LIST = MASTER, STARTD, SCHEDD, VIEW_SERVER
For this change to take effect, you must re-start the
condor_ master on this host (which you can do with the
condor_ restart command, if you run the command from a machine with
administrator access to your pool.
(See section 3.7.5 on
page for full details of IP/host-based
security in Condor).
NOTE: Before you spawn the CondorView server by restarting your condor_ master, you should make sure CONDOR_VIEW_HOST is defined in your configuration (as described in the following section).
For the CondorView server to function, you must configure your pool to send updates to it. You do this by configuring your existing condor_ collector to forward its updates to the CondorView server. All the Condor daemons in the pool send their ClassAd updates to the regular condor_ collector, which in turn will forward them onto the CondorView server.
You do this by defining the following setting in your configuration file:
CONDOR_VIEW_HOST = full.hostname
where full.hostname is the full hostname of the machine where you
are running your CondorView collector.
You should place this setting in your global configuration file, since it should be the same for both the main condor_ collector and the CondorView server. If you do not have a shared global configuration file for Condor, you should put the same value in the configuration files on both the main condor_ collector and the CondorView server host.
Once this setting is in place, you can finally restart the condor_ master at your CondorView server host (or spawn the condor_ master if it is not yet running). Once the CondorView server is running, you can finally send a condor_ reconfig to your main condor_ collector for the change to take effect so it will begin forwarding updates.
This section describes how to configure the condor_ startd for SMP (Symmetric Multi-Processor) machines. Beginning with Condor version 6.1, machines with more than one CPU can be configured to run more than one job at a time. As always, owners of the resources have great flexibility in defining the policy under which multiple jobs may run, suspend, vacate, etc.
The way SMP machines are represented to the Condor system is that the shared resources are broken up into individual virtual machines (each virtual machine is called a VM). Each VM can be matched and claimed by users. Each VM is represented by an individual ClassAd (see the ClassAd reference, section 4.1, for details). In this way, each SMP machine will appear to the Condor system as a collection of separate VMs. As an example, an SMP machine named vulture.cs.wisc.edu would appear to Condor as the multiple machines, named vm1@vulture.cs.wisc.edu, vm2@vulture.cs.wisc.edu, vm3@vulture.cs.wisc.edu, and so on.
The way that the condor_ startd breaks up the shared system resources into the different virtual machines is configurable. All shared system resources (like RAM, disk space, swap space, etc.) can either be divided evenly among all the virtual machines, with each CPU getting its own virtual machine, or you can define your own virtual machine types, so that resources can be unevenly partitioned. The following section gives details on how to configure Condor to divide the resources on an SMP machine into separate virtual machines.
This section describes the settings that allow you to define your own virtual machine types and to control how many virtual machines of each type are reported to Condor.
There are two main ways to go about partitioning an SMP machine:
Beginning with Condor version 6.1.6, the number of each type being reported can be changed at run-time, by issuing a reconfiguration command to the condor_ startd daemon (sending a SIGHUP or using condor_ reconfig). However, the definitions for the types themselves cannot be changed with reconfiguration. If you change any VM type definitions, you must use condor_ restart
condor_restart -startdfor that change to take effect.
To define your own virtual machine types, add configuration file parameters that list how much of each system resource you want in the given VM type. Do this by defining configuration variables of the form VIRTUAL_MACHINE_TYPE_<N> . The <N> represents an integer (for example, VIRTUAL_MACHINE_TYPE_1), which specifies the virtual machine type defined. This number is used to configure how many VMs of this type are advertised.
A type describes what share of the total system resources a given virtual machine has available to it.
The type can be defined by:
The attributes that specify the number of CPUs and the total amount of RAM in the SMP machine do not change. For these attributes, specify either absolute values or percentages of the total available amount. For example, in a machine with 128 Mbytes of RAM, all the following definitions result in the same allocation amount.
mem=64 mem=1/2 mem=50%
Other attributes are dynamic, such as disk space and swap space. For these, specify a percentage or fraction of the total value that is allocated to each VM, instead of specifying absolute values. As the total values of these resources change on your machine, each VM will take its fraction of the total and report that as its available amount.
The four attribute names are case insensitive when defining VM types. The first letter of the attribute name distinguishes between the attributes. The four attributes, with several examples of acceptable names for each are
As an example, consider a host of 4 CPUs and 256 megs of RAM. Here are valid example VM type definitions. Types 1-3 are all equivalent to each other, as are types 4-6
VIRTUAL_MACHINE_TYPE_1 = cpus=2, ram=128, swap=25%, disk=1/2
VIRTUAL_MACHINE_TYPE_2 = cpus=1/2, memory=128, virt=25%, disk=50%
VIRTUAL_MACHINE_TYPE_3 = c=1/2, m=50%, v=1/4, disk=1/2
VIRTUAL_MACHINE_TYPE_4 = c=25%, m=64, v=1/4, d=25%
VIRTUAL_MACHINE_TYPE_5 = 25%
VIRTUAL_MACHINE_TYPE_6 = 1/4
The number of virtual machines of each type is set with the configuration variable NUM_VIRTUAL_MACHINES_TYPE_<N> , where N is the type as given in the VIRTUAL_MACHINE_TYPE_<N>variable.
Note that it is possible to set the configuration variables such that they specify an impossible configuration. If this occurs, the condor_ startd daemon fails after writing a message to its log attempting to indicate the configuration requirements that it could not implement.
If you are not defining your own VM types, then all resources are divided equally among the VMs. The number of VMs within the SMP machine is the only attribute that needs to be defined. Its definition is accomplished by setting the configuration variable NUM_VIRTUAL_MACHINES to the integer number of machines desired. If variable NUM_VIRTUAL_MACHINES is not defined, it defaults to the number of CPUs within the SMP machine.
Section 3.6 details the Startd Policy Configuration. This section continues the discussion with respect to SMP machines.
Each virtual machine within an SMP machine is treated as an independent machine, each with its own view of its machine state. There is a single set of policy expressions for the SMP machine as a whole. This policy may consider the VM state(s) in its expressions. This makes some policies easy to set, but it makes other policies difficult or impossible to set.
An easy policy to set configures how many of the virtual machines notice console or tty activity on the SMP as a whole. VMs that are not configured to notice any activity will report ConsoleIdle and KeyboardIdle times from when the condor_ startd daemon was started, (plus a configurable number of seconds). With this, you can set up a multiple CPU machine with the default policy settings plus add that the keyboard and console noticed by only one virtual machine. Assuming a reasonable load average (see section 3.10.6 below on ``Load Average for SMP Machines''), only the one virtual machine will suspend or vacate its job when the owner starts typing at their machine again. The rest of the virtual machines could be matched with jobs and leave them running, even while the user was interactively using the machine. If the default policy is used, all virtual machines notice tty and console activity and currently running jobs would suspend or preempt.
This example policy is controlled with the following configuration variables.
These configuration variables are fully described in
section 3.3.8 on
page which lists all the
configuration file settings for the condor_ startd.
The configuration of virtual machines allows each VM to advertise its own machine ClassAd. Yet, there is only one set of policy expressions for the SMP machine as a whole. This makes the implementation of certain types of policies impossible. While evaluating the state of one VM (within the SMP machine), the state of other VMs (again within the SMP machine) are not available. Decisions for one VM cannot be based on what other machines within the SMP are doing.
Specifically, the evaluation of a VM policy expression works in the following way.
To set a different policy for the VMs within an SMP machine,
a (SUSPEND) policy will be of the form
SUSPEND = ( (VirtualMachineID == 1) && (PolicyForVM1) ) || \
( (VirtualMachineID == 2) && (PolicyForVM2) )
where (PolicyForVM1) and (PolicyForVM2) are the
desired expressions for each VM.
Most operating systems define the load average for an SMP machine as the total load on all CPUs. For example, if you have a 4-CPU machine with 3 CPU-bound processes running at the same time, the load would be 3.0 In Condor, we maintain this view of the total load average and publish it in all resource ClassAds as TotalLoadAvg.
Condor also provides a per-CPU load average for SMP machines. This nicely represents the model that each node on an SMP is a virtual machine, separate from the other nodes. All of the default, single-CPU policy expressions can be used directly on SMP machines, without modification, since the LoadAvg and CondorLoadAvg attributes are the per-virtual machine versions, not the total, SMP-wide versions.
The per-CPU load average on SMP machines is a Condor invention. No system call exists to ask the operating system for this value. Condor already computes the load average generated by Condor on each virtual machine. It does this by close monitoring of all processes spawned by any of the Condor daemons, even ones that are orphaned and then inherited by init. This Condor load average per virtual machine is reported as the attribute CondorLoadAvg in all resource ClassAds, and the total Condor load average for the entire machine is reported as TotalCondorLoadAvg. The total, system-wide load average for the entire machine is reported as TotalLoadAvg. Basically, Condor walks through all the virtual machines and assigns out portions of the total load average to each one. First, Condor assigns the known Condor load average to each node that is generating load. If there's any load average left in the total system load, it is considered an owner load. Any virtual machines Condor believes are in the Owner state (like ones that have keyboard activity), are the first to get assigned this owner load. Condor hands out owner load in increments of at most 1.0, so generally speaking, no virtual machine has a load average above 1.0. If Condor runs out of total load average before it runs out of virtual machines, all the remaining machines believe that they have no load average at all. If, instead, Condor runs out of virtual machines and it still has owner load remaining, Condor starts assigning that load to Condor nodes as well, giving individual nodes with a load average higher than 1.0.
This section describes how the condor_ startd daemon handles its debugging messages for SMP machines. In general, a given log message will either be something that is machine-wide (like reporting the total system load average), or it will be specific to a given virtual machine. Any log entrees specific to a virtual machine have an extra header printed out in the entry: vm#:. So, for example, here's the output about system resources that are being gathered (with D_ FULLDEBUG and D_ LOAD turned on) on a 2-CPU machine with no Condor activity, and the keyboard connected to both virtual machines:
11/25 18:15 Swap space: 131064 11/25 18:15 number of kbytes available for (/home/condor/execute): 1345063 11/25 18:15 Looking up RESERVED_DISK parameter 11/25 18:15 Reserving 5120 kbytes for file system 11/25 18:15 Disk space: 1339943 11/25 18:15 Load avg: 0.340000 0.800000 1.170000 11/25 18:15 Idle Time: user= 0 , console= 4 seconds 11/25 18:15 SystemLoad: 0.340 TotalCondorLoad: 0.000 TotalOwnerLoad: 0.340 11/25 18:15 vm1: Idle time: Keyboard: 0 Console: 4 11/25 18:15 vm1: SystemLoad: 0.340 CondorLoad: 0.000 OwnerLoad: 0.340 11/25 18:15 vm2: Idle time: Keyboard: 0 Console: 4 11/25 18:15 vm2: SystemLoad: 0.000 CondorLoad: 0.000 OwnerLoad: 0.000 11/25 18:15 vm1: State: Owner Activity: Idle 11/25 18:15 vm2: State: Owner Activity: Idle
If, on the other hand, this machine only had one virtual machine connected to the keyboard and console, and the other VM was running a job, it might look something like this:
11/25 18:19 Load avg: 1.250000 0.910000 1.090000 11/25 18:19 Idle Time: user= 0 , console= 0 seconds 11/25 18:19 SystemLoad: 1.250 TotalCondorLoad: 0.996 TotalOwnerLoad: 0.254 11/25 18:19 vm1: Idle time: Keyboard: 0 Console: 0 11/25 18:19 vm1: SystemLoad: 0.254 CondorLoad: 0.000 OwnerLoad: 0.254 11/25 18:19 vm2: Idle time: Keyboard: 1496 Console: 1496 11/25 18:19 vm2: SystemLoad: 0.996 CondorLoad: 0.996 OwnerLoad: 0.000 11/25 18:19 vm1: State: Owner Activity: Idle 11/25 18:19 vm2: State: Claimed Activity: Busy
As you can see, shared system resources are printed without the header (like total swap space), and VM-specific messages (like the load average or state of each VM) get the special header appended.
Beginning with Condor version 6.1.5, Condor can run on machines with multiple network interfaces. Here are some common scenarios that you might encounter and how you go about solving them.
Often users of Condor wish to set up ``compute farms'' where there is one machine with two network interface cards(one for the public internet, and one for the private net). It is convenient to set up the ``head'' node as a central manager in most cases and so here are the instructions required to do so.
Setting up the central manager on a machine with more than one NIC can be a little confusing because there are a few external variables that could make the process difficult. One of the biggest mistakes in getting this to work is that either one of the separate interfaces is not active, or the host/domain names associated with the interfaces are incorrectly configured.
Given that the interfaces are up and functioning, and they have good host/domain names associated with them here is how to configure Condor:
In this example, farm-server.farm.org maps to the private interface.
On your central manager's global(to the cluster) config file:
CONDOR_HOST = farm-server.farm.org
On your central manager's local configuration file:
NETWORK_INTERFACE = ip address of farm-server.farm.org
NEGOTIATOR = $(SBIN)/condor_negotiator
COLLECTOR = $(SBIN)/condor_collector
DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR, SCHEDD, STARTD
If your central manager and farm machines are all NT, then you only have vanilla universe and it will work now. However, if you have this setup for UNIX, then at this point, standard universe jobs should be able to function in the pool, but if you did not configure the UID_DOMAIN macro to be homogeneous across the farm machines, the standard universe jobs will run as nobody on the farm machines.
In order to get vanilla jobs and file server load balancing for standard universe jobs working(under unix), you need to do some more work both in the cluster you have put together and in Condor to make everything work. First, you need a file server(which could also be the central manager) to serve files to all of the farm machines. This could be NFS or AFS, it does not really matter to Condor. The mount point of the directories you wish your users to use must be the same across all of the farm machines. Now, configure UID_DOMAIN and FILESYSTEM_DOMAIN to be homogeneous across the farm machines and the central manager. Now, you will have to inform Condor that an NFS or AFS filesystem exists and that is done in this manner. In the global(to the farm) configuration file:
# If you have NFS USE_NFS = True # If you have AFS HAS_AFS = True USE_AFS = True # if you want both NFS and AFS, then enable both sets above
Now, if you've set up your cluster so that it is possible for a machine name to never have a domain name(for example: you've placed a machine name but no fully qualified domain name in /etc/hosts), you must configure DEFAULT_DOMAIN_NAME to be the domain that you wish to be added on to the end of your hostname.
If you have a client machine with two or more NICs, then there might be
a specific network interface with which you desire a client machine to
communicate with the rest of the Condor pool. In this case, in the local
configuration file for that machine, place:
NETWORK_INTERFACE = ip address of interface desired
If your Checkpoint Server is on a machine with multiple interfaces, the only way to get things to work is if your different interfaces have different hostnames associated with them, and you set CKPT_SERVER_HOST to the hostname that corresponds with the IP address you want to use in the global configuration file for your pool. You will still need to specify NETWORK_INTERFACE in the local config file for your Checkpoint Server.
CONDOR_HOST = machX.cs.wisc.edu COLLECTOR_HOST = $(CONDOR_HOST) NEGOTIATOR_HOST = $(CONDOR_HOST)the configuration might be
CONDOR_HOST = machX.cs.wisc.edu COLLECTOR_HOST = $(CONDOR_HOST):9650 NEGOTIATOR_HOST = $(CONDOR_HOST):9651
If a Condor pool is completely behind a firewall, then no special consideration is needed. However, if there is a firewall between the machines within a Condor pool, then configuration variables may be set to force the usage of specific ports and to utilize a specific range of ports.
By default, Condor uses port 9618 for the condor_ collector daemon, 9614 for the condor_ negotiator daemon, and system-assigned (apparently random) ports for everything else. See section 3.10.8, if non standard ports are to be used for the condor_ collector condor_ negotiator daemons.
The configuration variables HIGHPORT and LOWPORT facilitate setting a restricted range of ports that Condor will use. This may be useful if behind a firewall. The configuration macros HIGHPORT and LOWPORT only affect these system-assigned ports, but will restrict them to the range specified. These configuration variables are fully defined in section 3.3.2. Note that both HIGHPORT and LOWPORT must be at least 1024.
An experimental mechanism for allocating bandwidth for checkpointing and remote system calls has been introduced in Condor version 6.3.0. This mechanism enables the condor_ negotiator to limit job placements and preemptions to within configured bandwidth limits. If a bandwidth limit is reached for a host or network subnet, the condor_ negotiator won't schedule jobs that require additional bandwidth on that host or subnet. Instead, the condor_ negotiator will attempt to run the job on another host or network where bandwidth is available. If that is not possible, the job will remain idle until network load decreases below the configured limits.
Allocating bandwidth allows the system to perform more efficiently when the network is a bottleneck and avoids oversubscribing the capacity of networks and servers. Limiting Condor's bandwidth usage can also be a way to reserve bandwidth for other uses.
To allocate bandwidth, the condor_ negotiator must have information about jobs' bandwidth requirements and bandwidth usage. Condor is able to obtain information about the bandwidth requirements for checkpoint and executable transfers and remote system calls. Other network usage, such as NFS or AFS I/O is currently not monitored or allocated.
The condor_ negotiator allocates bandwidth using a sliding window in time. The size of the window defines the allocation granularity and is typically set to the condor_ negotiator's scheduling interval. For example, a bandwidth limit of 10 Mbps using the default window of 5 minutes will restrict bandwidth allocations to 375 MB every 5 minutes.
The following parameters must be defined in your central manager's configuration file to enable bandwidth allocation.
The following optional parameters may also be defined in your central manager's configuration file.
To enable collection of network usage information in the Condor pool,
MANAGE_BANDWIDTH = True
should be defined for all machines in the Condor pool (if possible).
If $(CKPT_SERVER_HOST) is defined, then
$(STARTD_EXPRS) should also include CkptServer:
CkptServer : "$(CKPT_SERVER_HOST)"
STARTD_EXPRS = CkptServer
Finally, $(STARTD_EXPRS) should contain the following
attributes:
STARTD_JOB_EXPRS = ImageSize, ExecutableSize, JobUniverse
The configuration file specified by the $(NETWORK_ROUTING_INFO) macro defines a network routing table for Condor's bandwidth allocation, allowing the condor_ negotiator to allocate bandwidth for network segments in addition to network hosts. To allocate bandwidth for a network transfer, the condor_ negotiator computes the transfer's route from the routing table and allocates bandwidth on each hop in the route.
The format of the configuration file is:
IP-ADDR SUBNET-MASK
--> NEXT-HOP IP-ADDR SUBNET-MASK
where IP-ADDR, SUBNET-MASK, and NEXT-HOP are all given in the standard
numbers-and-dots notation. The first line defines a network resource
and the "-->" lines that follow define hops from that network resource
to other network resources.
A rule applies to a network address when the subnet-masked bits of the
address match the rule's address.
If an address matches multiple rules, the routing algorithm chooses
the match with the most bits in the mask.
The simplest configuration is:
0.0.0.0 0.0.0.0
This configuration defines a single network segment connecting all
endpoints.
The SUBNET-MASK of 0.0.0.0 will match any IP address.
Any bandwidth limits defined for the 0.0.0.0 network will be applied
to all transfers between endpoints.
Bandwidth limits can also be set for specific endpoint addresses using
this configuration.
The following example defines a network with 2 subnets, connected to each other through a backbone network:
0.0.0.0 0.0.0.0
--> 128.105.101.0 128.105.101.0 255.255.255.0
--> 128.105.102.0 128.105.102.0 255.255.255.0
128.105.101.0 255.255.255.0
--> 0.0.0.0 0.0.0.0 0.0.0.0
128.105.102.0 255.255.255.0
--> 0.0.0.0 0.0.0.0 0.0.0.0
Some example routes that would be computed from this configuration
are:
128.105.101.5 --> 128.105.101.0 --> 0.0.0.0
--> 128.105.102.0 --> 128.105.102.3
128.105.101.5 --> 128.105.101.0 --> 128.105.101.7
128.105.101.5 --> 128.105.101.0 --> 0.0.0.0
--> 128.105.65.3
Depending on how you intend to use it, the routing table can be very detailed or may describe a very idealized representation of your network. There is no need to include endpoints in the table. The route always starts with the source address and ends with the destination address of a network flow.
The configuration file specified by the $(NETWORK_CAPACITY_INFO) parameter defines bandwidth limits for network segments and hosts in the network. An empty file defines no limits.
The format of the configuration file is:
IP-ADDR CAPACITY
where IP-ADDR indicates an endpoint IP address or a network resource
from the routing table configuration file in the standard
numbers-and-dots notation and CAPACITY is a floating-point number
indicating the network capacity (in Mbps) of the resource.
For example:
128.105.101.0 40.0
128.105.65.3 5.0
defines a 40 Mbps limit on the 128.105.101.0 subnet and a 5 Mbps limit
for the host 128.105.65.3.
Beginning with Condor version 6.3.0, users can submit applications to Condor which cannot be preempted and which require multiple resources. Condor's unique solution to this problem involves a combination of opportunistic scheduling and dedicated scheduling within a single system. Opportunistic scheduling involves placing jobs on non-dedicated resources under the assumption that the resources might not be available for the entire duration of the jobs. Dedicated scheduling assumes the constant availability of resources to compute fixed schedules. In other words, dedicated scheduling involves placing jobs on resources where it is assumed that the job can run to completion without interruption.
This section describes how to configure a Condor pool for scheduling
and running parallel jobs on dedicated resources.
The user manual, section 2.10 on page
contains information on how to submit MPI jobs under Condor.
To support dedicated applications, a Condor administrator must configure some resources in the pool to be dedicated resources. These are machines which will not preempt jobs. These resources are controlled by a dedicated scheduler, a single machine within the pool that runs a condor_ schedd daemon. In general, there is no limit on the number of dedicated schedulers within a Condor pool. However, each dedicated resource may only be managed by a single dedicated scheduler. Therefore, running multiple dedicated schedulers in a single pool results in a greater fragmentation of dedicated resources. This can create a situation where jobs will not run, because the jobs can not get needed resources.
After a condor_ schedd daemon has been selected as the dedicated scheduler for the pool and resources are configured to be dedicated, users submit MPI jobs to that condor_ schedd. When an idle MPI job is found in the queue, the dedicated scheduler performs its own scheduling algorithm to find appropriate resources for the job. The dedicated scheduler claims the resources and uses them to service the MPI job. When a resource can no longer be used to serve dedicated jobs, it is allowed to run opportunistic jobs.
We recommend that you select a single host to act as the
dedicated scheduler.
This is the host from which all users submit their MPI jobs.
If you have a dedicated cluster of compute nodes and a single
front-end machine from which users are supposed to submit jobs, that
machine would be a perfect choice for your dedicated
scheduler.
If your pool does not have an obvious choice for a submit machine,
choose a host that all of your users can log into, and one
that is likely to be up and running all the time.
All of the Condor's other resource requirements for a submit node apply to
this machine, such as having enough disk space in the spool
directory to hold jobs (see section 3.2.3 on
page for details on these issues).
Once you have selected a machine to serve as the dedicated scheduler, ensure that the machine is running version of the condor_ schedd and condor_ shadow daemons that support MPI jobs. These versions must be the same, and they should be at least 6.3.0. The default configuration files with Condor version 6.3.0 include all required settings.
To configure a dedicated resource under a given scheduler, the resource owner or administrator sets a few lines in the condor_ startd's configuration file. Starting with Condor version 6.3.0, all of these settings are supplied in an example local configuration file called condor_config.local.dedicated.resource which can be found in the etc directory once you unpack the Condor binaries.
Each dedicated resource advertises a special attribute in its ClassAd that says which dedicated scheduler it is willing to be managed by. This is accomplished by modifying the following lines within the local configuration file for any dedicated resource:
DedicatedScheduler = ``DedicatedScheduler@full.host.name'' STARTD_EXPRS = $(STARTD_EXPRS), DedicatedScheduler
Substitute the real host name of the dedicated scheduler machine.
All dedicated resources must have policy expressions which allow for
dedicated jobs to always run and to never be evicted.
The resource must also be configured to prefer jobs from the dedicated
scheduler over all other jobs.
A dedicated resource in Condor is simply configured so that the
dedicated scheduler of its choice has the highest rank.
See section 3.6 on
page
for more details on Condor's policy expressions.
It is worth noting that Condor puts no other requirements on a resource for it to be considered dedicated. If the owners of desk-top workstations were willing to allow their machines to be configured in this way, those workstations would be dedicated resources in Condor, and would behave exactly like the nodes in a Beowulf compute cluster.
To aid in the definition of the policy expressions, the dedicated scheduler adds an attribute to all resource request ClassAds it generates, the Scheduler attribute. This attribute identifies each ClassAd as a request of a particular dedicated scheduler. For example, if your dedicated scheduler were running on a host named front-end.cs.wisc.edu, the Scheduler attribute for all jobs submitted from there would be DedicatedScheduler@front-end.cs.wisc.edu.
The owners of the resources can easily define separate policies for dedicated and opportunistic jobs, simply by including two cases in each policy expression, one case for when the Scheduler attribute identifies the request as one belonging to the preferred dedicated scheduler, and one for if the Scheduler attribute is not defined or points to a different scheduler.
In the following sections, we will discuss a couple of different policy scenarios you might want to use for your dedicated resources and give you the exact policy expressions to put in your configuration file to implement them. The configuration settings for each scenario are provided in the condor_config.local.dedicated.resource file.
NOTE: You can configure different resources in your pool to have different dedicated policies. For example, you might have a cluster of machines in racks which have no interactive user and which can always run jobs, along-side desk-top machines that are willing to run dedicated jobs when necessary, but which will still preempt and evict non-dedicated jobs if the machine is being used by its owner. Both of these policy scenarios are discussed below, and both might be present in a single pool. In other words, the following policy scenarios are specific to a given machine, not to a whole pool.
One possible scenario for the use of dedicated resources is to only allow dedicated jobs to run on them. This is the most basic policy for dedicated resources. To enact this policy, the following expressions are used in the configuration file:
START = Scheduler =?= $(DedicatedScheduler) SUSPEND = False CONTINUE = True PREEMPT = False KILL = False WANT_SUSPEND = False WANT_VACATE = False RANK = Scheduler =?= $(DedicatedScheduler)
The START expression specifies that the Scheduler attribute in the job ClassAd must match the string specified for the DedicatedScheduler attribute in the machine ClassAd. The RANK expression specifies that a job with the Scheduler attribute appropriately defined will have the highest rank, which will prevent any other jobs from preempting it based on user priorities. The rest of the expressions disable all of the condor_ startd daemon's regular policies for evicting jobs when keyboard and CPU activity is discovered on the machine.
While the first example works nicely for dedicated jobs, it can lead to poor resource utilization if there are not enough dedicated jobs to keep the dedicated machines busy. A more sophisticated strategy allows the machines to run non-dedicated jobs when no dedicated jobs exist. The machine is configured to prefer dedicated jobs, yet run opportunistic jobs if no dedicated jobs are available. Note that those jobs that do not require a dedicated resource are executed as if they were dedicated jobs.
To implement this, configure machines as dedicated resources. Then, modify the START expression to be:
START = True
A third policy example allows both dedicated and non-dedicated jobs. It assumes resources that are not configured to prefer or always run dedicated jobs. These desk-top machines have a START expression that takes the machine owner's usage into account for non-dedicated jobs. The machine does not preempt jobs that must run on dedicated resources, while it will preempt other jobs based on a previously set policy for running jobs. So, the default pool policy is used for starting and stopping non-dedicated jobs, but dedicated jobs always start and are not preempted.
Allowing both dedicated and opportunistic jobs on the resources requires that an opportunistic policy is already defined. In the local configuration file for resources with this hybrid policy, a second case is added to policy expressions that overrides the initial policy expression specifically for dedicated jobs. The following represent the only settings that need to be modified to implement this policy.
SUSPEND = Scheduler =!= $(DedicatedScheduler) && ($(SUSPEND)) PREEMPT = Scheduler =!= $(DedicatedScheduler) && ($(PREEMPT)) RANK_FACTOR = 1000000 RANK = (Scheduler =?= $(DedicatedScheduler) * $(RANK_FACTOR)) + $(RANK) START = (Scheduler =?= $(DedicatedScheduler)) || ($(START))
NOTE: For everything to work, you MUST set MacroRANK_FACTOR to be a larger value than the maximum value your existing rank expression could possibly evaluate to. RANK is just a floating point value, so there is no harm in having a value that is very large.
TCP sockets are reliable, connection-based sockets that guarantee the delivery of any data sent. However, TCP sockets are fairly expensive to establish, and there is more network overhead involved in sending and receiving messages.
UDP sockets are datagrams, and are not reliable. There is very little overhead in establishing or using a UDP socket, but there is also no guarantee that the data will be delivered. All previous Condor versions used UDP sockets to send updates to the condor_ collector, and this did not cause problems.
Beginning with version 6.5.0, Condor can be configured to use TCP sockets to send updates to the condor_ collector instead of UDP datagrams. It is not intended for most sites. This feature is targeted at sites where UDP updates are lost because of the underlying network. Most Condor administrators that believe this is a good idea for their site are wrong. Do not enable this feature just because it sounds like a good idea. The only cases where an administrator would want this feature are if the ClassAd updates are consistently not getting to the condor_ collector. An example where this may happen is if the pool is comprised of machines across a wide area network (WAN) where UDP packets are frequently dropped.
Configuration variables are set to enable the use of TCP sockets. There are two variables that an administrator must define to enable this feature:
The use of a cache allows Condor to leave established TCP sockets open, facilitating much better performance. Subsequent updates can reuse an already open socket. The work to establish a TCP connection may be lengthy, including authentication and setting up encryption. Therefore, Condor requires that a socket cache be defined if TCP updates are to be used. TCP updates will be refused by the condor_ collector daemon if a cache is not enabled.
Each Condor daemon will have 1 socket open to the condor_ collector. So, in a pool with N machines, each of them running a condor_ master, condor_ schedd, and condor_ startd, the condor_ collector would need a socket cache that has at least 3*N entries. Machines running Personal Condor in the pool need an additional two entries (for the condor_ master and condor_ schedd) for each Personal Condor installation.
Every cache entry utilizes a file descriptor within the condor_ collector daemon. Therefore, be careful not to define a cache that is larger than the number of file descriptors the underlying operating system allocates for a single process.
NOTE: At this time, UPDATE_COLLECTOR_WITH_TCP, only affects the main condor_ collector for the site, not any sites that a condor_ schedd might flock to.