| Nevis Linux Cluster Disk Guide |
This is a guide to issues with disk storage on the
on the Linux cluster.
To find out how disks on one system can be access from another,
be sure to read the automount page.
Topics on this page include:
|
| How much disk space do I have? |
To find out how much disk space you have available, use the df command. You'll probably always want to use the -h option, so the sizes appear in human-readable form:
Here's the result of executing df -hl on the machine tanya on 28-Jan-2004:
On most of the systems in the cluster, /home is smaller than /data, sometimes much smaller; it's usually 4-6 GB for workstations and 15-35 GB for servers. It's intended that the /home be used for "source" files (program code, scientific papers, mail, etc.); /data should be used for large and re-creatable files (compiled binaries, data summaries, temporary work files, etc.).
These partitions also have different permissions set. Only a systems administrator can create a new directory in the /home partition; typically, such directories are for user's home directories. Anyone in the same group as the /data can create a file or directory there -- but after that, only the person who created the file/directory can delete it (the "sticky" bit is set).
An aside: Although you can do so, never turn on group-write permission on your home directory. You'll interfere with both ssh and mail (your ~/.forward file won't be recognized).
One final difference between /home and /data: the /home partition is backed up; /data is not. In fact, it goes one step further: the /data partition is always considered expendable for any type of system maintenance activity; if a system is being repaired, upgraded, or restored, the /data partition will probably be erased. (There's some advice on what do about this below.)
First, look to /data (or perhaps /home) partitions on other systems. If you use a workstation, the disk space on the server is always available to you. The /data partitions on all the systems that belong to a group are intended to be a shared resource; if you don't have enough space on /a/data/yourmachine, cd /a/data/othermachine in your group and see how much free space it has.
I strongly advise you to exercise common courtesy as you're scrounging for disk space. If I found someone had used a big chunk of my workstation's /data partition without asking, I might be annoyed. Also, be aware that a workstation user can reboot that system at any time; don't assume 100% availability of files stored on another user's workstation.
If you still don't have enough disk space on all your group's machines to satisfy your needs, you may have to request more disks be added to the existing systems (or buy a new box).
| Where should my home directory go? |
In a UNIX-based system, the home directory is your primary workspace. Your personal configuration files go here, your most important files will probably be stored here, and your home directory is key in the processing and storage of your e-mail.
When a systems administrator creates an account on the Nevis Linux cluster, he has a choice on where to place your home directory:
For users who have a system on their desk, this is usually the best solution. They don't usually have to share their /home partition with other users, and so have the freedom to manage their disk space. Also, although you can submit jobs on any system on which you can login, it's easier to submit jobs on your own workstation.
The disadvantage of this option is that workstations are usually less robust systems than the servers. Their disk drives, at least in theory, may not last as long as those in servers (although this has not been the case with newer workstations in recent years). Also, after a power outage, the servers will usually reboot automatically while a workstation may not -- or a workstation may boot faster than the servers, and not have access to some of the services.
(Before you're discouraged from this option, it may help to know that this is the option I choose.)
If you don't have a system on your desk, this will probably be the option that the systems administrator will choose. But even users with a desktop workstation have chosen this option. Typically, the issue is whether the desktop system reliably reboots after a power outage.
However, this option has its disadvantages. You have to share the /home partition with other users. Also, we have the speed paradox: because workstations are purchased more frequently than servers, the processor speed of a workstation is often faster than that of the workgroup's server. (The servers have other speed advantages, though: faster hard drives and dual processors.)
Generally, this option is reserved for those users who were never part of any research group at Nevis (e.g., the administrative staff), or who never made use of the Linux cluster.
Incidentally, home directories can be moved; if you're unhappy with its location, you can ask for a change.
| How often are backups made? |
The Nevis Linux cluster is normally backed up nightly onto shelley, the Nevis backup server. This includes the systems at the Nevis Annex. The Windows systems at Nevis are also backed up by this server.
Actually, we don't copy every file from every system; we use a program called rsync to copy over only those files that have changed since the day before.
While the entirety of the mail and archive directories (on franklin and archive) are backed up, we don't back up every file on every system on the cluster. The policy is: the /home partition is backed up; /data is not. There is a web page that contains the list of which partitions are backed up.
We maintain previous versions of old files on shelley. (Actually we do an incremental tar of the disk images after the rsync procedure has run for all the machines in the cluster.) This means we can recover old versions of files if necessary. However, there's a time limit: we only keep old file versions for three months. We cannot recover files that were deleted or overwritten prior to that.
At this point we usually get these questions:
We have vastly more disk storage on the Nevis cluster than we can hope to back up on any system that we can afford. As of 11-Dec-2006, of roughly 12TB of disk storage on the cluster, we back up ~1TB.
We therefore have to ask users to segregrate their files into key files that will be backed up, and re-creatable files that won't. The relative sizes of /home versus /data partitions help enforce this segregation.
The simplest thing to do is to make copies on other /data partitions in your workgroup's cluster. After all, to first order, that's all a backup is: a second copy of your files.
Another possibility (if the files are small enough) is to consider backing them up on CDs or DVDs, assuming that your workstation has a burner. For Linux systems, I recommend k3b, which is installed on most of the systems of the Nevis cluster.
We're doing what we can with the resources we have available. We don't have the disk space for a year's worth of backups.
You can supplement our backups with copies of your own. For example, I have my own private backup procedure for my critical source files. The procedure makes use of the rsync command; you can see it in ~seligman/bin/rsync.sh on the Linux cluster.
I run this script automatically a few times per day using cron. Here's a sample line from my crontab file:
As you look over my files, notice that I copy a subset of my home directory onto a data disk on another machine. Abusing this facility by backing gigabytes of files in your home directory onto someone else's system will probably get you yelled at.
| Long-term data storage |
For the purposes of this section, "long-term" means more than six months or so.
By the above definition, there is no long-term data storage at Nevis. As noted above:
If you need long-term storage for any of your files, I suggest you consider the facilities at BNL, FNAL, or CERN.
Once again, I offer the advice I gave above: If you want to assure that files stored in a /data directory will be preserved, make a copy of them on a second system's /data directory.
to the Nevis Linux cluster Page.
to the Nevis Computing Page.
to the Nevis Home Page.
Send any comments or questions to the
webmaster.