Skip to content
By Yiyan Hao · Last updated 2026-08-20

Submitting Jobs

Types of Jobs

Jobs can be submitted to the cluster in two modes: interactive or batch jobs. Interactive jobs are used for exploratory tasks, such as testing code or running small analyses. Batch jobs are used for tasks that cost relatively large computational resources and do not require user interaction. Array jobs are a special type of batch job that is ideal for running multiple instances of the same job with different parameters.

Interactive Jobs

The most common use of interactive jobs is to run an interactive session of a software (e.g., R, Python). This allows you to use the computational resources of the cluster while still having access to the user-friendly interface of the software. We will use R as an example below.

When you log into the cluster via a login node, interactive jobs can be submitted to an interactive queue (e.g., taki_interactive). The submission command will ask LSF to allocate a compute node for you:

ibash

Once the job starts, your terminal session will be transferred to an interactive shell on the compute node (e.g., pennsive01), where you can start an R session as usual by running R in the terminal. Refer to the pages for basics about R sessions and package management on the cluster.

Batch Jobs

While interactive jobs are useful for exploratory tasks, they are generally slower and prone to interruptions in the case of network issues. For computational tasks that are more resource-intensive and time-consuming, it is recommended to submit them via batch jobs. Unlike interactive jobs, a submitted batch job will not be affected should you be disconnected from the cluster, and you will be able to monitor its progress. Batch jobs are submitted to a normal queue (e.g., taki_normal).

As a quick example, run the following command to submit a batch job:

bsub sleep 60

This job simply runs the sleep command for 60 seconds. Once it's done, you will receive an email containing the job output.

Example Output

Here is an example of the output file generated by the above job script:

Job <sleep 60> was submitted from host <takim2> by user <yiyanhao> in cluster <PMACS-SCC> at Mon May 25 21:00:32 2026
Job was executed on host(s) <pennsive01>, in queue <taki_normal>, as user <yiyanhao> in cluster <PMACS-SCC> at Mon May 25 21:00:33 2026
</home/yiyanhao> was used as the home directory.
</home/yiyanhao> was used as the working directory.
Started at Mon May 25 21:00:33 2026
Terminated at Mon May 25 21:01:34 2026
Results reported at Mon May 25 21:01:34 2026

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
sleep 60
------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :                                   0.18 sec.
    Max Memory :                                 8 MB
    Average Memory :                             7.17 MB
    Total Requested Memory :                     -
    Delta Memory :                               -
    Max Swap :                                   -
    Max Processes :                              3
    Max Threads :                                4
    Run time :                                   61 sec.
    Turnaround time :                            62 sec.

The output (if any) follows:

Although it is possible to submit batch jobs directly from the command line (as we just did), it is more common to write a job script (.sh file) that contains all the necessary commands and parameters for the job. This allows for better organization and reproducibility of the job submission process.

For illustration purposes, we assume that you want to run an R script /home/your_username/scripts/test_script.R. You can create a job script test_script.sh with the following content:

#BSUB -J "test_job"
#BSUB -o /home/your_username/logs/test_job.%J.out
#BSUB -e /home/your_username/logs/test_job.%J.err
#BSUB -u your_email@pennmedicine.upenn.edu
#BSUB -n 2
#BSUB -q taki_normal
#BSUB -R "rusage[mem=10000]"
#BSUB -sp 40
#BSUB -m "pennsive01"

Rscript /home/your_username/scripts/test_script.R

Then, run the following command in terminal to submit the job:

bsub < /home/your_username/scripts/test_script.sh

In this submission file, the lines starting with #BSUB provide instructions for LSF on how to run the job; the last line is the actual command to run the R script. You may add additional bash commands as if you were running them on a terminal (e.g., changing directories, loading modules, etc.). Labels that we typically specify include:

  • -J: job name, which will be displayed in the job queue and used in the output file name.
  • -o: output file path, where the standard output of the job will be saved (%J will be replaced by the job ID assigned by LSF). This may be in your home directory, a project directory (in case too many output files are stored and cause memory issues in your home directory), or /scratch (if you don't expect to keep it after logging off). If not provided, the output will be sent via email.
  • -e: error file path, where the standard error of the job will be saved. If not provided, the error will be sent via email.
  • -u: email address, where notifications about the job status will be sent (e.g., when the job starts, ends, or fails).
  • -n: number of cores to use for the job. This is important for parallel computing, where setting -n to a value greater than 1 allows the job to utilize multiple CPU cores. Default is 1.
  • -q: queue name, which determines the resources allocated to the job. Default is a normal queue depending on your affiliation (e.g., taki_normal). You can check available queues via bqueues.
  • -R: resource requirements (in MB), which specify the amount of memory and other resources needed for the job. In this example, we request 10 GB of memory. It is recommended that you run a test job to determine the appropriate amount of memory needed for your job to avoid memory issues.
  • -sp: priority of the job, which determines the order in which jobs are scheduled. Higher priority is assigned to jobs with a larger number. Default is 50.
  • -m: host name, which specifies the compute node to run the job on. Specifying a host can ensure that all your jobs run on the same node, which can be useful for debugging and for jobs that require a large amount of memory. However, it may also lead to longer wait times if the specified node is busy. Default is any available node. You can check available nodes via bhosts to determine which host to use.

The IBM documentation page provides helpful tips on what additional parameters you can specify.

Array Jobs

An array job is helpful when you want to run the same batch job multiple times, each time with a slightly different parameter setup, but otherwise using the same code or logic. For example, you may want to apply the same image processing pipeline to 10 different subjects; or in a simulation study, you may want to run the same model with 10 different hyperparameter values. Instead of making 10 duplicates of the same job script with only one line changed, you can write a single job script and automate parameter sweeping via an array job. Keep in mind that in more complicated settings where simulations involve inner loops or parallelizations, it is important to plan ahead how you would like to set up an array job.

Similar to batch jobs, we need a job script to submit an array job, and the actual script itself to perform the task. To dictate the parameter value used for a given instance of the job, we will specify a sequence of job indices in the submission script, and query this environment variable in the actual job script. Here, we will demonstrate this with a simple example of simulating data with different sample sizes from the same standard normal distribution and calculating the sample mean.

First, we will write the job script /home/your_username/scripts/simulate_mean.R:

# simulate_mean.R

# Get array index from LSF (1-based index)
task_id <- as.integer(Sys.getenv("LSB_JOBINDEX"))

# Define one sample size per array task
sample_sizes <- c(10, 50, 100, 500, 1000)

# Use the array index to select the sample size
n <- sample_sizes[task_id]

set.seed(1000 + task_id)

# Simulate data from standard normal
x <- rnorm(n, mean = 0, sd = 1)

# Calculate sample mean
result <- data.frame(
  task_id = task_id,
  sample_size = n,
  sample_mean = mean(x)
)

cat("Finished task", task_id, "\n")
cat("Sample size:", n, "\n")
cat("Sample mean:", mean(x), "\n")

Then, write the submission script /home/your_username/scripts/submit_simulate_mean.sh:

#!/bin/bash
#BSUB -J "mean_sim[1-5]"
#BSUB -o /home/your_username/logs/simulate_mean.%J.out
#BSUB -e /home/your_username/logs/simulate_mean.%J.err
#BSUB -u your_email@pennmedicine.upenn.edu
#BSUB -q taki_normal
#BSUB -n 1
#BSUB -R "rusage[mem=1000]"
#BSUB -m "pennsive01"

Rscript /home/your_username/scripts/simulate_mean.R

Submit the job by:

bsub < /home/your_username/scripts/submit_simulate_mean.sh

Caution

Please always be mindful of the number of (array) jobs you submit. Some tips for avoiding overloading the cluster:

  • Test with toy examples or a single array: Always run the job with one array task (i.e. job_name[1] in -J) first to make sure that the code runs correctly before scaling up to multiple tasks.

  • Perform a dry run: Sometimes the submission script gets complicated (e.g., a bash script to submit jobs under for-loops). It is a good idea to check that jobs will be submitted as expected through a dry run, without executing them yet. For example, add cat() around the code chunk that contains bsub commands and check the printed output to avoid unintended nested for-loops.

  • Monitor the job queue: Periodically check the job queue with bjobs to see if your jobs are running as expected.

Monitoring Jobs

After submitting a job, you can monitor its status and progress using the following commands:

  • bjobs: shows the status of all your jobs in the queue, including their job ID, name, status (e.g., RUN, PEND), execution host, and other details. If you submitted an array job, you will see one line for each instance of the job in the queue.
  • bjobs -l <job_id>: provides detailed information about a specific job, including its memory usage, CPU usage, and other detailed information.
  • bkill <job_id>: allows you to terminate a running job at any time.
  • bkill -u username: allows you to terminate all jobs submitted by a specific user (e.g., yourself).

Some variants of the bjobs command that you may find helpful:

  • bjobs | head -n 50: shows the top 50 jobs in the queue.
  • bjobs -a -u all -noheader -o "jobid" | sort | uniq | tail -n 10: shows the most recently submitted 10 jobs.

You may also modify BSUB parameters of a submitted, pending job using the bmod command. For example, if you realize that you need to request more memory for a pending job, you can run bmod -R "rusage[mem=20000]" -sp 90 <job_id> to update the memory requirement to 20 GB and priority to 90.