README.md

# slurm-partitions-stats

This is a `sacct` python analysis program, made for complex partitions analysis.

For instance, you can create `virtual_partitions` which can aggregate `sacct` data for many partitions pointing to the same physical nodes.

Main purpose is to get the usage percentage of physical partitions, and the waiting times. You will also retrieve informations about cputime by partitions.

> Please note that you can see 100%+ partition percentage; this is due to long time running jobs that can overlap between months.

## installation

Clone this repository, create a virtualenv and install numpy.

```bash
git clone https://forgemia.inra.fr/isdm-meso/slurm-partitions-stats.git
python3 -m venv slurmstats_venv
source slurmstats_venv/bin/activate
cd slurm-partitions-stats
python3 -m pip install numpy
```

## Configuration

You need to configure the `config/clusterstats.conf.sample` according to your cluster configuration and rename it to `config/clusterstats.conf`.


## Usage

First, you need data from your cluster with the `sacct` command.

For example, you can do the following:

```bash
#!/bin/bash
mkdir csv
for i in {2..4};
do
    for j in {1..12};
    do
        cur_month=$(printf %02d $j)
        nb_days=$(cal ${j} 202${i}| awk 'NF {DAYS = $NF}; END {print DAYS}')
        /usr/bin/sacct --format CPUTimeRaw,ElapsedRaw,AllocCPUS,Partition,User,Submit,Start --state COMPLETED --parsable2 -S 202${i}-${cur_month}-01 -E 202${i}-${cur_month}-${nb_days} > csv/202${i}-${cur_month}.csv 2>&1
    done
done
/usr/bin/sacct --format CPUTimeRaw,ElapsedRaw,AllocCPUS,Partition,User,Submit,Start --state COMPLETED --parsable2 -S 2025-01-01 -E 2025-01-31 > csv/2025-01.csv 2>&1
/usr/bin/sacct --format CPUTimeRaw,ElapsedRaw,AllocCPUS,Partition,User,Submit,Start --state COMPLETED --parsable2 -S 2025-02-01 -E 2025-02-28 > csv/2025-02.csv 2>&1
/usr/bin/sacct --format CPUTimeRaw,ElapsedRaw,AllocCPUS,Partition,User,Submit,Start --state COMPLETED --parsable2 -S 2025-03-01 -E 2025-03-21 > csv/2025-03.csv 2>&1
```

You will get statistics by month from 2022/01/01 to 2025/03/21 in a CSV directory.

Then, copy your `csv` directory in the `slurm-partitions-stats` directory.

After that, you just need to launch the analysis program :

```bash
python compute_elapse_percentage.py
```


You will get results in your terminal and in the `./results/csv/*` folders.