script on Alán's blog

Calculate the cumulative sum of a column using DuckDB

Wed, 22 Oct 2025 20:34:00 -0400

Duckdb, the (tabular) data exploration tool I use supports window operations. I recently discovered that it can also perform cumulative sums in a very efficient manner.

Let us generate a toy dataset where we want to calculate the sum of one column relative to the order of another one.

 -- seeding for reproducibility, creating a table to hide output
CREATE OR REPLACE TABLE seed AS SELECT SETSEED(0.1);

-- Create a mock dataset with two integer columns
CREATE OR REPLACE TABLE my_table AS
SELECT
 #1 AS column_1,
 CAST(FLOOR(RANDOM() * 100) AS INT) AS column_2
FROM generate_series(1, 10); -- This generates 10 rows
SELECT * FROM my_table;

-- We write it to a csv for future use
COPY my_table TO my_table.csv;

┌──────────┬──────────┐
│ column_1 │ column_2 │
│ int64 │ int32 │
├──────────┼──────────┤
│ 1 │ 27 │
│ 2 │ 45 │
│ 3 │ 2 │
│ 4 │ 84 │
│ 5 │ 84 │
│ 6 │ 26 │
│ 7 │ 18 │
│ 8 │ 65 │
│ 9 │ 97 │
│ 10 │ 11 │
├──────────┴──────────┤
│ 10 rows 2 columns │
└─────────────────────┘

If we wanted to calculate the distribution of the cumulative sum of the table we could use the OVER clause to perform the sum of column_2 in the order defined by column_1.

SELECT *, sum(column_2) OVER (ORDER by column_1) AS cumulative_sum FROM my_table

┌──────────┬──────────┬────────────────┐
│ column_1 │ column_2 │ cumulative_sum │
│ int64 │ int32 │ int128 │
├──────────┼──────────┼────────────────┤
│ 1 │ 27 │ 27 │
│ 2 │ 45 │ 72 │
│ 3 │ 2 │ 74 │
│ 4 │ 84 │ 158 │
│ 5 │ 84 │ 242 │
│ 6 │ 26 │ 268 │
│ 7 │ 18 │ 286 │
│ 8 │ 65 │ 351 │
│ 9 │ 97 │ 448 │
│ 10 │ 11 │ 459 │
├──────────┴──────────┴────────────────┤
│ 10 rows 3 columns │
└──────────────────────────────────────┘

The cumulative sum can be pretty handy to get a general notion of a distribution. As a bonus tip, I’ll show how to use duckdb in a one-liner to plot the data directly in a terminal by using gnuplot.

duckdb -csv -c "
 SELECT *, sum(column_2) OVER (ORDER by column_1) AS cumulative_sum
 FROM read_csv('my_table.csv');" |
 gnuplot -e "
 set terminal dumb;
 set datafile separator ',';
 set style data histograms;
 set style fill solid 1.00 border -1;
 set xlabel 'Column 1';
 set ylabel 'CSum';
 set title 'Cumulative Sum of values';
 plot '-' using 3:xtic(1);" |
 tr -d '\014' # Remove a pesky ^L at the top



 Cumulative Sum of values
 500 +-----------------------------------------------------------------+
 | + + + + + + + + + ++ |
 450 |-+ '-' using 3:xtic+-+ +-||--+-|
 400 |-+ |#| || +-|
 | |#| || |
 350 |-+ ++ |#| || +-|
 | || |#| || |
 300 |-+ +-+ || |#| || +-|
 250 |-+ ++ |#| || |#| || +-|
CSum | +-+ || |#| || |#| || |
 200 |-+ |#| || |#| || |#| || +-|
 | |#| || |#| || |#| || |
 150 |-+ ++ |#| || |#| || |#| || +-|
 | || |#| || |#| || |#| || |
 100 |-+ +-+ || |#| || |#| || |#| || +-|
 50 |-+ ++ |#| || |#| || |#| || |#| || +-|
 | +-+ || |#| || |#| || |#| || |#| || |
 0 +-----------------------------------------------------------------+
 1 2 3 4 5 6 7 8 9 10
 Column 1

We get a cute ascii-like plot! That is a bit too long of a “one-liner”, I’ll go through the commands:

Run a duckdb command (-c) that reads the previously-saved table. The -csv flag at the starts converts the output to csv.
Run gnuplot with certain specifications:
- The flag -e Allows to pass a series of commands without an interactive session.
- set terminal dumb: it will send as plain text to stdout.
- set datafiler separator ",": The input is a CSV file.
- set style data histograms: Changes the plotting style into a barplot.
- set style fill solid ...: Visual adjustments to the bars for clarity.
- set xlabel ... Adds the axis labels. Similar for ylabel and title.
- plot '-' using 3:xtic(1): Use stdin data to Plot the columns 3 on the y axis (cumulative_sum) and the first column in the x-axis (column_1).
Lastly, use the tr command line tool to remove a ^L That appeared at the start of the output and was bothering me too much.

While there are a many other ways to wrangle tables such as via pandas or polars in Python, I find duckdb to be a powerful tool for exploratory analyses and data wrangling (often from within Python). It is flexible enough to be used by itself, via bindings in another language, or directly on the command line. Lastly, I showed that when used as a Command Line Interface (CLI) duckdb synergises with other tools for data visualisation from the comfort(?) of the terminal.

Run multiple python scripts in the background

Tue, 26 Aug 2025 14:29:00 -0400

To solve a multitude of challenges I have faced when processing high throughput microscopy data, have developed Nahual, a tool that allows me to move data across multiple Python environments that deploy deep learning models in the background. I usually keep these models “listening” in the background for the main analysis pipeline (aliby) to send them data to process. To be able to monitor what’s going on inside of these scripts I use GNU screen, which allows me to detach and reattach into these sessions whenever I need to. At some point I had to reboot my server and had rerun all these in independent screens. This rudimentary shell script did the job:

cd cellpose
screen -d -S cellpose1 -m bash -c 'nix develop . --command bash -c "python server.py ipc:///tmp/cellpose1.ipc"'
screen -d -S cellpose2 -m bash -c 'nix develop . --command bash -c "python server.py ipc:///tmp/cellpose2.ipc"'
cd ../trackastra
screen -d -S trackastra -m bash -c 'nix develop . --command bash -c "python server.py ipc:///tmp/trackastra.ipc"'
cd ..

Basically the screen runs my Nix environment and deploys the model (in this case, my fork of cellpose with Nix dependency management) while detached. This executes a server.py file within the Nix enviroment, it runs on a loop waiting to receive data and process it. Automatically deploying to multiple screens reduces the annoyance of having to the usual steps of (go to the folder -> run screen -> Nix environment -> run Python server -> Detach screen session). I just add more models if I want further deployments, put it in a bash script and call it a day.

To access any of these screens for inspection I just use the name indicated after the -S flag (e.g., screen -r cellpose1). This way I can check if any issue crops up in the main analysis script or pipeline.

Simple progress indicators with awk

Tue, 19 Aug 2025 18:58:00 -0400

I wanted a simple way to see the progress of a data processing pipeline, and the internal progress bar tools were messed up by threading. I thus decided to use the number of output files in each folder as an indicator of progress. In my case the output of tree . looks like this:

.
└── steps
 ├── A01_001
 │   ├── segment_nuclei
 │   │   ├── 0000.npz
 │   │   ├── 0001.npz
 │   │   ├── ...
 │   │   └── 0019.npz
 │   ├── tile
 │   │   ├── 0000.npz
 │   │   ├── 0001.npz
 │   │   ├── ...

I can get the info I need by counting the total number of files and the occurrences of the A01_001 -> P24_005 range (these are fields of view from a microscopy experiment). Using this simple find command we get all the files in the current folder.

find . -type f

which results in this:

./steps:
./steps/A01_003/tile/0007.npz
./steps/A01_003/tile/0009.npz
./steps/A01_003/tile/0018.npz
./steps/A01_003/tile/0016.npz
...

We could use wc -l to get the number of files per directory, we want a bunch of progress bars to get a better sense of change over time. For this I use awk, my swiss-army knife for text processing, and I write a short script that counts, sorts and prints the number of occurrences as a number of dots. I also added a conditional to only track after more than one file has been produced, for pipelines that produce save one file before actually running the whole pipeline.

# progress_bar.awk
{
 if (match($0,"([A-P][0-9]{2}_[0-9]{3})", capture)){
 count[capture[1]] += 1
 }

}
END{
 n=asorti(count, sorted)
 for (i=1; i<=n; i++){
 nfiles = count[sorted[i]]
 if (nfiles > 1){
 s = sprintf(key "%*s", nfiles, "");
 gsub(".", ".", s)
 print sorted[i] " " s
 }
 }
}

Running the find command and the awk script (find . -type f | awk -f progress_bar.awk) yields the following snapshot of the processing progess

A01_001 ...............................................................
A01_002 ...............................................................
A01_003 ...............................................................
A01_004 ...............................................................
A01_005 ...............................................................
A02_001 ...............................................................
A02_002 ...............................................................
A02_003 .................................................
A02_004 ..........................................
A02_005 ........................................
A03_001 ..............................................

Thus the last thing to do is to use `watch` to automatically refresh the status:

watch -dc --interval 1 'find . -type f | awk -f progress_bar.awk | tac'

The watch flag -d highlight the changes over time and -c enables intrepreting ANSI colours, in my terminal this makes the changes last stay longer, but YMMV. Finally, tac makes sure that the last lines are displayed at the top. I like to run this command somewhere in another terminal or in a `screen` terminal multiplexer. When the number of rows becomes too high it may be useful find a heuristic to remove uninformative lines.

Update figure numbering

Thu, 14 Aug 2025 15:42:00 -0400

I was editing some markdown and had to insert a new figure in the middle. The problem is that this document already has an explicit figure numbering (e.g., “Figure 5”), so changing tens of figures felt dull. I like to run small (GNU) awk scripts for this type of tasks.

# update_figures.awk
{
 if (match($0, "Figure ([0-9]+)", num)){
 if (num[1] > after)
 gsub("Figure ([0-9]+)", "Figure " num[1] + increase_by)
 };
 print $0
}

This changes Figure X into Figure X + increase_by starting after the variable “after”. And we can run it as follows:

awk -v after=4 -v increase_by=1 -f update_figures.awk input_file.md

To edit the file in-place add the -i flag.