Alán's blog

Exploring the MBTA public dataset using DuckDB

Sun, 12 Apr 2026 12:40:00 -0400

To showcase the real-life usefulness of Duckdb (and SQL-adjacent Domain Specific Languages in general) I decided to use the public datasets made available by the Massachusetts Bay Transport Authority (MBTA). I have lived in Boston for a couple of years and wanted to test if my intuition of the busy lines and stations lined up with their data.

There are multiple available (tabular) datasets:

Ridership by Trip, Route line and stop
Monthly ridership by month
Gated station entries
Passenger surveys

.maxrows 11

INSTALL httpfs;
LOAD httpfs;

CREATE OR REPLACE TABLE monthly_ridership AS (SELECT * FROM read_csv('https://hub.arcgis.com/api/v3/datasets/a2d15ddd86b34867a31cd4b8e0a83932_0/downloads/data?format=csv&spatialRefId=4326&where=1%3D1'));

SELECT column_name, column_type FROM (DESCRIBE monthly_ridership);

┌─────────────────────────────────┬──────────────────────────┐
│ column_name │ column_type │
│ varchar │ varchar │
├─────────────────────────────────┼──────────────────────────┤
│ service_date │ TIMESTAMP WITH TIME ZONE │
│ mode │ VARCHAR │
│ route_or_line │ VARCHAR │
│ total_monthly_weekday_ridership │ BIGINT │
│ average_monthly_weekday_ridersh │ BIGINT │
│ countofdates_weekday │ BIGINT │
│ total_monthly_ridership │ DOUBLE │
│ average_monthly_ridership │ BIGINT │
│ countofdates │ BIGINT │
│ ObjectId │ BIGINT │
├─────────────────────────────────┴──────────────────────────┤
│ 10 rows 2 columns │
└────────────────────────────────────────────────────────────┘

We first loaded the httpfs extension to pull the data directly from their website. I am using a local database but this should also work without writing to a file thanks to the in-memory database capabilities of duckdb.

Then we created a new table monthly_ridership by running a subquery (another valid SQL expression surrounded by parentheses). This will download and save the CSV table into a table in the database (in-memory or into a file that works as a database).

Lastly, we describe the table and I like to filter out other columns that are not informative. I predominantly care about the column names and data types. Here the ones we care about are either total_monthly_weekday_ridership or average_monthly_weekday_ridership alongside route_or_line.

We will thus group by route or line to see the average ridership per route.

SELECT route_or_line, CAST(MEAN(total_monthly_weekday_ridership) AS INTEGER) AS mean_monthly_weekday_ridership FROM monthly_ridership GROUP BY route_or_line ORDER BY mean_monthly_weekday_ridership DESC;

┌───────────────┬────────────────────────────────┐
│ route_or_line │ mean_monthly_weekday_ridership │
│ varchar │ int32 │
├───────────────┼────────────────────────────────┤
│ Bus │ 7395469 │
│ Red Line │ 5326706 │
│ Orange Line │ 4410355 │
│ Green Line │ 3893135 │
│ Commuter Rail │ 2684943 │
│ Blue Line │ 1392658 │
│ Silver Line │ 724622 │
│ The RIDE │ 138221 │
│ Boat-F1 │ 66248 │
│ Boat-F3 │ 23922 │
│ Boat-F4 │ 19975 │
├───────────────┴────────────────────────────────┤
│ 11 rows 2 columns │
└────────────────────────────────────────────────┘

We can see here all 5 lines of the metro system, in addition to commuter rail, buses, The RIDE (a door-to-door service for folks unable to ride the fixed routes) and several boat routes that cross the Boston Harbour.

Gated entries give info on specific entrances. We first fetch the table and print the schema.

CREATE OR REPLACE TABLE gated_entries AS (SELECT * FROM read_csv('https://hub.arcgis.com/api/v3/datasets/001c177f07594e7c99f193dde32284c9_0/downloads/data?format=csv&spatialRefId=4326&where=1%3D1'));
SELECT column_name, column_type FROM (DESCRIBE gated_entries);

┌───────────────┬──────────────────────────┐
│ column_name │ column_type │
│ varchar │ varchar │
├───────────────┼──────────────────────────┤
│ service_date │ TIMESTAMP WITH TIME ZONE │
│ time_period │ VARCHAR │
│ stop_id │ VARCHAR │
│ station_name │ VARCHAR │
│ route_or_line │ VARCHAR │
│ gated_entries │ DOUBLE │
│ ObjectId │ BIGINT │
└───────────────┴──────────────────────────┘

The schema makes sense in general, we care the most about station_name, route_or_line and gated_entries. Before aggregating, it is worth checking the time span the data covers.

SELECT MIN(service_date) AS start_date, MAX(service_date) AS end_date FROM gated_entries;

┌──────────────────────────┬──────────────────────────┐
│ start_date │ end_date │
│ timestamp with time zone │ timestamp with time zone │
├──────────────────────────┼──────────────────────────┤
│ 2024-08-25 00:00:00-04 │ 2026-02-28 00:00:00-05 │
└──────────────────────────┴──────────────────────────┘

So the dataset covers about a year and a half, from August 2024 to February 2026. We can perform “quality control” to check if there are stations with very few records.

SELECT days_recorded,
COUNT(*) AS num_stations,
FIRST(route_or_line) AS example_line
FROM (
 SELECT station_name, route_or_line,
 COUNT(DISTINCT service_date) AS days_recorded
 FROM gated_entries
 GROUP BY station_name, route_or_line
)
GROUP BY days_recorded
ORDER BY days_recorded DESC;

┌───────────────┬──────────────┬──────────────┐
│ days_recorded │ num_stations │ example_line │
│ int64 │ int64 │ varchar │
├───────────────┼──────────────┼──────────────┤
│ 553 │ 26 │ Red Line │
│ 552 │ 3 │ Orange Line │
│ 551 │ 4 │ Orange Line │
│ 550 │ 2 │ Blue Line │
│ 549 │ 4 │ Orange Line │
│ 548 │ 3 │ Blue Line │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ 536 │ 1 │ Green Line │
│ 526 │ 1 │ Green Line │
│ 523 │ 3 │ Green Line │
│ 518 │ 1 │ Green Line │
│ 512 │ 1 │ Green Line │
├───────────────┴──────────────┴──────────────┤
│ 22 rows (11 shown) 3 columns │
└─────────────────────────────────────────────┘

My first impression is that the Green Line has fewer records than the others. If you have lived in Boston this should make sense, since part of the Green Line runs like a tram/light rail, I would thus expect the logistics of data collection to be trickier in above-ground stations.

SELECT route_or_line, ROUND(AVG(days_recorded),1) AS avg_days_per_station
FROM (
 SELECT route_or_line, station_name, COUNT(DISTINCT service_date) AS days_recorded
 FROM gated_entries
 GROUP BY route_or_line, station_name
)
GROUP BY route_or_line
ORDER BY avg_days_per_station DESC;

┌───────────────┬──────────────────────┐
│ route_or_line │ avg_days_per_station │
│ varchar │ double │
├───────────────┼──────────────────────┤
│ Silver Line │ 551.7 │
│ Orange Line │ 549.7 │
│ Red Line │ 549.0 │
│ Blue Line │ 546.3 │
│ Green Line │ 538.3 │
│ Mattapan Line │ 538.0 │
└───────────────┴──────────────────────┘

Are the stations above ground the ones with fewer records?

SELECT station_name, COUNT(DISTINCT service_date) AS days_recorded
FROM gated_entries
WHERE route_or_line = 'Green Line'
GROUP BY station_name
ORDER BY days_recorded
LIMIT 10;

┌─────────────────┬───────────────┐
│ station_name │ days_recorded │
│ varchar │ int64 │
├─────────────────┼───────────────┤
│ Union Square │ 512 │
│ Magoun Square │ 518 │
│ Ball Square │ 523 │
│ Medford/Tufts │ 523 │
│ East Somerville │ 523 │
│ Gilman Square │ 526 │
│ Copley │ 536 │
│ Boylston │ 537 │
│ Kenmore │ 538 │
│ Arlington │ 541 │
├─────────────────┴───────────────┤
│ 10 rows 2 columns │
└─────────────────────────────────┘

Indeed all the stations under 530 days recorded are part of the Green Line extension, opened in 2022. We know that they were running at the time this dataset was collected, so it is a bit surprising that they have the most missing data (even if less than 10%). I am curious about the stations that people use the most. Let’s look at the top 10 stations with the most gated entries.

SELECT station_name, route_or_line, CAST(SUM(gated_entries) AS INT) AS gated_entries
FROM gated_entries
GROUP BY station_name, route_or_line
ORDER BY gated_entries DESC;

┌───────────────────┬───────────────┬───────────────┐
│ station_name │ route_or_line │ gated_entries │
│ varchar │ varchar │ int32 │
├───────────────────┼───────────────┼───────────────┤
│ Harvard │ Red Line │ 5359023 │
│ Back Bay │ Orange Line │ 4772945 │
│ Copley │ Green Line │ 4184767 │
│ North Station │ Orange Line │ 4099727 │
│ Central │ Red Line │ 4094223 │
│ Kendall/MIT │ Red Line │ 4054482 │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ Government Center │ Blue Line │ 258449 │
│ Union Square │ Green Line │ 186641 │
│ Ball Square │ Green Line │ 174971 │
│ Magoun Square │ Green Line │ 154011 │
│ East Somerville │ Green Line │ 92067 │
├───────────────────┴───────────────┴───────────────┤
│ 78 rows (11 shown) 3 columns │
└───────────────────────────────────────────────────┘

There are two issues with naive aggregation: First, some stations are part of multiple routes. We thus should remove the route_or_line grouping. Second, the raw sums are skewed if not all stations have records covering the same time span. Since the dataset has a service_date column, we can normalise by the number of distinct dates each station appears in to get a fairer average daily figure.

We thus adjust our query to make these changes: Aggregate data from different lines and normalize it by the number of days recorded for each station.

SELECT
 station_name,
 LIST(DISTINCT route_or_line) AS lines,
 CAST(SUM(gated_entries) / COUNT(DISTINCT service_date) AS INT) AS avg_daily_entries
FROM gated_entries
GROUP BY station_name
ORDER BY avg_daily_entries DESC;

┌───────────────────┬───────────────────────────┬───────────────────┐
│ station_name │ lines │ avg_daily_entries │
│ varchar │ varchar[] │ int32 │
├───────────────────┼───────────────────────────┼───────────────────┤
│ North Station │ [Orange Line, Green Line] │ 11315 │
│ South Station │ [Red Line, Silver Line] │ 10342 │
│ Harvard │ [Red Line] │ 9691 │
│ Downtown Crossing │ [Red Line, Orange Line] │ 9492 │
│ Back Bay │ [Orange Line] │ 8774 │
│ Park Street │ [Green Line, Red Line] │ 8017 │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ Medford/Tufts │ [Green Line] │ 540 │
│ Suffolk Downs │ [Blue Line] │ 483 │
│ Union Square │ [Green Line] │ 365 │
│ Ball Square │ [Green Line] │ 335 │
│ Magoun Square │ [Green Line] │ 297 │
│ East Somerville │ [Green Line] │ 176 │
├───────────────────┴───────────────────────────┴───────────────────┤
│ 71 rows (12 shown) 3 columns │
└───────────────────────────────────────────────────────────────────┘

This better matches my intuitive impression from being in those areas. Harvard, despite being a major transit hub for buses and the T, is outranked by North and South Station once considering all lines. Park Street jumped from 19th before all the way up to sixth. The new Green Line stations in Cambridge still rank at the bottom of daily entries. There is a stark difference between the most and least “popular” stations. For every person recording their entry on East Somerville there are 62 in North Station. East Somerville (and all the other Green Line stations in Cambridge) do not enforce checks on ticket purchases. The same can be said for stations west of Kenmore, but those are located in an area with much denser population, since multiple universities are based in that area.

Overall, I really like duckdb due to the flexibility and speed for data crunching analysis. It is fast, simple and it integrates well with notebook-like workflows and Command-Line Interface (CLI) usage. The translation from questions to queries to tables is seamless. I believe it is a tool worth mastering, since it provides a Swiss Army Knife for everyday data processing.

There are some caveats worth mentioning. For instance, it is unclear to me how they are differentiating the Red and the Green Line entries in Park Street, since it is a two-layered station with the green lines on top, and one can only access the Red line from the Green Line. While it seemed negligible for these questions, the frequency and data acquisition differs across stations. There may be a weekday vs weekend bias that we are not accounting for. That said, I’m glad that my intuition of the usage of stations and lines matches my mental model.

Meta conclusion: I used this post also to test literate programming to incrementally build a data crunching workflow. In this case I coupled with an org-mode notebook (that is how I generate this blog) to explore a public dataset for fun. When wrapping up, I had a couple of lingering questions and a notion of the necessary query, but not enough time. I used an agent to add those in the middle of the analysis, which I evaluated via org-babel in a quick feedback loop. It worked shockingly well. Turns out this is quite similar to the recently released marimo-pair, an extension for reproducible data analysis notebooks using agents. I want to further explore the potential of having an agent as an interface for data analysis, where I still review and check that the code fulfills its intended purpose. In the end the goal is a reproducible artifact that gives us new insights on the data we are processing, and I think this approach facilitates rapid and reproducible data exploration and analysis.

Set up email hosting and a personal website on personal domain

Mon, 24 Nov 2025 19:38:00 -0500

At some point I was struggling to get access to my Gmail account. Since I usually block unwanted scripts from running on my computer, Google likes to flag my Gmail login attempts as suspicious activity. This would be fine if it didn’t also make one of my only alternative identification methods an SMS. If I were to lose my phone or number I could be locked out of my account. Most accounts I have assume I have access to this email, thus a good chunk of modern life requires me accessing it, the prospect of becoming unable to log in seems quite realistic. This post (by an email-hosting company) builds a case against Gmail ground of privacy. It was at last time to get my own domain and control over my email. I also started this blog this year and it is pleasing to give it a nice .com home.

‘Yer a domain, Harry

Buy a domain from a domain provider. Common options are CloudFare, GoDaddy and NameCheap. I used CloudFare due to mixed comments online about customer service on the others. Also, NameCheap may pre-purchase domains that were looked up on domain search engines. If you are paranoid like me you can use whois (though it is being sunset, I’d suggest looking for an alternative). Once you find a domain name you like and is available just pay for it, you can get it for up to 10-years at a time.

Buy email hosting services and link them to the domain

Doing some research (mostly HackerNews and Reddit comments) I narrowed down options to either Migadu and MXroute. Migadu had a free trial, so I gave it a shot, but in the end I went for MXroute due to a promo they had at the time.

Make an email account. On Migadu it should be straightforward. In the case of MXroute, you have to pay at this point. the URL for management should be <SERVER>.mxrouting.net:2222, were <SERVER> is in the confirmation email. Once you have access to your email hosting server you can to link it to your domain.

Link email service to domain

The first step once you have an account is to add DNS (Domain Name Service) records to the website. They are instructions stored in servers that provide information about how to handle requests, mostly linking names to IPs.

Migadu

Export the BIND records into a file and download it
Load this file, add them CloudFare DNS records
If the DKMS and ARC keys fail, make sure to untoggle the proxy switch on their records

MXroute

Mostly follow the instructions sent upon registration, but it is basically the same as Migadu but without the import/export convenience. For instance, one of the MX records (with different fields separated by commas) is MX, <domain.com>, <server>.mxrouting.net. The ones I had to copy over from the email were:

TXT records: spf1 and DKIM1
MX records: <SERVER>.mxrouting.net and <SERVER>-relay.mxrouting.net

Copy them on Cloudfare, specifically on DNS records:

Cloudfare's domain home -> DNS Records -> Add record

Test the setup

Log-in to yor account on webmail.migadu.com or <SERVER>.mxrouting.net/roundcube
Create an account for your first user
Send an mail to yourself to validate that it works

Optional: Set subdomain for server access

To access my email from my phone’s Thunderbird App I had to set a subdomain. Only MXroute required this.

From the Control panel provided by MXroute the verification key
```
<SERVER>.mxrouting.net -> (sidebar) Account manager -> DNS record
```
And add it as a TXT record on CloudFare
Add mail.<domain.com> and/or webmail.<domain.com> subdomain(s) on CloudFare

Link Github Pages website to domain

If we have one, we can also link our website to the domain. Some companies like to use blog.<domain.com> for blogs and keep <domain.com> for their landing page, but since I am almost certainly a person I skipped the subdomain approach (and used the so-called apex domain). In my case I am using Github to host my website, so I followed their instructions:

Go to DNS records

Cloudfare's domain home -> DNS Records -> Add record

Add A records to CloudFare (e.g., A,<domain.com>,<185.ipv4.github.address>)
Add CNAME,www,<user>.github.io record. This step wasn’t specified on the Github docs but I found it to be necessary for some reason.it

Add cloudfare rules to redirect https

Cloudfare's domain home -> Cloudfare rules -> Templates -> Redirect http to HTTPS -> Deploy

Add custom domain to github pages:

Github repo -> settings -> pages on side panel -> Custom domain

Also on Github, the Enforce HTTPS checkbox, if the rest was properly configured it should work without a hitch

Conclusions

I like that I have control over my email, and I could even give my family and friends their personal emails (if any of them actually wants one). The first time I did all this it took me around three hours, mostly because Github could not find my DNS records. Since I have my domain now I can probably do more fun stuff, such as self-hosting tools to share with friends and family.

Use dired-do-shell to explore the parquet schema from Emacs

Thu, 23 Oct 2025 15:21:00 -0400

I use dired-do-shell command in Emacs to run CLI commands from within its file manager dired. This workflow makes it easy to perform batch operations on files that would be annoying otherwise. The trouble arose when trying to use the duckdb CLI to print the schema of a parquet file, as the notation for wildcards in emacs (* and ?) conflicts with duckdb’s usage of the former. Thus running the following after M-x dired-do-shell (bound to ! in dired-mode) did not work:

duckdb -c "DESCRIBE * FROM read_parquet('?')"

I am trying to get Emacs to substitute ? for the selected file(s) in dired and retain * as a normal duckdb wildcard (for which it means “get all columns”), but Emacs interprets the star as a wildcard and not the question mark. Thus Emacs yapped about it:

dired-do-shell-command: You can not combine ‘*’ and ‘?’ substitution marks

The trick is that we need to surround ? with backquotes (`), otherwise the spaces will be in the file name that read_parquet ingests. To fix the “naked star” problem (which Emacs will try to substitute) we can envelop it with the COLUMNS() command, and the lack of spaces around * will prevent Emacs from substituting its value. Thus the following command in the Emacs command line does the trick:

duckdb -c "DESCRIBE COLUMNS(*) FROM read_parquet('`?`')"

Which is a bit verbose but I can easily reuse from my command history to explore the schemas or parquets without leaving the dired buffer.

Calculate the cumulative sum of a column using DuckDB

Wed, 22 Oct 2025 20:34:00 -0400

Duckdb, the (tabular) data exploration tool I use supports window operations. I recently discovered that it can also perform cumulative sums in a very efficient manner.

Let us generate a toy dataset where we want to calculate the sum of one column relative to the order of another one.

 -- seeding for reproducibility, creating a table to hide output
CREATE OR REPLACE TABLE seed AS SELECT SETSEED(0.1);

-- Create a mock dataset with two integer columns
CREATE OR REPLACE TABLE my_table AS
SELECT
 #1 AS column_1,
 CAST(FLOOR(RANDOM() * 100) AS INT) AS column_2
FROM generate_series(1, 10); -- This generates 10 rows
SELECT * FROM my_table;

-- We write it to a csv for future use
COPY my_table TO my_table.csv;

┌──────────┬──────────┐
│ column_1 │ column_2 │
│ int64 │ int32 │
├──────────┼──────────┤
│ 1 │ 27 │
│ 2 │ 45 │
│ 3 │ 2 │
│ 4 │ 84 │
│ 5 │ 84 │
│ 6 │ 26 │
│ 7 │ 18 │
│ 8 │ 65 │
│ 9 │ 97 │
│ 10 │ 11 │
├──────────┴──────────┤
│ 10 rows 2 columns │
└─────────────────────┘

If we wanted to calculate the distribution of the cumulative sum of the table we could use the OVER clause to perform the sum of column_2 in the order defined by column_1.

SELECT *, sum(column_2) OVER (ORDER by column_1) AS cumulative_sum FROM my_table

┌──────────┬──────────┬────────────────┐
│ column_1 │ column_2 │ cumulative_sum │
│ int64 │ int32 │ int128 │
├──────────┼──────────┼────────────────┤
│ 1 │ 27 │ 27 │
│ 2 │ 45 │ 72 │
│ 3 │ 2 │ 74 │
│ 4 │ 84 │ 158 │
│ 5 │ 84 │ 242 │
│ 6 │ 26 │ 268 │
│ 7 │ 18 │ 286 │
│ 8 │ 65 │ 351 │
│ 9 │ 97 │ 448 │
│ 10 │ 11 │ 459 │
├──────────┴──────────┴────────────────┤
│ 10 rows 3 columns │
└──────────────────────────────────────┘

The cumulative sum can be pretty handy to get a general notion of a distribution. As a bonus tip, I’ll show how to use duckdb in a one-liner to plot the data directly in a terminal by using gnuplot.

duckdb -csv -c "
 SELECT *, sum(column_2) OVER (ORDER by column_1) AS cumulative_sum
 FROM read_csv('my_table.csv');" |
 gnuplot -e "
 set terminal dumb;
 set datafile separator ',';
 set style data histograms;
 set style fill solid 1.00 border -1;
 set xlabel 'Column 1';
 set ylabel 'CSum';
 set title 'Cumulative Sum of values';
 plot '-' using 3:xtic(1);" |
 tr -d '\014' # Remove a pesky ^L at the top



 Cumulative Sum of values
 500 +-----------------------------------------------------------------+
 | + + + + + + + + + ++ |
 450 |-+ '-' using 3:xtic+-+ +-||--+-|
 400 |-+ |#| || +-|
 | |#| || |
 350 |-+ ++ |#| || +-|
 | || |#| || |
 300 |-+ +-+ || |#| || +-|
 250 |-+ ++ |#| || |#| || +-|
CSum | +-+ || |#| || |#| || |
 200 |-+ |#| || |#| || |#| || +-|
 | |#| || |#| || |#| || |
 150 |-+ ++ |#| || |#| || |#| || +-|
 | || |#| || |#| || |#| || |
 100 |-+ +-+ || |#| || |#| || |#| || +-|
 50 |-+ ++ |#| || |#| || |#| || |#| || +-|
 | +-+ || |#| || |#| || |#| || |#| || |
 0 +-----------------------------------------------------------------+
 1 2 3 4 5 6 7 8 9 10
 Column 1

We get a cute ascii-like plot! That is a bit too long of a “one-liner”, I’ll go through the commands:

Run a duckdb command (-c) that reads the previously-saved table. The -csv flag at the starts converts the output to csv.
Run gnuplot with certain specifications:
- The flag -e Allows to pass a series of commands without an interactive session.
- set terminal dumb: it will send as plain text to stdout.
- set datafiler separator ",": The input is a CSV file.
- set style data histograms: Changes the plotting style into a barplot.
- set style fill solid ...: Visual adjustments to the bars for clarity.
- set xlabel ... Adds the axis labels. Similar for ylabel and title.
- plot '-' using 3:xtic(1): Use stdin data to Plot the columns 3 on the y axis (cumulative_sum) and the first column in the x-axis (column_1).
Lastly, use the tr command line tool to remove a ^L That appeared at the start of the output and was bothering me too much.

While there are a many other ways to wrangle tables such as via pandas or polars in Python, I find duckdb to be a powerful tool for exploratory analyses and data wrangling (often from within Python). It is flexible enough to be used by itself, via bindings in another language, or directly on the command line. Lastly, I showed that when used as a Command Line Interface (CLI) duckdb synergises with other tools for data visualisation from the comfort(?) of the terminal.

Run multiple python scripts in the background

Tue, 26 Aug 2025 14:29:00 -0400

To solve a multitude of challenges I have faced when processing high throughput microscopy data, have developed Nahual, a tool that allows me to move data across multiple Python environments that deploy deep learning models in the background. I usually keep these models “listening” in the background for the main analysis pipeline (aliby) to send them data to process. To be able to monitor what’s going on inside of these scripts I use GNU screen, which allows me to detach and reattach into these sessions whenever I need to. At some point I had to reboot my server and had rerun all these in independent screens. This rudimentary shell script did the job:

cd cellpose
screen -d -S cellpose1 -m bash -c 'nix develop . --command bash -c "python server.py ipc:///tmp/cellpose1.ipc"'
screen -d -S cellpose2 -m bash -c 'nix develop . --command bash -c "python server.py ipc:///tmp/cellpose2.ipc"'
cd ../trackastra
screen -d -S trackastra -m bash -c 'nix develop . --command bash -c "python server.py ipc:///tmp/trackastra.ipc"'
cd ..

Basically the screen runs my Nix environment and deploys the model (in this case, my fork of cellpose with Nix dependency management) while detached. This executes a server.py file within the Nix enviroment, it runs on a loop waiting to receive data and process it. Automatically deploying to multiple screens reduces the annoyance of having to the usual steps of (go to the folder -> run screen -> Nix environment -> run Python server -> Detach screen session). I just add more models if I want further deployments, put it in a bash script and call it a day.

To access any of these screens for inspection I just use the name indicated after the -S flag (e.g., screen -r cellpose1). This way I can check if any issue crops up in the main analysis script or pipeline.

Simple progress indicators with awk

Tue, 19 Aug 2025 18:58:00 -0400

I wanted a simple way to see the progress of a data processing pipeline, and the internal progress bar tools were messed up by threading. I thus decided to use the number of output files in each folder as an indicator of progress. In my case the output of tree . looks like this:

.
└── steps
 ├── A01_001
 │   ├── segment_nuclei
 │   │   ├── 0000.npz
 │   │   ├── 0001.npz
 │   │   ├── ...
 │   │   └── 0019.npz
 │   ├── tile
 │   │   ├── 0000.npz
 │   │   ├── 0001.npz
 │   │   ├── ...

I can get the info I need by counting the total number of files and the occurrences of the A01_001 -> P24_005 range (these are fields of view from a microscopy experiment). Using this simple find command we get all the files in the current folder.

find . -type f

which results in this:

./steps:
./steps/A01_003/tile/0007.npz
./steps/A01_003/tile/0009.npz
./steps/A01_003/tile/0018.npz
./steps/A01_003/tile/0016.npz
...

We could use wc -l to get the number of files per directory, we want a bunch of progress bars to get a better sense of change over time. For this I use awk, my swiss-army knife for text processing, and I write a short script that counts, sorts and prints the number of occurrences as a number of dots. I also added a conditional to only track after more than one file has been produced, for pipelines that produce save one file before actually running the whole pipeline.

# progress_bar.awk
{
 if (match($0,"([A-P][0-9]{2}_[0-9]{3})", capture)){
 count[capture[1]] += 1
 }

}
END{
 n=asorti(count, sorted)
 for (i=1; i<=n; i++){
 nfiles = count[sorted[i]]
 if (nfiles > 1){
 s = sprintf(key "%*s", nfiles, "");
 gsub(".", ".", s)
 print sorted[i] " " s
 }
 }
}

Running the find command and the awk script (find . -type f | awk -f progress_bar.awk) yields the following snapshot of the processing progess

A01_001 ...............................................................
A01_002 ...............................................................
A01_003 ...............................................................
A01_004 ...............................................................
A01_005 ...............................................................
A02_001 ...............................................................
A02_002 ...............................................................
A02_003 .................................................
A02_004 ..........................................
A02_005 ........................................
A03_001 ..............................................

Thus the last thing to do is to use `watch` to automatically refresh the status:

watch -dc --interval 1 'find . -type f | awk -f progress_bar.awk | tac'

The watch flag -d highlight the changes over time and -c enables intrepreting ANSI colours, in my terminal this makes the changes last stay longer, but YMMV. Finally, tac makes sure that the last lines are displayed at the top. I like to run this command somewhere in another terminal or in a `screen` terminal multiplexer. When the number of rows becomes too high it may be useful find a heuristic to remove uninformative lines.

Update figure numbering

Thu, 14 Aug 2025 15:42:00 -0400

I was editing some markdown and had to insert a new figure in the middle. The problem is that this document already has an explicit figure numbering (e.g., “Figure 5”), so changing tens of figures felt dull. I like to run small (GNU) awk scripts for this type of tasks.

# update_figures.awk
{
 if (match($0, "Figure ([0-9]+)", num)){
 if (num[1] > after)
 gsub("Figure ([0-9]+)", "Figure " num[1] + increase_by)
 };
 print $0
}

This changes Figure X into Figure X + increase_by starting after the variable “after”. And we can run it as follows:

awk -v after=4 -v increase_by=1 -f update_figures.awk input_file.md

To edit the file in-place add the -i flag.

Recursive search and replace

Tue, 12 Aug 2025 13:07:00 -0400

I needed to rename all occurrences of a pattern with another, where I knew there was no ambiguous situations. This uses ripgrep, xargs and GNU sed. source.

rg old_pattern --files-with-matches | xargs sed -i 's/old_pattern/new_pattern/g'

A workflow for bioimaging and data exploration

Wed, 30 Jul 2025 13:05:00 -0400

One of the common challenges when analysing large bioimaging datasets is to bring it all together in one place. I usually use tools like DuckDB for database querying and copairs for selecting statistically significant subsets of the data. For one of my recent projects I built a marimo interface to explore the result of large-scale (~2TB images, ~2GB feature profiles) image-based profiles, then performs dimensionality reduction of the data, and finally retrieves back the images. This I think is the ideal workflow, one where you can be nimble and pull up the images alongside statistical analyses to be able to interpret the data structure in the biological context. The code is not yet available to the public, but you can find the demo here.

Github code review on existing code base

Tue, 26 Nov 2024 13:06:00 -0500

Create an empty branch with one empty commit

Create new branch git checkout --orphan review-1-target
Reset git reset .
Clean branch git clean -df
Add empty commit git commit --allow-empty -m 'Empty commit'

Rebase a branch to put this commit at the root

Push to your fork git push -u origin review-1-target
Move to branch to review git checkout origin/main
Spin-off branch from here git checkout -b review-1
Rebase to empty branch git rebase -i review-1-target, the empty commit must be at the start
Push git push -u origin review-1

That should make a pull request possible, providing the code review tooling. source