From Notebook to Pipeline: Analysis You Can Re-Run

The notebook that only works on your laptop

Every analyst has one. It produced the number everybody quotes, it has forty cells, cell 12 is commented out, and the file path in cell 3 points at your Downloads folder. Someone asks for the same analysis on last month's data and the honest answer is "give me a day."

The problem isn't notebooks. Notebooks are excellent for exploring. The problem is that exploration code and production code have different jobs, and nobody ever schedules the hour it takes to move from one to the other.

Here is what that hour looks like.

Out-of-order execution is the root cause

A notebook's cells share one mutable namespace, and you can run them in any order. That's exactly what you want while thinking, and it means the notebook's output is not reproducible from its source.

You can prove this to yourself. Restart the kernel and run every cell top to bottom. If it fails, the notebook was never a description of your analysis — it was a transcript of your session. Variables defined in deleted cells, a dataframe reshaped by a cell you ran twice, a filter applied then loosened.

Restart-and-run-all is the only test that matters. Do it before you share any notebook. If it passes, everything else below is refinement. If it fails, nothing else you do can be trusted.

Push the logic into functions

The first real change is to move the thinking out of cell-level statements and into named functions. Not for elegance — because functions have explicit inputs and outputs, which means the dependency between steps becomes visible instead of implicit in cell order.

Before:

# cell 4
df = pd.read_csv('C:/Users/me/Downloads/orders_aug.csv')
# cell 5
df = df[df['status'] != 'cancelled']
df['revenue'] = df['qty'] * df['unit_price']
# cell 9
summary = df.groupby('region')['revenue'].sum().reset_index()

After:

def load_orders(path):
    return pd.read_csv(path, parse_dates=['order_date'])
 
def clean_orders(df):
    df = df[df['status'] != 'cancelled'].copy()
    df['revenue'] = df['qty'] * df['unit_price']
    return df
 
def summarise_by_region(df):
    return df.groupby('region', as_index=False)['revenue'].sum()

The notebook becomes three lines that call them. Two things follow immediately: the functions can be moved into a .py file and imported, and they can be tested. The .copy() is deliberate — filtering then assigning a column to a slice is how you earn a SettingWithCopyWarning and, occasionally, a silently discarded assignment.

Configuration belongs in one place

Hard-coded paths and dates scattered through cells are what make a notebook unrepeatable. Collect them at the top, then take them from the environment so the same code runs on your machine and on a server:

import os
from pathlib import Path
 
DATA_DIR = Path(os.environ.get('DATA_DIR', './data'))
REPORT_MONTH = os.environ.get('REPORT_MONTH', '2026-07')
OUTPUT_DIR = Path(os.environ.get('OUTPUT_DIR', './output'))
 
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

Use Path rather than string concatenation and your code stops caring whether it's running on Windows or Linux. Give every setting a sensible default so running it locally needs no setup, but make every one of them overridable so nothing is baked in.

The test for whether you've done this properly: could you run last quarter's version of this analysis by changing one environment variable? If not, there's still a constant hiding somewhere.

Make it safe to run twice

This is the step people skip, and it's the one that turns a script into something you can schedule.

An idempotent job produces the same end state whether it runs once or five times. The usual failure is appending: a job that appends 300 rows to a table produces 1,500 after a retry loop, and nobody notices until the totals drift.

Write full partitions rather than appending rows:

def write_month(df, month):
    """Replace the whole month's output rather than appending to it."""
    target = OUTPUT_DIR / f'revenue_{month}.parquet'
    tmp = target.with_suffix('.parquet.tmp')
    df.to_parquet(tmp, index=False)
    tmp.replace(target)          # atomic on the same filesystem

Writing to a temporary file and renaming it means a crash mid-write leaves the previous good output in place instead of a truncated file. For a database target, the same idea is a delete-then-insert inside one transaction, keyed on the partition you're rebuilding.

Fail loudly, in the right place

A pipeline that fails is annoying. A pipeline that succeeds with wrong data is expensive. Assert the things you're assuming:

def clean_orders(df):
    before = len(df)
    df = df[df['status'] != 'cancelled'].copy()
    df['revenue'] = df['qty'] * df['unit_price']
 
    assert df['order_id'].is_unique, 'order_id is not unique - check upstream join'
    assert df['revenue'].notna().all(), 'null revenue after calculation'
    assert len(df) > before * 0.5, f'lost more than half the rows: {before} -> {len(df)}'
 
    return df

That third assertion is the useful kind. Type checks catch bugs in your code; volume checks catch changes in your data, which is where the surprises actually come from.

Replace print with logging while you're here. It costs one line and gives you timestamps, levels, and output that survives a scheduled run:

import logging
 
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(message)s',
)
log = logging.getLogger(__name__)
 
log.info('loaded %d orders for %s', len(df), REPORT_MONTH)

Pin your dependencies

A notebook that worked in March and breaks in September usually hasn't changed at all — pandas has. Record what you ran with:

python -m venv .venv
source .venv/bin/activate     # .venv\Scripts\activate on Windows
pip install pandas pyarrow
pip freeze > requirements.txt

Commit requirements.txt. It's the difference between "it broke" and "it broke, and here's exactly what changed."

The shape it ends up in

project/
  README.md            # what this does, how to run it, who owns it
  requirements.txt
  src/
    config.py          # paths and parameters from the environment
    load.py            # read from source
    transform.py       # the actual logic, as pure functions
    write.py           # idempotent output
    main.py            # orchestration: load -> transform -> write
  tests/
    test_transform.py  # a handful of cases on small fixtures
  notebooks/
    exploration.ipynb  # kept, and clearly labelled as exploration

main.py reads as a summary of the whole job:

def main():
    log.info('starting run for %s', REPORT_MONTH)
    orders = load_orders(DATA_DIR / f'orders_{REPORT_MONTH}.csv')
    clean = clean_orders(orders)
    summary = summarise_by_region(clean)
    write_month(summary, REPORT_MONTH)
    log.info('wrote %d rows', len(summary))
 
if __name__ == '__main__':
    main()

Note that the notebook stays. It's still the right tool for the next question, and now it can import the same functions the pipeline uses — so exploration and production can't drift apart.

Do it in this order

You don't need all of this on day one, and doing it all at once on a working analysis is a good way to break it. In order of value per minute spent:

1. Restart and run all. Until this passes, nothing else counts.
2. Move constants to the top, then to environment variables.
3. Extract functions, one per logical step.
4. Add three assertions: grain uniqueness, no nulls where you expect none, row count sanity.
5. Make the write idempotent — replace, don't append.
6. Pin dependencies.
7. Then tests, then scheduling.

Steps 1 through 4 take about an hour and eliminate most of the risk. The rest is what turns a repeatable analysis into a maintained one — worth doing, but only once someone other than you depends on it running.