Streamlit in data migration – fast data visualization with parquet, free and open source

A large part of any migration is about looking at the data. How many records do we have per data object? How do volumes break down by month and group? Do the totals reconcile between source and target? Which records fall out of the quality checks? Those questions come up day after day – and the answers need to be fast, visual and easy to share with the business.

My favourite tool for exactly that is Streamlit.

What is Streamlit?

Streamlit is an open source Python framework that turns an ordinary script into an interactive web app. No HTML, CSS or JavaScript – I write Python, and Streamlit renders tables, charts, filters and KPIs in the browser. It's free, and since I build my ETL in Python with pandas anyway, it's the same language all the way from extract to visualization.

And parquet?

Migration extracts often land as parquet files. Parquet is a columnar, compressed file format for large datasets. Unlike CSV, data is stored by column rather than by row – which means you can quickly read just the fields you need, and the files are much smaller. With a modern compression algorithm such as Meta's zstd, a parquet file often takes only around a tenth of the space of an equivalent CSV. The format is language-agnostic and read directly by pandas:

import pandas as pd
df = pd.read_parquet("purchase_order_lines.parquet")

One line, and the entire extract is in memory, ready to be profiled, grouped and visualized. Parquet + pandas + Streamlit is a hard combination to beat when you need to understand an unfamiliar dataset quickly.

How I use it in a migration

I usually build one page per data object – purchase orders, transfer orders, customers, suppliers, sales orders and so on. On each page:

• Volumes – record counts by month, by group, by country. Reveals gaps and anomalies instantly.
• KPIs – coverage against the target system, share in foreign currency, totals in a common currency.
• Reconciliation – source against target, so you can see nothing is lost along the way.
• Data quality – records that break the rules, ready for business review.

The beauty is that it's interactive and self-service. Instead of emailing static Excel reports back and forth, business key users can click around, filter and drill down themselves – which makes the test and validation steps (steps 6 and 7 in my method) far smoother.

When you outgrow it

Streamlit doesn't replace a production data platform. At my most recent client we implemented Microsoft Fabric for analytics and reporting at scale. But that's exactly why Streamlit is so valuable: for getting started quickly and cheaply – profiling extracts, visualizing volumes and building trust in the data before a heavy platform is in place – it's hard to beat. And even with Fabric in place, I keep using Streamlit for fast, ad-hoc analysis throughout the migration.

In summary

If you're facing a migration and want to see your data without first building a whole BI stack: drop your extracts as parquet, read them with pandas and build a simple Streamlit app. It's free, open source and surprisingly capable.

Want to know how I set this up in practice? Get in touch – I'm happy to share.