Holden Karau Portfolio
Co-Founder, Fight Health Insurance
GitHub profile of Holden Karau, an Apache Spark committer and co-author of Learning Spark and High Performance Spark, maintaining spark-testing-base (1.6k stars, now tracking Spark 4.x and Spark Connect) among 300+ public repositories.
Location
San Francisco, CA, USA
Companies
Skills
Notable
- Apache Spark committer and PMC member (listed on spark.apache.org/committers.html)
- Co-author of Learning Spark, High Performance Spark, Kubeflow for ML and Scaling Python with Ray
- spark-testing-base: 1.6k GitHub stars, 357 forks
- IBM OTAA award and Google Open Source Peer Bonus for Apache Spark work, per holdenkarau.com
What makes it work
A breakdown of the choices that make this portfolio stand out.
Testing infrastructure for a tool most engineers only consume
spark-testing-base solves a problem most Spark users hit but rarely build tooling for (how do you actually unit-test a Spark job), which is why it sits at 1.6k stars, 357 forks and 832 commits and still ships builds for new Spark versions rather than being a niche utility.
Books that extend committer-level knowledge to a wider audience
Learning Spark and High Performance Spark translate deep, project-internals-level knowledge into material approachable for working data engineers, bridging a gap that pure API documentation doesn't close.
A cluster of small, complementary Spark utilities
Beyond the flagship testing library, smaller repos (a validator, a debugging flowchart, a project template) each solve one specific pain point in the Spark workflow rather than trying to be one big framework.
What Data Engineers can take from this
Specific, actionable tips to apply to your own portfolio — no generic advice.
Look for the unglamorous but universally needed problem in your ecosystem (like testing) — solving it well can outlast flashier projects.
If you have deep internals knowledge of a tool, consider writing it up for a broader audience; a book or long guide reaches people your code alone won't.
Ship a cluster of small, single-purpose utilities around a flagship project instead of trying to fold every feature into one repository.
Projects
The most-starred repo here answers a question every Spark user eventually asks and few build tooling for: how do you test this? Holden Karau's profile stays on that unglamorous beat throughout, with validators, upgrade comparison tooling and a debugging flowchart.
Base classes for testing Spark jobs, so users stop hand-writing local-mode Spark setup and teardown in every suite. Published to Maven as com.holdenkarau:spark-testing-base (versioned per Spark release) and as a Python package. Recent work adds Spark Connect support for DataFrame testing, approximate-equality tolerances for structs, timestamps and BigDecimal, and Spark 4.1.0 builds.
Testing is where most Spark codebases are weakest, and maintaining the library everyone reaches for across a decade of Spark versions is sustained maintainer work.
Artifacts versioned as [spark_version]_[library_version]
Cross-builds against multiple Spark versions
DataFrame comparisons with configurable tolerance
Tooling to help people upgrade to the latest version of Spark. It includes upgrade validation and a side-by-side mode that runs the same pipeline on two Spark versions and compares the output tables, aimed at pipelines without full test coverage. Planned work (marked not yet started) includes an Iceberg library to automate the comparison.
Version migrations are a real, expensive platform-team problem, and this attacks it with comparison tooling rather than hoping tests catch everything.
Side-by-side comparison for pipelines with incomplete tests
README is candid about which parts are not yet started
The companion code for the O'Reilly book Learning Spark, with examples across Java, Scala and Python plus a standalone mini-complete-example with a minimal build file.
It pairs the book with runnable code, the committer-level knowledge made hands-on.
Multi-language examples for the same concepts
Minimal-dependency starter project included
More Data Engineer portfolios
Ready to build your portfolio?
Follow the Data Engineer roadmap — skills, projects, and timeline to get hired.