Data Engineer

Holden Karau Portfolio

Co-Founder, Fight Health Insurance

GitHub profile of Holden Karau, an Apache Spark committer and co-author of Learning Spark and High Performance Spark, maintaining spark-testing-base (1.6k stars, now tracking Spark 4.x and Spark Connect) among 300+ public repositories.

Apache SparkData Pipeline TestingScalaOpen Source

Location

San Francisco, CA, USA

Companies

Fight Health InsuranceNetflixAppleGoogleIBMAlpine Data LabsDatabricksFoursquare

Skills

Apache SparkData Pipeline TestingScalaOpen Source

Notable

  • Apache Spark committer and PMC member (listed on spark.apache.org/committers.html)
  • Co-author of Learning Spark, High Performance Spark, Kubeflow for ML and Scaling Python with Ray
  • spark-testing-base: 1.6k GitHub stars, 357 forks
  • IBM OTAA award and Google Open Source Peer Bonus for Apache Spark work, per holdenkarau.com

What makes it work

A breakdown of the choices that make this portfolio stand out.

1

Testing infrastructure for a tool most engineers only consume

spark-testing-base solves a problem most Spark users hit but rarely build tooling for (how do you actually unit-test a Spark job), which is why it sits at 1.6k stars, 357 forks and 832 commits and still ships builds for new Spark versions rather than being a niche utility.

2

Books that extend committer-level knowledge to a wider audience

Learning Spark and High Performance Spark translate deep, project-internals-level knowledge into material approachable for working data engineers, bridging a gap that pure API documentation doesn't close.

3

A cluster of small, complementary Spark utilities

Beyond the flagship testing library, smaller repos (a validator, a debugging flowchart, a project template) each solve one specific pain point in the Spark workflow rather than trying to be one big framework.

What Data Engineers can take from this

Specific, actionable tips to apply to your own portfolio — no generic advice.

  • Look for the unglamorous but universally needed problem in your ecosystem (like testing) — solving it well can outlast flashier projects.

  • If you have deep internals knowledge of a tool, consider writing it up for a broader audience; a book or long guide reaches people your code alone won't.

  • Ship a cluster of small, single-purpose utilities around a flagship project instead of trying to fold every feature into one repository.

Projects

The most-starred repo here answers a question every Spark user eventually asks and few build tooling for: how do you test this? Holden Karau's profile stays on that unglamorous beat throughout, with validators, upgrade comparison tooling and a debugging flowchart.

Open SourceadvancedOngoing, 800+ commits

spark-testing-base

ScalaJavaPythonApache SparkSpark ConnectMaven/sbt

Base classes for testing Spark jobs, so users stop hand-writing local-mode Spark setup and teardown in every suite. Published to Maven as com.holdenkarau:spark-testing-base (versioned per Spark release) and as a Python package. Recent work adds Spark Connect support for DataFrame testing, approximate-equality tolerances for structs, timestamps and BigDecimal, and Spark 4.1.0 builds.

Testing is where most Spark codebases are weakest, and maintaining the library everyone reaches for across a decade of Spark versions is sustained maintainer work.

  • Artifacts versioned as [spark_version]_[library_version]

  • Cross-builds against multiple Spark versions

  • DataFrame comparisons with configurable tolerance

TooladvancedOngoing

Spark-Upgrade

ScalaApache SparkApache Iceberg

Tooling to help people upgrade to the latest version of Spark. It includes upgrade validation and a side-by-side mode that runs the same pipeline on two Spark versions and compares the output tables, aimed at pipelines without full test coverage. Planned work (marked not yet started) includes an Iceberg library to automate the comparison.

Version migrations are a real, expensive platform-team problem, and this attacks it with comparison tooling rather than hoping tests catch everything.

  • Side-by-side comparison for pipelines with incomplete tests

  • README is candid about which parts are not yet started

Open SourcebeginnerBook companion

Learning Spark examples

JavaScalaPythonApache Spark

The companion code for the O'Reilly book Learning Spark, with examples across Java, Scala and Python plus a standalone mini-complete-example with a minimal build file.

It pairs the book with runnable code, the committer-level knowledge made hands-on.

  • Multi-language examples for the same concepts

  • Minimal-dependency starter project included

Ready to build your portfolio?

Follow the Data Engineer roadmap — skills, projects, and timeline to get hired.

Data Engineer Roadmap