Work

• We Built an Open-Source Data Quality Testframework for PySpark

  How and why we built a data-quality test framework for PySpark, and what we learned releasing it as open source.

  Kind

  article

  Roles

  • data engineering
  • software engineering

  Tools

  • python
  • pyspark
  • databricks

  Evidence

  https://medium.com/data-science/we-built-an-open-source-data-quality-testframework-for-pyspark-2301b9d87127

• pyspark-testframework

  Data-quality tests for PySpark DataFrames, with results returned as a long-format DataFrame.

  Kind

  project

  Position

  Senior Data Scientist, Woonstad Rotterdam

  Roles

  • data engineering
  • software engineering

  Tools

  • python
  • pyspark
  • databricks

  Secondary tools

  • uv
  • ruff
  • pytest
  • pre-commit
  • github-actions

  Evidence

  https://github.com/woonstadrotterdam/pyspark-testframework

• Terraform for infrastructure-as-code deployments.

  Kind

  responsibility

  Position

  Software Engineer, Woonstad Rotterdam

  Roles

  • cloud engineering

  Tools

  • terraform
  • azure

  Evidence

  None