Senior Data Engineer

• Design, develop, and maintain scalable data pipelines that efficiently transform and transport large datasets from various sources to data storage systems. • Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets suitable for analysis and reporting. • Implement and optimize data models in data warehousing solutions, ensuring data integrity, accuracy, and accessibility for end-users. • Utilize cloud computing platforms (e.g., AWS, Azure, Google Cloud) to build and manage data infrastructure, leveraging services like ETL, data lakes, and data warehouses. • Experience level: 10+ years hands-on data engineering with production pipeline ownership; at least one platform or system owned end to end. Data transformation and modeling • dbt (data build tool): models, tests, macros, incremental models, project structure • Data modeling: dimensional modeling, star schema, medallion architecture (raw / curated / presentation, or bronze / silver / gold) • Data warehousing and lakehouse architecture concepts Orchestration • Apache Airflow: DAG design, task dependencies, triggers, sensors, parameterized runs, failure handling, scheduling • Workflow orchestration and monitoring of production pipelines Data ingestion and loading • Batch data ingestion and ETL / ELT pipeline development • Load patterns: full load, incremental load, upsert / merge, change data capture (CDC) • Multi-source ingestion: SFTP, REST APIs, relational databases, file-based ingestion • Data quality, data validation, schema evolution handling Query engine, storage, and table formats • Dremio (or equivalent lakehouse query engine: Trino, Presto, Starburst, Athena) • Apache Iceberg (or equivalent open table format: Delta Lake, Apache Hudi) • S3 / S3-compatible object storage (AWS S3, MinIO, Dell ECS) • Partitioning, table layout, and query performance tuning Programming and databases • Python (pipeline development, automation, scripting) • Advanced SQL, query optimization, performance tuning • PostgreSQL and/or SQL Server (relational databases) Non-relational / NoSQL databases (must-have, not just exposure) The platform ingests from and operates alongside several non-relational stores. The candidate must be able to work with them as a data engineer, meaning read from them, model their data into the lakehouse, and reason about their consistency and performance characteristics. Screen for hands-on experience with at least two of the following families: • Key-value / in-memory stores: Redis (caching, queues, state; TTL and eviction behaviour; using Redis as a pipeline coordination or lookup layer) • Document databases: MongoDB, Couchbase, or DocumentDB (nested JSON documents, schema-on-read, flattening documents into relational / lakehouse tables) • Wide-column stores: Cassandra, ScyllaDB, or HBase (partition-key design, read patterns, extracting at scale) • Search / analytics engines: Elasticsearch or OpenSearch (index design, querying, extracting data out of an index for analytics) Data formats and semi-structured data • Columnar formats: Apache Parquet (required), ORC; row formats: Avro • Text and semi-structured formats: JSON, JSON Lines, CSV / delimited files, XML; encoding, delimiter, and quoting edge cases • Compression codecs (Snappy, gzip, zstd) and their effect on storage and query • Handling nested and evolving schemas end to end (schema inference, drift detection, schema evolution: column add / delete / type change) Distributed processing • Apache Spark / PySpark (DataFrame API, partitioning, joins, skew handling, writing Iceberg / Parquet); or equivalent distributed engine • Understanding of parallelism, shuffles, memory tuning, and cost of distributed jobs Scripting, Linux, and connectivity • Bash / shell scripting and Linux command line (the deployment and operational tooling is script- based) • SFTP / SSH: key management, automated transfers, file-based handoffs (verify / result files, archiving, retries) • TLS certificates and secure connections (understanding certificate verification; never disabling it in production) • Connectivity troubleshooting: DNS, firewalls, proxies, connection timeouts, service-to-service networking inside Kubernetes / OpenShift • YAML and configuration management for pipelines, dbt, Airflow, and Kubernetes manifests Skills data engineering dbt Apache Airflow