Data Engineering

요약

데이터 엔지니어링 도메인에 대한 지식과 도구들을 정리한 맵입니다.

Tools

Workflow Orchestration

• Airflow
• Dagster · Prefect

Data Processing

• Spark · Flink

Messaging & Streaming

• Kafka

Search & Analytics

• Elasticsearch · Kibana

Data Transformation

• dbt

Lakehouse & Table Formats

• Lakehouse Architecture
• Delta Lake · Apache Iceberg

Data Platforms

• Palantir Foundry

Data Observability & Quality

• Data Observability · Monte Carlo
• OpenTelemetry

Concepts

아키텍처

• Batch vs Real-time
• Streaming Architecture
• Medallion Architecture

데이터 플랫폼 전략

• Data Fabric · Data Mesh

검색

• Information Retrieval · Lucene

도구 비교

• Dagster vs Prefect vs Airflow