This project showcases an end-to-end data engineering and hypermodern BI solution built on unstructured data sources. It creates a dashboard for tracking upcoming rocket launches, designed for Mooncoon's Raccoon Engineers to plan their next lunar infiltration.
This dashboard demonstrates a complete data pipeline:
- Extracting data from unstructured sources (Launch Library 2 API)
- Transforming and loading data using modern data engineering tools
- Presenting insights through a cutting-edge BI layer
It serves as an excellent example of integrating various data engineering and BI technologies to derive value from unstructured data.
- Data Extraction: Utilizes the Launch Library 2 API to fetch unstructured data on upcoming rocket launches.
- Data Loading: Employs
dltto load and structure data into a PostgreSQL database running in Docker. - Data Transformation: Uses dbt for creating a data mart, optimizing raw data for dashboard queries.
- Scheduling: Implements Dagster for orchestrating the ETL pipeline, ensuring data freshness.
- Visualization: Presents data through an Evidence dashboard, showcasing modern BI capabilities.
docker compose upInstall bun, and make sure that your node version complies with evidence's requirements ,then run:
cd evidence && bun install && bun run devAfter running both Dagster and Evidence, access the dashboard through the URL provided by the Evidence development server. This will display the processed and analyzed rocket launch data in an intuitive, interactive format.
