Create Your First Project
Start adding your projects to your portfolio. Click on "Manage Projects" to get started
Data Platform Design and Development
Customer
The client is a leading provider of temperature-controlled containers to the global pharmaceutical and life science industries
Detailed information about the client cannot be disclosed under the provisions of the NDA.
Role
</> Senior Data Engineer
Industry
Cold-chain solutions
Challenge
The client came to me with an existing BI platform characterized by non-scalability. This posed a challenge since they needed an efficient data pipeline for ingesting sensor data
Solution
Designed and developed a scalable data lake solution to process over 500 MB of sensor data daily, reducing data retrieval times by 30% for analytics purposes.
Core pipeline components:
>Azure Data Factory (ADF) - data ingestion and orchestration layer
>Azure Data Lake Storage Gen 2 - used for the storage using the Medallion Architecture
>Azure Databricks - data transformation/serving layer
>Power BI - data visualization or reports
>Azure DevOps - version control
Worked with stakeholders to design a scalable cloud-based data lake solution
