Processing di Big Data con Pipeline ETL su Databricks

In the current data-driven landscape, organizations face the challenge of efficiently processing and analyzing large volumes of data for analytics and machine learning purposes. This thesis focuses on the usage of Databricks as a tool for constructing the Extract, Transform, Load (ETL) pipelines to facilitate the entire data processing workflow, from raw data ingestion to final stage data quality processing. Firstly, the concept of Big Data is introduced, highlighting its significance and impact in the present-day environment. Then, we delve into an exploration of Hadoop, Spark, and Databricks as powerful platforms that enable the management and processing of big data. These frameworks are discussed in terms of their capabilities, functionalities, and relevance in addressing the complexities of large-scale data processing. Then, this thesis presents a practical application scenario that incorporates industry best practices. The focus lies on implementing data quality techniques for measuring custom Key Performance Indicators (KPIs) to ensure reliable data analysis. Additionally, a data observability solution is introduced, which enables graphical monitoring of the ETL pipeline to provide real-time insights into its performance and health. Finally, a machine learning solution for data quality is presented as a future improvement on the project.