Search for a command to run...
Articles tagged with #data-engineering
Hey Data Engineers! 馃憢 In this project, we build an end-to-end data pipeline using dbt (Data Build Tool) and Snowflake, simulating a Netflix-style dataset. Starting from raw data in Amazon S3, we load it into Snowflake, apply transformations using db...
Hey data enthusiasts! 馃憢 The Azure Sales Data Pipeline Automation project focuses on building an end-to-end automated data pipeline for processing sales order files dropped into a storage account by third parties. The goal is to validate these files,...
Hey data enthusiasts! 馃憢 In this week鈥檚 blog, we explore how to use Azure Data Factory's HTTP Connector to streamline data ingestion from HTTP endpoints into Azure Data Lake Storage (ADLS) Gen2. Through practical examples, we鈥檒l also demonstrate how ...
Hey there! 馃憢 Welcome to the second and final part of our Lending Club project. In this blog, we focus on calculating loan scores based on loan payment history, financial health, and loan defaulters' history. We'll also create permanent and temporary...
Introduction Welcome back, data enthusiasts! This week, we'll unravel the intricacies of schema inference and enforcement, data type handling, creating and refining dataframes, and removing duplicates. Let's get started right away! Schema inference, ...
Introduction Hey there, data enthusiasts! This week, we will explore the power of Spark's higher level APIs - Dataframes and Spark SQL. Let's begin using Spark like a pro and unveil the enhanced data processing capabilities of this dynamic duo. Task ...
