Data Mining and Preparation
This chapter covers acquiring data and preparing it for analysis. You will learn data acquisition and integration, the ETL and ELT patterns, combining datasets with joins and blends, cleansing techniques, and profiling to assess quality.
Acquisition and integration
Analysts gather data from many sources, including databases, files, APIs, and web services. Integration combines these sources into a unified dataset. Joins merge tables based on a shared key, such as a customer ID, so records from different tables can be analyzed together, while blending combines data from different sources at a summary level. Choosing the right method depends on the granularity and keys available.
ETL and ELT
Moving data into an analytical store typically follows extract, transform, load (ETL): data is pulled from sources, cleaned and reshaped, then loaded into a warehouse. A variant, ELT, loads raw data first and transforms it inside the target system, which suits modern cloud warehouses and data lakes. Both patterns aim to deliver consistent, analysis-ready data.
Data cleansing
Real-world data is messy, so cleansing is essential. Deduplication removes repeated records so entities are counted once. Imputation fills in missing values using strategies such as the mean or median. Parsing splits or extracts fields into usable pieces, and handling outliers decides whether extreme values are errors to correct or genuine signals to keep. Clean data prevents misleading results downstream.
Data profiling
Before deeper analysis, profiling examines a dataset's structure, value ranges, null counts, and data types to understand its content and assess quality. Profiling surfaces problems such as unexpected nulls, inconsistent formats, or out-of-range values early, when they are cheapest to fix, and informs the cleansing steps that follow.