Data cleaning

Data cleaning (also known as data preprocessing or data wrangling) is a critical step in data analysis and machine learning. The quality of your data has a direct impact on the quality of your analysis or model performance. Here’s a comprehensive list of techniques you need to learn for effective data cleaning:

1. Handling Missing Data

  • Identify Missing Data: Understand how to detect missing values (NaN, None, Null).
  • Imputation:
    • Mean/Median/Mode Imputation: Replace missing values with the mean, median, or mode of the column.
    • Forward/Backward Fill: Fill missing values with the previous/next value in the column.
    • Interpolation: Use methods like linear interpolation to fill in missing values.
    • K-Nearest Neighbors (KNN) Imputation: Estimate missing values based on similar observations.
  • Dropping Missing Values: Remove rows or columns with missing data if they represent too much noise.

2. Handling Outliers

  • Detecting Outliers:
    • Statistical Methods: Use Z-scores, IQR (Interquartile Range), or box plots.
    • Visual Methods: Scatter plots, histograms, or box plots.
  • Handling Outliers:
    • Truncation: Cap the outliers to a maximum or minimum value.
    • Transformation: Apply transformations like log or square root to reduce the impact of outliers.
    • Removal: Drop outliers if they represent noise and don't contribute valuable information.

3. Handling Duplicates

  • Identify Duplicates: Use functions to detect duplicate rows or records.
  • Remove Duplicates: Drop duplicates or merge them if necessary.

4. Handling Inconsistent Data

  • Standardizing Formats: Ensure consistency in formats (e.g., dates, addresses).
  • String Cleaning:
    • Remove Punctuation: Strip unnecessary punctuation from text fields.
    • Case Normalization: Convert all text to lowercase or uppercase for consistency.
    • Remove Whitespace: Trim leading/trailing spaces.
  • Correcting Inconsistent Labels: Ensure consistent category names (e.g., "Male" vs. "M").

5. Handling Incorrect Data

  • Data Validation: Check for errors in data such as impossible values (e.g., negative ages).
  • Correcting Errors: Fix incorrect entries based on rules or external validation.

6. Handling Categorical Data

  • Encoding:
    • Label Encoding: Convert categories to numeric labels.
    • One-Hot Encoding: Create binary columns for each category.
  • Handling High Cardinality: Consider grouping rare categories or using techniques like target encoding.

7. Scaling/Normalization

  • Standardization: Rescale data to have a mean of 0 and a standard deviation of 1.
  • Normalization: Rescale data to a range between 0 and 1.
  • Log Transformation: Reduce the skewness of data by applying logarithms.

8. Feature Engineering

  • Binning: Convert continuous variables into discrete bins (e.g., age groups).
  • Creating New Features: Derive new features from existing ones (e.g., date-time features like day of the week).
  • Polynomial Features: Create interaction terms or higher-order terms.
  • Dealing with Multicollinearity: Identify and remove/reduce highly correlated features.

9. Handling Date-Time Data

  • Parsing Dates: Convert strings to date-time formats.
  • Extracting Date Components: Extract features like year, month, day, hour, etc.
  • Handling Time Zones: Ensure consistent time zone handling.
  • Calculating Differences: Compute time deltas (e.g., time since the last event).

10. Handling Text Data

  • Tokenization: Split text into words or sentences.
  • Removing Stop Words: Remove common words that don’t add value to analysis.
  • Stemming/Lemmatization: Reduce words to their base or root form.
  • TF-IDF/Count Vectorization: Convert text to numerical features.

11. Handling Imbalanced Data

  • Resampling:
    • Oversampling: Increase the frequency of minority class examples (e.g., SMOTE).
    • Undersampling: Reduce the frequency of majority class examples.
  • Class Weighting: Adjust the weights of classes in algorithms.

12. Data Integration

  • Merging Data: Combine data from different sources.
  • Joining Datasets: Perform joins (inner, outer, left, right) to combine datasets.
  • Concatenating Data: Stack data vertically or horizontally.

13. Handling Anomalies

  • Detecting Anomalies: Identify unusual patterns that do not conform to expected behavior.
  • Handling Anomalies: Decide whether to remove or correct anomalies based on domain knowledge.

14. Feature Selection

  • Variance Thresholding: Remove features with low variance.
  • Correlation Matrix: Identify and drop highly correlated features.
  • Feature Importance: Use models like Random Forest to identify important features.

15. Handling Data Drift

  • Detecting Drift: Identify changes in data distribution over time.
  • Handling Drift: Adjust models or data preprocessing based on detected drift.

16. Automating Data Cleaning

  • Pipelines: Create pipelines to automate repetitive data cleaning steps.
  • Libraries: Use libraries like pandas, numpy, sklearn, or dplyr (in R) to streamline the process.

17. Data Documentation

  • Documenting Assumptions: Keep track of assumptions made during cleaning.
  • Version Control: Track changes in data cleaning scripts.

Comments

Post a Comment

Popular posts from this blog

I am starting a new course Course objective In this course, we are going to focus on the following learning objectives: 1. Understand the theory and intuition behind Multiple Linear Regression. 2. Import Key python libraries, dataset and perform data visualization 3. Perform exploratory data analysis and standardize the training and testing data. 4. Train and Evaluate linear regression model using Sci-kit Learn library. 5. Understand the difference between various regression models KPIs such as MSE, RMSE, MAE, R2, and adjusted R2. 6. Assess the performance of regression models and visualize the performance of the best model using various KPIs. 7. Understand the distribution and relationship of data Project Structure The hands on project on Life Expectancy Prediction Using Machine Learning is divided into following tasks: Task #1: Understand the Problem Statement and Business Case Task #2: Import Datasets and Libraries Task #3: Perform Data Visualization and Exploratory Data Analysis T...
Read more

Linear Models Battle

Image
Linear Models Battle: Who Wins Disaster Tweet Prediction? On day three of my ML journey, I'm exploring different linear models to see how they work and which ones are most accurate. Before I start coding, I want to understand the two main learning approaches in machine learning: Supervised Learning 1. 1 Linear model Let's start by understanding what a linear model is. It's like a special tool that helps us make guesses (or predictions) when one thing seems to be related to another. Imagine this: Your favorite basketball player, Steph Curry, is practicing his 3-pointers. You notice a pattern: Scenario 1: He takes 100 shots and makes 10 of them. Scenario 2: He takes 200 shots and makes 40 of them. Now, you want to guess how many 3-pointers he might make if he takes 250 shots. A linear model can help us with that! Think of it like this: The Dots: Each scenario (100 shots, 10 makes; 200 shots, 40 makes) is like a dot on a piece of graph paper. The Line: A linear model trie...
Read more

Day 3

 Lets start with Random Forest  1. It combines the output of multiple decision tree to reach the single result. 2. It handles both regression and classification problems so we wont be having problems we encountered on Ordinary Square Method. 3. it is made of many decision tree but I am yet to learn decision tree. Lets move back and learn decision tree first. 1. Similar to Random forest as it can handle both regression and classification.  Lets drive into some math before we start: 1. Entropy (Information Gain): Measure's the impurity or disorder of set of data. High entropy means the data is more mixed up (e.g., equal numbers of different classes), while low entropy means it's more pure (mostly one class). 2. Information Gain it is a decrease in entropy achieved by splitting the data on particular attribute. One of the main attribute of decision tree is that it gives highest information gain, as this leads to most information splits. Formula for Entropy: Entropy(S) = - Σ ...
Read more