Data Science - Recent Developments

 

2026

Data science in late 2026 is shifting away from isolated notebook experimentation toward agentic AI workflows, MLOps/engineering integration, and modular local/specialized models.

Key Trends & Updates

• Rise of Multi-Agent Systems & AI Agents: Frameworks like the OpenAI Agents SDK and open tools (such as Ollama and Mem0) allow data scientists to build multi-agent architectures, persistent memory layers, and tool-using pipelines in Python rather than static scripts.

• Shift from Notebooks to Production (MLOps): Pure "notebook-only" data science roles are contracting. Companies expect data scientists to understand model deployment, containerization, CI/CD, and monitoring using tools like Docker, FastAPI, and GitHub Actions.

• Specialized Small Models Over Giant LLMs: Organizations increasingly leverage sub-10-parameter open-weight specialist models tailored for single tasks (such as OCR, transcription, routing, and time-series forecasting) to cut compute costs.

• Automated Exploratory Data Analysis (EDA): Routine EDA tasks are rapidly being automated by AI assistants, raising the bar for practitioners to focus on architecture, business ROI, and system validation.

• Modern Python Stack: Tools like uv are rapidly replacing legacy venv and pip workflows for faster dependency management and reproducible lockfiles.

Data Science - Introduction

 

Chapter 1 Introduction

Data science is a collection of techniques used to extract value from data. It is a useful  tool for any organization that collects, stores, and processes data as part of its operations. Data science techniques rely on finding useful patterns, connections, and relationships within data.

The use of the term science in data science indicates that the methods are evidence based, and are built on empirical knowledge, more specifically historical observations.

 Vast majority of data science practitioners today use a handful of very powerful techniques to accomplish their objectives: decision trees, regression models, deep learning, and clustering. A majority of the data science activity can be accomplished using relatively few techniques. However,  the long tail, which is made up of a large number of specialized techniques, is where the value lies, and depending on what is needed, the best approach may be a relatively obscure technique or a combination of several not so commonly used procedures. 

Artificial intelligence is about giving machines the capability of mimicking

human behavior, particularly cognitive functions. Examples would be: facial

recognition, automated driving, sorting mail based on postal code. In some

cases, machines have far exceeded human capabilities (sorting thousands of

postal mails in seconds) and in other cases we have barely scratched the

surface (search “artificial stupidity”). There are quite a range of techniques

that fall under artificial intelligence: linguistics, natural language processing,

decision science, bias, vision, robotics, planning, etc. Learning is an important part of human capability. In fact, many other living organisms can

learn.

Machine learning can either be considered a sub-field or one of the tools of

artificial intelligence, is providing machines with the capability of learning

from experience. Experience for machines comes in the form of data. Data

that is used to teach machines is called training data. Machine learning turns

the traditional programing model upside down

Data science is the business application of machine learning, artificial intelligence, and other quantitative fields like statistics, visualization, and mathematics. It is an interdisciplinary field that extracts value from data. In the

context of how data science is used today, it relies heavily on machine learning and is sometimes called data mining. Examples of data science user cases

are: recommendation engines that can recommend movies for a particular

user, a fraud alert model that detects fraudulent credit card transactions, find

customers who will most likely churn next month, or predict revenue for the

next quarter

Data science problems can also be classified into tasks such as: classification,

regression, association analysis, clustering, anomaly detection, recommendation engines, feature selection, time series forecasting, deep learning, and text

mining. This book is organized around these data science tasks. 

DATA SCIENCE ALGORITHMS

An algorithm is a logical step-by-step procedure for solving a problem.

A classification task can be solved using many different learning algorithms such as decision trees, artificial neural networks, k-NN, and even

some regression algorithms. The choice of which algorithm to use depends

on the type of dataset, objective, structure of the data, presence of outliers,

available computational power, number of records, number of attributes,

and so on. It is up to the data science practitioner to decide which algorithm

(s) to use by evaluating the performance of multiple algorithms. There have

been hundreds of algorithms developed in the last few decades to solve data

science problems

 In order to focus the appropriate amount of time on data

and algorithms, data science tools or statistical programing tools, like R,

RapidMiner, Python, SAS Enterprise Miner, etc., which can implement these

algorithms with ease, can be leveraged. These data science tools offer a library

of algorithms as functions, which can be interfaced through programming

code or configurated through graphical user interfaces.

Various Data Science Tasks and Multiple Algorithms for Each.

Classification:  Predict if a data point belongs to one of the predefined classes. The prediction will be based on learning from a known dataset

Decision trees, neural

networks, Bayesian

models, induction rules,

k-nearest neighbors

Regression Predict the numeric target label of a data point. The prediction will be

based on learning from a known dataset

Linear regression, logistic

regression

Anomaly detection:  Predict if a data point is an outlier compared to other data points in

the dataset

Distance-based,

density-based, LOF

Time series forecasting: Predict the value of the target variable for a future timeframe

based on historical values

Exponential smoothing,

ARIMA, regression

Clustering:  Identify natural clusters within the dataset based on inherit properties within the dataset

k-Means, density-based clustering (e.g., DBSCAN)

Association analysis

FP-growth algorithm, a priori algorithm

Recommendation engines:   Predict the preference of an item for a user

Collaborative filtering,  content-based filtering, hybrid recommenders

Data Science Concepts and Practice - Vijay Kotu, Bala Deshpande - Book Information - Contents

 

Table of contents

1. Introduction

2. Data Science Process

3. Data Exploration

4. Classification

5. Deep Learning

6. Regression Methods

7. Association Analysis

8. Recommendation Engines

9. Clustering

10. Text Mining (renamed to: Natural Language Processing)

11. Time Series Forecasting

12. Anomaly Detection

13. Feature Selection

14. Model Evaluation

15. Efficient Model Execution

16. Getting Started with RapidMiner

Hui Lin and Ming Li - Practitioner’s Guide to Data Science - Book Information - Notes

Contents

List of Figures ix

List of Tables xiii

Preface xv

About the Authors xxiii

1 Introduction 1

1.1 A Brief History of Data Science . . . . . . . . . . 1

1.2 Data Science Role and Skill Tracks . . . . . . . . 5

1.2.1 Engineering . . . . . . . . . . . . . . . . . 7

1.2.2 Analysis . . . . . . . . . . . . . . . . . . . 8

1.2.3 Modeling/Inference . . . . . . . . . . . . . 10

1.3 What Kind of Questions Can Data Science Solve? 15

1.3.1 Prerequisites . . . . . . . . . . . . . . . . 15

1.3.2 Problem Type . . . . . . . . . . . . . . . 18

1.4 Structure of Data Science Team . . . . . . . . . 20

1.5 Data Science Roles . . . . . . . . . . . . . . . . . 24

2 Soft Skills for Data Scientists 31

2.1 Comparison between Statistician and Data Scientist . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.2 Beyond Data and Analytics . . . . . . . . . . . . 33

2.3 Three Pillars of Knowledge . . . . . . . . . . . . 35

2.4 Data Science Project Cycle . . . . . . . . . . . . 36

2.4.1 Types of Data Science Projects . . . . . . 36

2.4.2 Problem Formulation and Project Planning

Stage . . . . . . . . . . . . . . . . . . . . 38

2.4.3 Project Modeling Stage . . . . . . . . . . 40

2.4.4 Model Implementation and Post Production Stage . . . . . . . . . . . . . . . . . . 41

2.4.5 Project Cycle Summary . . . . . . . . . . 42

2.5 Common Mistakes in Data Science . . . . . . . . 43

2.5.1 Problem Formulation Stage . . . . . . . . 43

2.5.2 Project Planning Stage . . . . . . . . . . . 44

2.5.3 Project Modeling Stage . . . . . . . . . . 45

2.5.4 Model Implementation and Post Production Stage . . . . . . . . . . . . . . . . . . 46

2.5.5 Summary of Common Mistakes . . . . . . 47

3 Introduction to the Data 49

3.1 Customer Data for a Clothing Company . . . . . 49

3.2 Swine Disease Breakout Data . . . . . . . . . . . 51

3.3 MNIST Dataset . . . . . . . . . . . . . . . . . . 53

3.4 IMDB Dataset . . . . . . . . . . . . . . . . . . . 53

4 Big Data Cloud Platform 57

4.1 Power of Cluster of Computers . . . . . . . . . . 58

4.2 Evolution of Cluster Computing . . . . . . . . . 59

4.2.1 Hadoop . . . . . . . . . . . . . . . . . . . 59

4.2.2 Spark . . . . . . . . . . . . . . . . . . . . 60

4.3 Introduction of Cloud Environment . . . . . . . 60

4.3.1 Open Account and Create a Cluster . . . 61

4.3.2 R Notebook . . . . . . . . . . . . . . . . . 62

4.3.3 Markdown Cells . . . . . . . . . . . . . . 63

4.4 Leverage Spark Using R Notebook . . . . . . . . 64

4.5 Databases and SQL . . . . . . . . . . . . . . . . 71

4.5.1 History . . . . . . . . . . . . . . . . . . . 71

4.5.2 Database, Table and View . . . . . . . . . 72

4.5.3 Basic SQL Statement . . . . . . . . . . . 74

4.5.4 Advanced Topics in Database . . . . . . . 78

5 Data Pre-processing 79

5.1 Data Cleaning . . . . . . . . . . . . . . . . . . . 81

5.2 Missing Values . . . . . . . . . . . . . . . . . . . 84

5.2.1 Impute missing values with median/mode 85

5.2.2 K-nearest neighbors . . . . . . . . . . . . 86

5.2.3 Bagging Tree . . . . . . . . . . . . . . . . 88

5.3 Centering and Scaling . . . . . . . . . . . . . . . 88

5.4 Resolve Skewness . . . . . . . . . . . . . . . . . 90

5.5 Resolve Outliers . . . . . . . . . . . . . . . . . . 93

5.6 Collinearity . . . . . . . . . . . . . . . . . . . . . 97

5.7 Sparse Variables . . . . . . . . . . . . . . . . . . 100

5.8 Re-encode Dummy Variables . . . . . . . . . . . 101

6 Data Wrangling 105

6.1 Summarize Data . . . . . . . . . . . . . . . . . . 107

6.1.1 dplyr package . . . . . . . . . . . . . . . . 107

6.1.2 apply(), lapply() and sapply() in base R . . 116

6.2 Tidy and Reshape Data . . . . . . . . . . . . . . 120

7 Model Tuning Strategy 125

7.1 Variance-Bias Trade-Off . . . . . . . . . . . . . . 126

7.2 Data Splitting and Resampling . . . . . . . . . . 134

7.2.1 Data Splitting . . . . . . . . . . . . . . . 135

7.2.2 Resampling . . . . . . . . . . . . . . . . . 145

8 Measuring Performance 151

8.1 Regression Model Performance . . . . . . . . . . 151

8.2 Classification Model Performance . . . . . . . . . 155

8.2.1 Confusion Matrix . . . . . . . . . . . . . . 157

8.2.2 Kappa Statistic . . . . . . . . . . . . . . . 159

8.2.3 ROC . . . . . . . . . . . . . . . . . . . . . 161

8.2.4 Gain and Lift Charts . . . . . . . . . . . . 163

9 Regression Models 167

9.1 Ordinary Least Square . . . . . . . . . . . . . . . 168

9.1.1 The Magic P-value . . . . . . . . . . . . . 173

9.1.2 Diagnostics for Linear Regression . . . . . 176

9.2 Principal Component Regression and Partial Least

Square . . . . . . . . . . . . . . . . . . . . . . . 180

10 Regularization Methods 189

10.1 Ridge Regression . . . . . . . . . . . . . . . . . . 190

10.2 LASSO . . . . . . . . . . . . . . . . . . . . . . . 195

10.3 Elastic Net . . . . . . . . . . . . . . . . . . . . . 199

10.4 Penalized Generalized Linear Model . . . . . . . 201

10.4.1 Introduction to glmnet package . . . . . . 201

10.4.2 Penalized logistic regression . . . . . . . . 206

11 Tree-Based Methods 217

11.1 Tree Basics . . . . . . . . . . . . . . . . . . . . . 217

11.2 Splitting Criteria . . . . . . . . . . . . . . . . . . 221

11.2.1 Gini impurity . . . . . . . . . . . . . . . . 222

11.2.2 Information Gain (IG) . . . . . . . . . . . 223

11.2.3 Information Gain Ratio (IGR) . . . . . . . 224

11.2.4 Sum of Squared Error (SSE) . . . . . . . . 226

11.3 Tree Pruning . . . . . . . . . . . . . . . . . . . . 228

11.4 Regression and Decision Tree Basic . . . . . . . . 232

11.4.1 Regression Tree . . . . . . . . . . . . . . . 232

11.4.2 Decision Tree . . . . . . . . . . . . . . . . 236

11.5 Bagging Tree . . . . . . . . . . . . . . . . . . . . 241

11.6 Random Forest . . . . . . . . . . . . . . . . . . . 245

11.7 Gradient Boosted Machine . . . . . . . . . . . . 249

11.7.1 Adaptive Boosting . . . . . . . . . . . . . 250

11.7.2 Stochastic Gradient Boosting . . . . . . . 252

12 Deep Learning 259

12.1 Feedforward Neural Network . . . . . . . . . . . 263

12.1.1 Logistic Regression as Neural Network . . 263

12.1.2 Stochastic Gradient Descent . . . . . . . . 265

12.1.3 Deep Neural Network . . . . . . . . . . . 266

12.1.4 Activation Function . . . . . . . . . . . . 270

12.1.5 Optimization . . . . . . . . . . . . . . . . 274

12.1.6 Deal with Overfitting . . . . . . . . . . . . 282

12.1.7 Image Recognition Using FFNN . . . . . . 284

12.2 Convolutional Neural Network . . . . . . . . . . 298

12.2.1 Convolution Layer . . . . . . . . . . . . . 299

12.2.2 Padding Layer . . . . . . . . . . . . . . . 303

12.2.3 Pooling Layer . . . . . . . . . . . . . . . . 304

12.2.4 Convolution Over Volume . . . . . . . . . 308

12.2.5 Image Recognition Using CNN . . . . . . 311

12.3 Recurrent Neural Network . . . . . . . . . . . . 317

12.3.1 RNN Model . . . . . . . . . . . . . . . . . 320

12.3.2 Long Short Term Memory . . . . . . . . . 323

12.3.3 Word Embedding . . . . . . . . . . . . . . 326

12.3.4 Sentiment Analysis Using RNN . . . . . . 328

Appendix 337

13 Handling Large Local Data 339

13.1 readr . . . . . . . . . . . . . . . . . . . . . . . . 339

13.2 data.table— enhanced data.frame . . . . . . . . . 347

14 R code for data simulation 359

14.1 Customer Data for Clothing Company . . . . . . 359

14.2 Swine Disease Breakout Data . . . . . . . . . . . 364

Bibliography 369

Index 37

1 Introduction 1

1.1 A Brief History of Data Science . . . . . . . . . . 1

1.2 Data Science Role and Skill Tracks . . . . . . . . 5

1.2.1 Engineering . . . . . . . . . . . . . . . . . 7

1.2.2 Analysis . . . . . . . . . . . . . . . . . . . 8

1.2.3 Modeling/Inference . . . . . . . . . . . . . 10

1.3 What Kind of Questions Can Data Science Solve? 15

1.3.1 Prerequisites . . . . . . . . . . . . . . . . 15

1.3.2 Problem Type . . . . . . . . . . . . . . . 18

1.4 Structure of Data Science Team . . . . . . . . . 20

1.5 Data Science Roles . . . . . . . . . . . . . . . . . 24

2 Soft Skills for Data Scientists 31

2.1 Comparison between Statistician and Data Scientist . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.2 Beyond Data and Analytics . . . . . . . . . . . . 33

2.3 Three Pillars of Knowledge . . . . . . . . . . . . 35

2.4 Data Science Project Cycle . . . . . . . . . . . . 36

2.4.1 Types of Data Science Projects . . . . . . 36

2.4.2 Problem Formulation and Project Planning

Stage . . . . . . . . . . . . . . . . . . . . 38

2.4.3 Project Modeling Stage . . . . . . . . . . 40

iii

iv Contents

2.4.4 Model Implementation and Post Production Stage . . . . . . . . . . . . . . . . . . 41

2.4.5 Project Cycle Summary . . . . . . . . . . 42

2.5 Common Mistakes in Data Science . . . . . . . . 43

2.5.1 Problem Formulation Stage . . . . . . . . 43

2.5.2 Project Planning Stage . . . . . . . . . . . 44

2.5.3 Project Modeling Stage . . . . . . . . . . 45

2.5.4 Model Implementation and Post Production Stage . . . . . . . . . . . . . . . . . . 46

2.5.5 Summary of Common Mistakes . . . . . . 47

3 Introduction to the Data 49

3.1 Customer Data for a Clothing Company . . . . . 49

3.2 Swine Disease Breakout Data . . . . . . . . . . . 51

3.3 MNIST Dataset . . . . . . . . . . . . . . . . . . 53

3.4 IMDB Dataset . . . . . . . . . . . . . . . . . . . 53

4 Big Data Cloud Platform 57

4.1 Power of Cluster of Computers . . . . . . . . . . 58

4.2 Evolution of Cluster Computing . . . . . . . . . 59

4.2.1 Hadoop . . . . . . . . . . . . . . . . . . . 59

4.2.2 Spark . . . . . . . . . . . . . . . . . . . . 60

4.3 Introduction of Cloud Environment . . . . . . . 60

4.3.1 Open Account and Create a Cluster . . . 61

4.3.2 R Notebook . . . . . . . . . . . . . . . . . 62

4.3.3 Markdown Cells . . . . . . . . . . . . . . 63

4.4 Leverage Spark Using R Notebook . . . . . . . . 64

4.5 Databases and SQL . . . . . . . . . . . . . . . . 71

4.5.1 History . . . . . . . . . . . . . . . . . . . 71

4.5.2 Database, Table and View . . . . . . . . . 72

4.5.3 Basic SQL Statement . . . . . . . . . . . 74

4.5.4 Advanced Topics in Database . . . . . . . 78

5 Data Pre-processing 79

5.1 Data Cleaning . . . . . . . . . . . . . . . . . . . 81

5.2 Missing Values . . . . . . . . . . . . . . . . . . . 84

5.2.1 Impute missing values with median/mode 85

5.2.2 K-nearest neighbors . . . . . . . . . . . . 86

Contents v

5.2.3 Bagging Tree . . . . . . . . . . . . . . . . 88

5.3 Centering and Scaling . . . . . . . . . . . . . . . 88

5.4 Resolve Skewness . . . . . . . . . . . . . . . . . 90

5.5 Resolve Outliers . . . . . . . . . . . . . . . . . . 93

5.6 Collinearity . . . . . . . . . . . . . . . . . . . . . 97

5.7 Sparse Variables . . . . . . . . . . . . . . . . . . 100

5.8 Re-encode Dummy Variables . . . . . . . . . . . 101

6 Data Wrangling 105

6.1 Summarize Data . . . . . . . . . . . . . . . . . . 107

6.1.1 dplyr package . . . . . . . . . . . . . . . . 107

6.1.2 apply(), lapply() and sapply() in base R . . 116

6.2 Tidy and Reshape Data . . . . . . . . . . . . . . 120

7 Model Tuning Strategy 125

7.1 Variance-Bias Trade-Off . . . . . . . . . . . . . . 126

7.2 Data Splitting and Resampling . . . . . . . . . . 134

7.2.1 Data Splitting . . . . . . . . . . . . . . . 135

7.2.2 Resampling . . . . . . . . . . . . . . . . . 145

8 Measuring Performance 151

8.1 Regression Model Performance . . . . . . . . . . 151

8.2 Classification Model Performance . . . . . . . . . 155

8.2.1 Confusion Matrix . . . . . . . . . . . . . . 157

8.2.2 Kappa Statistic . . . . . . . . . . . . . . . 159

8.2.3 ROC . . . . . . . . . . . . . . . . . . . . . 161

8.2.4 Gain and Lift Charts . . . . . . . . . . . . 163

9 Regression Models 167

9.1 Ordinary Least Square . . . . . . . . . . . . . . . 168

9.1.1 The Magic P-value . . . . . . . . . . . . . 173

9.1.2 Diagnostics for Linear Regression . . . . . 176

9.2 Principal Component Regression and Partial Least

Square . . . . . . . . . . . . . . . . . . . . . . . 180

10 Regularization Methods 189

10.1 Ridge Regression . . . . . . . . . . . . . . . . . . 190

10.2 LASSO . . . . . . . . . . . . . . . . . . . . . . . 195

vi Contents

10.3 Elastic Net . . . . . . . . . . . . . . . . . . . . . 199

10.4 Penalized Generalized Linear Model . . . . . . . 201

10.4.1 Introduction to glmnet package . . . . . . 201

10.4.2 Penalized logistic regression . . . . . . . . 206

11 Tree-Based Methods 217

11.1 Tree Basics . . . . . . . . . . . . . . . . . . . . . 217

11.2 Splitting Criteria . . . . . . . . . . . . . . . . . . 221

11.2.1 Gini impurity . . . . . . . . . . . . . . . . 222

11.2.2 Information Gain (IG) . . . . . . . . . . . 223

11.2.3 Information Gain Ratio (IGR) . . . . . . . 224

11.2.4 Sum of Squared Error (SSE) . . . . . . . . 226

11.3 Tree Pruning . . . . . . . . . . . . . . . . . . . . 228

11.4 Regression and Decision Tree Basic . . . . . . . . 232

11.4.1 Regression Tree . . . . . . . . . . . . . . . 232

11.4.2 Decision Tree . . . . . . . . . . . . . . . . 236

11.5 Bagging Tree . . . . . . . . . . . . . . . . . . . . 241

11.6 Random Forest . . . . . . . . . . . . . . . . . . . 245

11.7 Gradient Boosted Machine . . . . . . . . . . . . 249

11.7.1 Adaptive Boosting . . . . . . . . . . . . . 250

11.7.2 Stochastic Gradient Boosting . . . . . . . 252

12 Deep Learning 259

12.1 Feedforward Neural Network . . . . . . . . . . . 263

12.1.1 Logistic Regression as Neural Network . . 263

12.1.2 Stochastic Gradient Descent . . . . . . . . 265

12.1.3 Deep Neural Network . . . . . . . . . . . 266

12.1.4 Activation Function . . . . . . . . . . . . 270

12.1.5 Optimization . . . . . . . . . . . . . . . . 274

12.1.6 Deal with Overfitting . . . . . . . . . . . . 282

12.1.7 Image Recognition Using FFNN . . . . . . 284

12.2 Convolutional Neural Network . . . . . . . . . . 298

12.2.1 Convolution Layer . . . . . . . . . . . . . 299

12.2.2 Padding Layer . . . . . . . . . . . . . . . 303

12.2.3 Pooling Layer . . . . . . . . . . . . . . . . 304

12.2.4 Convolution Over Volume . . . . . . . . . 308

12.2.5 Image Recognition Using CNN . . . . . . 311

Contents vii

12.3 Recurrent Neural Network . . . . . . . . . . . . 317

12.3.1 RNN Model . . . . . . . . . . . . . . . . . 320

12.3.2 Long Short Term Memory . . . . . . . . . 323

12.3.3 Word Embedding . . . . . . . . . . . . . . 326

12.3.4 Sentiment Analysis Using RNN . . . . . . 328

Appendix 337

13 Handling Large Local Data 339

13.1 readr . . . . . . . . . . . . . . . . . . . . . . . . 339

13.2 data.table— enhanced data.frame . . . . . . . . . 347

14 R code for data simulation 359

14.1 Customer Data for Clothing Company . . . . . . 359

14.2 Swine Disease Breakout Data . . . . . . . . . . . 364

Ud. 9.11.2026

Pub. 16.11.2023

Qualitative Thinking and Research in Data Science

 

Why the Data Revolution Needs Qualitative Thinking

by Anissa Tanweer, Emily Kalah Gade, P.M. Krafft, and Sarah Dreier

Published on

Jul 30, 2021

Harvard Data Science Review

https://hdsr.mitpress.mit.edu/pub/u9s6f22y/release/4

In this article, we focus on a set of concepts that are intrinsically informed by particular epistemological and ontological positions common in qualitative social sciences—positions that seek to understand the contingently and subjectively constructed nature of the social world. We refer to these concepts as ‘sensibilities’ because we intend them to intervene on methodology in a sensitizing rather than prescriptive way. The three sensibilities we discuss, have certain kinds of methodological practices, and they can be coupled with multiple modes of data collection and analysis.

Sensibility

Interpretivism

Working definition

An epistemological approach probing the multiple and contingent ways that meaning is ascribed to objects, actions, and situations.

Example of related methods

Trace ethnography (Geiger & Ribes, 2011; Geiger & Halfaker, 2017)

Sensibility

Abductive reasoning

Working definition

A mode of inference that updates and builds upon preexisting assumptions based on new observations in order to generate a novel explanation for a phenomenon.

Example of related methods

Iterations of open coding, theoretical coding, and selective coding (Thornberg & Charmaz, 2013)

Sensibility

Reflexivity

Working definition

A process by which researchers systematically reflect upon their own positions relative to their object, context, and method of inquiry.

Example of related methods

Brain dumps, situational mapping, and toolkit critiques (Markham, 2017)

Abduction

Abduction is often described as “inference to the best explanation” (Douven, 2011). 

Abductive reasoning updates and builds upon preexisting assumptions (in other words, theories) based on new observations in order to generate a novel explanation for a phenomenon. As such, it demarks “a creative outcome which engenders a new idea,"

When using abductive reasoning, qualitative researchers have developed ways of addressing the relationships between prior assumptions, new observations, and newly derived explanations. This can be incorporated in data science. 

The labeling of data in qualitative methods (what qualitative researchers would instead call ‘coding’) is not a matter of mere assumption, but rather a systematic part of the theory-building process.

LinkedIn AI article

What are the most common reasoning frameworks for data science?  Interesting content.

https://www.linkedin.com/advice/0/what-most-common-reasoning-frameworks-data-science-41fif

I made a contribution to the above article*.

The above article in is decision making series

https://www.linkedin.com/pulse/topics/soft-skills-s2976/decision-making-s2506/

From Big Data to Thick Data: Why Qualitative Insights Are the New Gold

Tiffany Perkins-Munn, Ph.D.

Tiffany Perkins-Munn, Ph.D.

C-suite & Board Advisor | Data Science, ML, AI | Global Data Power Women 2025 | Top 100 Most Influential People in Data | AI–100 | ex-BLK, ML, Citadel

https://www.linkedin.com/pulse/from-big-data-thick-why-qualitative-insights-new-gold-tiffany-srqne/

Understanding Qualitative Data and Its Application in Data Science

Zahid Butt

Zahid Butt

Consultancy Services (Data Analyst)

May 16, 2025

https://www.linkedin.com/pulse/understanding-qualitative-data-its-application-science-zahid-butt-iturc/

Jan 11, 2023

AI concepts for beginners - Exploring abductive reasoning in AI

https://indiaai.gov.in/article/exploring-abductive-reasoning-in-ai

Qualitative Data Analysis Methodologies and Methods

Last Updated : 23 Jul, 2025

https://www.geeksforgeeks.org/data-analysis/qualitative-data-analysis-methodologies-and-methods/

Leveraging LLM and Data Science Methods to Quantify Qualitative Data: A Workflow and Method Example

https://medium.com/the-whispers-of-a-data-analyst/leveraging-llm-and-data-science-methods-to-quantify-qualitative-data-a-workflow-and-method-example-5dcb86f70dfe

Making Sense of Data: AI-based NLP Tools for Education Research

by Prof.Dr.Ergul Demir | Sep 5, 2025

https://blog.eera-ecer.de/nlp-tools-for-education-research/

20 October 2023

15 Pages

Thematic Analysis of Big Data in Financial Institutions Using NLP Techniques with a Cloud Computing Perspective: A Systematic Literature Review

Full paper

Ratnesh Kumar Sharma, et al.

1

School of Computer Science, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney 2007, Australia

https://www.mdpi.com/2078-2489/14/10/577

A Detailed Guide about Natural Language Processing and NLP Techniques Every Data Scientist Should Know

Dejan Markovic

https://medium.com/@dejanmarkovic_53716/a-detailed-guide-about-natural-language-processing-and-nlp-techniques-every-data-scientist-should-08bfc0ca7ec3

The goal of NLP is to achieve human-level language understanding, allowing software to “read” enormous volumes of documents, analyze sentiment and topics, generate summaries, translate between languages, and even create original text.

some of the most common NLP techniques leveraged today:

Tokenization — Splitting text into sentences and words

Text classification — Categorize and label documents

Stemming — Reduce words to their root form

Sentiment analysis — Detect emotional tone

Keyword extraction — Identify important terms

Semantic search — Understand the meaning and intent behind search queries

Information extraction — Structured data from unstructured text

Language detection — Identify the origin language of the text

Leading NLP Solutions

BERT — Breakthrough NLP model from GoogleAI focusing on language representation

GPT-3 — Advanced language generation model from OpenAI

Amazon Comprehend — AWS service for NLP and natural language understanding

Microsoft LUIS — Azure cognitive service to build language understanding into apps

 Powerful Rules-Based Methods

Rules-based NLP techniques apply structured knowledge and logic flows to decipher language:

Parsing involves applying grammar templates and vocabulary to systematically break down sentence constructs and relate words accordingly. This allows proper interpretation of complex linguistics like irony that are often misclassified by statistical models.

Part-of-speech tagging carefully uses lexical rules and dictionaries to classify words as nouns, verbs, adjectives, etc. This structured output provides clarity for downstream processes.

Semantic analysis relies on hierarchical ontologies and semantic networks to encode real-world knowledge and eliminate ambiguity. For instance, determining appropriate synonyms based on a term’s meaning.

Statistical and Neural Machine Learning Methods

Machine learning NLP techniques automatically learn to transform, classify, or simulate language from large datasets:

Text generation models like OpenAI’s celebrated GPT-3 employ multi-layer transformer neural networks trained on enormous unlabeled text corpora to learn contextual relationships within the language and mimic its fluidity with striking fluency. However, their potential for propagating harmful biases due to ingesting entire Internet scrapes remains hotly debated.

Sentence embedding techniques like Google’s BERT representation model set performance records across various language tasks by innovating bidirectional conditioning to capture semantic meaning within phrases for downstream predictive modeling deeply. However, their intense computational requirements yielding an unprecedented 175 billion parameter models have sparked sustainability implications.

Mature sentiment analysis classifiers like VADER leverage logistic regression on top of heuristically defined linguistic rule sets to categorize the effects of diverse text from tweets to lengthy articles to speeches. They now power a plethora of critical applications — from monitoring population mental health to political polling to guiding business

Open access

Research article

First published online November 8, 2023

Natural Language Processing Enhanced Qualitative Methods: An Opportunity to Improve Health Outcomes

R. David Parker https://orcid.org/0000-0002-1955-7445 david.parker@alaska.edu, Karen Mancini, and Marissa D. Abram

https://journals.sagepub.com/doi/full/10.1177/16094069231214144

Using Natural Language Processing (NLP) to predict the future

Published

November 15, 2025·

Updated

October 13, 2026·

Alex Nikic

https://fastdatascience.com/natural-language-processing/predict-future/

Natural language processing for data analysis: an application on the well-being

Open access

Published: 10 June 2026

Volume 60, pages 18253–18287 (2026)

https://link.springer.com/article/10.1007/s11135-026-02894-9

Bringing Data Science to Qualitative Analysis

Year: 2018, Pages: 325-326

DOI Bookmark: 10.1109/eScience.2018.00076

Authors

You-Wei Cheah, Lawrence Berkeley Nat. Lab., Berkeley, CA, USA  

Drew Paine, Lawrence Berkeley Nat. Lab., Berkeley, CA, USA  

Devarshi Ghoshal, Lawrence Berkeley Nat. Lab., Berkeley, CA, USA  

Lavanya Ramakrishnan, Lawrence Berkeley Nat. Lab., Berkeley, CA, USA

https://www.computer.org/csdl/proceedings-article/e-science/2018/915600a325/17D45W2WyAq

2026 Guide to Generative AI: Techniques, Tools & Trends

https://hatchworks.com/blog/gen-ai/generative-ai/

Methods to Integrate Natural Language Processing Into Qualitative Research

SAGE Publications Inc

International Journal of Qualitative Methods

December 202019(8)

https://www.researchgate.net/publication/347973721_Methods_to_Integrate_Natural_Language_Processing_Into_Qualitative_Research

Review Article

Open Access

The Utilization of Natural Language Processing for Analyzing Social Media Data in Nursing Research: A Scoping Review

Zhenrong Wang, Yulin Ma, Yuanyuan Song, Yao Huang, Guopeng Liang, Xi Zhong

First published: 30 December 2024

https://onlinelibrary.wiley.com/doi/10.1155/jonm/2857497

ud. 8.11.2026

pub. 22.11.2023

Software Quality Assurance

 

The Future of Software Quality Assurance

2019

Page count:257

Published:19 November 2019

Format:ebook

Publisher:Springer International Publishing

Language:English

Editor:Stephan Goericke

This open access book, published to mark the 15th anniversary of the International Software Quality Institute (iSQI), is intended to raise the profile of software testers and their profession. It gathers contributions by respected software testing experts in order to highlight the state of the art as well as future challenges and trends. In addition, it covers current and emerging technologies like test automation, DevOps, and artificial intelligence methodologies used for software testing, before taking a look into the future.

The contributing authors answer questions like: "How is the profession of tester currently changing? What should testers be prepared for in the years to come, and what skills will the next generation need? What opportunities are available for further training...

Source: Publisher

https://www.google.co.in/books/edition/The_Future_of_Software_Quality_Assurance/YRS_DwAAQBAJ

Software Engineering Intelligence Platforms

 

https://www.infoworld.com/article/3711372/how-software-engineering-intelligence-platforms-boost-developer-productivity.amp.html