%md

# Financial Fraud Detection using Decision Tree Machine Learning Models

This notebooks is the accompaniment to the following databricks blog:

- https://databricks.com/blog/2019/05/02/detecting-financial-fraud-at-scale-with-decision-trees-and-mlflow-on-databricks.html

This is an exercise in self-learning. Here you will learn to use mlflow in pyspark to keep track of your experiments.

Also you may have to adapt it for Spark 3.x if needed.

In this notebook, we will showcase the use of decision tree ML models to perform financial fraud detection.

<br/>

### Source Data

PaySim simulates mobile money transactions based on a sample of real transactions extracted from one month of financial logs from a mobile money service implemented in an African country. The original logs were provided by a multinational company, who is the provider of the mobile financial service which is currently running in more than 14 countries all around the world.

This [synthetic dataset](https://www.kaggle.com/ntnu-testimon/paysim1) is scaled down 1/4 of the original dataset and it is created just for Kaggle.  To load the dataset yourself, please download it to your local machine from Kaggle and then import the data via **Import Data**: [Azure](https://docs.azuredatabricks.net/user-guide/importing-data.html#import-data) | [AWS](https://docs.databricks.com/user-guide/importing-data.html#import-data).

<br/>

### Dictionary

This is the column definition of the referenced sythentic dataset.

| Column Name | Description |

| ----------- | ----------- | 

| step | maps a unit of time in the real world. In this case 1 step is 1 hour of time. Total steps 744 (30 days simulation).|

| type | CASH-IN, CASH-OUT, DEBIT, PAYMENT and TRANSFER. |

| amount |  amount of the transaction in local currency. |

| nameOrig | customer who started the transaction |

| oldbalanceOrg | initial balance before the transaction |

| newbalanceOrig | new balance after the transaction |

| nameDest | customer who is the recipient of the transaction |

| oldbalanceDest | initial balance recipient before the transaction. Note that there is not information for customers that start with M (Merchants). |

| newbalanceDest | new balance recipient after the transaction. Note that there is not information for customers that start with M (Merchants). |

<br/>

import warnings

warnings.filterwarnings("ignore")

%md

### Calculate Differences between Originating and Destination Balanaces

With the following PySpark DataFrame query, we will calculate the following columns:

| New Column | Definition |

| ---------- | ---------- |

| orgDiff | Difference between the originating balance |

| destDiff | Difference between the destination balance |

%md ### Let's review the data and ask some simple questions

%md #### What are the type of transactions?

%md #### How much money are we talking about (synthetically)?

%md

## Rules-based Model: Create a set of rules to identify fraud based on known cases

The following `where` clause are a set of rules to identify know fraud-based cases using SQL; i.e. rules-based model.

* Often, financial fraud analytics start with with clauses like the `where` clause below

* Note, in reality, rules are often much larger and more complicated 

%md ### How much fraud are we talking about?

Based on the existing rules, while 4% of the transactions are fraudulent, it takes into account of the 11% of the total amount.   

%md #### Top Origination / Destination Difference Pairs (>$1M TotalDestDiff)

Each bar represents a pair of entities performing a transaction

%md ### What type of transactions are associated with fraud?

Reviewing the rules-based model, it appears that most fraudulent transactions are in the category of `Transfer` and `Cash_Out`.

%md # Rules vs. ML model

Instead of creating specific rules that will change over time, can we be more precise and go to production faster by creating a ML model?

%md

## Decision Trees

[Decision trees](http://en.wikipedia.org/wiki/Decision_tree_learning) and their ensembles are popular methods for the machine learning tasks of classification and regression. Decision trees are widely used since they are easy to interpret, handle categorical features, extend to the multiclass classification setting, do not require feature scaling, and are able to capture non-linearities and feature interactions. Tree ensemble algorithms such as random forests and boosting are among the top performers for classification and regression tasks.  

Because of these facets, decision trees often perform well on top of rules-based models and are often a good starting point for fraud detection.

![](http://cdn.iopscience.com/images/1749-4699/5/1/015004/Full/csd422281fig1.jpg)

Source: [The Wise Old Tree](https://pallav-routh.netlify.com/post/the-wise-old-tree/)