ALS_Recommender_System (EN).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems Project\n",
    "\n",
    "## Requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random, time\n",
    "import matplotlib.pyplot as plt\n",
    "from typing import List, Tuple\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Collection and Preprocessing\n",
    "\n",
    "- Download the MovieLens and IMDb datasets.\n",
    "- Merge datasets to include movie ratings, genres, and metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets \n",
    "\n",
    "The Datasets used are:\n",
    "- **MovieLens Dataset**: [MovieLens](https://grouplens.org/datasets/movielens/)\n",
    "    - movies.dat\n",
    "    - ratings.dat\n",
    "    - users.dat\n",
    "\n",
    "- **IMDb Dataset**: [IMDb Datasets](https://www.imdb.com/interfaces/)\n",
    "    - title.basics.tsv\n",
    "    - name.basics.tsv\n",
    "\n",
    "## Loading and visualization (MovieLens)\n",
    "\n",
    "In this step, we are first loading the MovieLens dataset. We will use the pandas library to read this dataset into a DataFrame for easier manipulation and analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to get the summary of a Dataframe if the case where analysis=True\n",
    "def get_summary(df, name, analysis=False):\n",
    "    print('Name:', name, ', Length:', len(df))\n",
    "    display(df)\n",
    "\n",
    "    if not analysis:\n",
    "        return\n",
    "    df_summary = pd.DataFrame({\n",
    "        'dtype': df.dtypes,\n",
    "        'unique':df.nunique().values,\n",
    "        'missing': df.isna().sum().values,\n",
    "        'duplicates': df.duplicated().sum(),\n",
    "    })\n",
    "    return df_summary.style.background_gradient(cmap='viridis_r',low=0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load MovieLens data\n",
    "df_users = pd.read_csv(\"../data/ml-1m/users.dat\", sep=\"::\", engine='python', header=None)\n",
    "df_users.columns = ['UserId', 'Gender', 'Age', 'Occupation', 'Zip-code']\n",
    "df_users.set_index(\"UserId\", inplace=True)\n",
    "\n",
    "df_movies = pd.read_csv(\"../data/ml-1m/movies.dat\", sep=\"::\", engine='python', header=None, encoding='latin1')\n",
    "df_movies.columns = ['MovieId', 'Title', 'Genres']\n",
    "df_movies.set_index(\"MovieId\", inplace=True)\n",
    "\n",
    "df_ratings = pd.read_csv(\"../data/ml-1m/ratings.dat\", sep=\"::\", engine='python', header=None)\n",
    "df_ratings.columns = ['UserId', 'MovieId', 'Rating', 'Timestamp']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: df_users , Length: 6040\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Occupation</th>\n",
       "      <th>Zip-code</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UserId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6036</th>\n",
       "      <td>F</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>32603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6037</th>\n",
       "      <td>F</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "      <td>76006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6038</th>\n",
       "      <td>F</td>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>14706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6039</th>\n",
       "      <td>F</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>01060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6040</th>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>11106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6040 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Gender  Age  Occupation Zip-code\n",
       "UserId                                 \n",
       "1           F    1          10    48067\n",
       "2           M   56          16    70072\n",
       "3           M   25          15    55117\n",
       "4           M   45           7    02460\n",
       "5           M   25          20    55455\n",
       "...       ...  ...         ...      ...\n",
       "6036        F   25          15    32603\n",
       "6037        F   45           1    76006\n",
       "6038        F   56           1    14706\n",
       "6039        F   45           0    01060\n",
       "6040        M   25           6    11106\n",
       "\n",
       "[6040 rows x 4 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_f8afb_row0_col1, #T_f8afb_row1_col1 {\n",
       "  background-color: #1f9e89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_f8afb_row0_col2, #T_f8afb_row0_col3, #T_f8afb_row1_col2, #T_f8afb_row1_col3, #T_f8afb_row2_col2, #T_f8afb_row2_col3, #T_f8afb_row3_col2, #T_f8afb_row3_col3 {\n",
       "  background-color: #fde725;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_f8afb_row2_col1 {\n",
       "  background-color: #1e9d89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_f8afb_row3_col1 {\n",
       "  background-color: #440154;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_f8afb\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_f8afb_level0_col0\" class=\"col_heading level0 col0\" >dtype</th>\n",
       "      <th id=\"T_f8afb_level0_col1\" class=\"col_heading level0 col1\" >unique</th>\n",
       "      <th id=\"T_f8afb_level0_col2\" class=\"col_heading level0 col2\" >missing</th>\n",
       "      <th id=\"T_f8afb_level0_col3\" class=\"col_heading level0 col3\" >duplicates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_f8afb_level0_row0\" class=\"row_heading level0 row0\" >Gender</th>\n",
       "      <td id=\"T_f8afb_row0_col0\" class=\"data row0 col0\" >object</td>\n",
       "      <td id=\"T_f8afb_row0_col1\" class=\"data row0 col1\" >2</td>\n",
       "      <td id=\"T_f8afb_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "      <td id=\"T_f8afb_row0_col3\" class=\"data row0 col3\" >244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_f8afb_level0_row1\" class=\"row_heading level0 row1\" >Age</th>\n",
       "      <td id=\"T_f8afb_row1_col0\" class=\"data row1 col0\" >int64</td>\n",
       "      <td id=\"T_f8afb_row1_col1\" class=\"data row1 col1\" >7</td>\n",
       "      <td id=\"T_f8afb_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "      <td id=\"T_f8afb_row1_col3\" class=\"data row1 col3\" >244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_f8afb_level0_row2\" class=\"row_heading level0 row2\" >Occupation</th>\n",
       "      <td id=\"T_f8afb_row2_col0\" class=\"data row2 col0\" >int64</td>\n",
       "      <td id=\"T_f8afb_row2_col1\" class=\"data row2 col1\" >21</td>\n",
       "      <td id=\"T_f8afb_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "      <td id=\"T_f8afb_row2_col3\" class=\"data row2 col3\" >244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_f8afb_level0_row3\" class=\"row_heading level0 row3\" >Zip-code</th>\n",
       "      <td id=\"T_f8afb_row3_col0\" class=\"data row3 col0\" >object</td>\n",
       "      <td id=\"T_f8afb_row3_col1\" class=\"data row3 col1\" >3439</td>\n",
       "      <td id=\"T_f8afb_row3_col2\" class=\"data row3 col2\" >0</td>\n",
       "      <td id=\"T_f8afb_row3_col3\" class=\"data row3 col3\" >244</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x72e4f0b81360>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_summary(df_users, 'df_users', analysis=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: df_movies , Length: 3883\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MovieId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3948</th>\n",
       "      <td>Meet the Parents (2000)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3949</th>\n",
       "      <td>Requiem for a Dream (2000)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3950</th>\n",
       "      <td>Tigerland (2000)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3951</th>\n",
       "      <td>Two Family House (2000)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3952</th>\n",
       "      <td>Contender, The (2000)</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      Title                        Genres\n",
       "MovieId                                                                  \n",
       "1                          Toy Story (1995)   Animation|Children's|Comedy\n",
       "2                            Jumanji (1995)  Adventure|Children's|Fantasy\n",
       "3                   Grumpier Old Men (1995)                Comedy|Romance\n",
       "4                  Waiting to Exhale (1995)                  Comedy|Drama\n",
       "5        Father of the Bride Part II (1995)                        Comedy\n",
       "...                                     ...                           ...\n",
       "3948                Meet the Parents (2000)                        Comedy\n",
       "3949             Requiem for a Dream (2000)                         Drama\n",
       "3950                       Tigerland (2000)                         Drama\n",
       "3951                Two Family House (2000)                         Drama\n",
       "3952                  Contender, The (2000)                Drama|Thriller\n",
       "\n",
       "[3883 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_b06a3_row0_col1 {\n",
       "  background-color: #440154;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_b06a3_row0_col2, #T_b06a3_row0_col3, #T_b06a3_row1_col2, #T_b06a3_row1_col3 {\n",
       "  background-color: #fde725;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_b06a3_row1_col1 {\n",
       "  background-color: #1f9e89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_b06a3\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_b06a3_level0_col0\" class=\"col_heading level0 col0\" >dtype</th>\n",
       "      <th id=\"T_b06a3_level0_col1\" class=\"col_heading level0 col1\" >unique</th>\n",
       "      <th id=\"T_b06a3_level0_col2\" class=\"col_heading level0 col2\" >missing</th>\n",
       "      <th id=\"T_b06a3_level0_col3\" class=\"col_heading level0 col3\" >duplicates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_b06a3_level0_row0\" class=\"row_heading level0 row0\" >Title</th>\n",
       "      <td id=\"T_b06a3_row0_col0\" class=\"data row0 col0\" >object</td>\n",
       "      <td id=\"T_b06a3_row0_col1\" class=\"data row0 col1\" >3883</td>\n",
       "      <td id=\"T_b06a3_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "      <td id=\"T_b06a3_row0_col3\" class=\"data row0 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b06a3_level0_row1\" class=\"row_heading level0 row1\" >Genres</th>\n",
       "      <td id=\"T_b06a3_row1_col0\" class=\"data row1 col0\" >object</td>\n",
       "      <td id=\"T_b06a3_row1_col1\" class=\"data row1 col1\" >301</td>\n",
       "      <td id=\"T_b06a3_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "      <td id=\"T_b06a3_row1_col3\" class=\"data row1 col3\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x72e4ebe01d80>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_summary(df_movies, 'df_movies', analysis=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: df_ratings , Length: 1000209\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserId</th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000204</th>\n",
       "      <td>6040</td>\n",
       "      <td>1091</td>\n",
       "      <td>1</td>\n",
       "      <td>956716541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000205</th>\n",
       "      <td>6040</td>\n",
       "      <td>1094</td>\n",
       "      <td>5</td>\n",
       "      <td>956704887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000206</th>\n",
       "      <td>6040</td>\n",
       "      <td>562</td>\n",
       "      <td>5</td>\n",
       "      <td>956704746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000207</th>\n",
       "      <td>6040</td>\n",
       "      <td>1096</td>\n",
       "      <td>4</td>\n",
       "      <td>956715648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000208</th>\n",
       "      <td>6040</td>\n",
       "      <td>1097</td>\n",
       "      <td>4</td>\n",
       "      <td>956715569</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000209 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         UserId  MovieId  Rating  Timestamp\n",
       "0             1     1193       5  978300760\n",
       "1             1      661       3  978302109\n",
       "2             1      914       3  978301968\n",
       "3             1     3408       4  978300275\n",
       "4             1     2355       5  978824291\n",
       "...         ...      ...     ...        ...\n",
       "1000204    6040     1091       1  956716541\n",
       "1000205    6040     1094       5  956704887\n",
       "1000206    6040      562       5  956704746\n",
       "1000207    6040     1096       4  956715648\n",
       "1000208    6040     1097       4  956715569\n",
       "\n",
       "[1000209 rows x 4 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_d0256_row0_col1 {\n",
       "  background-color: #1e9c89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_d0256_row0_col2, #T_d0256_row0_col3, #T_d0256_row1_col2, #T_d0256_row1_col3, #T_d0256_row2_col2, #T_d0256_row2_col3, #T_d0256_row3_col2, #T_d0256_row3_col3 {\n",
       "  background-color: #fde725;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_d0256_row1_col1 {\n",
       "  background-color: #1e9d89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_d0256_row2_col1 {\n",
       "  background-color: #1f9e89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_d0256_row3_col1 {\n",
       "  background-color: #440154;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_d0256\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_d0256_level0_col0\" class=\"col_heading level0 col0\" >dtype</th>\n",
       "      <th id=\"T_d0256_level0_col1\" class=\"col_heading level0 col1\" >unique</th>\n",
       "      <th id=\"T_d0256_level0_col2\" class=\"col_heading level0 col2\" >missing</th>\n",
       "      <th id=\"T_d0256_level0_col3\" class=\"col_heading level0 col3\" >duplicates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_d0256_level0_row0\" class=\"row_heading level0 row0\" >UserId</th>\n",
       "      <td id=\"T_d0256_row0_col0\" class=\"data row0 col0\" >int64</td>\n",
       "      <td id=\"T_d0256_row0_col1\" class=\"data row0 col1\" >6040</td>\n",
       "      <td id=\"T_d0256_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "      <td id=\"T_d0256_row0_col3\" class=\"data row0 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d0256_level0_row1\" class=\"row_heading level0 row1\" >MovieId</th>\n",
       "      <td id=\"T_d0256_row1_col0\" class=\"data row1 col0\" >int64</td>\n",
       "      <td id=\"T_d0256_row1_col1\" class=\"data row1 col1\" >3706</td>\n",
       "      <td id=\"T_d0256_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "      <td id=\"T_d0256_row1_col3\" class=\"data row1 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d0256_level0_row2\" class=\"row_heading level0 row2\" >Rating</th>\n",
       "      <td id=\"T_d0256_row2_col0\" class=\"data row2 col0\" >int64</td>\n",
       "      <td id=\"T_d0256_row2_col1\" class=\"data row2 col1\" >5</td>\n",
       "      <td id=\"T_d0256_row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "      <td id=\"T_d0256_row2_col3\" class=\"data row2 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d0256_level0_row3\" class=\"row_heading level0 row3\" >Timestamp</th>\n",
       "      <td id=\"T_d0256_row3_col0\" class=\"data row3 col0\" >int64</td>\n",
       "      <td id=\"T_d0256_row3_col1\" class=\"data row3 col1\" >458455</td>\n",
       "      <td id=\"T_d0256_row3_col2\" class=\"data row3 col2\" >0</td>\n",
       "      <td id=\"T_d0256_row3_col3\" class=\"data row3 col3\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x72e4ebed3610>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_summary(df_ratings, 'df_ratings', analysis=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABdIAAAPdCAYAAACOcJpIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAACldklEQVR4nOzdd3zV1f0/8HdCIKywl0xBBAUUF+JiabUq7tlW62oddfxatbXV1ln3qm21Wv266taqraNuwQUqIFZxsASRgjIEwkogyef3R5prQpJPEgjD+nw+HvcBuZ/zOZ9zb869uff1OZ9zspIkSQIAAAAAAKhS9sZuAAAAAAAAbMoE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAPxPuOSSSyIrKyuysrI2dlP4jjvhhBMiKysrNt98843dFAD+h82cOTPz2eeee+7Z2M35n3PPPfdknt+ZM2du7OYAsAkQpAP8Dyv/BWtdbnxj3rx5ce+998aZZ54Zu+22W/Ts2TPy8vIiNzc3Nttss/j+978ft956ayxfvrzaOtbm91JTKLtixYq49tprY9CgQdGmTZto1qxZbLXVVnHuuefG559/XqfHOGvWrLj44otjp512ivbt20fjxo2jW7duMWTIkLjoooti0qRJdaqvvOHDh1d4XJdddlmt9rviiisq7Dd8+PC1bgP1b/ny5XHLLbfEXnvtFV26dInc3Nzo2LFj7LDDDnHWWWfFiy++WGMdkyZNilNPPTW22GKLaNKkSbRv3z6GDBkSt912WxQVFa1Vu1asWBG9evWq9euo/H719XqqSvnXQW3p+/Xriy++iMcffzx+85vfxJ577hktW7bMPMeXXHJJnet76KGHYp999olOnTpF48aNo0ePHnHsscfG2LFja11HffW7zz//PM4999zYaqutolmzZtGmTZsYNGhQXHfddbFixYrUfVevXh3PP/98nH322bHbbrtFu3btomHDhtGqVavYYYcd4le/+lV89tlntW5LUVFR3HbbbTFkyJBo3759NGnSJLbYYos49dRT46OPPqpVHcXFxfHAAw/EyJEjo1OnTtGoUaPo2LFjDB8+PP7617+u9ftDeeVPxpe/ZWdnR4sWLaJv375x7LHHxgsvvLDOx2LDmDt3btx0001xwAEHxBZbbBEtWrSIRo0aRfv27WOHHXaIn/zkJ/Hoo4/W+JoAgAoSAP5nzZgxI4mIdb6tjbvvvjuz/4wZM+r3gVXh4osvXqf21tYdd9xRq+esR48eyfjx46usY21+L/vss0+1bZo6dWqy5ZZbVrtvixYtkqeffrpWj+9Pf/pT0qxZs9S2/PznP1+bpy5JkiQZNmxYhbr69OlTq/369u1bYb9hw4atdRvWt+OPPz7TB74LXn311aRHjx6pfWbgwIGpddx+++1Jo0aNqt1/5513TubPn1/ntp177rmVXpc1qc/XU3XKvw5q69vQ978tZs6cmdpfL7744lrXtWLFimT//fevtq7s7OzkkksuqbGe+up3Tz31VNKiRYtq6+nTp08yderUKvedN29e0rZt2xr/HjVq1Ci56aabamzL/Pnzk0GDBlVbT25ubnLHHXek1jFnzpxkl112SW3PjjvumMyZM6fG9qQp/xmiptthhx2WFBQUrNPxalLbvlj+88Tdd9+9Xtv0bVFQUJD88pe/TJo0aVKr32deXl5y4YUXJsuXL69U14b+LAvApi8nAPif1aVLl/jwww+r3b7NNttERMROO+0Ud99994Zq1rdaVlZWbLnlljF8+PDYfvvto0uXLrHZZptFQUFBfP7553H//ffHCy+8EJ9//nnsvffeMWnSpOjcuXOFOmr6vZS56qqr4sEHH4yIiOOPP77KMkuXLo2RI0fG1KlTIyLi5JNPjh/84AfRpEmTGDVqVFx11VWRn58fRx99dLz11lux3XbbVXu8yy+/PC688MKIiOjTp0+cfPLJMWjQoGjZsmUsXLgwJk6cGE8++WRkZ6/7BW2NGzeOgoKCmDJlSrzzzjsxePDgasuOGzcuJk+eXGG/Tdk999zznbnE/uWXX44DDzwwCgoKolWrVnHaaafF8OHDo0OHDrFixYr45JNP4plnnomvvvqq2jr+9a9/xWmnnRYlJSXRsWPH+O1vfxuDBw+Or7/+Ou6444544okn4t13341DDz00Ro8eHQ0aNKhV2yZOnBg33XRTNG7cOBo2bBhLly6tcZ/6fD2x6UqSJPP/rKys2GKLLaJz587x+uuv17muk046Kf71r39FRMSIESPi5z//eXTu3Dk+/PDDuPLKK2P69OlxySWXxGabbRannHJKlXXUV7+bOHFiHH300bFy5cpo3rx5nH/++TFixIhYuXJlPPzww3HHHXfElClTYuTIkTF+/PjIy8ursH9hYWEsXLgwIiK22267OPjgg2Pw4MHRsWPHWLJkSTz33HPx5z//OQoKCuIXv/hFNGnSpNrHVFxcHIceemiMGzcuIiIOO+ywOPnkk6NNmzbxzjvvxOWXXx7z5s2LU089Nbp06RL77bdfpTpWrlwZ+++/f7z//vsREfG9730vTjvttOjZs2csXLgw/v73v8f//d//xYQJE+KAAw6It956Kxo3bpz+C6uFu+66KwYNGhQRESUlJfHFF1/EmDFj4g9/+EOsXLkynnjiiTjnnHPilltuWedjravNN9+8Qn/+rluwYEEcdNBBmStBmjVrFkcddVTsueee0b1798jLy4uFCxfGtGnT4qWXXornnnsuli5dGr///e9j//33j1122aVCfSeccEKccMIJG+GRALDJ2thJPgAbT6zHEY7/qyPSV69eXWOZP/zhD5m2nH322Wt1nKKioqRz586Z0VIrVqyostyFF16YOda1115baftbb72V5OTk1Ph7fvnllzP1HHfcccmqVauqLVtYWFjnx1OmbCRujx49kp133jmJiOSMM85I3eess85KIiIZPHhwZuSzUbkbX/nRq9ttt13y5ZdfVlu2uj6zatWqpFevXpkRt9OmTatU5vTTT6/ziMuioqJkxx13TCIiueyyyzL9pqYR6fX1eqqJEekb14IFC5LLL788efHFF5Ovv/46SZIkGTVqVJ1HpL/yyiuZfQ488MCkqKiowvb58+cn3bt3TyIiadWqVeZYa6qvfjdkyJAkIpKcnJxkzJgxlbZfe+21qY9x9uzZyd57752MHTu22mO8/fbbmZG+LVu2TPLz86ssd+edd2aOdfrpp1faPnXq1MzI+d69e1f5t/W6667L1HHiiScmJSUllcrcddddmTLXXXddte2uSfnPEKNGjaqyzHvvvZf5PeTk5CRz585d6+PVpK59kdL3/eHDh2eeu4MOOij56quvUvf58ssvk3PPPTdp0KBBar8HgDKCdIDvMEH6+rF69eqkefPmSUQkO+2001rV8fzzz1cIEKqyatWqpGXLlklEJFtvvXVSXFxcZblTTz01U9e7775baXtxcXFmSoGBAwfW6mTB2iofpN98881JRCTt2rWrNrhfvXp10r59+yQikltuuUWQvgn5yU9+kkRE0rRp02TmzJlrVccjjzyS6ZtXXXVVlWWWL1+etG7dOomIpF+/frWq94YbbkgiIunbt29SWFhYqyC9vl5PtSFI3/SsTZC+3377ZULVL774osoyDz30UGpIXl/97p133slsP/XUU6uso7i4ONl6660zwX7aCdM05adMevzxx6ssU3acNm3aVDllRpIkyVVXXZWp59FHH620vX///klEJM2aNUsWL15cbXt23333JCKSDh06VDqZUVu1CdKTJEmOPfbYTLmHHnporY5VG4L0uit/4uWAAw6o9rVUlTFjxlR5IhcA1mSxUQCqVVJSEvfff3/sv//+mQW+2rdvHyNGjIi//OUvsWrVqkr7jB49OrKysuLEE0/M3NezZ89KC3iNHj26wn5vv/12/O53v4vhw4dnjtWiRYvo169f/OxnP4uPP/54fT/cepOTk5O5vHxtpyH529/+lvl/ddO6jBo1KpYsWZIpU92UK+UvS37yyScrbX/xxRczUwr8+te/jpycDTPz29FHHx0NGzaMBQsWxHPPPVdlmeeeey7mz58fDRs2jKOPPrrWdX/44YdxyimnxJZbbhlNmzaNvLy86N+/f5x99tkxc+bMKvc56aSTIisrK5o0aVKraUD69u0bWVlZsfPOO1e4/4QTTqjVwpZLliyJq666Knbfffdo3759NGrUKDbbbLM48MAD4+9//3uNl+s/+eSTccghh0TXrl0jNzc38vLyolevXjFkyJC48MIL4913363xMaytRYsWZaYdOvbYY6NHjx5rVc8//vGPzP+ru3y+adOmcdRRR0VExMcffxxTpkxJrfPzzz+Piy66KCIibrvttmjUqFGt2lJfr6eNZfHixXHFFVfErrvuGq1bt46GDRtG+/bto1+/fnHooYfGrbfemjrFTkFBQdx8882x1157Zd6DO3ToEN/73vfizjvvTF3QcfPNN4+srKzMczNhwoQ44YQTomfPnpGbm1tpUdV1beumYOnSpfHKK69EROmUI127dq2y3GGHHRYtWrSIiKr7S331u/KvpfJ/f8vLzs6O4447LiJKfwejRo2qslxNRowYkfn/9OnTK22fMmVKfPLJJxERcdRRR0XTpk2rrCftMa1cuTKzGOmuu+4aLVu2rLY9++67b0SULgb+xhtv1O5BrKWyKfEiShesrcqiRYvi7rvvjmOPPTb69esXzZs3j0aNGkWnTp3i+9//ftx+++1Vfn6K+Oa1VObSSy+t9Pmp/PNWfvHyqqYUK7+IakTp6/y6666LHXbYIfLy8iIvLy923nnnuPnmm2u1aOubb74Zhx9+eGZR3V69esVpp50W06ZNi4hvFlKublHkgoKC+NOf/hTDhw+P9u3bR8OGDaNNmzbRt2/f2G+//eLGG2+s9m90msLCwrjhhhsiovRvxp133lmnaeh23XXX2GKLLSrdf88992Sev/Ltev311zP333HHHTXWf9VVV2XKV/d59r333ovTTjst+vbtG82bN49mzZpF375942c/+1nq370121hSUhK333577LbbbtG6deto1qxZbLvttnHFFVdYWBWgPmzsJB+AjSdSRjguXLgwM8qrutvWW29daSRq+VF9abfyI77Kj16v7tagQYPklltuqfax1DQivfyCXOt7RGf5aVKOOOKIOu+fn5+fNG3aNImIZPPNN6/ycvYkqTgdQNolyatXr87UN3To0ErbTzrppCQikqysrAqX6S9cuDCZMmVKsnDhwjo/huqUH5GeJEly0EEHpT5PRx55ZBIRycEHH5wkSVKrEelXXnllkp2dXW1fys3NTe69995K+5X/vd1zzz2pj2PcuHGZsmsuulebxUZffvnlGhf123///ZOlS5dW2reoqCjzvKTddtxxxyqPXX409NpeLfK3v/0tU8czzzyTuX/58uXJ1KlTk7lz51bbb8vr1q1bZuR4mgcffDBzvLvuuiu1bNnCjz/+8Y8z99VmRHp9vZ5qo75HpH/88ceZqaDSbn/+85+rrPv999+vccHYQYMGVTt9T9m+xx9/fHLrrbdmpr8of6uPtq7P9/G6jkgvP61LdVdTlNlnn32SiNKR62uOAq+vflc2rUuzZs1SryoaM2ZM5ngXXXRRDY+yao8//nimjhtuuKHS9vLTutQ0artPnz5JRCTdu3evcP/s2bMzdRxzzDGpddx+++2ZspdeemndH1BS+xHp5adu+8Mf/lBlmZpeSxGRbL/99lVODVObfY8//vhM+ZoWGy3/uL788stku+22q7beAw88MHUU99VXX51kZWVVuW9eXl7ywgsvZN7bqnp9zpkzJ+nXr1+Nj+/cc8+ttg3V+ec//5nZ/6STTqrz/tWp7urKkpKSzLRNw4cPr7GeAQMGJBGl06Ctqbi4ODn77LOrfW7L3jv++te/1tjGjz76KNlrr72qrWfnnXdOli1bttbPBwBGpANQheLi4szCXRERw4YNi8ceeyzGjx8fTz31VBxyyCEREfHJJ5/EXnvtFcuWLcvsO2jQoPjwww/j8ssvz9z3wgsvxIcffljhVraQV0REUVFRtG7dOk444YS466674o033oj33nsvnnnmmbjsssuiXbt2UVxcHGeeeWa8+uqrG+ZJqKOlS5fGxx9/HJdcckkceuihmft//vOf17muv//975lRQz/+8Y8rjeYsU35U01ZbbVVtfTk5OdG7d++IiMwowfLefvvtiCgdCZeXlxcPPvhgbLPNNtG2bdvo06dPtG3bNvr27RvXX399FBYW1vnxpPnxj38cERFPP/10LF68uMK2JUuWxNNPP12hXE3+8pe/xAUXXBAlJSXRvn37uP7662Ps2LHx5ptvxiWXXBLNmjWLwsLCOOGEEzILBJYZMWJEZmHYBx54IPU4ZaOxGzRoED/4wQ9q1bYyb731Vuy3336xcOHC6NixY1x++eXx9NNPx4QJE+Lpp5+OY489NiJKF+Gs6mqEW2+9NR577LGIiNhjjz3innvuybxmXnrppbjhhhti7733rvWinGujrM9ElI7QHDduXOyzzz6Rl5cXW265ZWy22WbRsWPHOPPMM6sdWbxs2bLMiM60/rvm9qr6cJmHH344/vWvf0Xr1q0zoxNrq75eTxvDj3/845gzZ040bNgwTj/99Hj66adj3Lhx8c4778Tjjz8ev/rVrzJtXtO0adNi2LBh8fnnn0eLFi3i/PPPjyeffDLGjx8fL7zwQpxxxhmRk5MT48aNi4MPPjhWr15dbTvGjRsXZ555ZnTt2jVuvvnmePvtt+PNN9+Mq666ql7auimpbX8pv72oqChz9U9d66mp35Xd17t379Srimr7Wkrz2muvZf6/9dZbV9q+Ns/NF198EcuXL8/c37x588z/y0bsV6f89vV99Vr556y6q46Ki4tj8ODB8fvf/z6eeeaZGDduXLz11ltx//33Z0bPT5w4scq/HS+++GKFxch/9rOfVfr8dMUVV6xV2w877LD4+OOP4//9v/8XL730UkyYMCEefPDBzO/w6aefrnZ09aOPPhq/+c1vIkmSaNOmTVxzzTUxZsyYGDNmTFxzzTWRk5MTP/jBD2Lu3LnVHv+ss87K/H6OPfbYeOKJJ+Ltt9+OcePGxVNPPRUXXXRRDBw4cK0eW/k+WfYcr09ZWVnxwx/+MCJKR6f/5z//qbbsBx98EJMmTYqIiGOOOabS9rPOOiv+8Ic/RJIkMXTo0Ljrrrti9OjR8e6778Ydd9wR/fv3j6Kiojj11FPjqaeeSm3XySefHKNGjYrjjz8+nn322ZgwYUI8+eSTseuuu0ZExLvvvlvh8zkAa2FjJ/kAbDxRzci+srmrI0oXnqxqZOkFF1yQKXPeeedV2l6XOdJnz55d7RyqSZIkixcvTrbddtskIpI99tijyjIbY0R6+WOueWvQoEFy8803r1W95RfLmjp1arXlBg8enBmBWJORI0dm6iwoKMjcX1xcnBm9PWjQoOT//b//lzpSbLfddksWLVq0Vo8rSSqPSC8oKEhatWqVRERy++23Vyh7xx13JBGRtG7dOtPmtBHp8+bNy4zY7Ny5czJr1qxKZd57772kWbNmSUQkXbp0qTQ69Jxzzsn8/qobfVtcXJwZUbvPPvtU2p42In3VqlXJ5ptvnkREsu+++1bb78uPsHzxxRcrbCsbeTp48ODUkafVXUlQHyPSy/fRe+65p8oRyGW3Tp06Je+//36lOj755JNMmZoWnJ0/f36m7A9+8IMqy3z99ddJx44dk4ioNHKvNiPS6+P1VFv1OSJ9+vTpmW3VjThPktIRlFUtdrnbbrtlRsjOnz+/yn2fe+65zPvEmq/TJKk4inabbbap9j1iXdu6KY1I//Wvf50pP27cuNSy5edufv755ytsq49+t3Llysz9I0eOrLGesvfAXXbZpcaya5ozZ06Sl5eXRETSvn37ZOXKlZXKHH300Zn2VNenypxxxhmZsp9++mmFbZtttlkSUbqORtoi1wceeGCmjl133bXOjylJajcifdasWZnnrlWrVlVeMZQkSTJlypTUY5VfIPXll1+uskxt+2JdRqQ3bNiwyse2cOHCzHvntttuW2l7QUFBZnu7du2q/FwyefLkpE2bNtW+PleuXJk0bNgwiah5xPnaXAX3ve99L3Ps+pzrPO2z7AcffJDZlrbQbdl7RXZ2djJ79uwK21588cVMHf/3f/9X5f4rV65M9txzz8zfsDX/7q95Ved9991XqY6CgoLMqPi2bduu17VwAP7XGZEOQCW33HJLRES0b98+br755ipHRF966aWZkWR33HHHOo1U7tKlS7VzqEZEtGzZMi677LKIKJ2fc+HChWt9rA1hr732ikmTJsUZZ5xR531nzZqVGVm12267pY7MLJvHu/zIveo0a9Ys8//yVxAsWbIkSkpKIqJ0XvE//elPsdlmm8X9998fX3/9daxYsSJee+212GWXXSIiYsyYMXHSSSfV+XFVJzc3NzP/9f33319h23333RcRpXPs5ubm1ljX3XffnRnJf+ONN0a3bt0qldl+++3j/PPPj4iI//znPxXmFY74ZrRYcXFxPPzww1UeZ9SoUTFnzpwK5Wvr4YcfjpkzZ0bjxo3jb3/7W7X9/uSTT87Mvb7mvLdffvllRJT2j7SRp23atKlT2+ri66+/zvz/tNNOi6ysrLj88stj1qxZUVhYGB999FFmHt8vv/wyDjnkkMjPz69QR/l56Gvqw9X13/J+9atfxVdffRW77rprnHzyyXV9SPXyetoYyvpDRMTQoUOrLZeVlRWtW7eucN8bb7wRY8aMiYiIe++9N9q1a1flvvvuu28cccQREVG5P67plltuiVatWtV7Wzc19dV/66Pf1aUt5eupa99NkiROPfXUzPEuvPDCzHog5dXXc3PQQQdFRMSCBQuqvcLkzTffjGeffbbKY9eHkpKSmDVrVjz44IOxxx57ZEbN//73v6/2sW255ZapdZ544omx3XbbRURU+hu0Pp111llVzl3epk2bzLz6H374YaUrAP7xj39kriy65JJLqvxc0qdPn7j44ourPfbXX3+duZol7bVf1p66WrBgQeb/7du3r7bcwoULY9KkSVXeZsyYUadjbrPNNpk586u7ii1JknjooYciovTqzi5dulTYfvXVV0dExOGHHx4/+clPqqyjcePGcfPNN0dE6RogaWsbHHbYYZmr2srLzc2NM888MyJKn4Nv07pDAJsaQToAFcyZM6fCImF5eXlVlsvJycl88Vq0aFG899579daG5cuXx8yZM+Ojjz7KfMFp2LBhZvu///3vOte5+eabR5IkkSRJpYVO19bpp5+eudT67bffjrvvvjtGjBgRr7zyShx99NHxzjvv1LnO+++/P7PIZNmicNUpW8i0Nosplg+iV65cmfl/+UvpCwoKomnTpjFq1Kg45phjonXr1tGkSZMYOnRovPrqq5lLrp988sm1emzVKZu25Y033ojPP/88Ikq/LJYtGlfbaV1efvnliIho1apVHHbYYdWW++lPf1ppnzI77LBD5gRR2fQtayq7v0mTJhWm8amNssuyhw0blvplP+KbsGHs2LEV7t9ss80iovQy/PLhQW2NHj0681qoaUHU6qzZb+6888747W9/G926dYtGjRpFv3794u67745TTjklIkoXxLv11lsr1FF+Id6a+nB1/bfM66+/HnfddVfk5OTEbbfdVu10SGnq4/W0MZT1h4iaQ+41lfXHvn37VlhEsSpl/XHcuHHVLkrYrVu3GDJkyHppa8T6eR9fW/XVf+uj39WlLeXrqWvfvfLKKzPTbY0YMaLak8X19dz8+te/znwG+e1vfxtnn312TJ06NVavXh1ffvll3HLLLbH//vtXOKFYH6/HESNGZBZvbNCgQfTo0SOOOeaYmDVrVnTu3Dn+7//+LxNK1iRJkvjyyy9jypQpFULbskB1bT7PrK20E7877rhjRJS2d81AuezvZHZ2dmodxx57bLXvvW3bts30hfvuu69WC5vWRfkTKOVPzqzpvvvuywTga96qW6Q3Tdnz8f7771c5VdKbb74Zs2bNqlC2TH5+fuZ9rOxEZXW23nrrzInONT8TVNWeqpT9jiMiPvvss9TjAVA9QToAFZTN4xgRMXjw4NSy5beX329tLFiwIC644ILo27dv5OXlRc+ePWPAgAGZLzgjR46sUHZT0KFDhxgwYEAMGDAgBg8eHCeccEK8+uqrcfnll8cHH3wQw4cPjxdffLFOdZaNws7NzY2jjz46tWzZSMBVq1bVWG/5KwaaNGlSqY4yP/3pT6Nv376V9m/SpEmFeVkfeeSRGo9ZW3vssUf06tUrkiTJjEq/7777IkmS2GKLLWL33XevVT1lfXCHHXaocOJlTR07dswEyFX127Ivou+++25MmzatwrbCwsJ44oknIqJ0tGR1J5qqM378+IgoXTegLKip7nb99ddHRMVRvBGRmTd92rRp0bt37zjppJPioYceitmzZ9epLeuifL/Zdtttqz3ZceWVV2aCsjX7TPk6aurD1fXfsm2nnHJKJEkSP//5z2Pbbbet3YNYQ328njaGnj17ZsLrP/zhD9G/f/+46KKL4tVXX81coVGdsv44efLkGvtjWXC4evXqClcklFfTc78ubd3U1Ff/re/38brUU5e++8ADD8SFF14YEaW/xwcffDCys6v+Kllfz03Pnj3jkUceiebNm0eSJHHTTTdFnz59olGjRrHZZpvFmWeeGStWrMiM1o2IOr8n19V+++1XqyuRnn322TjggAOiZcuWsdlmm2VOVpXdykbRb8jPM2nz1ZcfBb7mqP6yv5O9evWq9mqTsjp69epV5bbyn2n+/ve/R+/eveO8886Lf/3rX5XWR1kb5X/v5U/0rm8//OEPMycPqhqVXnbiPTc3Nw4//PAK2yZOnJi5IrCsnrRbWV9Z8zNBeWv7Owag9gTpAFRQPiDp0KFDatlOnTpVuV9dTZgwIbbaaqu46qqrYsqUKZkR2dXZ2CNAa/Lb3/42dt555ygoKIiTTz651iOv3n333fj0008jojSkTfvCGvHNF8faXJ5f3SJua4YO++yzT7V17LXXXpmRf+PGjavxmHVRdily2YmEsn+rukS5OmV9sKZ+G/FN362q3/7oRz/K/H/NL8bPPvts5kt/Xad1iYiYN29enfdZs7+fdNJJccEFF0ROTk4sWbIk7r777vjRj34U3bp1i969e8e555673keble83aX2mbdu2sdNOO0VE6cjL8qFa+Tpq6sPV9d+IiCuuuCImT54c3bp1i0svvbR2D6AK9fF6qq3yozZrer9bs0xVIz4feuihzGJyH3/8cfz+97+PvfbaK1q1ahVDhw6N2267rcIo4TJr0x8jotrQuzbTsaxtWzc19dV/6/t9vC711LbvPvvss3HiiSdGkiTRqVOneOmllyr8/V9TfT03EaXB9XvvvRfHHXdchb+JWVlZMWLEiHjjjTcyU8BE1K4P1uSuu+7KXG327rvvxoMPPhjDhg2LiIg777wzjjjiiGpft0mSxE9/+tM44IAD4tlnn60xsNyQn2fSptArf1KkuLi4wrZFixZFRPqUKWXSytx8881x4IEHRkTpVWfXXXddjBw5Mtq2bRuDBg2K6667rsaFZavTtm3bzP/nz59fbblf/OIXmataym7ronv37pmTg2texbZ69erMwuAjR46s9Jmuvt9/I9b+dwxA7QnSAajW2kyPUFerVq2Ko446KhYuXBgNGzaMc845J1577bWYO3duFBQUZL7oTJ8+PbPPun7x2RAOPvjgiCid8/zdd9+t1T5/+9vfMv+vaVqXiIiuXbtGRGkQUdOIri+++CIiSr/klr+UPjc3t8IX36rmFS/TuHHjzKXFaV9U10bZiObJkyfHzTffHFOmTKlwf12sa7/t1atXJuhb84tx2c9t27aNfffdt851l3153W+//TJBTW1ua7riiiti2rRpccUVV8See+6Z+fI8ffr0uPHGG2OrrbaK2267rc7tq63y/SStz5TfXlJSUuHERfm5YmsaTV/Wf6s63jXXXBMREd/73vfi6aefjocffrjSrSysW758eea+V199tUI99fF6qq3yI29rE6SVDxurmragS5cuMWbMmHj55Zfj9NNPj/79+0dWVlasXr063njjjfjZz34WAwYMyLyuypT1x4EDB9apP645z2+ZBg0a1PhY1ratm5qy/hKxbv23Pvpd48aNM0FiTW1ZtGhRpj/V9NqNKJ0K6ogjjojVq1dH69at44UXXogtttgidZ+1eW6ysrIq7FfelltuGffee28sXLgwZs+eHVOmTImlS5fGq6++GrvuumtMnTo1U7Z///41PqaalF0RN2DAgBg0aFD88Ic/jFGjRmWm/nj22WfjpptuqnLfu+66K+68886IiNhuu+3innvuiU8++STy8/OjqKgo85mm7G/bt+HzTH1p0aJFPPXUU/HOO+/EueeeGzvuuGM0aNAgSkpKYvz48XHeeedFnz59UqcuqU7Z1HMRpSO9N6Syk+ozZsyo0PYXXnghs6ZPVSfey4fZf/3rX2v9/lv+6kAANrzqV6gC4Dup/KWfZYtLVaf85aVru7Dhq6++mhk9+5e//KXC/NXlrcuI942hfDj9+eefx2677ZZafvXq1ZnFLTt06FCrkLZfv37x+OOPR0TEp59+mlkQdE1FRUWZExFbb711pe39+/fPzNNZ0yilsu1pi1yujd69e8euu+4aY8eOjV/96lcRUbqYZk2BTXlt2rSJuXPn1thvI77pu9X122OOOSbGjh0bU6ZMifHjx8dOO+0U+fn5mcvxjzzyyNTpY6rTtm3bmDNnTqxatSoGDBhQ5/3L69GjR1xwwQVxwQUXxOrVq2PcuHHx6KOPxl//+tcoKCiI008/PQYPHhzbb7/9Oh2nKv3798+MtKttn4mo2G/y8vKiW7du8cUXX2SuxKhO+e1r9uGyUe5333133H333an1LFiwIH74wx9GROk89XvuuWdmW329nmqjfL/78ssvq50OoXyZqvZd01577RV77bVXRJQuKPfyyy/H7bffHq+++mpMnz49jj766AohU1n4umzZsnXuj3VV17Zuavr165f5f237b05OTqWFKOur3/Xr1y/eeOONmDZtWhQVFVX7Hp32WlrTu+++GwceeGAUFBRE8+bN47nnnqvV1ElrPjdlC2umtadbt26pc1tHlI6oreokzoQJEzL/L1ukub5lZWXFzTffHK+88krMmjUrLr300jj++OMrvR7vuOOOiCj9mzZmzJhqp8/5Nn2mKRvlX5sT6LUps/POO2d+T0uXLo3Ro0fHPffcE0888UTMmzcvDj/88Jg+fXqdph4aNmxY3HjjjRER8fzzz9c453h9OvLII+Oss86KVatWxQMPPFDpRHzLli0rTE9Ypvwo+qZNm27w92AA1o4R6QBUUP6DfE0LSpYfab3mF4Dajgr+6KOPMv9PmxO8bC7fb4v//Oc/mf/X5vL5Z599NjNy6Uc/+lGtguo99tgj8//XXnut2nLjx4/PjECsar7xskUEI9IXoMrPz8/M0VndiNR1UTZCr2xah7qORi/rg++9917qdDrz5s3LLGpa3RfXo446KvM7KPsy/Pjjj2fatjbTukREJtQeP358reYyrq2GDRvGbrvtFjfddFOmvUmSxN///vd6O0Z5te0zEZEJ/xo3blwpdCrrw5MnT06d97V8/67tnPl1VV+vp9ooH0aWDwGrU34x59rOAd+2bds4+uij45VXXslMffH+++9XGL1b1h8/++yz1Od/fatNWzc1gwYNyiyemNZfVq1aFW+//XZmnzVPwNVXvyurZ/ny5al9qravpQ8++CD23XffWLZsWTRu3DiefvrpGtdNWbMtax5vTWWLcNbUlpqUndRr0qRJZuqQ9aFp06Zx0UUXRUTEkiVL4tprr61UpuwzzUEHHVRtEJwkSb0u0L6+lY3y/+yzzzLTvFTl66+/rvO0Ynl5eXHggQfG448/Hv/v//2/iIiYO3duvPnmm3Wq5/vf/3507NgxIiIefvjher9qLk3r1q1jv/32i4iIRx99NIqKimL58uXxz3/+MyJKFxKt6sql7bbbLvNZ+a233tpg7QVg3QjSAaigc+fOmVFqjz76aLXzmxYXF8c999wTEaVfInbYYYcK28svNlZ+MbE1lQ88q1sgqqSkJDPK69ugpKQkM8IwImKbbbapcZ/y07qULSZZk+HDh0fLli0jIuLee++t9hLxst9TRMShhx5aaXv5BbCefPLJao/35JNPZo5RNidofTr66KMjLy8vcnNzIy8vL4466qg67f+9730vIiIWL16cWRC0KnfeeWfmcZTts6b27dtn5v5++OGHo6SkJBNQ9+jRY61Dn7KQsGxu8/WhbJRvxPpbyG7o0KGZqy6efvrpakelz5gxI95///2IKA3K1lyc8JBDDsn8v3w/LW/FihXx6KOPRkTpSNc+ffpU2L7mfLdV3Xr06BERpb+7svvKrsIoU1+vp9oo/ztac/qgqpQtwrvmvmtzvPJ9oqw/JkkSf/zjH+tc7/qwIfpvfcjLy8u09eWXX652CpMnnngi8vPzI6Lq/lJf/a78a6m695aSkpLM35pWrVrFiBEjqiw3ZcqU2GeffWLRokXRsGHDePzxx2P48OFVlq1Knz59KnyOqG5O5/p4Lb344ouZEPKYY46pcW2RdXXcccdF9+7dI6L0Kro1R5aXfaZJW/Dyn//8Z8ydOzf1OGWfodI+P20oZf28/N/Bqtx///3rNFXNurz2c3Nz45xzzomI0uf+5JNPzizkuSGUnVyfP39+vPTSS/GPf/wj0++rO/Hevn37zBUoDz744AYN/wFYBwkA31kRkUREMmzYsAr333zzzZltJ554YpX7XnjhhZky5513XqXtr732Wmb7s88+W20bHn/88Uy5q666qsoy5513XqZMRCR33313pTIXX3xxZntVZsyYUe3jrYvbb789KSoqqnZ7cXFx8otf/CJzrCFDhtRY58KFC5NGjRolEZFss802dWpP+d/DtddeW2n7mDFjkpycnBof93777ZdERJKdnZ28/PLLlbbPnTs36dq1axIRSaNGjZLZs2fXqZ1lhg0blkRE0qNHj7Xav0ePHtU+lnnz5iVNmzZNIiLp2rVrlW18//33k+bNmycRkXTp0iVZtWpVtcd64IEHMs/tfffdlzRo0CCJiOT8889PbePxxx9f7WMsKChIunXrlkRE0rx58+S1115LreuNN95IRo8eXeG+++67L1m9enW1+zz22GOpr6my30FEJDNmzEg9fpprrrkmU8/FF19cafvq1auTfffdN1Pmscceq1Rm1apVSa9evZKISFq0aJFMmzatUpnTTz899bVfG2X9pqZ+V1+vp9rYddddM8d68MEHqy33t7/9LVNujz32qLR94sSJycSJE6vdv6SkJDnwwAOTiEiysrKSOXPmVNi+8847JxGRNGjQIHnkkUdS2/zBBx8kTz31VKX7y57f448/PnX/dW1rfb2PV2XUqFGp/bkqr7zySmafgw46qNLfhvnz5yfdu3dPIiJp1apV8vXXX1dZT331uyFDhiQRkeTk5CRjxoyptP3aa6+t8TF+/vnnmfeoBg0aVPm6rY0777wzc6wzzjij0vZp06YlLVq0SCIi6d27d7XvaWl/az744IOkffv2SUQkbdu2TebNm7dWbU2Sip8hRo0alVq2/Gekiy66qMK2bbbZJomIpHPnzsnChQsr7Ttt2rSkc+fOmf2re0/q2bNnEhHJkUcemdqW8q+JtflsVKZ8/1/z8a9cuTLp0KFDEhFJu3btqnyfnjJlStKmTZtqX5/Tp0+v9LdsTWeccUZm/7Fjx6aWrcrq1auToUOHZuo49NBDk/nz56fu8/XXX6e+p9x99921+nu5cuXKTH8+5phjkv333z/zOaO4uLja/Z599tlM/UOHDk0WLVpUbdmCgoLk5ptvTlauXLlWbayprwBQO4J0gO+w6r48FBUVVQh59txzz+Tvf/97MmHChOSZZ55JDjvssMy2LbbYIlm6dGmluvPz85PGjRsnEZHssMMOyYsvvphMnjw5mTp1ajJ16tRkxYoVSZIkybJlyzJf0Bo0aJCceuqpyfPPP5+MHz8+efjhh5O99toriYhk99133ySC9LIvRj//+c+Thx56KBkzZkzy/vvvJ6NGjUpuvPHGZNttt80cp0WLFskHH3xQY5233HJLZp/rr7++Tu3Jz89P+vTpk9n/lFNOSV599dVk7NixyZVXXpkJjZs0aZIaYE2ePDlp1apVEhFJ48aNk9/85jfJ66+/nowbNy655ZZbMiF6RCTXXHNNndpY3voM0pOk4nPZsWPH5A9/+EPyzjvvJG+99VZy6aWXZp6PrKys1BM8SVLaN5s1a5YJwcrqnTRpUup+aUF6kiTJ2LFjk9zc3EyfP+aYY5LHHnssGT9+fPLuu+8m//znP5OLLrooE8j8+c9/rrB/2WP72c9+ltx3333JmDFjkvfeey957rnnknPOOSdp0qRJJqifNWtWpePXV5C+cuXKZIcddsjU9YMf/CB57rnnkgkTJiSPPvpohfeQ/fffPykpKamynmeffTbJzs7OPK4///nPyTvvvJM8//zzyeGHH14hRE47iZWmtkF6fb2eauPDDz/M/K6ysrKSI488MnnkkUeSd999Nxk3blzyyCOPJEceeWSSlZWVRETSrFmz5OOPP65UT1mIMmjQoOSyyy5LnnnmmWT8+PHJ2LFjkwcffDDZe++9M4/n4IMPrrT/tGnTKgRgBx54YHL//fcn77zzTjJ+/PjkX//6V3LFFVcku+yySxIRybnnnlupjtoG6eva1voM0p977rnk7rvvztx+/etfVzh2+W1pYfIPfvCDzH4jRoxI/vnPfybjxo1L7rrrrmSLLbbIbPvrX/9abR311e/ee++9Cq//K6+8Mhk7dmzy6quvJqecckqm/j59+iT5+fmV9l+wYEGy5ZZbZsqdd955yYcffph6qy7oLioqqvB3+/DDD0+ef/755J133kn+/Oc/Z/7uZ2dnJ//617+qfUzbbLNNstdeeyV/+tOfktdffz157733kmeeeSY5/fTTM++jjRs3Tl588cVq66iNugTpK1euTDp16pRERNK6desKz+V1111X4Xm+8847k3feeSd57bXXkosvvjhp2bJl0rhx48x7Z3XvScccc0wSEUlubm5y2223JR9++GHm89NXX32VKbchgvQkSZIHH3wws71t27bJtddem4wdOzYZO3Zscs011yRt2rRJWrVqlek/w4cPr7L+fv36Jb/97W+TJ598Mnn33XeTd999N3n88ceTo446KlP/dtttV+3fi5p89dVXmZODZa+Dn/70p8kDDzyQvPHGG8n777+fvPXWW8l9992XnHzyyZnwOyKS/fbbr1J9tQ2pkyRJTjjhhMx7dcOGDZOISH75y1/W2Oaf//znmWN06tQpueSSS5KXX345mThxYvLmm28m99xzT/KTn/wkad26dRIRlT5zC9IBNixBOsB3WFogsXDhwgpfgqu6bb311snMmTOrrX/NkeTlb+W/qD3//POZ0L2q2/Dhw5NJkyat05fF+gzSa3PbeuutkwkTJtSqzsGDB2dC1blz59a5TVOnTq0Qfqx5a9GiRfL000/XWM8bb7yRdOzYsdp6srKykt/97nd1bl956ztIT5IkueKKKzLBbFW33Nzc5N57763V8crCjLLbwIEDa9ynpiA9SUrD9LJRnzXd1mxrbfZp2bJl8txzz1V57LKQoWHDhlWOmKyLOXPmJDvuuGNqW/bff/8qQ7vybr/99sxVGVXddt555xpHFqapbZCeJPX3eqqNMWPGJF26dKnx99mlS5fk7bffrrKO8iFK2m233XZLFixYUGUdkydPTgYMGFCrei699NJK+9c1SF/btn788ceZMocddljNT3CK8ieUarql9ZsVK1ZkRp9WdcvOzq7VCPf66ndPPfVUhXBwzVufPn2SqVOnVrlv+TC1tre03/n8+fOTQYMGpb4X33HHHamPp3///qnH79atW/LKK6/U+LzUpC5BepJUHN1/5ZVXZu5ftWpVss8++1Tb3iZNmiSPPvpojX8nJk6cmDlRkPacb6ggPUmS5PLLL8+c2Fvz1rRp0+TZZ5/NXBWx7777Vlt/2m2rrbZKPvvss9S21mTlypXJ2Wefnfq5cs3X1gUXXJAsW7asUl11CdJfeumlSnXX5oRrSUlJcumll2auOkm7NWvWLDMQpa5tFKQD1A9BOsB3WNkH6upCyeLi4uRvf/tbsu+++yYdO3ZMGjZsmLRt2zYZPnx4cvPNNyeFhYWp9ZeUlCR33HFHMmTIkKRNmzaZqTGq+qI2adKk5Nhjj006d+6cNGzYMGnfvn0ybNiw5Pbbb0+Ki4vX+ctifQXpEyZMSC6//PLkgAMOSPr375+0a9cuycnJSVq0aJFstdVWyY9+9KPkscceS50ypLwpU6Zk2rXmF8+6WLZsWXLNNdckO+20U9KqVaukadOmSd++fZOzzz479WTHmhYsWJBcfPHFycCBA5MWLVokjRs3Tnr27JmceOKJyXvvvbfW7SuzIYL0JEmSf//738nJJ5+cbLHFFkmTJk2SZs2aJVtvvXXy85//vE6jsP/1r39V+BJb1bQLa6pNkJ4kpZdp33bbbcnIkSOTzp07J40aNUoaN26cdOvWLdlnn32SK664Ivn0008r7Tdp0qTkmmuuSQ488MCkX79+Sdu2bZMGDRokrVq1SnbZZZfk4osvTr788ssqj7ly5cpMYP2Tn/ykVs9BTVavXp3cdtttybBhw5L27dsnDRs2TDp16pQcdNBByRNPPFHrej788MPk5JNPTnr16pU0btw4adu2bbLHHnskt956a+pUNrVRlyA9Serv9VQbK1asSG699dbkgAMOSLp06ZI0btw4ady4cdKlS5dk5MiRya233lrpUv7yCgoKkn/961/J2Wefneyxxx5Jz549k6ZNmyaNGjVKunbtmhx00EHJAw88kDq9QJKUjiB+8MEHk8MPPzzp3r170qRJk6RRo0bJZpttlgwfPjz53e9+V+3JwdoG6eva1vLThbzxxhupx6pJfQXpZR544IFk7733Tjp06JA0atQo6datW/KjH/2oyilWqlNf/W7mzJnJ2WefnfTp0ydp2rRp0qpVq2SnnXZKrrnmmmT58uXV7lffQXqSlL4//OUvf0n22GOPpG3btknjxo2TXr16JSeffHKNV/ckSemVA6effnoycODACu8vw4cPT/70pz9VGX6ujboG6UuXLs1cydG+ffsKz+vq1auTP/3pT8lOO+2UNG3aNGnSpEnSu3fv5LTTTks++eSTJElq93fivffeS374wx8m3bt3rxCqb6wgPUmS5PXXX08OOeSQpEOHDklubm7So0eP5KSTTspcLTNw4MAkIpKjjz66wn5FRUXJ6NGjk/PPPz8ZMWJE0rt37yQvLy9p2LBh0rFjx2SfffZJbrvttqSgoCC1nXUxZ86c5IYbbkj233//ZPPNN0+aN2+e+Xw5cODA5KSTTkruv//+1NdEXYL04uLiZLPNNsuU79evX53a+9lnnyXnnXdestNOO2U+M+fl5SX9+vVLjjnmmOTee++t8qS0IB1gw8pKknVYEQQAgG+N0aNHx4gRIyInJycmT54cvXr12thNglo74YQT4t57740RI0bEq6++urGbA5SzevXqaNmyZaxcuTJ+97vfxe9///uN3SQAqHfZG7sBAABsGK+99lpERBxzzDFCdL51yvrvRRddtJFbAqzpH//4R6xcuTIiInbZZZeN3BoAWD+MSAcA+I7Ya6+94rXXXotPPvkkttxyy43dHKi12bNnR7du3WLIkCHx+uuvb+zmwHfOtGnTonfv3lVumzlzZgwZMiRmz54dHTt2jNmzZ0dOTs4GbiEArH+CdAAAAKBaOTk5sf/++8cBBxwQ/fv3j2bNmsW8efNi1KhRcdttt8XixYsjIuK+++6LY489duM2FgDWE0E6AAAAUK2srKzU7dnZ2XH55ZfH+eefv4FaBAAbnuutAAAAgGo9/fTT8dxzz8WYMWPiq6++ioULF0Zubm506dIlhg8fHmeccUYMGDBgYzcTANar/7kR6SUlJTFnzpzIy8ur8aw5AAAAAADfTUmSxNKlS6Nz586RnZ2dWvZ/bkT6nDlzolu3bhu7GQAAAAAAfAt88cUX0bVr19Qy/3NBel5eXkSUPvgWLVps5NYAAAAAALApys/Pj27dumUy5TT/c0F62XQuLVq0EKQDAAAAAJCqNlOEp0/8AgAAAAAA33GCdAAAAAAASCFIBwAAAACAFIJ0AAAAAABIIUgHAAAAAIAUgnQAAAAAAEghSAcAAAAAgBSCdAAAAAAASCFIBwAAAACAFIJ0AAAAAABIIUgHAAAAAIAUgnQAAAAAAEghSAcAAAAAgBSCdAAAAAAASCFIBwAAAACAFIJ0AAAAAABIIUgHAAAAAIAUgnQAAAAAAEghSAcAAAAAgBSCdAAAAAAASCFIBwAAAACAFIJ0AAAAAABIIUgHAAAAAIAUgnQAAAAAAEghSAcAAAAAgBSCdAAAAAAASCFIBwAAAACAFIJ0AAAAAABIIUgHAAAAAIAUgnQAAAAAAEghSAcAAAAAgBSCdAAAAAAASCFIBwAAAACAFIJ0AAAAAABIIUgHAAAAAIAUgnQAAAAAAEghSAcAAAAAgBSCdAAAAAAASCFIBwAAAACAFIJ0AAAAAABIIUgHAAAAAIAUgnQAAAAAAEiRs7EbsL4MuPiFyM5turGbAf/TZl49cmM3AQAAAADWOyPSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSCdVDOvHhkzrx4ZR+zYdWM3BQAAAABgo8jZ2A2g7h4+ZZfYpVfbzM9FxSWxaMWqeHfGorjquU9i9qKVdarviB27xvVHDoyIiM1/82yFbRNnLYqIiIXLVq1jq2tvyw7N45ff7xsDu7aKTi0bR0TEraOnxTXPT17rOhtkZ8VP9ugZh+3QJXq2bRaFxSXx+YIVceNLU2LU5HmRnRXxu5H94qDtOkdOdla88um8uPAfk2LFquKIiMjLzYmXzx0W946ZGX8ZPb1eHicAAAAA8O0gSP8WKywqjo/n5EfLJg2jV/vmMXLbzaJ3h+bx/Zter7djHPqXMfVWV231bNcs9t66Y3y2YHm91XnbsTvE3v06RUTEzAXLY8WqoujWpkn079IiRk2eF0ft1C1O2qNn/Orv/465iwvi/p8OjrmLC+L6F0vD+/P33zq+Xr4q/vr6Z/XWJgAAAADg22GTC9Jff/31uO6662LChAkxd+7cePLJJ+OQQw7Z2M3aJM1fWpgJum84amAcvkPX6NspL1o1bRiLV6wuvf/IgbFDj9bRPi83GjXIjnlLC+Klj7+KG16cEssKi+L6I7eNI3bslqlz5tUjIyLippenxE0vT838/MvH/h1/nzC7wuj1H97+dvx25NbRu0PzmDZvWVz4j0kx8YvFmbqO27VHnD68d7RokhMvfvRVfDB7cVx0YP+IiNjjmlerHTk/dvrC2PbSF2NZYVHm+FUpa/vsRStij2tGVVvuwG03i737dYrlhUXx4zvfjff+O8o+IqJpowYREdGvc4uIiBg34+v4z+LSdm29WV5EROzcs00cuVPXOPzWMVFcklR7HAAAAADgf9MmN0f68uXLY+DAgXHLLbds7KZ8K+WvXB3LCooyP+/dr2O0bNIwZi1cHnOXrIyurZvGibv3jGsO3zYiIj5fuCI+X/jNyO+JsxbFxFmLYu6SghqPdc+Jg6JJwwbRIDsrBnRpGX/+0fbRIDsrIiL22rpDXHbwgOjUsnGsXFUcg3q2iXP36Vurx7C0sCiWFRbVXLCWDhjYOSIiZn29In71/b4x6dLvx2u/Gh5nf2/LWF1cEhERH8/Jj4iIQT3bxOCepdPmfDJ3aTRqkB1XHbpN3DtmZnwwe0m9tQkAAAAA+PbY5Eak77fffrHffvvVunxhYWEUFhZmfs7Pz18fzdoktc/LjSdP3y0ztcui5aviN098EEXlRk0fffvY+GTu0szP5+7TJ87ac8vYp3/HyM3Jjj+/Oi3mLinIjDKvy1QuVz73adw7ZmacsNvmcclB/aNr66axedumMX3+8jh16BYRETFr4YrY74+vR0FRSTzw08EV5nZfV/PyC2P6vGXxZX566N+rXbOIiNh6sxaRX7A6vlpSEFt0aB4//16faNW0UVz81Efx6Pgvok/HvPjNvltFToPsePy92fGX0dPirL16R6Oc7Lh37Mz46493jME928SX+QVx1b8+jdemzK+3xwIAAAAAbLo2uSC9rq666qq49NJLN3YzNorcnAaxfffWmZ+nzlsW42cuqlBm997t4qajt48ebZtG44YNMvc3bJAdbZo1qtXI8+o8+d7s/x73m6C+XfPcmD5/efTp2DwiIkZPmRfL/7tg53OTvqzXIP3aFybHtS/UvABp2Sj5ouKS2P+Pb8TsRSvjmsO3jaMHdYsf7tw9fv/Mx1FUksRlz3wclz3zcWa/vh3z4tShW8RP7h0Xv/7+VrHnVh3iZ/dPiNOGbRG3HLNDDL12VHy9fMMtwgoAAAAAbByb3NQudXX++efHkiVLMrcvvvhiYzdpg5m9aEX0PP/ZOO7Od6JwdXHs3LNNXH34NpntB2/XOX43sl/07ZQXS1aujvdnLaowjUtZwLy28v87hUz5ecOz1q3K9eKr/NIrFr5eviozL/u//zuXe6Oc7OjUsnGlfbKyIq45fJt4+oM58cbUBbF773bx6dyl8fIn8+LpD+ZG89yc2L57qw31EAAAAACAjehbH6Tn5uZGixYtKty+S5Ik4vWpC+Jvb38eERF79+sU23ZtGRGRGa2+tGB1DLlmVBzylzHxxtQFlepY+d8R4xERTcqNWl8Xk78qHaW+R+920aRhg8jOiti3f6d6qbvMed/vG6+cMywe+Ong1HJvTit9zG2aNYourZpERMQ2/32OlhcWxbz8wkr7nLDb5tG1TdP4/X9HqGdlRWY+9aL//gsAAAAAfDd866d2odQdr38Wx+3aI3JzGsQZI3rHqfdNiE/nls4Xn9e4Ybx+3ohYVVQSeY0r/8qnz1+W+f9L5wyN+UsL4/JnP4kJny+qVLa2bn/9sxjcs230at+89NjFJdG6acNa7btdt1Zx09HbVbjvR4N7xH4DNosv8wviB7e/HRERHVrkxhYdmkduw/TzQfeNnRk/3LlbdG3dNJ77+ZCYt7QwenconXrmttemx6o1gvHOLRvHufv0jQue+DAWr1gdEaVh/Ii+HaJr6yYxtE/7WF5YFO/PWlyrxwMAAAAAfLt960ekU2re0sJ4cuJ/IiJi7607xpYdmscj47+IO974LBYuK4xmuQ3i7RkL48aXplTa99Mvl8YfX5ka85cWRNfWTWP77q2jZZPahd7VeeWTeXHRPyfFV/kF0Sy3QUz8fFH8ZfT0zPaC1cXV7tu4YXZs3q5ZbP7fRUIjIlo2aRibt2sWXVs3qXNb8guK4sjbxsY/3/9PFJcksVnLxvHh7CXxi4ffjz+/Oq1S+SsO3SbenfF1PPXvOZn7Ln3q43j7s4Xx/C+GRu/2zePMByfGQvOjAwAAAMB3QlaSJEnNxTacZcuWxbRppeHm9ttvHzfeeGOMGDEi2rRpE927d69x//z8/GjZsmV0+8WjkZ3bdH03l2rkZGdFp5aNM3OSZ2dF3H3CoBjWt0N8lV8Qg698ZSO3kPow8+qRG7sJAAAAALBWyrLkJUuW1Dhl+CY3tcv48eNjxIgRmZ/POeeciIg4/vjj45577tlIraKumjZqEKN/OTw+/M+SmL+0MPp2yosebUtHmN/wYuVR8QAAAAAAm6pNLkgfPnx4bGKD5FkLhUUlMWry/BjYtWX079wyClYXx9jpC+PONz+Llz+Zt7GbBwAAAABQa5tckM7/hsKikjj5b+M3djMAAAAAANaZxUYBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABS5GzsBqwvky79frRo0WJjNwMAAAAAgG85I9IBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUuRs7AasLwMufiGyc5tu7GYA8B018+qRG7sJAAAAQD0xIh0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgHQAAAAAAUgjSAQAAAAAghSAdAAAAAABSCNIBAAAAACCFIB0AAAAAAFII0gEAAAAAIIUgne+EmVePjJlXj4wjduy6sZsCAAAAAHzL5GzsBvDd9fApu8Quvdpmfi4qLolFK1bFuzMWxVXPfRKzF62sU31H7Ng1rj9yYEREbP6bZytsmzhrUURELFy2ah1bXbf2HLjtZtGnU160adooFq1YHe9/sTj++MqU+GTu0jrX1z4vNy4+oF8M7NYqurVpGhERT/97Tpz10MQK5Xbu2SZ+NmyL2LZry2jbPDciIn775IfxwDuzMmWaNmoQvz9kQOy1VYcoKkniqffnxOXPfhwlSen2rq2bxAu/GBrnP/FhPPXvOWv5DAAAAADA/4ZNbkT6VVddFYMGDYq8vLzo0KFDHHLIITF58uSN3SzWo8Ki4pg4a1HM+npFtM9rHCO33SzuPH5QvR7j0L+MiUP/MiZGTZ5Xr/WmOWLHrjGsb4dYVVQSsxetjE4tG8e+AzrF30/bLbq2blLn+to3z40DBnaOJIkoWF1cbbkBnVvEHlu2i8UrV1db5vThvePwHbrGzx54L657YXKctEfPOGqnbpntVx66Tbwz42shOgAAAADEJjgi/bXXXoszzjgjBg0aFEVFRXHBBRfEPvvsEx9//HE0a9ZsYzeP9WD+0sI49C9jIiLihqMGxuE7dI2+nfKiVdOGsXhFaRh8w5EDY4ceraN9Xm40apAd85YWxEsffxU3vDgllhUWxfVHbhtH7PhNEDzz6pEREXHTy1PippenZn7+5WP/jr9PmF1h9PoPb387fjty6+jdoXlMm7csLvzHpJj4xeJMXcft2iNOH947WjTJiRc/+io+mL04Ljqwf0RE7HHNq9WOnH/xoy/jt09Oiunzl0VExIm7bx4XH9g/muXmxPf7d4o735wREZFp++xFK2KPa0ZV+zxNn78strvsxVi8YnW8+esR0bV10yrLPTHxP/HAO7OifV5uvPnrPass069zi4iIGDfj6/hySUFERGy9Wel9h27fJXbo0Tr2ufG1atsCAAAAAN8lm1yQ/vzzz1f4+Z577okOHTrEhAkTYujQoRupVWxo+StXx7KCoszPe/frGEUlScxauDya5eZEj7bN4sTde0aHvMZxxoPvxecLV8TnC5dHj7alJ1vKpnKZ+9+QOM09Jw6K2YtWRoPsrBjQpWX8+Ufbx7DrRkdxSRJ7bd0hLjt4QERELFxWGIN6tom9+3Ws1WO4662ZFX5+d8bXmf8XFpXUqo7yCotKarVf2cmHNB/PyY89t+oQg3q2ie7/nSbmk7n50bppw/jdyK3j+hcmx5xaPHcAAAAA8F2wyQXpa1qyZElERLRp06bK7YWFhVFYWJj5OT8/f4O0i/rTPi83njx9t2jZpGH0at88Fi1fFb954oMoKpuwOyKOvn1shXnFz92nT5y155axT/+OkZuTHX9+dVrMXVKQGWVeNsK9Nq587tO4d8zMOGG3zeOSg/pH19ZNY/O2TWP6/OVx6tAtIiJi1sIVsd8fX4+CopJ44KeDK8ztXlvH7tIjIiIWLV8Vz0+am7l/Xn5hTJ+3LL7M33DB9V9GT4vNWjWO247dMYqKS+KuN2fEo+O/iBuO2i4+X7giXp8yPx746eDYpkvL+GzB8rjon5Pig9lLNlj7AAAAAGBTskkH6SUlJfGLX/widt999xgwYECVZa666qq49NJLN3DLqE+5OQ1i++6tMz9Pnbcsxs9cVKHM7r3bxU1Hbx892jaNxg0bZO5v2CA72jRrVKuR59V58r3Z/z3uN0F9u+a5MX3+8ujTsXlERIyeMi+Wryqdl/y5SV/WKUhvkJ0Vvz94QPxw5+6xrLAoTr1/Qiwot+jptS9Mjmtf2LDrAKxYVRznPvrvCvcN69M+9t+mUxz45zfjxqMGxubtmsXPHpgQVxyyTdx67I4x/LpRsbo4qaZGAAAAAPjftcktNlreGWecEZMmTYqHH3642jLnn39+LFmyJHP74osvNmALqQ+zF62Inuc/G8fd+U4Uri6OnXu2iasP3yaz/eDtOsfvRvaLvp3yYsnK1fH+rEXx+cLlme0NsrPW6fj5/51CprjcCPisdasyo1mjBnHn8TvFjwZ3j/lLC+KHt79dYYqXTUWThg3iikMHxG2vfRb/WbQytuveOsZ+tjDemrYwXvrkq+jSqkn0atd8YzcTAAAAADaKTTZIP/PMM+OZZ56JUaNGRdeuXastl5ubGy1atKhw49snSSJen7og/vb25xERsXe/TrFt15YREZnR6ksLVseQa0bFIX8ZE29MXVCpjpX/HTEeURoM14fJX5WOUt+jd7to0rBBZGdF7Nu/U6327dgiNx47bdcY3rdDTPlqaRxyy5j48D+Vp0c57/t945VzhsUDPx1cL21eG7/8fp9Yuao4bnl1WmT99yzC6v/Ox15UXPf53AEAAADgf8kmN7VLkiRx1llnxZNPPhmjR4+Onj17buwmsQHd8fpncdyuPSI3p0GcMaJ3nHrfhPh0bum893mNG8br542IVUUlkde4ctedPn9Z5v8vnTM05i8tjMuf/SQmfL6oUtnauv31z2Jwz7bRq33z0mMXl0Trpg1rte91RwyMfp1LTwZkRcTNP9o+s+3VT+fFn1+dFhERHVrkxhYdmkduw/TzWh1b5MYjp+z63/83joiIEVt1iNG/HB4REcOvHx0REd/v3ynO32+ryGnwzbD6s/fuEycP6RXvf7E4fvHI+xXq3bZryzhu183jB7e/HauKS2JVcUm8/8Xi2KFH62jXvFHstkW7mLN4ZXy2YFkAAAAAwHfRJjci/Ywzzoj7778/HnzwwcjLy4svv/wyvvzyy1i5cuXGbhobwLylhfHkxP9ERMTeW3eMLTs0j0fGfxF3vPFZLFxWGM1yG8TbMxbGjS9NqbTvp18ujT++MjXmLy2Irq2bxvbdW0fLJrULvavzyifz4qJ/Toqv8guiWW6DmPj5ovjL6OmZ7QWri6vdt1HONy+vLTvmxfbdW2duPdo2rXNbGjbIjs3bNYvN2zWLhg1K626em5O5r0xe49L7urb+5hjtmufG5u2aRaeWjSvU2SA7K645fNt4+N0vKpxwOOeR9+M/i1fGa78aEQ2ys+KMB94zPzoAAAAA31lZSZJsUulYVjWTU999991xwgkn1Lh/fn5+tGzZMrr94tHIzq17WAnl5WRnRaeWjWP2otITOdlZEXefMCiG9e0QX+UXxOArX9nILQQ2VTOvHrmxmwAAAACkKMuSlyxZUuOU4Zvk1C6wqWjaqEGM/uXw+PA/S2L+0sLo2ykverQtHf19w4uVR8UDAAAAAP97NrkgHTYlhUUlMWry/BjYtWX079wyClYXx9jpC+PONz+Llz+Zt7GbBwAAAABsAIJ0SFFYVBIn/238xm4GAAAAALARbXKLjQIAAAAAwKZEkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQImdjN2B9mXTp96NFixYbuxkAAAAAAHzLGZEOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkEKQDgAAAAAAKQTpAAAAAACQQpAOAAAAAAApBOkAAAAAAJBCkA4AAAAAACkE6QAAAAAAkCJnYzdgfRlw8QuRndt0YzcDAAAAWAczrx65sZsAAEakAwAAAABAGkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApFinID07OzsaNGiQesvJyYk2bdrErrvuGtdee22sXLmyvtoOAAAAAADr3ToF6UOHDo1tt902kiSJ7Ozs2HzzzWPw4MGx+eabR3Z2diRJEttss0107do1Pvjggzj//PNjl112ifz8/PpqPwAAAAAArFfrFKTff//9sWjRojjuuONixowZMX369BgzZkxMnz49Zs6cGccff3wsXrw4nnvuufjqq6/i5JNPjg8//DCuvPLK+mo/AAAAAACsV1lJkiRru/MPfvCDmDVrVowZM6baMrvvvnt07949HnrooSgpKYltt902Vq9eHZMnT17bw6bKz8+Pli1bRrdfPBrZuU3XyzEAAACADWPm1SM3dhMA+B9VliUvWbIkWrRokVp2nUakv/zyyzFixIjUMsOGDYuXX3659GDZ2TFkyJCYNWvWuhwWAAAAAAA2mHUK0gsKCmLu3LmpZebOnVthgdG8vLzIyclZl8MCAAAAAMAGs05B+g477BAPP/xwjB07tsrt77zzTjzyyCOx4447Zu777LPPomPHjutyWAAAAAAA2GDWaWj473//+9h7771jyJAhcdBBB8Xuu+8eHTp0iHnz5sVbb70VTz/9dGRnZ8dll10WERHLli2LF154IY466qh6aTwAAAAAAKxv6xSkDxs2LJ555pk45ZRT4h//+Ef84x//iKysrChbv7R79+5x2223xbBhwyKidI70N998M7p06bLuLQcAAAAAgA1gnScr32effeKzzz6LN998M/79739Hfn5+tGjRIgYOHBh77LFHZGd/M3tM06ZNY+DAget6SAAAAAAA2GDqZdXP7OzsGDp0aAwdOrQ+qgMAAAAAgE3GOi02CgAAAAAA/+vWeUT6/Pnz4+67745x48bF4sWLo7i4uFKZrKyseOWVV9b1UAAAAAAAsMGtU5D+wQcfxJ577hmLFi3KLDBalaysrHU5DAAAAAAAbDTrNLXLueeeG19//XX89re/jRkzZsTq1aujpKSk0q2qUeoAAAAAAPBtsE4j0seOHRuHHHJIXHbZZfXVHgAAAAAA2KSs04j0Ro0axRZbbFFfbQEAAAAAgE3OOgXpw4YNi/Hjx9dXWwAAAAAAYJOzTkH69ddfH5MmTYrrr7++vtoDAAAAAACblHWaI/2KK66IAQMGxK9//eu47bbbYrvttosWLVpUKpeVlRV33nnnuhwKAAAAAAA2iqwkSZK13Tk7u3YD2rOysqK4uHhtD1Mn+fn50bJly+j2i0cjO7fpBjkm8N0z8+qRERHxy8f+HX+fMHsjtwYAAP53lX32BoD6VpYlL1mypMoB4uWt04j0GTNmrMvuAPXq4VN2iV16tc38XFRcEotWrIp3ZyyKq577JGYvWlmn+o7YsWtcf+TAiIjY/DfPVtg2cdaiiIhYuGzVOra69g4a2DlO2qNn9GjTNJrl5sTiFaviozn5cetr0+PdGV/Xub4jduwaB267WfTplBdtmjaKRStWx/tfLI4/vjIlPpm7tELZrq2bxC++t2UM3bJ9tGraKJasXB0f/mdJ/PyhibG0sCh6tWsWVxy6TWzbtWUsXLYq/vTq1AonGEZus1lcf+TA+P5Nr8esr1es83MBAAAAsCGtU5Deo0eP+moHQL0pLCqOj+fkR8smDaNX++YxctvNoneH5vH9m16vt2Mc+pcx9VZXbW3XrVV0bd0kvswviIiI3h2ax4itOsSuW7SN79342lqdKNilV9v4fOHymL1oZWzRoXns27JTDNmyXXz/ptcz9fVs1ywe/9lu0aZZo1ixqiimz18WDRtkxx6920Wz3JxYWlgU1x6xbfRq3zyGXz86frPvVnHN4dvGxFmLYvr85dGiSU5cclC/uOnlKUJ0AAAA4FtpnYL09eHWW2+NW2+9NWbOnBkREf3794+LLroo9ttvv43bMOBbY/7SwkzQfcNRA+PwHbpG30550appw1i8YnXp/UcOjB16tI72ebnRqEF2zFtaEC99/FXc8OKUWFZYFNcfuW0csWO3TJ1ll5Pe9PKUuOnlqZWmdik/ev2Ht78dvx25dfTu0DymzVsWF/5jUkz8YnGmruN27RGnD+8dLZrkxIsffRUfzF4cFx3YPyIi9rjm1WoD8Wue/zQue+bjzM9H7dQtrj1i22jcsEEM6NIys1/ZyPy3P1sYP7j97Wqfpxc/+jJ+++SkmD5/WUREnLj75nHxgf2jWW5OfL9/p7jzzdKrji45sF+0adYoxkxfEKfdNyHyC4oiIiI3JzuKSkpnB+vXuUV8PCc/5i8tjPGffx2H79g1+nTMi+nzl8dv9+8XX+UXxv+96SomAAAA4NupTkH666+Xjubceeedo3Hjxpmfa2Po0KG1Kte1a9e4+uqrY8stt4wkSeLee++Ngw8+OCZOnBj9+/evS3MBMvJXro5l/w2AIyL27tcxikqSmLVweTTLzYkebZvFibv3jA55jeOMB9+LzxeuiM8XLo8ebZtFxDdTucxdUlDjse45cVDMXrQyGmRnxYAuLePPP9o+hl03OopLkthr6w5x2cEDIiJi4bLCGNSzTezdr2OtHkNhUUls361VXHRgv2jcsEFs0b55REQUrC6OD2cvqdPzERFx11szK/xcfnqYwqKSiIho0SQnhmzZPiIilqxcHU+duUe0y8uNqV8tjRtenBJvTlsQEREfz8mPXu2bR/u83NipR5soLkliyldLY9debeOwHbrEIbe8FcUla70kBwAAAMBGVacgffjw4ZGVlRWffPJJ9OnTJ/NzbdR2sdEDDzywws9XXHFF3HrrrfH2229XGaQXFhZGYWFh5uf8/PxaHQf439U+LzeePH23zNQui5avit888UFm9HRExNG3j60wD/i5+/SJs/bcMvbp3zFyc7Ljz69Oi7lLCjKjzOsylcuVz30a946ZGSfstnlcclD/6Nq6aWzetmlMn788Th26RUREzFq4Ivb74+tRUFQSD/x0cIW53dO0aNIwtu/eOvPzgmWFcfoD78V/Fn8ziv0/i1fG9HnLKtxXG8fuUjpd16Llq+L5SXMjIqJn22aRnV36Pr/fgM1i1sIVUbi6OLbv3jruOXFQHHHb2Hj/i8Vx3t8/iCsP2yZe+9XwWLB0Vfzm8Q9i9qKV8X/HD4o735wR7ZrnxvO/GBKdWjSOd2Z8Hb998sNYsAHnlwcAAABYF3UK0i+66KLIysqKdu3aVfh5fSkuLo7HHnssli9fHrvuumuVZa666qq49NJL11sbgG+f3JwGFcLmqfOWxfiZiyqU2b13u7jp6O2jR9um0bhhg8z9DRtkR5tmjWo18rw6T743+7/H/Saob9c8N6bPXx59OpaOIh89ZV4sX1V6gvG5SV/WOkh/bcr82Pw3z0a75o3i9OG946Q9esZNR28XR9w6Jub8t83nPvrvOrW3QXZW/P7gAfHDnbvHssKiOPX+CZmQu0F2dqbcG1Pnx4/vfDfycnPi9fNGROtmjeLYXbrH+18sjs8WLK80jcyv9+0bWRFx39jP44Wzh8bkL5fGuY/9O247dse4+MD+cdZDE+vUTgAAAICNpU5B+iWXXJL6c3358MMPY9ddd42CgoJo3rx5PPnkk9GvX78qy55//vlxzjnnZH7Oz8+Pbt26VVkW+G6YvWhFDLl2VAzp3S7uOG6n2Llnm7j68G3i5L9NiIiIg7frHL8bWfqe8lV+QXw6Nz9aN2uUmcalQfa6nSAsm0O8/FQm9X3OccGyVXHjS1PipD16RudWTeKYXXrEdS9MrnM9zRo1iFuO2SGG9+0Q85cWxEn3jI8P//PNNDFf5X9zQqFs+pilhUUxY8HyaN2sUXRt3bTKevtt1iJ+skevOOHud2OrzfKieW5OPPXvOfHKJ/Pi07lLY4/e7ercVgAAAICNJbvmItWbNWtWjVOpLF26NGbNmlWnevv27Rvvv/9+vPPOO/Gzn/0sjj/++Pj444+rLJubmxstWrSocANIkojXpy6Iv739eURE7N2vU2zbtWVERGa0+tKC1THkmlFxyF/GxBtTF1SqY+Wqb6akalJu1Pq6mPxV6Sj1PXq3iyYNG0R2VsS+/TvVat8f79KjQjv23KpD5v9NG31z/w1HDYxXzhkWNxw1MLW+ji1y47HTdo3hfTvElK+WxiG3jKkQokeUThPz2X8XIx3QpfT5a56bEz3blZ50mLFgeaV6s7Mirjpsm/jH+/+JMdMXRtk5hNX/nXe9qKSkVo8XAAAAYFNRpxHpa+rZs2dccsklceGFF1Zb5k9/+lNcdNFFtZ4jPSKiUaNG0bt374iI2HHHHWPcuHHxxz/+Mf7617+uS3OB76A7Xv8sjtu1R+TmNIgzRvSOU++bEJ/OLT0BmNe4Ybx+3ohYVVQSeY0rvx1O/2+AHBHx0jlDY/7Swrj82U9iwueLKpWtrdtf/ywG92wbvdo3Lz12cUm0btqwVvv+/pAB8bsDto5ZC1dEToPsTJi9urgk/vn+nEy5Lq2axBYdmsf8ZYXVVRUREdcdMTD6dS4Nx7Mi4uYfbZ/Z9uqn8+LPr06LiIhrnp8ctx6zQwzt0z5e+9XwaJabE62bNYrlhUXxf2/MqFRv6Sj5xvHju96JiIiJsxbH8sKiGNKnXbw+dX5s1alFvPLpV7V6zAAAAACbgnUakZ4kSSRJUmOZdVVSUlJhQVGA2pq3tDCenPifiIjYe+uOsWWH5vHI+C/ijjc+i4XLCqNZboN4e8bCuPGlKZX2/fTLpfHHV6bG/KUF0bV109i+e+to2aR2oXd1XvlkXlz0z0nxVX5BNMttEBM/XxR/GT09s71gdfUnHR8b/0XMXrQyOrdqEl1bN4l5+QXx/KQv46j/LvhZV41yvvkTsGXHvNi+e+vMrUfbb6ZseeGjL+OU+8bH+18sjo4tGkeSJPHCR1/GQTe/WeFkQ0RE19ZN4py9+8QlT30c+StLp7hZuHxVnPngxOjTIS+e/8XQGDN9QVz6VNVXGQEAAABsirKSdUi6s7Oz45JLLomLLrqo2jKnnXZaPPzww7F48eJa1Xn++efHfvvtF927d4+lS5fGgw8+GNdcc0288MILsffee9e4f35+frRs2TK6/eLRyM6teu5egI0lJzsrOrVsHLMXrYyI0mlQ7j5hUAzr2yG+yi+IwVe+spFbCAAAm5aZV4/c2E0A4H9UWZa8ZMmSGqcMr/PULpdddlmFn0ePHl1lueLi4vjiiy/i4Ycfjl122aXW9c+bNy+OO+64mDt3brRs2TK23XbbWofoAJu6po0axOhfDo8P/7Mk5i8tjL6d8jKLnN7wYuVR8QAAAABsfHUekZ6d/c1UAFlZWTVO3dK5c+d48sknY9CgQWvXwjoyIh3YlOXmZMfNP9ohBnZtGa2aNoqC1cXx0Zz8uPPNz+LlT+Zt7OYBAMAmx4h0ANaX9ToifdSoURFROvf5nnvuGSeccEIcf/zxlco1aNAg2rRpE1tttVWF8B3gu6ywqCRO/tv4jd0MAAAAAOqgzkH6sGHDMv+/+OKLY8SIETF06NB6bRQAAAAAAGwq6hykl3fxxRfXVzsAAAAAAGCTtE5BenlffPFFzJkzJwoLC6vcbtQ6AAAAAADfRuscpD/99NPxq1/9KqZOnZparri4eF0PBQAAAAAAG9w6rQI6evToOPTQQ2PZsmVx5plnRpIkMXTo0DjllFOiX79+kSRJjBw5Mi666KL6ai8AAAAAAGxQ6xSkX3311dG8efOYMGFC/PGPf4yIiBEjRsStt94aH374YVxxxRXxyiuvxMEHH1wvjQUAAAAAgA1tnYL0cePGxSGHHBIdO3bM3FdSUpL5//nnnx/bb7+9EekAAAAAAHxrrVOQvmLFiujSpUvm59zc3MjPz69QZpdddom33nprXQ4DAAAAAAAbzToF6Z06dYr58+dnfu7SpUt89NFHFcosXLjQQqMAAAAAAHxrrVOQPnDgwJg0aVLm5xEjRsSoUaPioYceiuXLl8cLL7wQjz76aGy77bbr3FAAAAAAANgY1ilIP+igg+L999+Pzz//PCIiLrjggmjevHkce+yx0aJFi9h///2jqKgoLr/88nppLAAAAAAAbGhZSZIk9Vnh9OnT48Ybb4zPPvssevToEaeddlpst9129XmIVPn5+dGyZcvo9otHIzu36QY7LgAAAFD/Zl49cmM3AYD/UWVZ8pIlS6JFixapZXPq++BbbLFF3HLLLZmfZ8yYESeccELcc8899X0oAAAAAABY79Zpapc0s2bNipNPPjm22mqruO+++9bXYQAAAAAAYL1aqyD9zTffjBEjRkSLFi2iTZs2cfDBB8fkyZMjImLFihVxzjnnRJ8+feLOO++M9u3bx5/+9Kd6bTQAAAAAAGwodZ7aZcKECfG9730vVq1albnv6aefjvHjx8cbb7wRBx10UHz88cfRuXPn+PWvfx2nnHJK5Obm1mujAQAAAABgQ6nziPRrr702Vq1aFVdddVXMmzcv5s2bF1dccUXMnTs3hgwZEp9++mn87ne/i2nTpsVZZ50lRAcAAAAA4FutziPS33rrrdhzzz3j17/+dea+888/P15++eUYPXp0XHfddXHOOefUayMBAAAAAGBjqfOI9Hnz5sWOO+5Y6f6y+44//vh1bxUAAAAAAGwi6hykFxUVRbNmzSrdX3Zf27Zt171VAAAAAACwiahzkA4AAAAAAN8ldZ4jPSLi/vvvj7fffrvCfdOmTYuIiP33379S+aysrHj22WfX5lAAAAAAALBRrVWQPm3atExwvqbnn3++0n1ZWVlrcxgAAAAAANjo6hykz5gxY320AwAAAAAANkl1DtJ79OixPtoBAAAAAACbJIuNAgAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQIqcjd2A9WXSpd+PFi1abOxmAAAAAADwLWdEOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAAAAAKQQpAMAAAAAQApBOgAAAAAApBCkAwAAAABACkE6AAAAAACkEKQDAAAAAEAKQToAAPz/9u49zqq60P//e5gZhvtF7qggF8UUEa+ImpdE0XhUdjFTH6aejqXZ92tZllYn9WRh1jH7VXaq7zlRfT2Z9s3sHM27SCqaN1QUURAEjZsjMFwHhlm/P4idE7jE64zwfD4e+/GYvddnr/0Z3B/XPF6zZm0AAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlKhp7Qm8XUZedEva1XVq7WkAAAAAALQZcy+b0NpTeFdyRjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdgDZj7mUTMveyCfnYfju19lQAAAAAKmpaewIAtG3XfPqgHDS0V+V+04bmLF29Ln+ZszQT/zQjLyxd87r297H9dsr3Ttg7SbLLBTe22PbovKVJkvqV697krN+YC4/bPZ85fFhlLh++6r7XvY9DhvfKuUftlmF9Oqdrh9o0rF2fWYtX5j/umZPbnlpUGfe9E0blY/vtvNnzFyxfk7ET70yS9OlSl29/ZGQOGtorKxub8uupz+eqybMrY/fZuUeuPWtsPvGz+/Pw80tf91wBAACArSOkA7BVGps25Km/NqR7x9oM7dMlE0YNyPC+XTL+yilv2Wu8kXD9Vhk7rFfOfO/QN72fEf26ZkT/rlmwfE0WNqzNsD5dctDQXjlglx1ywr9PzSPzWgbvBcvXZOHytZX7L73ilwhfm/CeHDGib469cko+NHrHfPnY3fP4C8tzz6yXUtOuKhM/uleu+ct8ER0AAADeZm06pF922WW58MILc+655+bKK69s7ekAbNeWrGishO5/+/je+ei+O2VE/67p0ak2y1av3/j4CXtn38E906drXdpXt8viFWtz21OL8m+3PpOVjU2bnYU997IJSZIrb38mV97+bOX+l657LL97+IUWZ6+f9LP787UJ78nwvl0ya/HK/MsfpufR+csq+/rk2MH57BHD061jTW59clEef2FZvvGBPZMkh37nztIz57t3rM0VH987815enRVrm7LXTt03G7Np7i8sXZ1Dv3PXq+7r/94/L/9579zK/YOH9cp/nXlQqttVZd/BPTYL6b99cH6uvP3ZLe5rj4HdUr9yXWYvWZUH576cJHnPgG65Z9ZLOfuIYenesTbfufnpV50LAAAA8NZosyH9wQcfzE9/+tOMGjWqtacCwKtoWLM+K9c2Ve4fvUe/NDUXmVe/Kp3rajK4V+eccciQ9O3aIef81yN5vn51nq9flcG9Oif5+6VcFrzijOxXM+mMA/LC0jWpbleVkTt2zw9P3ieHf3dyNjQXOeo9ffOvHxqZJKlf2ZgDhuyQo/fot9Xfx8SP7JXeXepywr9PzQXH7f56/gk2s25Dc3bs0TE/OnmftK9pl6G9uyRJNjQXeWQLZ47/0yFDcvYRw1K/cl0efn5pvnvLzMx7eXWS5Km/NmTCqAEZ2rtzDthlhyTJjAUNGdancz535PCc81+PZGVj02b7BAAAAN5abTKkr1y5Mqecckp+/vOf59JLLy0d29jYmMbGxsr9hoaGt3t6ANulPl3rcv1nD65c2mXpqnW54PePp6m5qIw58WdTM2PBisr9Lx6zW/7X+3bNMXv2S11Nu/zwzllZsHxt5Szz13Mpl2//6en88r65Of3gXXLxB/fMTj07ZZdenTJ7yap85rCN1zWfV786x/1gStY2Nefqfx7T4trur+bj+++c9+81IN+9ZWamveIM93+0uKExsxevzMKG147+HWrbZZ9BPSv3VzU25fzfPZZH5rXcf2PThixe0Zja6qoM7tU5A3t0zHt37Z3xV07JoobGfOvGGelcV5MbPndIVjVuyOU3P517Zr2U337moNw2Y1EWr2jMH845JEN7d84TLy7PV69/Is/Xr37N+QEAAACvT5sM6eecc04mTJiQcePGvWZInzhxYi655JJ3aGYA26+6muoWcfjZxSvz0NyWZ1gfMrx3rjxxnwzu1Skdaqsrj9dWt8sOndtv1Znnr+b6R1742+v+PdT37lKX2UtWZbd+G8/6nvzM4qxatyFJ8qfpC18zpA/o3iHf+MAeeeC5+lw1eVbp2MtvmZnLb5m5VXOdvWRVdrngxnTvWJuTDtw5Fxz3nkz8yKg8X786T/514y98fzbluXzjhiez+m/zPfnAQfn2R/ZKj07tc8L+O+dHd87KkpWNOfNXD7XY9yljBmVEv675wjXTct3ZB2d9U3POvvrh/PjkfXPliaNb9TrzAAAAsK1q19oT+EfXXHNNHnnkkUycOHGrxl944YVZvnx55TZ//vy3eYYA26cXlq7OkAtvzCf/44E0rt+QA4fskMs+uldl+4dGD8zXJ+yREf27Zvma9Zk2b2mer19V2V7drupNvX7D3y4hs+EVZ8BXvbldZnCvTulSV5PRO/fI9IvH58lLxlcuoTJqpx558pLxGdGv6xve//I16/Pvdz+XZavXpXvH2nz6sL9/mOkzi1ZWInqS/GHai5WvB3bvuMX99e1al68cu3u+ddOMdO1Qmx17dMxtMxbl3ln1mfpcffYZ1DOd21dv8bkAAADAG9emQvr8+fNz7rnn5uqrr06HDh226jl1dXXp1q1bixsAb4+iSKY8+1J+df/zSZKj9+ifUX/7YM5NZ6uvWLs+7/3OXTn+qvvy52df2mwfa14RjzvWvjXRd+aijWepHzq8dzrWVqddVXLsnv23+vl1tdXpXFeTznU1leBf3a6qxf0vjx+RO847PFf/85jSfZ14wM7p3rG2cn/fQT3TrcPG+51eEbm/MG7X7NC5feX+B/YeWPn6haVbvjzLN48fmSdeXJ7rHnqh8kuE9U3NSZKmDcUWnwMAAAC8eW3q0i4PP/xwFi9enH333bfy2IYNGzJlypT86Ec/SmNjY6qrnWkH0Np+PuW5fHLs4NTVVOecI4fnM79+OE8v2HjJkq4dajPly0dmXVNzunbY/DAze8nKyte3nXdYlqxozKU3zsjDW/ggzq31synPZcyQXhnap8vG197QnJ6dal/zefc/93J2ueDGFo9d8+mDctDQXnl03tIWl0np260uw/p2SV1t+e+g/9f7hudbx4/M/KVr0rShOcP6dEm7v8X4//fI3886P3fcbvnc+3bNvJdXpyrJLr03fgDr4oa1+e2Dm/911bEj++ewXftk/JVTkmz8d/zrsjU5eFiv9O7SPvsN7plp85dVLm0DAAAAvHXa1BnpRx11VJ544olMmzatctt///1zyimnZNq0aSI6QBuxeEVjrn90YxQ++j39smvfLvntQ/Pz8z8/l/qVjelcV53759Tnitue2ey5Ty9ckR/c8WyWrFibnXp2yj6DerY4g/uNuGPG4nzjhulZ1LA2neuq8+jzS3PV5NmV7WvXv3Nx+b8f+2tmLVmZXl3aZ0jvzlm6el3unrk4p//nX3Lz9IWVcZff/HQefn5putTVpH/3Dpnz0qr83/ufzwd+dE/qV61rsc+udTW55IN75srbn8m8lzeerb5+Q5Fzrn4kNdXtcvf5R2b+0tX5wm+nvWPfJwAAAGxPqoqiaNN/C37EEUdk9OjRufLKK7dqfENDQ7p3756dP39t2tV1ensnB0CbUNOuKv27d8gLS9ckSdpVJb84/YAcPqJvFjWszZhv39HKMwQAAIC2Ye5lE1p7Cm3Gppa8fPny17xkeJu6tAsAvBGd2ldn8peOyBMvLs+SFY0Z0b9rBvfaeKmUf7t187PiAQAAAF6PNh/SJ0+e3NpTAKCNa2xqzl0zl2Tvnbpnz4Hds3b9hkydXZ//uOe53D5jcWtPDwAAAHiXa/MhHQBeS2NTc8781UOtPQ0AAABgG9WmPmwUAAAAAADaGiEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASNa09gbfL9EvGp1u3bq09DQAAAAAA3uWckQ4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoERNa0/g7TLyolvSrq5Ta08DAACA1zD3sgmtPQUAgFLOSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR14V5l72YTMvWxCPrbfTq09FQAAAAC2EzWtPQHg3e+aTx+Ug4b2qtxv2tCcpavX5S9zlmbin2bkhaVrXtf+PrbfTvneCXsnSXa54MYW2x6dtzRJUr9y3Zuc9Rubzz86/Lt35fn61a9rf0e9p29O3H/n7DGwW3p3qcuKtU2ZsaAhP75rVh6Y83Jl3D8dsktO2H/n7NijY+pq2+XlVevyyPPL8sM7n83TC1ckSfp0qcu3PzIyBw3tlZWNTfn11Odz1eTZlX3ss3OPXHvW2HziZ/fn4eeXvoHvHgAAAIA2d0b6xRdfnKqqqha33XffvbWnBWyFxqYNeXTe0sx7eXX6dO2QCaMG5D9OO+AtfY0PX3VfPnzVfblr5uK3dL9bY8Xa9Xl03tIWt8b1za97P8eN7J9j9uyfdlVVmVu/Kr06t89hu/XJ1f88JvsO6lEZN2Zor+zQuX3mvbw68+pXp+/f/k1/c+ZB6VhbnST52oT35IgRfXP8j+/NdQ+9kC8fu3sOHd47SVLTrioTP7pXrvnLfBEdAAAA4E1ok2ek77nnnrn99tsr92tq2uQ0gX+wZEVjPnzVfUmSf/v43vnovjtlRP+u6dGpNstWr9/4+Al7Z9/BPdOna13aV7fL4hVrc9tTi/Jvtz6TlY1N+d4Jo/Kx/Xau7HPuZROSJFfe/kyuvP3Zyv0vXfdYfvfwCy3OFj/pZ/fnaxPek+F9u2TW4pX5lz9Mz6Pzl1X29cmxg/PZI4anW8ea3Prkojz+wrJ84wN7JkkO/c6dr3nm/JN/bcgnfnb/q27fNPcXlq7Ood+561XH/WXO0vx66vN57IXlSZKj9+iXn39y/9RUt8sH9h6YR+ZtnPP//s2jaWz6e6g/7+jd8r+P2jU9O7fPsL6dM/3FhuwxsFvqV67L7CWr8uDcjWezv2dAt9wz66WcfcSwdO9Ym+/c/HTp9wUAAABAuTZZqGtqatK/f//WngbwFmhYsz4r1zZV7h+9R780NReZV78qnetqMrhX55xxyJD07doh5/zXI3m+fnWer1+Vwb06J/n7pVwWLF/7mq816YwD8sLSNaluV5WRO3bPD0/eJ4d/d3I2NBc56j19868fGpkkqV/ZmAOG7JCj9+j3ur6XvXfqkScvGZ/GpubMXLgi/98dz2bqc/Wvax9Jcu1D81vc/8srLufyynDe2NSc8Xv2y1mHD0uXupoM7dMlSfLSysbMWbIqSfLUXxsyYdSADO3dOQfsskOSZMaChgzr0zmfO3J4zvmvR7KysSkAAAAAvHFtMqQ/++yzGThwYDp06JCxY8dm4sSJGTRo0BbHNjY2prGxsXK/oaHhnZom8A/6dK3L9Z89ON071mZony5ZumpdLvj942lqLipjTvzZ1MxYsKJy/4vH7Jb/9b5dc8ye/VJX0y4/vHNWFixfWznLfNMZ7lvj2396Or+8b25OP3iXXPzBPbNTz07ZpVenzF6yKp85bFiSZF796hz3gylZ29Scq/95TItru5dpbi7y0srGrF63IcP6dM7YYb0yZsgO+dQvH6pcZmZxQ2NmL16ZhQ2vHf1f6dSxg5Mkjes35PePvNBiW+8uddlnUM/K/Xn1q/OpXz6YVes2JEm+deOMdK6ryQ2fOySrGjfk8pufzj2zXspvP3NQbpuxKItXNOYP5xySob0754kXl+er1z/xuq/pDgAAALC9a3MhfcyYMZk0aVJGjBiRBQsW5JJLLsl73/veTJ8+PV27dt1s/MSJE3PJJZe0wkyBf1RXU90i+j67eGUemtvy2tyHDO+dK0/cJ4N7dUqHv13nO0lqq9tlh87tt+rM81dz/d8i9LOL/x7qe3epy+wlq7Jbv41nc09+ZnElQv9p+sKtCun3zXopB028I4tXbPyl3R4DuuX/nX1wOravzqcOHVIJ6ZffMjOX3zLzdc35fx81POcdPSLrmppz3rWP5ZlFK1tsv/qBebn6gXkZ2L1DLnz/e/KBvQfmRyfvm49cdW9WrduQJSsbc+avHmrxnFPGDMqIfl3zhWum5bqzD876puacffXD+fHJ++bKE0e/rl9OAAAAANAGP2z0uOOOywknnJBRo0Zl/Pjxuemmm7Js2bJce+21Wxx/4YUXZvny5ZXb/PnztzgOePu9sHR1hlx4Yz75Hw+kcf2GHDhkh1z20b0q2z80emC+PmGPjOjfNcvXrM+0eUvzfP2qyvbqdlVv6vUb/nYJmQ2vOAO+6s3tMkny1+VrKxE9SZ5a0FCJ9QN7dHhD+6xpV5XvfmxUzjt6RFY2NuXTv3ooNz6xoHQOP75rVpJkRP+u+eDogVsc17drXb5y7O751k0z0rVDbXbs0TG3zViUe2fVZ+pz9dlnUM90bl+9xecCAAAAsGVtLqT/ox49emS33XbLrFmztri9rq4u3bp1a3EDWk9RJFOefSm/uv/5JMnRe/TPqJ26J0nlbPUVa9fnvd+5K8dfdV/+/OxLm+1jzd/OGE+SjrVvTfSduWhj+D50eO90rK1Ou6rk2D237rMYTj1ocIb37VK5v3v/rtm178a/kHnlB5R+efyI3HHe4bn6n8eU7q9rXU0mnXFgTth/5yxYviYf//epmfzMkhZjenSqzYf32TG11X//TcCRI/pWvu7Ufst/UPTN40fmiReX57qHXqj8EmH936673rSh2OJzAAAAACjX5i7t8o9WrlyZ2bNn59RTT23tqQCvw8+nPJdPjh2cuprqnHPk8Hzm1w/n6QUbP8Oga4faTPnykVnX1JyuHTb/39DsJX+/vMlt5x2WJSsac+mNM/Lw80s3G7u1fjbluYwZ0itD+3TZ+NobmtOzU+1WPXfCqAH55vEjs6hhbZauXpdhfbqktrpd1m9ozk/unl0Z17dbXYb17ZK62vLfUV74/t1z6K69kyTrmprzrQ+PrGyb/mJD/uWG6elSV5Pvnzg63/7wXnn+5VWVs8uTjb+IuHn6ws32e+zI/jls1z4Zf+WUJBv/Hf+6bE0OHtYrvbu0z36De2ba/GWVS9sAAAAAsHXa3BnpX/rSl3L33Xdn7ty5ue+++/LhD3841dXVOemkk1p7asDrsHhFY65/9MUkydHv6Zdd+3bJbx+an5//+bnUr2xM57rq3D+nPlfc9sxmz3164Yr84I5ns2TF2uzUs1P2GdQz3TtuXfR+NXfMWJxv3DA9ixrWpnNddR59fmmumvz3CL52/avH5V/eNze3PbUoG5qLDOnVOS+tbMxtTy3MR39yX6bOrn/dc2lf8/f/9Q7u1Tn7DOpZue36t2u5N6xZnz9OezGLV6zN4B06p2/Xury4bE1+/8gLOf7H9+XFZWta7LNrXU0u+eCeufL2ZzLv5Y0fJrp+Q5Fzrn4kNdXtcvf5R2b+0tX5wm+nve75AgAAAGzvqoqiaFN/6/+JT3wiU6ZMSX19ffr06ZNDDz003/rWtzJs2LCten5DQ0O6d++enT9/bdrVdXqbZwu8W9S0q0r/7h0ql2JpV5X84vQDcviIvlnUsDZjvn1HK88QAGD7NfeyCa09BQBgO7SpJS9fvvw1Lxne5i7tcs0117T2FIBtUKf21Zn8pSPyxIvLs2RFY0b075rBvTonSf7t1s3PigcAAACATdpcSAd4OzQ2NeeumUuy907ds+fA7lm7fkOmzq7Pf9zzXG6fsbi1pwcAAABAGyakA9uFxqbmnPmrh1p7GgAAAAC8C7W5DxsFAAAAAIC2REgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUqGntCbxdpl8yPt26dWvtaQAAAAAA8C7njHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAlhHQAAAAAACghpAMAAAAAQAkhHQAAAAAASgjpAAAAAABQQkgHAAAAAIASQjoAAAAAAJQQ0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAoIaQDAAAAAEAJIR0AAAAAAEoI6QAAAAAAUEJIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABK1LT2BN5qRVEkSRoaGlp5JgAAAAAAtFWbGvKmplxmmwvp9fX1SZKdd965lWcCAAAAAEBbt2LFinTv3r10zDYX0nfYYYckybx5817zm4dtWUNDQ3beeefMnz8/3bp1a+3pQKuxFsA6gE2sBdjIWgDrADbZ3tdCURRZsWJFBg4c+Jpjt7mQ3q7dxsu+d+/efbv8jw//qFu3btYCxFqAxDqATawF2MhaAOsANtme18LWnoztw0YBAAAAAKCEkA4AAAAAACW2uZBeV1eXiy66KHV1da09FWhV1gJsZC2AdQCbWAuwkbUA1gFsYi1svaqiKIrWngQAAAAAALRV29wZ6QAAAAAA8FYS0gEAAAAAoISQDgAAAAAAJYR0AAAAAAAosc2F9B//+MfZZZdd0qFDh4wZMyZ/+ctfWntKsFUuvvjiVFVVtbjtvvvule1r167NOeeck169eqVLly756Ec/mkWLFrXYx7x58zJhwoR06tQpffv2zfnnn5+mpqYWYyZPnpx99903dXV1GT58eCZNmrTZXKwj3klTpkzJBz7wgQwcODBVVVX5wx/+0GJ7URT5xje+kQEDBqRjx44ZN25cnn322RZjXn755Zxyyinp1q1bevTokU996lNZuXJlizGPP/543vve96ZDhw7Zeeedc/nll282l+uuuy677757OnTokL322is33XTT654LvFGvtRZOP/30zY4Txx57bIsx1gLvdhMnTswBBxyQrl27pm/fvjn++OMzc+bMFmPa0s9EWzMXeL22Zh0cccQRmx0TzjrrrBZjrAPe7X7yk59k1KhR6datW7p165axY8fmT3/6U2W74wHbi9daC44J76BiG3LNNdcU7du3L/7zP/+zePLJJ4szzzyz6NGjR7Fo0aLWnhq8posuuqjYc889iwULFlRuS5YsqWw/66yzip133rm44447ioceeqg46KCDioMPPriyvampqRg5cmQxbty44tFHHy1uuummonfv3sWFF15YGfPcc88VnTp1Ks4777ziqaeeKn74wx8W1dXVxc0331wZYx3xTrvpppuKr33ta8Xvf//7Iklx/fXXt9h+2WWXFd27dy/+8Ic/FI899ljxwQ9+sBgyZEixZs2ayphjjz222HvvvYv777+/+POf/1wMHz68OOmkkyrbly9fXvTr16845ZRTiunTpxe/+c1vio4dOxY//elPK2Puvffeorq6urj88suLp556qvj6179e1NbWFk888cTrmgu8Ua+1Fk477bTi2GOPbXGcePnll1uMsRZ4txs/fnzxi1/8opg+fXoxbdq04v3vf38xaNCgYuXKlZUxbelnoteaC7wRW7MODj/88OLMM89scUxYvnx5Zbt1wLbgj3/8Y3HjjTcWzzzzTDFz5sziq1/9alFbW1tMnz69KArHA7Yfr7UWHBPeOdtUSD/wwAOLc845p3J/w4YNxcCBA4uJEye24qxg61x00UXF3nvvvcVty5YtK2pra4vrrruu8tiMGTOKJMXUqVOLotgYYNq1a1csXLiwMuYnP/lJ0a1bt6KxsbEoiqL48pe/XOy5554t9n3iiScW48ePr9y3jmhN/xgPm5ubi/79+xff/e53K48tW7asqKurK37zm98URVEUTz31VJGkePDBBytj/vSnPxVVVVXFiy++WBRFUVx11VVFz549K2uhKIriK1/5SjFixIjK/Y9//OPFhAkTWsxnzJgxxWc+85mtngu8VV4tpH/oQx961edYC2yLFi9eXCQp7r777qIo2tbPRFszF3gr/OM6KIqN0eTcc8991edYB2yrevbsWfyf//N/HA/Y7m1aC0XhmPBO2mYu7bJu3bo8/PDDGTduXOWxdu3aZdy4cZk6dWorzgy23rPPPpuBAwdm6NChOeWUUzJv3rwkycMPP5z169e3eH/vvvvuGTRoUOX9PXXq1Oy1117p169fZcz48ePT0NCQJ598sjLmlfvYNGbTPqwj2po5c+Zk4cKFLd6T3bt3z5gxY1q893v06JH999+/MmbcuHFp165dHnjggcqYww47LO3bt6+MGT9+fGbOnJmlS5dWxpStj62ZC7zdJk+enL59+2bEiBE5++yzU19fX9lmLbAtWr58eZJkhx12SNK2fibamrnAW+Ef18EmV199dXr37p2RI0fmwgsvzOrVqyvbrAO2NRs2bMg111yTVatWZezYsY4HbLf+cS1s4pjwzqhp7Qm8VV566aVs2LChxZsiSfr165enn366lWYFW2/MmDGZNGlSRowYkQULFuSSSy7Je9/73kyfPj0LFy5M+/bt06NHjxbP6devXxYuXJgkWbhw4Rbf/5u2lY1paGjImjVrsnTpUuuINmXTe3dL78lXvq/79u3bYntNTU122GGHFmOGDBmy2T42bevZs+erro9X7uO15gJvp2OPPTYf+chHMmTIkMyePTtf/epXc9xxx2Xq1Kmprq62FtjmNDc35/Of/3wOOeSQjBw5Mkna1M9EWzMXeLO2tA6S5OSTT87gwYMzcODAPP744/nKV76SmTNn5ve//30S64BtxxNPPJGxY8dm7dq16dKlS66//vrssccemTZtmuMB25VXWwuJY8I7aZsJ6fBud9xxx1W+HjVqVMaMGZPBgwfn2muvTceOHVtxZgC0BZ/4xCcqX++1114ZNWpUhg0blsmTJ+eoo45qxZnB2+Occ87J9OnTc88997T2VKDVvNo6+PSnP135eq+99sqAAQNy1FFHZfbs2Rk2bNg7PU1424wYMSLTpk3L8uXL87vf/S6nnXZa7r777taeFrzjXm0t7LHHHo4J76Bt5tIuvXv3TnV19WafBLto0aL079+/lWYFb1yPHj2y2267ZdasWenfv3/WrVuXZcuWtRjzyvd3//79t/j+37StbEy3bt3SsWNH64g2Z9P7ruw92b9//yxevLjF9qamprz88stvyfp45fbXmgu8k4YOHZrevXtn1qxZSawFti2f+9zn8j//8z+56667stNOO1Ueb0s/E23NXODNeLV1sCVjxoxJkhbHBOuAbUH79u0zfPjw7Lfffpk4cWL23nvv/OAHP3A8YLvzamthSxwT3j7bTEhv37599ttvv9xxxx2Vx5qbm3PHHXe0uGYQvFusXLkys2fPzoABA7Lffvultra2xft75syZmTdvXuX9PXbs2DzxxBMtIsptt92Wbt26Vf7cZ+zYsS32sWnMpn1YR7Q1Q4YMSf/+/Vu8JxsaGvLAAw+0eO8vW7YsDz/8cGXMnXfemebm5soPEGPHjs2UKVOyfv36ypjbbrstI0aMSM+ePStjytbH1swF3kkvvPBC6uvrM2DAgCTWAtuGoijyuc99Ltdff33uvPPOzS5F1JZ+JtqaucAb8VrrYEumTZuWJC2OCdYB26Lm5uY0NjY6HrDd27QWtsQx4W3U2p92+la65pprirq6umLSpEnFU089VXz6058uevTo0eJTaaGt+uIXv1hMnjy5mDNnTnHvvfcW48aNK3r37l0sXry4KIqiOOuss4pBgwYVd955Z/HQQw8VY8eOLcaOHVt5flNTUzFy5MjimGOOKaZNm1bcfPPNRZ8+fYoLL7ywMua5554rOnXqVJx//vnFjBkzih//+MdFdXV1cfPNN1fGWEe801asWFE8+uijxaOPPlokKa644ori0UcfLZ5//vmiKIrisssuK3r06FHccMMNxeOPP1586EMfKoYMGVKsWbOmso9jjz222GeffYoHHniguOeee4pdd921OOmkkyrbly1bVvTr16849dRTi+nTpxfXXHNN0alTp+KnP/1pZcy9995b1NTUFN/73veKGTNmFBdddFFRW1tbPPHEE5UxWzMXeKPK1sKKFSuKL33pS8XUqVOLOXPmFLfffnux7777Frvuumuxdu3ayj6sBd7tzj777KJ79+7F5MmTiwULFlRuq1evroxpSz8TvdZc4I14rXUwa9as4l//9V+Lhx56qJgzZ05xww03FEOHDi0OO+ywyj6sA7YFF1xwQXH33XcXc+bMKR5//PHiggsuKKqqqopbb721KArHA7YfZWvBMeGdtU2F9KIoih/+8IfFoEGDivbt2xcHHnhgcf/997f2lGCrnHjiicWAAQOK9u3bFzvuuGNx4oknFrNmzapsX7NmTfHZz3626NmzZ9GpU6fiwx/+cLFgwYIW+5g7d25x3HHHFR07dix69+5dfPGLXyzWr1/fYsxdd91VjB49umjfvn0xdOjQ4he/+MVmc7GOeCfdddddRZLNbqeddlpRFEXR3Nxc/Mu//EvRr1+/oq6urjjqqKOKmTNntthHfX19cdJJJxVdunQpunXrVpxxxhnFihUrWox57LHHikMPPbSoq6srdtxxx+Kyyy7bbC7XXnttsdtuuxXt27cv9txzz+LGG29ssX1r5gJvVNlaWL16dXHMMccUffr0KWpra4vBgwcXZ5555ma/5LQWeLfb0hpI0uLnlbb0M9HWzAVer9daB/PmzSsOO+ywYocddijq6uqK4cOHF+eff36xfPnyFvuxDni3+6d/+qdi8ODBRfv27Ys+ffoURx11VCWiF4XjAduPsrXgmPDOqiqKonjnzn8HAAAAAIB3l23mGukAAAAAAPB2ENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQCgDXj44YfzqU99Krvuums6d+6cjh07ZtiwYTn11FNz2223tfb0XtXcuXNTVVWV008/vbWnAgAAbxshHQAAWlFzc3POO++87L///vnVr36VoUOH5qyzzsq5556b/fbbLzfeeGOOOeaYfPOb32ztqQIAwHarprUnAAAA27Ovf/3r+f73v5/Ro0fnd7/7XYYNG9Zi+5o1a/KjH/0o9fX1rTRDAADAGekAANBKZs2alcsvvzy9evXKzTffvFlET5KOHTvm/PPPzyWXXFJ57KWXXsrnP//5DBkyJHV1denbt28+/vGPZ/r06Zs9/4gjjkhVVdUWX//0009PVVVV5s6dW3ls0qRJqaqqyqRJk3Lrrbfm4IMPTqdOndKrV6+cdtppLYL+pEmTMmTIkCTJL3/5y1RVVVVukydPfoP/KgAA0PY4Ix0AAFrJpEmTsmHDhnzmM59Jv379SsfW1dUlSZYsWZKxY8dm9uzZOeKII/KJT3wic+bMye9+97vceOONueWWW3LooYe+6bn98Y9/zI033pgPfOADOfjggzNlypT86le/yuzZs3PPPfckSUaPHp1zzz03P/jBD7L33nvn+OOPrzx/l112edNzAACAtkJIBwCAVnLvvfcmSd73vvdt9XO+8pWvZPbs2bnwwgvz7W9/u/L4TTfdlAkTJuSMM87IzJkz067dm/vj0//+7//O5MmTc8ghhyRJNmzYkHHjxmXy5Mm5//77c9BBB2X06NH5/Oc/nx/84AcZPXp0Lr744jf1mgAA0Fa5tAsAALSShQsXJkl22mmnrRq/bt26/OY3v0mvXr3y9a9/vcW297///Tn66KMza9asSqB/M04++eRKRE+S6urqnHbaaUmSBx988E3vHwAA3k2EdAAAeJd4+umns3bt2hx44IHp1KnTZtuPPPLIJMm0adPe9Gvtt99+mz22KfgvW7bsTe8fAADeTYR0AABoJf3790+SvPjii1s1vqGhIUle9XrqAwYMaDHuzejWrdtmj9XUbLwy5IYNG970/gEA4N1ESAcAgFay6dIpd9xxx1aN3xS3Fy1atMXtmy4V88oIvula6U1NTZuNX758+dZPFgAAtmNCOgAAtJLTTz891dXV+dnPfpYlS5aUjm1sbMzuu++eDh065MEHH8zq1as3GzN58uQkyejRoyuP9ezZM8nmZ703Nzfnsccee3PfQDZeOz1xljoAANs2IR0AAFrJ8OHD8+UvfzkvvfRSjjvuuMyZM2ezMWvXrs0VV1yRiy++OO3bt89JJ52Ul156KRMnTmwx7uabb84tt9yS4cOHt/iQ0AMOOCBJMmnSpBbjr7jiii2+3uvVs2fPVFVVZf78+W96XwAA0FbVtPYEAABge3bppZdm7dq1+f73v58RI0bkfe97X0aOHJna2trMmTMnt99+e+rr63PppZcmSb7zne/k7rvvzqWXXpr77rsvY8aMydy5c3PdddelU6dO+cUvflG5nEuSnHHGGbn88stz8cUXZ9q0aRk2bFgeeuihTJ8+PYcffnjuvvvuNzX/Ll265IADDsiUKVNy6qmnZtddd027du1y6qmnZvDgwW9q3wAA0FYI6QAA0IratWuXK664IieffHJ+8pOfZMqUKZkyZUqam5szYMCAjB8/PmeccUbGjRuXJOnTp08eeOCBfPOb38wNN9yQP//5z+nevXuOP/74XHTRRRk5cmSL/ffr1y933XVXvvjFL+bWW29NTU1NjjzyyNx///259NJL33RIT5Jf//rX+cIXvpD/+Z//yfLly1MURQ499FAhHQCAbUZVURRFa08CAAAAAADaKtdIBwAAAACAEkI6AAAAAACUENIBAAAAAKCEkA4AAAAAACWEdAAAAAAAKCGkAwAAAABACSEdAAAAAABKCOkAAAAAAFBCSAcAAAAAgBJCOgAAAAAAlBDSAQAAAACghJAOAAAAAAAl/n+95ZQAs3CffAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1500x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Calculate rating counts\n",
    "rating_counts = df_ratings['Rating'].value_counts().sort_index()\n",
    "\n",
    "unique_movie_count = df_ratings['MovieId'].nunique()\n",
    "unique_user_count = df_ratings['UserId'].nunique()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15, 10))\n",
    "rating_counts.plot(kind='barh', legend=False, ax=ax)\n",
    "\n",
    "total_rating_count = df_ratings.shape[0]\n",
    "plt.title(f'Total: {unique_movie_count} Movies; {unique_user_count} Users; {total_rating_count} Ratings Given', fontsize=20)\n",
    "plt.xlabel('Count', fontsize=14)\n",
    "plt.ylabel('Rating', fontsize=14)\n",
    "ax.invert_yaxis()\n",
    "\n",
    "for i, (rating, count) in enumerate(rating_counts.items()):\n",
    "    percentage = count * 100 / total_rating_count\n",
    "    ax.text(count / 4, i, f'Rating {rating}: {percentage:.0f}%', color='white', weight='bold', va='center')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing (MovieLens)\n",
    "\n",
    "First, we will remove columns from the dataframes that could introduce biases into our models. Specifically, we will delete the column `Timestamp` from **df_ratings** (we are not going to make analysis based on recent/old ratings) and `Zip-code` (it will not be usefull for our use case)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ratings.drop(columns=['Timestamp'], inplace=True)\n",
    "df_users.drop(columns=['Zip-code'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To continue with **df_movies**, we are going to: \n",
    "\n",
    "1. Extract the release year from the movie titles and add it as a separate column (`Date`).\n",
    "\n",
    "For example :\n",
    "\n",
    "`Jumanji (1995)` will become:\n",
    "\n",
    "- **Title**: Jumanji\n",
    "- **Date**: 1995\n",
    "2. Correct the Titles patterns to make them 'match' with the one inside imdb\n",
    "\n",
    "Indeed, some titles have a structure where a definite article is placed at the end, for example, \"American President, The\". Additionally, we will remove any unnecessary information within parentheses, such as alternative titles or additional descriptors, to match the format used in IMDb.\n",
    "\n",
    "For example:\n",
    "\n",
    "- `Contender, The` will become `The Contender`\n",
    "- `Silence of the Palace, The (Saimt el Qusur)` will become `The Silence of the Palace`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: df_movies , Length: 3883\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "      <td>1995.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "      <td>1995.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "      <td>1995.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>1995.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1995.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>2000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "      <td>2000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title                        Genres  \\\n",
       "0           1                    Toy Story   Animation|Children's|Comedy   \n",
       "1           2                      Jumanji  Adventure|Children's|Fantasy   \n",
       "2           3             Grumpier Old Men                Comedy|Romance   \n",
       "3           4            Waiting to Exhale                  Comedy|Drama   \n",
       "4           5  Father of the Bride Part II                        Comedy   \n",
       "...       ...                          ...                           ...   \n",
       "3878     3948             Meet the Parents                        Comedy   \n",
       "3879     3949          Requiem for a Dream                         Drama   \n",
       "3880     3950                    Tigerland                         Drama   \n",
       "3881     3951             Two Family House                         Drama   \n",
       "3882     3952                The Contender                Drama|Thriller   \n",
       "\n",
       "        Date  \n",
       "0     1995.0  \n",
       "1     1995.0  \n",
       "2     1995.0  \n",
       "3     1995.0  \n",
       "4     1995.0  \n",
       "...      ...  \n",
       "3878  2000.0  \n",
       "3879  2000.0  \n",
       "3880  2000.0  \n",
       "3881  2000.0  \n",
       "3882  2000.0  \n",
       "\n",
       "[3883 rows x 4 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_movies.reset_index(inplace=True)\n",
    "df_movies['Date'] = df_movies['Title'].str.extract(r'\\((\\d{4})\\)')\n",
    "df_movies['Title'] = df_movies['Title'].str.replace(r'\\s\\(\\d{4}\\)', '', regex=True)\n",
    "df_movies['Title'] = df_movies['Title'].str.replace(r'\\s*\\([^)]*\\)', '', regex=True)\n",
    "df_movies['Title'] = df_movies['Title'].str.replace(r\"(.+),\\s(The|Les|An|A|L'|La|Das|Il|Le|Der|Die|I|El)$\", r'\\2 \\1', regex=True)\n",
    "df_movies['Date'] = df_movies['Date'].astype(float)\n",
    "get_summary(df_movies, 'df_movies')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and visualization (IMDb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the IMDb dataset. The dataset is in a tab-separated values (TSV) file format. We will use the pandas library to read this dataset into a DataFrame for easier manipulation and analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load IMDb dataset\n",
    "imdb_title_basics = pd.read_csv('../data/imdb/title.basics.tsv', sep=\"\\t\", na_values=\"\\\\N\", low_memory=False)\n",
    "imdb_name_basics = pd.read_csv('../data/imdb/name.basics.tsv', sep=\"\\t\", na_values=\"\\\\N\", low_memory=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: imdb_title_basics , Length: 10904346\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tconst</th>\n",
       "      <th>titleType</th>\n",
       "      <th>primaryTitle</th>\n",
       "      <th>originalTitle</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>startYear</th>\n",
       "      <th>endYear</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tt0000001</td>\n",
       "      <td>short</td>\n",
       "      <td>Carmencita</td>\n",
       "      <td>Carmencita</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1894.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Documentary,Short</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>tt0000002</td>\n",
       "      <td>short</td>\n",
       "      <td>Le clown et ses chiens</td>\n",
       "      <td>Le clown et ses chiens</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1892.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "      <td>Animation,Short</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tt0000003</td>\n",
       "      <td>short</td>\n",
       "      <td>Pauvre Pierrot</td>\n",
       "      <td>Pauvre Pierrot</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1892.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "      <td>Animation,Comedy,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tt0000004</td>\n",
       "      <td>short</td>\n",
       "      <td>Un bon bock</td>\n",
       "      <td>Un bon bock</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1892.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12</td>\n",
       "      <td>Animation,Short</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tt0000005</td>\n",
       "      <td>short</td>\n",
       "      <td>Blacksmith Scene</td>\n",
       "      <td>Blacksmith Scene</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1893.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>Comedy,Short</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904341</th>\n",
       "      <td>tt9916848</td>\n",
       "      <td>tvEpisode</td>\n",
       "      <td>Episode #3.17</td>\n",
       "      <td>Episode #3.17</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Action,Drama,Family</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904342</th>\n",
       "      <td>tt9916850</td>\n",
       "      <td>tvEpisode</td>\n",
       "      <td>Episode #3.19</td>\n",
       "      <td>Episode #3.19</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Action,Drama,Family</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904343</th>\n",
       "      <td>tt9916852</td>\n",
       "      <td>tvEpisode</td>\n",
       "      <td>Episode #3.20</td>\n",
       "      <td>Episode #3.20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Action,Drama,Family</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904344</th>\n",
       "      <td>tt9916856</td>\n",
       "      <td>short</td>\n",
       "      <td>The Wind</td>\n",
       "      <td>The Wind</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27</td>\n",
       "      <td>Short</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904345</th>\n",
       "      <td>tt9916880</td>\n",
       "      <td>tvEpisode</td>\n",
       "      <td>Horrid Henry Knows It All</td>\n",
       "      <td>Horrid Henry Knows It All</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "      <td>Adventure,Animation,Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10904346 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             tconst  titleType               primaryTitle  \\\n",
       "0         tt0000001      short                 Carmencita   \n",
       "1         tt0000002      short     Le clown et ses chiens   \n",
       "2         tt0000003      short             Pauvre Pierrot   \n",
       "3         tt0000004      short                Un bon bock   \n",
       "4         tt0000005      short           Blacksmith Scene   \n",
       "...             ...        ...                        ...   \n",
       "10904341  tt9916848  tvEpisode              Episode #3.17   \n",
       "10904342  tt9916850  tvEpisode              Episode #3.19   \n",
       "10904343  tt9916852  tvEpisode              Episode #3.20   \n",
       "10904344  tt9916856      short                   The Wind   \n",
       "10904345  tt9916880  tvEpisode  Horrid Henry Knows It All   \n",
       "\n",
       "                      originalTitle  isAdult  startYear  endYear  \\\n",
       "0                        Carmencita      0.0     1894.0      NaN   \n",
       "1            Le clown et ses chiens      0.0     1892.0      NaN   \n",
       "2                    Pauvre Pierrot      0.0     1892.0      NaN   \n",
       "3                       Un bon bock      0.0     1892.0      NaN   \n",
       "4                  Blacksmith Scene      0.0     1893.0      NaN   \n",
       "...                             ...      ...        ...      ...   \n",
       "10904341              Episode #3.17      0.0     2009.0      NaN   \n",
       "10904342              Episode #3.19      0.0     2010.0      NaN   \n",
       "10904343              Episode #3.20      0.0     2010.0      NaN   \n",
       "10904344                   The Wind      0.0     2015.0      NaN   \n",
       "10904345  Horrid Henry Knows It All      0.0     2014.0      NaN   \n",
       "\n",
       "         runtimeMinutes                      genres  \n",
       "0                     1           Documentary,Short  \n",
       "1                     5             Animation,Short  \n",
       "2                     5    Animation,Comedy,Romance  \n",
       "3                    12             Animation,Short  \n",
       "4                     1                Comedy,Short  \n",
       "...                 ...                         ...  \n",
       "10904341            NaN         Action,Drama,Family  \n",
       "10904342            NaN         Action,Drama,Family  \n",
       "10904343            NaN         Action,Drama,Family  \n",
       "10904344             27                       Short  \n",
       "10904345             10  Adventure,Animation,Comedy  \n",
       "\n",
       "[10904346 rows x 9 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_7abf6_row0_col1, #T_7abf6_row6_col2 {\n",
       "  background-color: #440154;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_7abf6_row0_col2, #T_7abf6_row1_col1, #T_7abf6_row1_col2, #T_7abf6_row2_col2, #T_7abf6_row3_col2, #T_7abf6_row4_col1, #T_7abf6_row4_col2, #T_7abf6_row5_col1, #T_7abf6_row6_col1, #T_7abf6_row7_col1, #T_7abf6_row8_col1 {\n",
       "  background-color: #1f9e89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_7abf6_row0_col3, #T_7abf6_row1_col3, #T_7abf6_row2_col3, #T_7abf6_row3_col3, #T_7abf6_row4_col3, #T_7abf6_row5_col3, #T_7abf6_row6_col3, #T_7abf6_row7_col3, #T_7abf6_row8_col3 {\n",
       "  background-color: #fde725;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_7abf6_row2_col1 {\n",
       "  background-color: #34618d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_7abf6_row3_col1 {\n",
       "  background-color: #34608d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_7abf6_row5_col2 {\n",
       "  background-color: #228c8d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_7abf6_row7_col2 {\n",
       "  background-color: #443a83;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_7abf6_row8_col2 {\n",
       "  background-color: #1f978b;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_7abf6\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_7abf6_level0_col0\" class=\"col_heading level0 col0\" >dtype</th>\n",
       "      <th id=\"T_7abf6_level0_col1\" class=\"col_heading level0 col1\" >unique</th>\n",
       "      <th id=\"T_7abf6_level0_col2\" class=\"col_heading level0 col2\" >missing</th>\n",
       "      <th id=\"T_7abf6_level0_col3\" class=\"col_heading level0 col3\" >duplicates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row0\" class=\"row_heading level0 row0\" >tconst</th>\n",
       "      <td id=\"T_7abf6_row0_col0\" class=\"data row0 col0\" >object</td>\n",
       "      <td id=\"T_7abf6_row0_col1\" class=\"data row0 col1\" >10904346</td>\n",
       "      <td id=\"T_7abf6_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "      <td id=\"T_7abf6_row0_col3\" class=\"data row0 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row1\" class=\"row_heading level0 row1\" >titleType</th>\n",
       "      <td id=\"T_7abf6_row1_col0\" class=\"data row1 col0\" >object</td>\n",
       "      <td id=\"T_7abf6_row1_col1\" class=\"data row1 col1\" >11</td>\n",
       "      <td id=\"T_7abf6_row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "      <td id=\"T_7abf6_row1_col3\" class=\"data row1 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row2\" class=\"row_heading level0 row2\" >primaryTitle</th>\n",
       "      <td id=\"T_7abf6_row2_col0\" class=\"data row2 col0\" >object</td>\n",
       "      <td id=\"T_7abf6_row2_col1\" class=\"data row2 col1\" >4907276</td>\n",
       "      <td id=\"T_7abf6_row2_col2\" class=\"data row2 col2\" >18</td>\n",
       "      <td id=\"T_7abf6_row2_col3\" class=\"data row2 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row3\" class=\"row_heading level0 row3\" >originalTitle</th>\n",
       "      <td id=\"T_7abf6_row3_col0\" class=\"data row3 col0\" >object</td>\n",
       "      <td id=\"T_7abf6_row3_col1\" class=\"data row3 col1\" >4931053</td>\n",
       "      <td id=\"T_7abf6_row3_col2\" class=\"data row3 col2\" >18</td>\n",
       "      <td id=\"T_7abf6_row3_col3\" class=\"data row3 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row4\" class=\"row_heading level0 row4\" >isAdult</th>\n",
       "      <td id=\"T_7abf6_row4_col0\" class=\"data row4 col0\" >float64</td>\n",
       "      <td id=\"T_7abf6_row4_col1\" class=\"data row4 col1\" >41</td>\n",
       "      <td id=\"T_7abf6_row4_col2\" class=\"data row4 col2\" >1</td>\n",
       "      <td id=\"T_7abf6_row4_col3\" class=\"data row4 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row5\" class=\"row_heading level0 row5\" >startYear</th>\n",
       "      <td id=\"T_7abf6_row5_col0\" class=\"data row5 col0\" >float64</td>\n",
       "      <td id=\"T_7abf6_row5_col1\" class=\"data row5 col1\" >151</td>\n",
       "      <td id=\"T_7abf6_row5_col2\" class=\"data row5 col2\" >1414728</td>\n",
       "      <td id=\"T_7abf6_row5_col3\" class=\"data row5 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row6\" class=\"row_heading level0 row6\" >endYear</th>\n",
       "      <td id=\"T_7abf6_row6_col0\" class=\"data row6 col0\" >float64</td>\n",
       "      <td id=\"T_7abf6_row6_col1\" class=\"data row6 col1\" >96</td>\n",
       "      <td id=\"T_7abf6_row6_col2\" class=\"data row6 col2\" >10778284</td>\n",
       "      <td id=\"T_7abf6_row6_col3\" class=\"data row6 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row7\" class=\"row_heading level0 row7\" >runtimeMinutes</th>\n",
       "      <td id=\"T_7abf6_row7_col0\" class=\"data row7 col0\" >object</td>\n",
       "      <td id=\"T_7abf6_row7_col1\" class=\"data row7 col1\" >906</td>\n",
       "      <td id=\"T_7abf6_row7_col2\" class=\"data row7 col2\" >7517622</td>\n",
       "      <td id=\"T_7abf6_row7_col3\" class=\"data row7 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_7abf6_level0_row8\" class=\"row_heading level0 row8\" >genres</th>\n",
       "      <td id=\"T_7abf6_row8_col0\" class=\"data row8 col0\" >object</td>\n",
       "      <td id=\"T_7abf6_row8_col1\" class=\"data row8 col1\" >2374</td>\n",
       "      <td id=\"T_7abf6_row8_col2\" class=\"data row8 col2\" >484495</td>\n",
       "      <td id=\"T_7abf6_row8_col3\" class=\"data row8 col3\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x72e4e9b5d570>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_summary(imdb_title_basics, 'imdb_title_basics', analysis=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: imdb_name_basics , Length: 13616649\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nconst</th>\n",
       "      <th>primaryName</th>\n",
       "      <th>birthYear</th>\n",
       "      <th>deathYear</th>\n",
       "      <th>primaryProfession</th>\n",
       "      <th>knownForTitles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>nm0000001</td>\n",
       "      <td>Fred Astaire</td>\n",
       "      <td>1899.0</td>\n",
       "      <td>1987.0</td>\n",
       "      <td>actor,miscellaneous,producer</td>\n",
       "      <td>tt0072308,tt0050419,tt0053137,tt0027125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>nm0000002</td>\n",
       "      <td>Lauren Bacall</td>\n",
       "      <td>1924.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>actress,soundtrack,archive_footage</td>\n",
       "      <td>tt0037382,tt0075213,tt0117057,tt0038355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>nm0000003</td>\n",
       "      <td>Brigitte Bardot</td>\n",
       "      <td>1934.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>actress,music_department,producer</td>\n",
       "      <td>tt0057345,tt0049189,tt0056404,tt0054452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>nm0000004</td>\n",
       "      <td>John Belushi</td>\n",
       "      <td>1949.0</td>\n",
       "      <td>1982.0</td>\n",
       "      <td>actor,writer,music_department</td>\n",
       "      <td>tt0072562,tt0077975,tt0080455,tt0078723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>nm0000005</td>\n",
       "      <td>Ingmar Bergman</td>\n",
       "      <td>1918.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>writer,director,actor</td>\n",
       "      <td>tt0050986,tt0083922,tt0050976,tt0069467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616644</th>\n",
       "      <td>nm9993714</td>\n",
       "      <td>Romeo del Rosario</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>animation_department,art_department</td>\n",
       "      <td>tt11657662,tt14069590,tt2455546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616645</th>\n",
       "      <td>nm9993716</td>\n",
       "      <td>Essias Loberg</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616646</th>\n",
       "      <td>nm9993717</td>\n",
       "      <td>Harikrishnan Rajan</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cinematographer</td>\n",
       "      <td>tt8736744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616647</th>\n",
       "      <td>nm9993718</td>\n",
       "      <td>Aayush Nair</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cinematographer</td>\n",
       "      <td>tt8736744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616648</th>\n",
       "      <td>nm9993719</td>\n",
       "      <td>Andre Hill</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>writer</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13616649 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             nconst         primaryName  birthYear  deathYear  \\\n",
       "0         nm0000001        Fred Astaire     1899.0     1987.0   \n",
       "1         nm0000002       Lauren Bacall     1924.0     2014.0   \n",
       "2         nm0000003     Brigitte Bardot     1934.0        NaN   \n",
       "3         nm0000004        John Belushi     1949.0     1982.0   \n",
       "4         nm0000005      Ingmar Bergman     1918.0     2007.0   \n",
       "...             ...                 ...        ...        ...   \n",
       "13616644  nm9993714   Romeo del Rosario        NaN        NaN   \n",
       "13616645  nm9993716       Essias Loberg        NaN        NaN   \n",
       "13616646  nm9993717  Harikrishnan Rajan        NaN        NaN   \n",
       "13616647  nm9993718         Aayush Nair        NaN        NaN   \n",
       "13616648  nm9993719          Andre Hill        NaN        NaN   \n",
       "\n",
       "                            primaryProfession  \\\n",
       "0                actor,miscellaneous,producer   \n",
       "1          actress,soundtrack,archive_footage   \n",
       "2           actress,music_department,producer   \n",
       "3               actor,writer,music_department   \n",
       "4                       writer,director,actor   \n",
       "...                                       ...   \n",
       "13616644  animation_department,art_department   \n",
       "13616645                                  NaN   \n",
       "13616646                      cinematographer   \n",
       "13616647                      cinematographer   \n",
       "13616648                               writer   \n",
       "\n",
       "                                   knownForTitles  \n",
       "0         tt0072308,tt0050419,tt0053137,tt0027125  \n",
       "1         tt0037382,tt0075213,tt0117057,tt0038355  \n",
       "2         tt0057345,tt0049189,tt0056404,tt0054452  \n",
       "3         tt0072562,tt0077975,tt0080455,tt0078723  \n",
       "4         tt0050986,tt0083922,tt0050976,tt0069467  \n",
       "...                                           ...  \n",
       "13616644          tt11657662,tt14069590,tt2455546  \n",
       "13616645                                      NaN  \n",
       "13616646                                tt8736744  \n",
       "13616647                                tt8736744  \n",
       "13616648                                      NaN  \n",
       "\n",
       "[13616649 rows x 6 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_fe1aa_row0_col1, #T_fe1aa_row3_col2 {\n",
       "  background-color: #440154;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_fe1aa_row0_col2, #T_fe1aa_row1_col2, #T_fe1aa_row2_col1, #T_fe1aa_row3_col1 {\n",
       "  background-color: #1f9e89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_fe1aa_row0_col3, #T_fe1aa_row1_col3, #T_fe1aa_row2_col3, #T_fe1aa_row3_col3, #T_fe1aa_row4_col3, #T_fe1aa_row5_col3 {\n",
       "  background-color: #fde725;\n",
       "  color: #000000;\n",
       "}\n",
       "#T_fe1aa_row1_col1 {\n",
       "  background-color: #472d7b;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_fe1aa_row2_col2 {\n",
       "  background-color: #46075a;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_fe1aa_row4_col1 {\n",
       "  background-color: #1e9d89;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_fe1aa_row4_col2 {\n",
       "  background-color: #25838e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_fe1aa_row5_col1 {\n",
       "  background-color: #32658e;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "#T_fe1aa_row5_col2 {\n",
       "  background-color: #218f8d;\n",
       "  color: #f1f1f1;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_fe1aa\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_fe1aa_level0_col0\" class=\"col_heading level0 col0\" >dtype</th>\n",
       "      <th id=\"T_fe1aa_level0_col1\" class=\"col_heading level0 col1\" >unique</th>\n",
       "      <th id=\"T_fe1aa_level0_col2\" class=\"col_heading level0 col2\" >missing</th>\n",
       "      <th id=\"T_fe1aa_level0_col3\" class=\"col_heading level0 col3\" >duplicates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_fe1aa_level0_row0\" class=\"row_heading level0 row0\" >nconst</th>\n",
       "      <td id=\"T_fe1aa_row0_col0\" class=\"data row0 col0\" >object</td>\n",
       "      <td id=\"T_fe1aa_row0_col1\" class=\"data row0 col1\" >13616649</td>\n",
       "      <td id=\"T_fe1aa_row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "      <td id=\"T_fe1aa_row0_col3\" class=\"data row0 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fe1aa_level0_row1\" class=\"row_heading level0 row1\" >primaryName</th>\n",
       "      <td id=\"T_fe1aa_row1_col0\" class=\"data row1 col0\" >object</td>\n",
       "      <td id=\"T_fe1aa_row1_col1\" class=\"data row1 col1\" >10479239</td>\n",
       "      <td id=\"T_fe1aa_row1_col2\" class=\"data row1 col2\" >57</td>\n",
       "      <td id=\"T_fe1aa_row1_col3\" class=\"data row1 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fe1aa_level0_row2\" class=\"row_heading level0 row2\" >birthYear</th>\n",
       "      <td id=\"T_fe1aa_row2_col0\" class=\"data row2 col0\" >float64</td>\n",
       "      <td id=\"T_fe1aa_row2_col1\" class=\"data row2 col1\" >541</td>\n",
       "      <td id=\"T_fe1aa_row2_col2\" class=\"data row2 col2\" >12997476</td>\n",
       "      <td id=\"T_fe1aa_row2_col3\" class=\"data row2 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fe1aa_level0_row3\" class=\"row_heading level0 row3\" >deathYear</th>\n",
       "      <td id=\"T_fe1aa_row3_col0\" class=\"data row3 col0\" >float64</td>\n",
       "      <td id=\"T_fe1aa_row3_col1\" class=\"data row3 col1\" >484</td>\n",
       "      <td id=\"T_fe1aa_row3_col2\" class=\"data row3 col2\" >13385906</td>\n",
       "      <td id=\"T_fe1aa_row3_col3\" class=\"data row3 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fe1aa_level0_row4\" class=\"row_heading level0 row4\" >primaryProfession</th>\n",
       "      <td id=\"T_fe1aa_row4_col0\" class=\"data row4 col0\" >object</td>\n",
       "      <td id=\"T_fe1aa_row4_col1\" class=\"data row4 col1\" >22906</td>\n",
       "      <td id=\"T_fe1aa_row4_col2\" class=\"data row4 col2\" >2622472</td>\n",
       "      <td id=\"T_fe1aa_row4_col3\" class=\"data row4 col3\" >0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_fe1aa_level0_row5\" class=\"row_heading level0 row5\" >knownForTitles</th>\n",
       "      <td id=\"T_fe1aa_row5_col0\" class=\"data row5 col0\" >object</td>\n",
       "      <td id=\"T_fe1aa_row5_col1\" class=\"data row5 col1\" >5687871</td>\n",
       "      <td id=\"T_fe1aa_row5_col2\" class=\"data row5 col2\" >1510252</td>\n",
       "      <td id=\"T_fe1aa_row5_col3\" class=\"data row5 col3\" >0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x72e4ebd54f40>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_summary(imdb_name_basics, 'imdb_name_basics', analysis=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing (IMDb)\n",
    "### imdb_title_basics\n",
    "\n",
    "`tilteType` can be :\n",
    "\n",
    "- short\n",
    "- tvEpisode\n",
    "- tvMovie\n",
    "- movie\n",
    "\n",
    "In this step, we are goind to first filter the dataset to include only the movies `titleType` = `movie`.\n",
    "\n",
    "We will then clean the dataset by removing unnecessary columns, such as:\n",
    "\n",
    "- titleType\n",
    "- originalTitle\n",
    "- endYear\n",
    "\n",
    "These columns are either not relevant to our analysis or contain redundant information. By removing these columns, we simplify our dataset, making it easier to work with and focus on the essential information for our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tconst</th>\n",
       "      <th>primaryTitle</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>startYear</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>tt0000009</td>\n",
       "      <td>Miss Jerry</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1894.0</td>\n",
       "      <td>45</td>\n",
       "      <td>Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>tt0000147</td>\n",
       "      <td>The Corbett-Fitzsimmons Fight</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>100</td>\n",
       "      <td>Documentary,News,Sport</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>498</th>\n",
       "      <td>tt0000502</td>\n",
       "      <td>Bohemios</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1905.0</td>\n",
       "      <td>100</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>570</th>\n",
       "      <td>tt0000574</td>\n",
       "      <td>The Story of the Kelly Gang</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1906.0</td>\n",
       "      <td>70</td>\n",
       "      <td>Action,Adventure,Biography</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>587</th>\n",
       "      <td>tt0000591</td>\n",
       "      <td>The Prodigal Son</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1907.0</td>\n",
       "      <td>90</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904237</th>\n",
       "      <td>tt9916622</td>\n",
       "      <td>Rodolpho Teóphilo - O Legado de um Pioneiro</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>57</td>\n",
       "      <td>Documentary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904264</th>\n",
       "      <td>tt9916680</td>\n",
       "      <td>De la ilusión al desconcierto: cine colombiano...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>100</td>\n",
       "      <td>Documentary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904276</th>\n",
       "      <td>tt9916706</td>\n",
       "      <td>Dankyavar Danka</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904286</th>\n",
       "      <td>tt9916730</td>\n",
       "      <td>6 Gunn</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>116</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10904296</th>\n",
       "      <td>tt9916754</td>\n",
       "      <td>Chico Albuquerque - Revelações</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>49</td>\n",
       "      <td>Documentary</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>685353 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             tconst                                       primaryTitle  \\\n",
       "8         tt0000009                                         Miss Jerry   \n",
       "144       tt0000147                      The Corbett-Fitzsimmons Fight   \n",
       "498       tt0000502                                           Bohemios   \n",
       "570       tt0000574                        The Story of the Kelly Gang   \n",
       "587       tt0000591                                   The Prodigal Son   \n",
       "...             ...                                                ...   \n",
       "10904237  tt9916622        Rodolpho Teóphilo - O Legado de um Pioneiro   \n",
       "10904264  tt9916680  De la ilusión al desconcierto: cine colombiano...   \n",
       "10904276  tt9916706                                    Dankyavar Danka   \n",
       "10904286  tt9916730                                             6 Gunn   \n",
       "10904296  tt9916754                     Chico Albuquerque - Revelações   \n",
       "\n",
       "          isAdult  startYear runtimeMinutes                      genres  \n",
       "8             0.0     1894.0             45                     Romance  \n",
       "144           0.0     1897.0            100      Documentary,News,Sport  \n",
       "498           0.0     1905.0            100                         NaN  \n",
       "570           0.0     1906.0             70  Action,Adventure,Biography  \n",
       "587           0.0     1907.0             90                       Drama  \n",
       "...           ...        ...            ...                         ...  \n",
       "10904237      0.0     2015.0             57                 Documentary  \n",
       "10904264      0.0     2007.0            100                 Documentary  \n",
       "10904276      0.0     2013.0            NaN                      Comedy  \n",
       "10904286      0.0     2017.0            116                       Drama  \n",
       "10904296      0.0     2013.0             49                 Documentary  \n",
       "\n",
       "[685353 rows x 6 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb_title_basics = imdb_title_basics[imdb_title_basics['titleType'] == 'movie']\n",
    "imdb_title_basics.drop(columns=['endYear', 'titleType', 'originalTitle'], inplace=True)\n",
    "imdb_title_basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to merge our two datasets, **imdb_title_basics** and **df_movies**. The goal is to enrich **df_movies** with additional data from **imdb_title_basics** to have higher quality data for the future.\n",
    "\n",
    "The merge will be performed on the movie title and release date for both datasets, using the following keys:\n",
    "- `Title` and `Date` for **df_movies**\n",
    "- `primaryTitle` and `startYear` for **imdb_title_basics**\n",
    "\n",
    "The additional key `Date` is used to avoid mistakenly including other films with the same title but from different years in the merge (to avoid including erroneous data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>tconst</th>\n",
       "      <th>primaryTitle</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>startYear</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>81</td>\n",
       "      <td>Adventure,Animation,Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>104</td>\n",
       "      <td>Adventure,Comedy,Family</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>101</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>124</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>106</td>\n",
       "      <td>Comedy,Family,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3900</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0212338</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>108</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3901</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0180093</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>102</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3902</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0170691</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>101</td>\n",
       "      <td>Drama,War</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3903</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0202641</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>108</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3904</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0208874</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>126</td>\n",
       "      <td>Drama,Thriller</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3905 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title                        Genres  \\\n",
       "0           1                    Toy Story   Animation|Children's|Comedy   \n",
       "1           2                      Jumanji  Adventure|Children's|Fantasy   \n",
       "2           3             Grumpier Old Men                Comedy|Romance   \n",
       "3           4            Waiting to Exhale                  Comedy|Drama   \n",
       "4           5  Father of the Bride Part II                        Comedy   \n",
       "...       ...                          ...                           ...   \n",
       "3900     3948             Meet the Parents                        Comedy   \n",
       "3901     3949          Requiem for a Dream                         Drama   \n",
       "3902     3950                    Tigerland                         Drama   \n",
       "3903     3951             Two Family House                         Drama   \n",
       "3904     3952                The Contender                Drama|Thriller   \n",
       "\n",
       "        Date     tconst                 primaryTitle  isAdult  startYear  \\\n",
       "0     1995.0  tt0114709                    Toy Story      0.0     1995.0   \n",
       "1     1995.0  tt0113497                      Jumanji      0.0     1995.0   \n",
       "2     1995.0  tt0113228             Grumpier Old Men      0.0     1995.0   \n",
       "3     1995.0  tt0114885            Waiting to Exhale      0.0     1995.0   \n",
       "4     1995.0  tt0113041  Father of the Bride Part II      0.0     1995.0   \n",
       "...      ...        ...                          ...      ...        ...   \n",
       "3900  2000.0  tt0212338             Meet the Parents      0.0     2000.0   \n",
       "3901  2000.0  tt0180093          Requiem for a Dream      0.0     2000.0   \n",
       "3902  2000.0  tt0170691                    Tigerland      0.0     2000.0   \n",
       "3903  2000.0  tt0202641             Two Family House      0.0     2000.0   \n",
       "3904  2000.0  tt0208874                The Contender      0.0     2000.0   \n",
       "\n",
       "     runtimeMinutes                      genres  \n",
       "0                81  Adventure,Animation,Comedy  \n",
       "1               104     Adventure,Comedy,Family  \n",
       "2               101              Comedy,Romance  \n",
       "3               124        Comedy,Drama,Romance  \n",
       "4               106       Comedy,Family,Romance  \n",
       "...             ...                         ...  \n",
       "3900            108              Comedy,Romance  \n",
       "3901            102                       Drama  \n",
       "3902            101                   Drama,War  \n",
       "3903            108        Comedy,Drama,Romance  \n",
       "3904            126              Drama,Thriller  \n",
       "\n",
       "[3905 rows x 10 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_movies = df_movies.merge(imdb_title_basics, left_on=['Title', 'Date'], right_on=['primaryTitle', 'startYear'], how='left')\n",
    "df_movies.drop(columns=['startYear', 'primaryTitle'])\n",
    "df_movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite merging on both keys, there are films with the same names released in the same year. Consequently, there are multiple rows with the same MovieId.\n",
    "\n",
    "To address this issue, we will calculate a similarity score (**match_score**) based on the similarity between the Genres of the film according to df_movies and IMDb (columns `Genres` and `genres`). The rows where the Genres are least similar will then be removed from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to calculate genre match score\n",
    "def genre_match_score(genres1, genres2):\n",
    "    if pd.isna(genres1) or pd.isna(genres2):\n",
    "        return 0\n",
    "    set1 = set(genres1.split('|'))\n",
    "    set2 = set(genres2.split(','))\n",
    "    return len(set1 & set2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Duplicated values and their counts:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>tconst</th>\n",
       "      <th>primaryTitle</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>startYear</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>genres</th>\n",
       "      <th>match_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>48</td>\n",
       "      <td>Pocahontas</td>\n",
       "      <td>Animation|Children's|Musical|Romance</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0114148</td>\n",
       "      <td>Pocahontas</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>81</td>\n",
       "      <td>Adventure,Animation,Drama</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>48</td>\n",
       "      <td>Pocahontas</td>\n",
       "      <td>Animation|Children's|Musical|Romance</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt9047432</td>\n",
       "      <td>Pocahontas</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>63</td>\n",
       "      <td>Animation</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>150</td>\n",
       "      <td>Apollo 13</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0112384</td>\n",
       "      <td>Apollo 13</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>140</td>\n",
       "      <td>Adventure,Drama,History</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>150</td>\n",
       "      <td>Apollo 13</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt6590506</td>\n",
       "      <td>Apollo 13</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>49</td>\n",
       "      <td>Documentary</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>297</td>\n",
       "      <td>Panther</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0114084</td>\n",
       "      <td>Panther</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>124</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>297</th>\n",
       "      <td>297</td>\n",
       "      <td>Panther</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt1363478</td>\n",
       "      <td>Panther</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>83</td>\n",
       "      <td>Action</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>776</th>\n",
       "      <td>783</td>\n",
       "      <td>The Hunchback of Notre Dame</td>\n",
       "      <td>Animation|Children's|Musical</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>tt0116583</td>\n",
       "      <td>The Hunchback of Notre Dame</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>91</td>\n",
       "      <td>Animation,Drama,Family</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>777</th>\n",
       "      <td>783</td>\n",
       "      <td>The Hunchback of Notre Dame</td>\n",
       "      <td>Animation|Children's|Musical</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>tt9799878</td>\n",
       "      <td>The Hunchback of Notre Dame</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>65</td>\n",
       "      <td>Animation</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1202</th>\n",
       "      <td>1216</td>\n",
       "      <td>The Big Blue</td>\n",
       "      <td>Adventure|Romance</td>\n",
       "      <td>1988.0</td>\n",
       "      <td>tt0094738</td>\n",
       "      <td>The Big Blue</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1988.0</td>\n",
       "      <td>100</td>\n",
       "      <td>Drama</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1203</th>\n",
       "      <td>1216</td>\n",
       "      <td>The Big Blue</td>\n",
       "      <td>Adventure|Romance</td>\n",
       "      <td>1988.0</td>\n",
       "      <td>tt0095250</td>\n",
       "      <td>The Big Blue</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1988.0</td>\n",
       "      <td>168</td>\n",
       "      <td>Adventure,Drama,Sport</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1389</th>\n",
       "      <td>1407</td>\n",
       "      <td>Scream</td>\n",
       "      <td>Horror|Thriller</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>tt0117571</td>\n",
       "      <td>Scream</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>111</td>\n",
       "      <td>Horror,Mystery</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1390</th>\n",
       "      <td>1407</td>\n",
       "      <td>Scream</td>\n",
       "      <td>Horror|Thriller</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>tt0176904</td>\n",
       "      <td>Scream</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Documentary</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2186</th>\n",
       "      <td>2249</td>\n",
       "      <td>My Blue Heaven</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>tt0100212</td>\n",
       "      <td>My Blue Heaven</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>97</td>\n",
       "      <td>Comedy,Crime</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2187</th>\n",
       "      <td>2249</td>\n",
       "      <td>My Blue Heaven</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>tt0100213</td>\n",
       "      <td>My Blue Heaven</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>95</td>\n",
       "      <td>Drama</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2190</th>\n",
       "      <td>2252</td>\n",
       "      <td>Hero</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>1992.0</td>\n",
       "      <td>tt0104412</td>\n",
       "      <td>Hero</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1992.0</td>\n",
       "      <td>119</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2191</th>\n",
       "      <td>2252</td>\n",
       "      <td>Hero</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>1992.0</td>\n",
       "      <td>tt0250422</td>\n",
       "      <td>Hero</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1992.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Musical,Romance</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2261</th>\n",
       "      <td>2322</td>\n",
       "      <td>Soldier</td>\n",
       "      <td>Action|Adventure|Sci-Fi|Thriller|War</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>tt0120157</td>\n",
       "      <td>Soldier</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>99</td>\n",
       "      <td>Action,Drama,Sci-Fi</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2262</th>\n",
       "      <td>2322</td>\n",
       "      <td>Soldier</td>\n",
       "      <td>Action|Adventure|Sci-Fi|Thriller|War</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>tt0211634</td>\n",
       "      <td>Soldier</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>156</td>\n",
       "      <td>Action,Drama,Musical</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2449</th>\n",
       "      <td>2509</td>\n",
       "      <td>Eight Days a Week</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>tt0119054</td>\n",
       "      <td>Eight Days a Week</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>92</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2450</th>\n",
       "      <td>2509</td>\n",
       "      <td>Eight Days a Week</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>tt7822002</td>\n",
       "      <td>Eight Days a Week</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>45</td>\n",
       "      <td>Documentary</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2534</th>\n",
       "      <td>2593</td>\n",
       "      <td>The Monster</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1994.0</td>\n",
       "      <td>tt0110570</td>\n",
       "      <td>The Monster</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1994.0</td>\n",
       "      <td>112</td>\n",
       "      <td>Comedy,Crime,Mystery</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2535</th>\n",
       "      <td>2593</td>\n",
       "      <td>The Monster</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1994.0</td>\n",
       "      <td>tt0260155</td>\n",
       "      <td>The Monster</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1994.0</td>\n",
       "      <td>132</td>\n",
       "      <td>Comedy,Horror</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2624</th>\n",
       "      <td>2682</td>\n",
       "      <td>Limbo</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>tt0164085</td>\n",
       "      <td>Limbo</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>126</td>\n",
       "      <td>Adventure,Drama,Thriller</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2625</th>\n",
       "      <td>2682</td>\n",
       "      <td>Limbo</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>tt5874116</td>\n",
       "      <td>Limbo</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>58</td>\n",
       "      <td>Horror</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2633</th>\n",
       "      <td>2690</td>\n",
       "      <td>An Ideal Husband</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>tt0122541</td>\n",
       "      <td>An Ideal Husband</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>97</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2634</th>\n",
       "      <td>2690</td>\n",
       "      <td>An Ideal Husband</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>tt0160395</td>\n",
       "      <td>An Ideal Husband</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>90</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2764</th>\n",
       "      <td>2820</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>tt0058126</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>140</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2765</th>\n",
       "      <td>2820</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>tt0058174</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>120</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2766</th>\n",
       "      <td>2820</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>tt0058175</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>191</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2767</th>\n",
       "      <td>2820</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>tt0058177</td>\n",
       "      <td>Hamlet</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2955</th>\n",
       "      <td>3008</td>\n",
       "      <td>Last Night</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>tt0156729</td>\n",
       "      <td>Last Night</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>95</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2956</th>\n",
       "      <td>3008</td>\n",
       "      <td>Last Night</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>tt0285283</td>\n",
       "      <td>Last Night</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3065</th>\n",
       "      <td>3117</td>\n",
       "      <td>Ride with the Devil</td>\n",
       "      <td>Drama|Romance|War</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>tt0134154</td>\n",
       "      <td>Ride with the Devil</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>138</td>\n",
       "      <td>Drama,Romance,War</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3066</th>\n",
       "      <td>3117</td>\n",
       "      <td>Ride with the Devil</td>\n",
       "      <td>Drama|Romance|War</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>tt0240867</td>\n",
       "      <td>Ride with the Devil</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>88</td>\n",
       "      <td>Horror</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3234</th>\n",
       "      <td>3285</td>\n",
       "      <td>The Beach</td>\n",
       "      <td>Adventure|Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0163978</td>\n",
       "      <td>The Beach</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>119</td>\n",
       "      <td>Adventure,Drama,Romance</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3235</th>\n",
       "      <td>3285</td>\n",
       "      <td>The Beach</td>\n",
       "      <td>Adventure|Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0326506</td>\n",
       "      <td>The Beach</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Documentary</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3503</th>\n",
       "      <td>3553</td>\n",
       "      <td>Gossip</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0176783</td>\n",
       "      <td>Gossip</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>90</td>\n",
       "      <td>Drama,Mystery,Thriller</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3504</th>\n",
       "      <td>3553</td>\n",
       "      <td>Gossip</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0238251</td>\n",
       "      <td>Gossip</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>134</td>\n",
       "      <td>Comedy,Drama</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3708</th>\n",
       "      <td>3757</td>\n",
       "      <td>Asylum</td>\n",
       "      <td>Horror</td>\n",
       "      <td>1972.0</td>\n",
       "      <td>tt0068230</td>\n",
       "      <td>Asylum</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1972.0</td>\n",
       "      <td>88</td>\n",
       "      <td>Horror</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3709</th>\n",
       "      <td>3757</td>\n",
       "      <td>Asylum</td>\n",
       "      <td>Horror</td>\n",
       "      <td>1972.0</td>\n",
       "      <td>tt0775421</td>\n",
       "      <td>Asylum</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1972.0</td>\n",
       "      <td>95</td>\n",
       "      <td>Documentary</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3888</th>\n",
       "      <td>3937</td>\n",
       "      <td>Runaway</td>\n",
       "      <td>Sci-Fi|Thriller</td>\n",
       "      <td>1984.0</td>\n",
       "      <td>tt0088024</td>\n",
       "      <td>Runaway</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1984.0</td>\n",
       "      <td>99</td>\n",
       "      <td>Action,Crime,Sci-Fi</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3889</th>\n",
       "      <td>3937</td>\n",
       "      <td>Runaway</td>\n",
       "      <td>Sci-Fi|Thriller</td>\n",
       "      <td>1984.0</td>\n",
       "      <td>tt6525802</td>\n",
       "      <td>Runaway</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1984.0</td>\n",
       "      <td>100</td>\n",
       "      <td>Drama</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title  \\\n",
       "47         48                   Pocahontas   \n",
       "48         48                   Pocahontas   \n",
       "149       150                    Apollo 13   \n",
       "150       150                    Apollo 13   \n",
       "296       297                      Panther   \n",
       "297       297                      Panther   \n",
       "776       783  The Hunchback of Notre Dame   \n",
       "777       783  The Hunchback of Notre Dame   \n",
       "1202     1216                 The Big Blue   \n",
       "1203     1216                 The Big Blue   \n",
       "1389     1407                       Scream   \n",
       "1390     1407                       Scream   \n",
       "2186     2249               My Blue Heaven   \n",
       "2187     2249               My Blue Heaven   \n",
       "2190     2252                         Hero   \n",
       "2191     2252                         Hero   \n",
       "2261     2322                      Soldier   \n",
       "2262     2322                      Soldier   \n",
       "2449     2509            Eight Days a Week   \n",
       "2450     2509            Eight Days a Week   \n",
       "2534     2593                  The Monster   \n",
       "2535     2593                  The Monster   \n",
       "2624     2682                        Limbo   \n",
       "2625     2682                        Limbo   \n",
       "2633     2690             An Ideal Husband   \n",
       "2634     2690             An Ideal Husband   \n",
       "2764     2820                       Hamlet   \n",
       "2765     2820                       Hamlet   \n",
       "2766     2820                       Hamlet   \n",
       "2767     2820                       Hamlet   \n",
       "2955     3008                   Last Night   \n",
       "2956     3008                   Last Night   \n",
       "3065     3117          Ride with the Devil   \n",
       "3066     3117          Ride with the Devil   \n",
       "3234     3285                    The Beach   \n",
       "3235     3285                    The Beach   \n",
       "3503     3553                       Gossip   \n",
       "3504     3553                       Gossip   \n",
       "3708     3757                       Asylum   \n",
       "3709     3757                       Asylum   \n",
       "3888     3937                      Runaway   \n",
       "3889     3937                      Runaway   \n",
       "\n",
       "                                    Genres    Date     tconst  \\\n",
       "47    Animation|Children's|Musical|Romance  1995.0  tt0114148   \n",
       "48    Animation|Children's|Musical|Romance  1995.0  tt9047432   \n",
       "149                                  Drama  1995.0  tt0112384   \n",
       "150                                  Drama  1995.0  tt6590506   \n",
       "296                                  Drama  1995.0  tt0114084   \n",
       "297                                  Drama  1995.0  tt1363478   \n",
       "776           Animation|Children's|Musical  1996.0  tt0116583   \n",
       "777           Animation|Children's|Musical  1996.0  tt9799878   \n",
       "1202                     Adventure|Romance  1988.0  tt0094738   \n",
       "1203                     Adventure|Romance  1988.0  tt0095250   \n",
       "1389                       Horror|Thriller  1996.0  tt0117571   \n",
       "1390                       Horror|Thriller  1996.0  tt0176904   \n",
       "2186                                Comedy  1990.0  tt0100212   \n",
       "2187                                Comedy  1990.0  tt0100213   \n",
       "2190                          Comedy|Drama  1992.0  tt0104412   \n",
       "2191                          Comedy|Drama  1992.0  tt0250422   \n",
       "2261  Action|Adventure|Sci-Fi|Thriller|War  1998.0  tt0120157   \n",
       "2262  Action|Adventure|Sci-Fi|Thriller|War  1998.0  tt0211634   \n",
       "2449                                Comedy  1997.0  tt0119054   \n",
       "2450                                Comedy  1997.0  tt7822002   \n",
       "2534                                Comedy  1994.0  tt0110570   \n",
       "2535                                Comedy  1994.0  tt0260155   \n",
       "2624                                 Drama  1999.0  tt0164085   \n",
       "2625                                 Drama  1999.0  tt5874116   \n",
       "2633                                Comedy  1999.0  tt0122541   \n",
       "2634                                Comedy  1999.0  tt0160395   \n",
       "2764                                 Drama  1964.0  tt0058126   \n",
       "2765                                 Drama  1964.0  tt0058174   \n",
       "2766                                 Drama  1964.0  tt0058175   \n",
       "2767                                 Drama  1964.0  tt0058177   \n",
       "2955                              Thriller  1998.0  tt0156729   \n",
       "2956                              Thriller  1998.0  tt0285283   \n",
       "3065                     Drama|Romance|War  1999.0  tt0134154   \n",
       "3066                     Drama|Romance|War  1999.0  tt0240867   \n",
       "3234                       Adventure|Drama  2000.0  tt0163978   \n",
       "3235                       Adventure|Drama  2000.0  tt0326506   \n",
       "3503                        Drama|Thriller  2000.0  tt0176783   \n",
       "3504                        Drama|Thriller  2000.0  tt0238251   \n",
       "3708                                Horror  1972.0  tt0068230   \n",
       "3709                                Horror  1972.0  tt0775421   \n",
       "3888                       Sci-Fi|Thriller  1984.0  tt0088024   \n",
       "3889                       Sci-Fi|Thriller  1984.0  tt6525802   \n",
       "\n",
       "                     primaryTitle  isAdult  startYear runtimeMinutes  \\\n",
       "47                     Pocahontas      0.0     1995.0             81   \n",
       "48                     Pocahontas      0.0     1995.0             63   \n",
       "149                     Apollo 13      0.0     1995.0            140   \n",
       "150                     Apollo 13      0.0     1995.0             49   \n",
       "296                       Panther      0.0     1995.0            124   \n",
       "297                       Panther      0.0     1995.0             83   \n",
       "776   The Hunchback of Notre Dame      0.0     1996.0             91   \n",
       "777   The Hunchback of Notre Dame      0.0     1996.0             65   \n",
       "1202                 The Big Blue      0.0     1988.0            100   \n",
       "1203                 The Big Blue      0.0     1988.0            168   \n",
       "1389                       Scream      0.0     1996.0            111   \n",
       "1390                       Scream      0.0     1996.0            NaN   \n",
       "2186               My Blue Heaven      0.0     1990.0             97   \n",
       "2187               My Blue Heaven      0.0     1990.0             95   \n",
       "2190                         Hero      0.0     1992.0            119   \n",
       "2191                         Hero      0.0     1992.0            NaN   \n",
       "2261                      Soldier      0.0     1998.0             99   \n",
       "2262                      Soldier      0.0     1998.0            156   \n",
       "2449            Eight Days a Week      0.0     1997.0             92   \n",
       "2450            Eight Days a Week      0.0     1997.0             45   \n",
       "2534                  The Monster      0.0     1994.0            112   \n",
       "2535                  The Monster      0.0     1994.0            132   \n",
       "2624                        Limbo      0.0     1999.0            126   \n",
       "2625                        Limbo      0.0     1999.0             58   \n",
       "2633             An Ideal Husband      0.0     1999.0             97   \n",
       "2634             An Ideal Husband      0.0     1999.0             90   \n",
       "2764                       Hamlet      0.0     1964.0            140   \n",
       "2765                       Hamlet      0.0     1964.0            120   \n",
       "2766                       Hamlet      0.0     1964.0            191   \n",
       "2767                       Hamlet      0.0     1964.0            NaN   \n",
       "2955                   Last Night      0.0     1998.0             95   \n",
       "2956                   Last Night      0.0     1998.0            NaN   \n",
       "3065          Ride with the Devil      0.0     1999.0            138   \n",
       "3066          Ride with the Devil      0.0     1999.0             88   \n",
       "3234                    The Beach      0.0     2000.0            119   \n",
       "3235                    The Beach      0.0     2000.0            NaN   \n",
       "3503                       Gossip      0.0     2000.0             90   \n",
       "3504                       Gossip      0.0     2000.0            134   \n",
       "3708                       Asylum      0.0     1972.0             88   \n",
       "3709                       Asylum      0.0     1972.0             95   \n",
       "3888                      Runaway      0.0     1984.0             99   \n",
       "3889                      Runaway      0.0     1984.0            100   \n",
       "\n",
       "                         genres  match_score  \n",
       "47    Adventure,Animation,Drama            1  \n",
       "48                    Animation            1  \n",
       "149     Adventure,Drama,History            1  \n",
       "150                 Documentary            0  \n",
       "296                       Drama            1  \n",
       "297                      Action            0  \n",
       "776      Animation,Drama,Family            1  \n",
       "777                   Animation            1  \n",
       "1202                      Drama            0  \n",
       "1203      Adventure,Drama,Sport            1  \n",
       "1389             Horror,Mystery            1  \n",
       "1390                Documentary            0  \n",
       "2186               Comedy,Crime            1  \n",
       "2187                      Drama            0  \n",
       "2190       Comedy,Drama,Romance            2  \n",
       "2191            Musical,Romance            0  \n",
       "2261        Action,Drama,Sci-Fi            2  \n",
       "2262       Action,Drama,Musical            1  \n",
       "2449             Comedy,Romance            1  \n",
       "2450                Documentary            0  \n",
       "2534       Comedy,Crime,Mystery            1  \n",
       "2535              Comedy,Horror            1  \n",
       "2624   Adventure,Drama,Thriller            1  \n",
       "2625                     Horror            0  \n",
       "2633             Comedy,Romance            1  \n",
       "2634             Comedy,Romance            1  \n",
       "2764                      Drama            1  \n",
       "2765                      Drama            1  \n",
       "2766                      Drama            1  \n",
       "2767                      Drama            1  \n",
       "2955       Comedy,Drama,Romance            0  \n",
       "2956                        NaN            0  \n",
       "3065          Drama,Romance,War            3  \n",
       "3066                     Horror            0  \n",
       "3234    Adventure,Drama,Romance            2  \n",
       "3235                Documentary            0  \n",
       "3503     Drama,Mystery,Thriller            2  \n",
       "3504               Comedy,Drama            1  \n",
       "3708                     Horror            1  \n",
       "3709                Documentary            0  \n",
       "3888        Action,Crime,Sci-Fi            1  \n",
       "3889                      Drama            0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "value_counts = df_movies['MovieId'].value_counts()\n",
    "duplicated_values = value_counts[value_counts > 1]\n",
    "\n",
    "print('Duplicated values and their counts:')\n",
    "duplicated_indices = df_movies[df_movies.duplicated(subset=['MovieId'], keep=False)].index\n",
    "duplicated_values_id = df_movies.loc[duplicated_indices]\n",
    "duplicated_values_id['match_score'] = duplicated_values_id.apply(lambda row: genre_match_score(row['Genres'], row['genres']), axis=1)\n",
    "duplicated_values_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataframe shows the duplicated `MovieId` that needs to be processed in order to retrieve only one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on matching scores, index to delete are computed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  48  150  297  777 1202 1390 2187 2191 2262 2450 2535 2625 2634 2765\n",
      " 2766 2767 2956 3066 3235 3504 3709 3889]\n"
     ]
    }
   ],
   "source": [
    "# Retrieve the rows with the highest matching genres\n",
    "id_to_keep = duplicated_values_id.groupby('MovieId')['match_score'].idxmax()\n",
    "\n",
    "# Retrieve the id of the rows to delete\n",
    "id_to_delete = duplicated_values_id.index.difference(id_to_keep)\n",
    "id_to_delete = np.array(id_to_delete)\n",
    "print(id_to_delete)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>tconst</th>\n",
       "      <th>primaryTitle</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>startYear</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>81</td>\n",
       "      <td>Adventure,Animation,Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>104</td>\n",
       "      <td>Adventure,Comedy,Family</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>101</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>124</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>106</td>\n",
       "      <td>Comedy,Family,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3900</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0212338</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>108</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3901</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0180093</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>102</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3902</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0170691</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>101</td>\n",
       "      <td>Drama,War</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3903</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0202641</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>108</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3904</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>tt0208874</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>126</td>\n",
       "      <td>Drama,Thriller</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title                        Genres  \\\n",
       "0           1                    Toy Story   Animation|Children's|Comedy   \n",
       "1           2                      Jumanji  Adventure|Children's|Fantasy   \n",
       "2           3             Grumpier Old Men                Comedy|Romance   \n",
       "3           4            Waiting to Exhale                  Comedy|Drama   \n",
       "4           5  Father of the Bride Part II                        Comedy   \n",
       "...       ...                          ...                           ...   \n",
       "3900     3948             Meet the Parents                        Comedy   \n",
       "3901     3949          Requiem for a Dream                         Drama   \n",
       "3902     3950                    Tigerland                         Drama   \n",
       "3903     3951             Two Family House                         Drama   \n",
       "3904     3952                The Contender                Drama|Thriller   \n",
       "\n",
       "        Date     tconst                 primaryTitle  isAdult  startYear  \\\n",
       "0     1995.0  tt0114709                    Toy Story      0.0     1995.0   \n",
       "1     1995.0  tt0113497                      Jumanji      0.0     1995.0   \n",
       "2     1995.0  tt0113228             Grumpier Old Men      0.0     1995.0   \n",
       "3     1995.0  tt0114885            Waiting to Exhale      0.0     1995.0   \n",
       "4     1995.0  tt0113041  Father of the Bride Part II      0.0     1995.0   \n",
       "...      ...        ...                          ...      ...        ...   \n",
       "3900  2000.0  tt0212338             Meet the Parents      0.0     2000.0   \n",
       "3901  2000.0  tt0180093          Requiem for a Dream      0.0     2000.0   \n",
       "3902  2000.0  tt0170691                    Tigerland      0.0     2000.0   \n",
       "3903  2000.0  tt0202641             Two Family House      0.0     2000.0   \n",
       "3904  2000.0  tt0208874                The Contender      0.0     2000.0   \n",
       "\n",
       "     runtimeMinutes                      genres  \n",
       "0                81  Adventure,Animation,Comedy  \n",
       "1               104     Adventure,Comedy,Family  \n",
       "2               101              Comedy,Romance  \n",
       "3               124        Comedy,Drama,Romance  \n",
       "4               106       Comedy,Family,Romance  \n",
       "...             ...                         ...  \n",
       "3900            108              Comedy,Romance  \n",
       "3901            102                       Drama  \n",
       "3902            101                   Drama,War  \n",
       "3903            108        Comedy,Drama,Romance  \n",
       "3904            126              Drama,Thriller  \n",
       "\n",
       "[3883 rows x 10 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the match_score column for final output\n",
    "df_movies = df_movies[~df_movies.index.isin(id_to_delete)]\n",
    "df_movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see there is no more duplicated `MovieId`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Duplicated values and their counts:\n",
      "Series([], Name: count, dtype: int64)\n"
     ]
    }
   ],
   "source": [
    "value_counts = df_movies['MovieId'].value_counts()\n",
    "duplicated_values = value_counts[value_counts > 1]\n",
    "\n",
    "print('Duplicated values and their counts:')\n",
    "print(duplicated_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The indexes are finally updated andwe will then clean the dataset by removing columns that aren't usefull anymore, such as:\n",
    "\n",
    "- primaryTitle (data already inside the column `Title`)\n",
    "- startYear (data already inside the column `Date`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_166881/1104113974.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_movies.drop(columns=['index', 'primaryTitle', 'startYear'], inplace=True)\n",
      "/tmp/ipykernel_166881/1104113974.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_movies['Date'] = df_movies['Date'].astype(int)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>tconst</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81</td>\n",
       "      <td>Adventure,Animation,Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104</td>\n",
       "      <td>Adventure,Comedy,Family</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106</td>\n",
       "      <td>Comedy,Family,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0212338</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "      <td>Comedy,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0180093</td>\n",
       "      <td>0.0</td>\n",
       "      <td>102</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0170691</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "      <td>Drama,War</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0202641</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "      <td>Comedy,Drama,Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0208874</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126</td>\n",
       "      <td>Drama,Thriller</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title                        Genres  \\\n",
       "0           1                    Toy Story   Animation|Children's|Comedy   \n",
       "1           2                      Jumanji  Adventure|Children's|Fantasy   \n",
       "2           3             Grumpier Old Men                Comedy|Romance   \n",
       "3           4            Waiting to Exhale                  Comedy|Drama   \n",
       "4           5  Father of the Bride Part II                        Comedy   \n",
       "...       ...                          ...                           ...   \n",
       "3878     3948             Meet the Parents                        Comedy   \n",
       "3879     3949          Requiem for a Dream                         Drama   \n",
       "3880     3950                    Tigerland                         Drama   \n",
       "3881     3951             Two Family House                         Drama   \n",
       "3882     3952                The Contender                Drama|Thriller   \n",
       "\n",
       "      Date     tconst  isAdult runtimeMinutes                      genres  \n",
       "0     1995  tt0114709      0.0             81  Adventure,Animation,Comedy  \n",
       "1     1995  tt0113497      0.0            104     Adventure,Comedy,Family  \n",
       "2     1995  tt0113228      0.0            101              Comedy,Romance  \n",
       "3     1995  tt0114885      0.0            124        Comedy,Drama,Romance  \n",
       "4     1995  tt0113041      0.0            106       Comedy,Family,Romance  \n",
       "...    ...        ...      ...            ...                         ...  \n",
       "3878  2000  tt0212338      0.0            108              Comedy,Romance  \n",
       "3879  2000  tt0180093      0.0            102                       Drama  \n",
       "3880  2000  tt0170691      0.0            101                   Drama,War  \n",
       "3881  2000  tt0202641      0.0            108        Comedy,Drama,Romance  \n",
       "3882  2000  tt0208874      0.0            126              Drama,Thriller  \n",
       "\n",
       "[3883 rows x 8 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_movies.reset_index(inplace=True)\n",
    "df_movies.drop(columns=['index', 'primaryTitle', 'startYear'], inplace=True)\n",
    "df_movies['Date'] = df_movies['Date'].astype(int)\n",
    "df_movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To ensure we capture all possible genres from both datasets, we will merge the genres from MovieLens and IMDb datasets.\n",
    "\n",
    "For example:\n",
    "\n",
    "**Jumanji**\n",
    "- `Genres`: Adventure|Children's|Fantasy\n",
    "- `genres`: Adventure,Comedy,Family\n",
    "\n",
    "Gives the new merged column `Genres`:\n",
    "- `Genres`: Adventure|Children's|Fantasy|Comedy|Family"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def merge_genres(row):\n",
    "    genres_set = set()\n",
    "\n",
    "    if pd.notna(row['Genres']):\n",
    "        genres_set.update(row['Genres'].split('|'))\n",
    "\n",
    "    if pd.notna(row['genres']):\n",
    "        genres_set.update(row['genres'].split(','))\n",
    "\n",
    "    return '|'.join(genres_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_166881/2986576834.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_movies['Genres'] = df_movies.apply(merge_genres, axis=1)\n",
      "/tmp/ipykernel_166881/2986576834.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_movies.drop(columns=['genres'], inplace=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>tconst</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Children's|Animation|Adventure|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Comedy|Fantasy|Children's|Family</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Romance|Drama|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Family|Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0212338</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0180093</td>\n",
       "      <td>0.0</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>War|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0170691</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Comedy|Romance|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0202641</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Thriller|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0208874</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title  \\\n",
       "0           1                    Toy Story   \n",
       "1           2                      Jumanji   \n",
       "2           3             Grumpier Old Men   \n",
       "3           4            Waiting to Exhale   \n",
       "4           5  Father of the Bride Part II   \n",
       "...       ...                          ...   \n",
       "3878     3948             Meet the Parents   \n",
       "3879     3949          Requiem for a Dream   \n",
       "3880     3950                    Tigerland   \n",
       "3881     3951             Two Family House   \n",
       "3882     3952                The Contender   \n",
       "\n",
       "                                          Genres  Date     tconst  isAdult  \\\n",
       "0          Children's|Animation|Adventure|Comedy  1995  tt0114709      0.0   \n",
       "1     Adventure|Comedy|Fantasy|Children's|Family  1995  tt0113497      0.0   \n",
       "2                                 Romance|Comedy  1995  tt0113228      0.0   \n",
       "3                           Romance|Drama|Comedy  1995  tt0114885      0.0   \n",
       "4                          Family|Romance|Comedy  1995  tt0113041      0.0   \n",
       "...                                          ...   ...        ...      ...   \n",
       "3878                              Romance|Comedy  2000  tt0212338      0.0   \n",
       "3879                                       Drama  2000  tt0180093      0.0   \n",
       "3880                                   War|Drama  2000  tt0170691      0.0   \n",
       "3881                        Comedy|Romance|Drama  2000  tt0202641      0.0   \n",
       "3882                              Thriller|Drama  2000  tt0208874      0.0   \n",
       "\n",
       "     runtimeMinutes  \n",
       "0                81  \n",
       "1               104  \n",
       "2               101  \n",
       "3               124  \n",
       "4               106  \n",
       "...             ...  \n",
       "3878            108  \n",
       "3879            102  \n",
       "3880            101  \n",
       "3881            108  \n",
       "3882            126  \n",
       "\n",
       "[3883 rows x 7 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the function to merge genres\n",
    "df_movies['Genres'] = df_movies.apply(merge_genres, axis=1)\n",
    "df_movies.drop(columns=['genres'], inplace=True)\n",
    "df_movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sadly, after all the preprocessing and merging, there are still **639 movies** that didn't get enriched with the IMDb data (mainly due to non-matching names between the datasets)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "632\n"
     ]
    }
   ],
   "source": [
    "value_counts = df_movies['runtimeMinutes'].isna()\n",
    "duplicated_values = value_counts[value_counts > 0]\n",
    "\n",
    "print(len(duplicated_values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: df_movies , Length: 3883\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>tconst</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Children's|Animation|Adventure|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Comedy|Fantasy|Children's|Family</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Romance|Drama|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Family|Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0212338</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0180093</td>\n",
       "      <td>0.0</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>War|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0170691</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Comedy|Romance|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0202641</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Thriller|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0208874</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title  \\\n",
       "0           1                    Toy Story   \n",
       "1           2                      Jumanji   \n",
       "2           3             Grumpier Old Men   \n",
       "3           4            Waiting to Exhale   \n",
       "4           5  Father of the Bride Part II   \n",
       "...       ...                          ...   \n",
       "3878     3948             Meet the Parents   \n",
       "3879     3949          Requiem for a Dream   \n",
       "3880     3950                    Tigerland   \n",
       "3881     3951             Two Family House   \n",
       "3882     3952                The Contender   \n",
       "\n",
       "                                          Genres  Date     tconst  isAdult  \\\n",
       "0          Children's|Animation|Adventure|Comedy  1995  tt0114709      0.0   \n",
       "1     Adventure|Comedy|Fantasy|Children's|Family  1995  tt0113497      0.0   \n",
       "2                                 Romance|Comedy  1995  tt0113228      0.0   \n",
       "3                           Romance|Drama|Comedy  1995  tt0114885      0.0   \n",
       "4                          Family|Romance|Comedy  1995  tt0113041      0.0   \n",
       "...                                          ...   ...        ...      ...   \n",
       "3878                              Romance|Comedy  2000  tt0212338      0.0   \n",
       "3879                                       Drama  2000  tt0180093      0.0   \n",
       "3880                                   War|Drama  2000  tt0170691      0.0   \n",
       "3881                        Comedy|Romance|Drama  2000  tt0202641      0.0   \n",
       "3882                              Thriller|Drama  2000  tt0208874      0.0   \n",
       "\n",
       "     runtimeMinutes  \n",
       "0                81  \n",
       "1               104  \n",
       "2               101  \n",
       "3               124  \n",
       "4               106  \n",
       "...             ...  \n",
       "3878            108  \n",
       "3879            102  \n",
       "3880            101  \n",
       "3881            108  \n",
       "3882            126  \n",
       "\n",
       "[3883 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "get_summary(df_movies, 'df_movies')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now check for NaN values in the DataFrame to ensure data completeness and integrity.\n",
    "We handle missing values in the runtimeMinutes column by filling them with the mean runtime and ensure the isAdult column is filled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_166881/588665147.py:1: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  df_movies['runtimeMinutes'].fillna(100.0, inplace=True)\n",
      "/tmp/ipykernel_166881/588665147.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_movies['runtimeMinutes'].fillna(100.0, inplace=True)\n",
      "/tmp/ipykernel_166881/588665147.py:2: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  df_movies['isAdult'].fillna(0.0, inplace=True)\n",
      "/tmp/ipykernel_166881/588665147.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_movies['isAdult'].fillna(0.0, inplace=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>tconst</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Children's|Animation|Adventure|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Comedy|Fantasy|Children's|Family</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Romance|Drama|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Family|Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0212338</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0180093</td>\n",
       "      <td>0.0</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>War|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0170691</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Comedy|Romance|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0202641</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Thriller|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>tt0208874</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title  \\\n",
       "0           1                    Toy Story   \n",
       "1           2                      Jumanji   \n",
       "2           3             Grumpier Old Men   \n",
       "3           4            Waiting to Exhale   \n",
       "4           5  Father of the Bride Part II   \n",
       "...       ...                          ...   \n",
       "3878     3948             Meet the Parents   \n",
       "3879     3949          Requiem for a Dream   \n",
       "3880     3950                    Tigerland   \n",
       "3881     3951             Two Family House   \n",
       "3882     3952                The Contender   \n",
       "\n",
       "                                          Genres  Date     tconst  isAdult  \\\n",
       "0          Children's|Animation|Adventure|Comedy  1995  tt0114709      0.0   \n",
       "1     Adventure|Comedy|Fantasy|Children's|Family  1995  tt0113497      0.0   \n",
       "2                                 Romance|Comedy  1995  tt0113228      0.0   \n",
       "3                           Romance|Drama|Comedy  1995  tt0114885      0.0   \n",
       "4                          Family|Romance|Comedy  1995  tt0113041      0.0   \n",
       "...                                          ...   ...        ...      ...   \n",
       "3878                              Romance|Comedy  2000  tt0212338      0.0   \n",
       "3879                                       Drama  2000  tt0180093      0.0   \n",
       "3880                                   War|Drama  2000  tt0170691      0.0   \n",
       "3881                        Comedy|Romance|Drama  2000  tt0202641      0.0   \n",
       "3882                              Thriller|Drama  2000  tt0208874      0.0   \n",
       "\n",
       "     runtimeMinutes  \n",
       "0                81  \n",
       "1               104  \n",
       "2               101  \n",
       "3               124  \n",
       "4               106  \n",
       "...             ...  \n",
       "3878            108  \n",
       "3879            102  \n",
       "3880            101  \n",
       "3881            108  \n",
       "3882            126  \n",
       "\n",
       "[3883 rows x 7 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_movies['runtimeMinutes'].fillna(100.0, inplace=True)\n",
    "df_movies['isAdult'].fillna(0.0, inplace=True)\n",
    "df_movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### imdb_name_basics\n",
    "\n",
    "For the preprocessing of the imdb_name_basics, we will clean the dataset by removing unnecessary columns, such as:\n",
    "\n",
    "- primaryProfession\n",
    "- birthYear\n",
    "- deathYear\n",
    "- nconst\n",
    "\n",
    "These columns are either not relevant to our analysis or contain redundant information. By removing these columns, we simplify our dataset, making it easier to work with and focus on the essential information for our analysis.\n",
    "\n",
    "We will then create a dictionary mapping each IMDb title (tconst) inside **df_movies** to the primary names of actors and directors associated with that title. Then, we will add this information to our movie dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: imdb_name_basics , Length: 12106391\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primaryName</th>\n",
       "      <th>knownForTitles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fred Astaire</td>\n",
       "      <td>tt0072308,tt0050419,tt0053137,tt0027125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lauren Bacall</td>\n",
       "      <td>tt0037382,tt0075213,tt0117057,tt0038355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brigitte Bardot</td>\n",
       "      <td>tt0057345,tt0049189,tt0056404,tt0054452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>John Belushi</td>\n",
       "      <td>tt0072562,tt0077975,tt0080455,tt0078723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ingmar Bergman</td>\n",
       "      <td>tt0050986,tt0083922,tt0050976,tt0069467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616639</th>\n",
       "      <td>Lu Bevins</td>\n",
       "      <td>tt17717854,tt11772904,tt11772812,tt11697102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616643</th>\n",
       "      <td>Sambit Mishra</td>\n",
       "      <td>tt20319332,tt27191658,tt10709066,tt15134202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616644</th>\n",
       "      <td>Romeo del Rosario</td>\n",
       "      <td>tt11657662,tt14069590,tt2455546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616646</th>\n",
       "      <td>Harikrishnan Rajan</td>\n",
       "      <td>tt8736744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13616647</th>\n",
       "      <td>Aayush Nair</td>\n",
       "      <td>tt8736744</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12106391 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 primaryName                               knownForTitles\n",
       "0               Fred Astaire      tt0072308,tt0050419,tt0053137,tt0027125\n",
       "1              Lauren Bacall      tt0037382,tt0075213,tt0117057,tt0038355\n",
       "2            Brigitte Bardot      tt0057345,tt0049189,tt0056404,tt0054452\n",
       "3               John Belushi      tt0072562,tt0077975,tt0080455,tt0078723\n",
       "4             Ingmar Bergman      tt0050986,tt0083922,tt0050976,tt0069467\n",
       "...                      ...                                          ...\n",
       "13616639           Lu Bevins  tt17717854,tt11772904,tt11772812,tt11697102\n",
       "13616643       Sambit Mishra  tt20319332,tt27191658,tt10709066,tt15134202\n",
       "13616644   Romeo del Rosario              tt11657662,tt14069590,tt2455546\n",
       "13616646  Harikrishnan Rajan                                    tt8736744\n",
       "13616647         Aayush Nair                                    tt8736744\n",
       "\n",
       "[12106391 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "imdb_name_basics.drop(columns=['primaryProfession', 'birthYear', 'deathYear', 'nconst'], inplace=True)\n",
    "imdb_name_basics.dropna(inplace=True)\n",
    "get_summary(imdb_name_basics, 'imdb_name_basics')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering\n",
    "\n",
    "- Extract relevant features\n",
    "- Create a user-item interaction matrix for collaborative filtering.\n",
    "- Understand how to combine user data to get data for the couple of users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dictionary where keys are tconst and values are lists of primary names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12106391/12106391 [06:14<00:00, 32289.81it/s]\n"
     ]
    }
   ],
   "source": [
    "# Create a dictionary where keys are tconst and values are lists of primary names\n",
    "from tqdm import tqdm\n",
    "\n",
    "tconst_to_names = {}\n",
    "for _, row in tqdm(imdb_name_basics.iterrows(), total=imdb_name_basics.shape[0]):\n",
    "    titles = row['knownForTitles'].split(',')\n",
    "    for title in titles:\n",
    "        if title not in tconst_to_names:\n",
    "            tconst_to_names[title] = []\n",
    "        tconst_to_names[title].append(row['primaryName'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `cast` feature encompasses all individuals involved in a movie's production, including actors, directors, producers, and other key contributors. By integrating cast information into our dataset, we can improve our analysis and recommendations, as the cast (in the case it is used) can impact a movie's popularity and user preferences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_166881/223325489.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df_movies['cast'] = df_movies['tconst'].map(lambda x: ','.join(tconst_to_names.get(x, [])))\n"
     ]
    }
   ],
   "source": [
    "# Add a new column to df_movies with the primary names\n",
    "df_movies['cast'] = df_movies['tconst'].map(lambda x: ','.join(tconst_to_names.get(x, [])))\n",
    "df_movies = df_movies.assign(cast=df_movies['cast'].str.split(','))\n",
    "df_movies.drop(columns='tconst', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>cast</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Children's|Animation|Adventure|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81</td>\n",
       "      <td>[Tim Allen, John Ratzenberger, Jim Varney, Joh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Comedy|Fantasy|Children's|Family</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104</td>\n",
       "      <td>[Bonnie Hunt, Bebe Neuwirth, Adam Hann-Byrd, R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "      <td>[James Andelin, Richard C. Berman, Les Brown, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Romance|Drama|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124</td>\n",
       "      <td>[Whitney Houston, Kenneth 'Babyface' Edmonds, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Family|Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106</td>\n",
       "      <td>[Rodrigo Botero, William Akey, Sam Anderson, D...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "      <td>[Blythe Danner, Teri Polo, Tom Houghton, Jon A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>102</td>\n",
       "      <td>[Jennifer Connelly, Ellen Burstyn, Jared Leto,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>War|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101</td>\n",
       "      <td>[Karolyn Arnold, Arian Ash, Rhynell Brumfield,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Comedy|Romance|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108</td>\n",
       "      <td>[Jeffrey L. Baker, Sally Bonython, Adam Carr, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Thriller|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126</td>\n",
       "      <td>[Arno Andrischok, Bev Appleton, Ken Arrington,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title  \\\n",
       "0           1                    Toy Story   \n",
       "1           2                      Jumanji   \n",
       "2           3             Grumpier Old Men   \n",
       "3           4            Waiting to Exhale   \n",
       "4           5  Father of the Bride Part II   \n",
       "...       ...                          ...   \n",
       "3878     3948             Meet the Parents   \n",
       "3879     3949          Requiem for a Dream   \n",
       "3880     3950                    Tigerland   \n",
       "3881     3951             Two Family House   \n",
       "3882     3952                The Contender   \n",
       "\n",
       "                                          Genres  Date  isAdult  \\\n",
       "0          Children's|Animation|Adventure|Comedy  1995      0.0   \n",
       "1     Adventure|Comedy|Fantasy|Children's|Family  1995      0.0   \n",
       "2                                 Romance|Comedy  1995      0.0   \n",
       "3                           Romance|Drama|Comedy  1995      0.0   \n",
       "4                          Family|Romance|Comedy  1995      0.0   \n",
       "...                                          ...   ...      ...   \n",
       "3878                              Romance|Comedy  2000      0.0   \n",
       "3879                                       Drama  2000      0.0   \n",
       "3880                                   War|Drama  2000      0.0   \n",
       "3881                        Comedy|Romance|Drama  2000      0.0   \n",
       "3882                              Thriller|Drama  2000      0.0   \n",
       "\n",
       "     runtimeMinutes                                               cast  \n",
       "0                81  [Tim Allen, John Ratzenberger, Jim Varney, Joh...  \n",
       "1               104  [Bonnie Hunt, Bebe Neuwirth, Adam Hann-Byrd, R...  \n",
       "2               101  [James Andelin, Richard C. Berman, Les Brown, ...  \n",
       "3               124  [Whitney Houston, Kenneth 'Babyface' Edmonds, ...  \n",
       "4               106  [Rodrigo Botero, William Akey, Sam Anderson, D...  \n",
       "...             ...                                                ...  \n",
       "3878            108  [Blythe Danner, Teri Polo, Tom Houghton, Jon A...  \n",
       "3879            102  [Jennifer Connelly, Ellen Burstyn, Jared Leto,...  \n",
       "3880            101  [Karolyn Arnold, Arian Ash, Rhynell Brumfield,...  \n",
       "3881            108  [Jeffrey L. Baker, Sally Bonython, Adam Carr, ...  \n",
       "3882            126  [Arno Andrischok, Bev Appleton, Ken Arrington,...  \n",
       "\n",
       "[3883 rows x 7 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make my work easier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "csv_file_path = 'preprocessed_movies.csv'\n",
    "df_movies.to_csv(csv_file_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Date</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>cast</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Children's|Animation|Adventure|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>['Tim Allen', 'John Ratzenberger', 'Jim Varney...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Adventure|Comedy|Fantasy|Children's|Family</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>['Bonnie Hunt', 'Bebe Neuwirth', 'Adam Hann-By...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>['James Andelin', 'Richard C. Berman', 'Les Br...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Romance|Drama|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>['Whitney Houston', \"Kenneth 'Babyface' Edmond...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Family|Romance|Comedy</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>['Rodrigo Botero', 'William Akey', 'Sam Anders...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>Romance|Comedy</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>['Blythe Danner', 'Teri Polo', 'Tom Houghton',...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>['Jennifer Connelly', 'Ellen Burstyn', 'Jared ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>War|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>['Karolyn Arnold', 'Arian Ash', 'Rhynell Brumf...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>Comedy|Romance|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>['Jeffrey L. Baker', 'Sally Bonython', 'Adam C...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>Thriller|Drama</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>['Arno Andrischok', 'Bev Appleton', 'Ken Arrin...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title  \\\n",
       "0           1                    Toy Story   \n",
       "1           2                      Jumanji   \n",
       "2           3             Grumpier Old Men   \n",
       "3           4            Waiting to Exhale   \n",
       "4           5  Father of the Bride Part II   \n",
       "...       ...                          ...   \n",
       "3878     3948             Meet the Parents   \n",
       "3879     3949          Requiem for a Dream   \n",
       "3880     3950                    Tigerland   \n",
       "3881     3951             Two Family House   \n",
       "3882     3952                The Contender   \n",
       "\n",
       "                                          Genres  Date  isAdult  \\\n",
       "0          Children's|Animation|Adventure|Comedy  1995      0.0   \n",
       "1     Adventure|Comedy|Fantasy|Children's|Family  1995      0.0   \n",
       "2                                 Romance|Comedy  1995      0.0   \n",
       "3                           Romance|Drama|Comedy  1995      0.0   \n",
       "4                          Family|Romance|Comedy  1995      0.0   \n",
       "...                                          ...   ...      ...   \n",
       "3878                              Romance|Comedy  2000      0.0   \n",
       "3879                                       Drama  2000      0.0   \n",
       "3880                                   War|Drama  2000      0.0   \n",
       "3881                        Comedy|Romance|Drama  2000      0.0   \n",
       "3882                              Thriller|Drama  2000      0.0   \n",
       "\n",
       "      runtimeMinutes                                               cast  \n",
       "0               81.0  ['Tim Allen', 'John Ratzenberger', 'Jim Varney...  \n",
       "1              104.0  ['Bonnie Hunt', 'Bebe Neuwirth', 'Adam Hann-By...  \n",
       "2              101.0  ['James Andelin', 'Richard C. Berman', 'Les Br...  \n",
       "3              124.0  ['Whitney Houston', \"Kenneth 'Babyface' Edmond...  \n",
       "4              106.0  ['Rodrigo Botero', 'William Akey', 'Sam Anders...  \n",
       "...              ...                                                ...  \n",
       "3878           108.0  ['Blythe Danner', 'Teri Polo', 'Tom Houghton',...  \n",
       "3879           102.0  ['Jennifer Connelly', 'Ellen Burstyn', 'Jared ...  \n",
       "3880           101.0  ['Karolyn Arnold', 'Arian Ash', 'Rhynell Brumf...  \n",
       "3881           108.0  ['Jeffrey L. Baker', 'Sally Bonython', 'Adam C...  \n",
       "3882           126.0  ['Arno Andrischok', 'Bev Appleton', 'Ken Arrin...  \n",
       "\n",
       "[3883 rows x 7 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_file_path = 'preprocessed_movies.csv'\n",
    "df_movies = pd.read_csv(csv_file_path, sep=',', engine='python')\n",
    "df_movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now free some memory by deleting no longer useful heavy dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "del imdb_title_basics\n",
    "del imdb_name_basics\n",
    "del tconst_to_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To allow easier analysis and manipulation of genre data, and to enable one-hot encoding for machine learning models, we will transform the Genres column into multiple binary columns, each representing a specific genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieId</th>\n",
       "      <th>Title</th>\n",
       "      <th>Date</th>\n",
       "      <th>isAdult</th>\n",
       "      <th>runtimeMinutes</th>\n",
       "      <th>cast</th>\n",
       "      <th>Musical</th>\n",
       "      <th>Thriller</th>\n",
       "      <th>Adventure</th>\n",
       "      <th>War</th>\n",
       "      <th>...</th>\n",
       "      <th>History</th>\n",
       "      <th>Family</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Animation</th>\n",
       "      <th>Fantasy</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Sci-Fi</th>\n",
       "      <th>Film-Noir</th>\n",
       "      <th>Biography</th>\n",
       "      <th>Mystery</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>['Tim Allen', 'John Ratzenberger', 'Jim Varney...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>['Bonnie Hunt', 'Bebe Neuwirth', 'Adam Hann-By...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>['James Andelin', 'Richard C. Berman', 'Les Br...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>['Whitney Houston', \"Kenneth 'Babyface' Edmond...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>1995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>['Rodrigo Botero', 'William Akey', 'Sam Anders...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>3948</td>\n",
       "      <td>Meet the Parents</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>['Blythe Danner', 'Teri Polo', 'Tom Houghton',...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3879</th>\n",
       "      <td>3949</td>\n",
       "      <td>Requiem for a Dream</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>['Jennifer Connelly', 'Ellen Burstyn', 'Jared ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>3950</td>\n",
       "      <td>Tigerland</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>['Karolyn Arnold', 'Arian Ash', 'Rhynell Brumf...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3881</th>\n",
       "      <td>3951</td>\n",
       "      <td>Two Family House</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>['Jeffrey L. Baker', 'Sally Bonython', 'Adam C...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>3952</td>\n",
       "      <td>The Contender</td>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>['Arno Andrischok', 'Bev Appleton', 'Ken Arrin...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3883 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MovieId                        Title  Date  isAdult  runtimeMinutes  \\\n",
       "0           1                    Toy Story  1995      0.0            81.0   \n",
       "1           2                      Jumanji  1995      0.0           104.0   \n",
       "2           3             Grumpier Old Men  1995      0.0           101.0   \n",
       "3           4            Waiting to Exhale  1995      0.0           124.0   \n",
       "4           5  Father of the Bride Part II  1995      0.0           106.0   \n",
       "...       ...                          ...   ...      ...             ...   \n",
       "3878     3948             Meet the Parents  2000      0.0           108.0   \n",
       "3879     3949          Requiem for a Dream  2000      0.0           102.0   \n",
       "3880     3950                    Tigerland  2000      0.0           101.0   \n",
       "3881     3951             Two Family House  2000      0.0           108.0   \n",
       "3882     3952                The Contender  2000      0.0           126.0   \n",
       "\n",
       "                                                   cast  Musical  Thriller  \\\n",
       "0     ['Tim Allen', 'John Ratzenberger', 'Jim Varney...        0         0   \n",
       "1     ['Bonnie Hunt', 'Bebe Neuwirth', 'Adam Hann-By...        0         0   \n",
       "2     ['James Andelin', 'Richard C. Berman', 'Les Br...        0         0   \n",
       "3     ['Whitney Houston', \"Kenneth 'Babyface' Edmond...        0         0   \n",
       "4     ['Rodrigo Botero', 'William Akey', 'Sam Anders...        0         0   \n",
       "...                                                 ...      ...       ...   \n",
       "3878  ['Blythe Danner', 'Teri Polo', 'Tom Houghton',...        0         0   \n",
       "3879  ['Jennifer Connelly', 'Ellen Burstyn', 'Jared ...        0         0   \n",
       "3880  ['Karolyn Arnold', 'Arian Ash', 'Rhynell Brumf...        0         0   \n",
       "3881  ['Jeffrey L. Baker', 'Sally Bonython', 'Adam C...        0         0   \n",
       "3882  ['Arno Andrischok', 'Bev Appleton', 'Ken Arrin...        0         1   \n",
       "\n",
       "      Adventure  War  ...  History  Family  Romance  Animation  Fantasy  \\\n",
       "0             1    0  ...        0       0        0          1        0   \n",
       "1             1    0  ...        0       1        0          0        1   \n",
       "2             0    0  ...        0       0        1          0        0   \n",
       "3             0    0  ...        0       0        1          0        0   \n",
       "4             0    0  ...        0       1        1          0        0   \n",
       "...         ...  ...  ...      ...     ...      ...        ...      ...   \n",
       "3878          0    0  ...        0       0        1          0        0   \n",
       "3879          0    0  ...        0       0        0          0        0   \n",
       "3880          0    1  ...        0       0        0          0        0   \n",
       "3881          0    0  ...        0       0        1          0        0   \n",
       "3882          0    0  ...        0       0        0          0        0   \n",
       "\n",
       "      Drama  Sci-Fi  Film-Noir  Biography  Mystery  \n",
       "0         0       0          0          0        0  \n",
       "1         0       0          0          0        0  \n",
       "2         0       0          0          0        0  \n",
       "3         1       0          0          0        0  \n",
       "4         0       0          0          0        0  \n",
       "...     ...     ...        ...        ...      ...  \n",
       "3878      0       0          0          0        0  \n",
       "3879      1       0          0          0        0  \n",
       "3880      1       0          0          0        0  \n",
       "3881      1       0          0          0        0  \n",
       "3882      1       0          0          0        0  \n",
       "\n",
       "[3883 rows x 29 columns]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_movies['Genres'] = df_movies['Genres'].str.split('|')\n",
    "all_genres = set(genre for sublist in df_movies['Genres'] for genre in sublist)\n",
    "\n",
    "# Create new columns for each genre\n",
    "for genre in all_genres:\n",
    "    df_movies[genre] = df_movies['Genres'].apply(lambda x: 1 if genre in x else 0)\n",
    "\n",
    "df_movies.drop(columns=['Genres'], inplace=True)\n",
    "df_movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_movies['runtimeMinutes'] = df_movies['runtimeMinutes'].astype(float)\n",
    "df_movies['cast'] = df_movies['cast'].astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ratings Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>MovieId</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>3943</th>\n",
       "      <th>3944</th>\n",
       "      <th>3945</th>\n",
       "      <th>3946</th>\n",
       "      <th>3947</th>\n",
       "      <th>3948</th>\n",
       "      <th>3949</th>\n",
       "      <th>3950</th>\n",
       "      <th>3951</th>\n",
       "      <th>3952</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UserId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6036</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6037</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6038</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6039</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6040</th>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6040 rows × 3706 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "MovieId  1     2     3     4     5     6     7     8     9     10    ...  \\\n",
       "UserId                                                               ...   \n",
       "1         5.0   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "2         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "3         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "4         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "5         NaN   NaN   NaN   NaN   NaN   2.0   NaN   NaN   NaN   NaN  ...   \n",
       "...       ...   ...   ...   ...   ...   ...   ...   ...   ...   ...  ...   \n",
       "6036      NaN   NaN   NaN   2.0   NaN   3.0   NaN   NaN   NaN   NaN  ...   \n",
       "6037      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "6038      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "6039      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "6040      3.0   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  ...   \n",
       "\n",
       "MovieId  3943  3944  3945  3946  3947  3948  3949  3950  3951  3952  \n",
       "UserId                                                               \n",
       "1         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "2         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "3         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "4         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "5         NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "...       ...   ...   ...   ...   ...   ...   ...   ...   ...   ...  \n",
       "6036      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "6037      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "6038      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "6039      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "6040      NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN   NaN  \n",
       "\n",
       "[6040 rows x 3706 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interaction_matrix = df_ratings.pivot(index='UserId', columns='MovieId', values='Rating')\n",
    "interaction_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to combine collaborative filtering with content-based filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Development\n",
    "\n",
    "Implement a recommender system algorithm (e.g., matrix factorization) to predict the rating of a movie by a couple of users.\n",
    "\n",
    "**The initial objective was to build an ALS model and integrate it with an NNEncoder or Bayesian Matrix Factorization (in order to fully uses dataframes). However, due to time constraints and issues with poor recommendations (refer to deprecated notebook), I decided to utilize only the ALS model.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql import functions as F\n",
    "from pyspark.sql.functions import col, explode\n",
    "\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "from pyspark.ml.recommendation import ALS\n",
    "\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ALS (Alternating Least Squares)\n",
    "\n",
    "I chosed the ALS (Alternating Least Squares) algorithm due to its effectiveness in handling large-scale collaborative filtering problems (with the usage of Spark).\n",
    "\n",
    "### Why did I chose ALS?\n",
    "\n",
    "- **Efficient**: Dfficient collaborative filtering algorithm.\n",
    "- **Parallel Computation**: Can be parallelized for faster computation in distributed environments.\n",
    "- **Scalability**: Efficiently handles large datasets, suitable for the current needs.\n",
    "- **Simplicity**: Easy to implement and understand compared to more complex algorithms.\n",
    "\n",
    "**As seen in the course:**\n",
    "\n",
    "### ALS Pros\n",
    "\n",
    "- **Flexibility**: Supports both explicit and implicit feedback.\n",
    "- **Performance**: Good prediction accuracy and computational efficiency, especially on large datasets.\n",
    "- **Convergence**: Guarantees convergence to a local minimum, ensuring stable results.\n",
    "- **Simplicity**: Easy to implement and tune, with a well-understood optimization process.\n",
    "\n",
    "### ALS Cons\n",
    "\n",
    "- **Cold Start Problem**: Struggles with new users or items with few interactions.\n",
    "- **Local Minima**: Convergence to a local, not global, minimum can affect recommendation quality.\n",
    "- **Overfitting**: Requires careful tuning of hyperparameters to avoid overfitting.\n",
    "- **Sparsity**: Extremely sparse data can reduce recommendation accuracy.\n",
    "\n",
    "Overall, ALS is a robust choice, balancing ease of implementation, scalability, and performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/07/07 20:22:33 WARN Utils: Your hostname, alex-ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.1.32 instead (on interface wlp4s0)\n",
      "24/07/07 20:22:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "24/07/07 20:22:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
     ]
    }
   ],
   "source": [
    "spark = SparkSession.builder.appName(\"ALSMatrixFactorisation\").getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convertion of Pandas dataframe to Spark dataframes (Spark RDD with schemas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/07/07 20:22:37 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n",
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------------------+----+-------+--------------+--------------------+-------+--------+---------+---+-------+-----+-----------+-----+-----+------+------+------+----------+-------+------+-------+---------+-------+-----+------+---------+---------+-------+\n",
      "|MovieId|               Title|Date|isAdult|runtimeMinutes|                cast|Musical|Thriller|Adventure|War|Western|Sport|Documentary|Crime|Music|Comedy|Horror|Action|Children's|History|Family|Romance|Animation|Fantasy|Drama|Sci-Fi|Film-Noir|Biography|Mystery|\n",
      "+-------+--------------------+----+-------+--------------+--------------------+-------+--------+---------+---+-------+-----+-----------+-----+-----+------+------+------+----------+-------+------+-------+---------+-------+-----+------+---------+---------+-------+\n",
      "|      1|           Toy Story|1995|    0.0|          81.0|['Tim Allen', 'Jo...|      0|       0|        1|  0|      0|    0|          0|    0|    0|     1|     0|     0|         1|      0|     0|      0|        1|      0|    0|     0|        0|        0|      0|\n",
      "|      2|             Jumanji|1995|    0.0|         104.0|['Bonnie Hunt', '...|      0|       0|        1|  0|      0|    0|          0|    0|    0|     1|     0|     0|         1|      0|     1|      0|        0|      1|    0|     0|        0|        0|      0|\n",
      "|      3|    Grumpier Old Men|1995|    0.0|         101.0|['James Andelin',...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     1|     0|     0|         0|      0|     0|      1|        0|      0|    0|     0|        0|        0|      0|\n",
      "|      4|   Waiting to Exhale|1995|    0.0|         124.0|['Whitney Houston...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     1|     0|     0|         0|      0|     0|      1|        0|      0|    1|     0|        0|        0|      0|\n",
      "|      5|Father of the Bri...|1995|    0.0|         106.0|['Rodrigo Botero'...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     1|     0|     0|         0|      0|     1|      1|        0|      0|    0|     0|        0|        0|      0|\n",
      "|      6|                Heat|1995|    0.0|         170.0|['Ashley Judd', '...|      0|       1|        0|  0|      0|    0|          0|    1|    0|     0|     0|     1|         0|      0|     0|      0|        0|      0|    1|     0|        0|        0|      0|\n",
      "|      7|             Sabrina|1995|    0.0|         127.0|['Julia Ormond', ...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     1|     0|     0|         0|      0|     0|      1|        0|      0|    1|     0|        0|        0|      0|\n",
      "|      8|        Tom and Huck|1995|    0.0|          97.0|['Jonathan Taylor...|      0|       0|        1|  0|      0|    0|          0|    0|    0|     1|     0|     0|         1|      0|     0|      0|        0|      0|    1|     0|        0|        0|      0|\n",
      "|      9|        Sudden Death|1995|    0.0|         111.0|['Powers Boothe',...|      0|       1|        0|  0|      0|    0|          0|    1|    0|     0|     0|     1|         0|      0|     0|      0|        0|      0|    0|     0|        0|        0|      0|\n",
      "|     10|           GoldenEye|1995|    0.0|         130.0|['Famke Janssen',...|      0|       1|        1|  0|      0|    0|          0|    0|    0|     0|     0|     1|         0|      0|     0|      0|        0|      0|    0|     0|        0|        0|      0|\n",
      "|     11|The American Pres...|1995|    0.0|         114.0|['David Paymer', ...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     1|     0|     0|         0|      0|     0|      1|        0|      0|    1|     0|        0|        0|      0|\n",
      "|     12|Dracula: Dead and...|1995|    0.0|          88.0|['Lysette Anthony...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     1|     1|     0|         0|      0|     0|      0|        0|      1|    0|     0|        0|        0|      0|\n",
      "|     13|               Balto|1995|    0.0|          78.0|['Jan Richter-Fri...|      0|       0|        1|  0|      0|    0|          0|    0|    0|     0|     0|     0|         1|      0|     0|      0|        1|      0|    1|     0|        0|        0|      0|\n",
      "|     14|               Nixon|1995|    0.0|         192.0|['Anthony Hopkins...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     0|     0|     0|         0|      1|     0|      0|        0|      0|    1|     0|        0|        1|      0|\n",
      "|     15|    Cutthroat Island|1995|    0.0|         124.0|['Matthew Modine'...|      0|       0|        1|  0|      0|    0|          0|    0|    0|     1|     0|     1|         0|      0|     0|      1|        0|      0|    0|     0|        0|        0|      0|\n",
      "|     16|              Casino|1995|    0.0|         178.0|['Sharon Stone', ...|      0|       1|        0|  0|      0|    0|          0|    1|    0|     0|     0|     0|         0|      0|     0|      0|        0|      0|    1|     0|        0|        0|      0|\n",
      "|     17|Sense and Sensibi...|1995|    0.0|         136.0|['Emma Thompson',...|      0|       0|        0|  0|      0|    0|          0|    0|    0|     0|     0|     0|         0|      0|     0|      1|        0|      0|    1|     0|        0|        0|      0|\n",
      "|     18|          Four Rooms|1995|    0.0|          98.0|['Jennifer Beals'...|      0|       1|        0|  0|      0|    0|          0|    0|    0|     1|     0|     0|         0|      0|     0|      0|        0|      0|    0|     0|        0|        0|      0|\n",
      "|     19|Ace Ventura: When...|1995|    0.0|          90.0|['Robert Folk', '...|      0|       0|        1|  0|      0|    0|          0|    1|    0|     1|     0|     0|         0|      0|     0|      0|        0|      0|    0|     0|        0|        0|      0|\n",
      "|     20|         Money Train|1995|    0.0|         110.0|['Gloria Estefan'...|      0|       0|        0|  0|      0|    0|          0|    1|    0|     1|     0|     1|         0|      0|     0|      0|        0|      0|    0|     0|        0|        0|      0|\n",
      "+-------+--------------------+----+-------+--------------+--------------------+-------+--------+---------+---+-------+-----+-----------+-----+-----+------+------+------+----------+-------+------+-------+---------+-------+-----+------+---------+---------+-------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sparkdf_movies = spark.createDataFrame(df_movies)\n",
    "sparkdf_movies.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+------+\n",
      "|UserId|MovieId|Rating|\n",
      "+------+-------+------+\n",
      "|     1|   1193|     5|\n",
      "|     1|    661|     3|\n",
      "|     1|    914|     3|\n",
      "|     1|   3408|     4|\n",
      "|     1|   2355|     5|\n",
      "|     1|   1197|     3|\n",
      "|     1|   1287|     5|\n",
      "|     1|   2804|     5|\n",
      "|     1|    594|     4|\n",
      "|     1|    919|     4|\n",
      "|     1|    595|     5|\n",
      "|     1|    938|     4|\n",
      "|     1|   2398|     4|\n",
      "|     1|   2918|     4|\n",
      "|     1|   1035|     5|\n",
      "|     1|   2791|     4|\n",
      "|     1|   2687|     3|\n",
      "|     1|   2018|     4|\n",
      "|     1|   3105|     5|\n",
      "|     1|   2797|     4|\n",
      "+------+-------+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sparkdf_ratings = spark.createDataFrame(df_ratings)\n",
    "sparkdf_ratings.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Training and Test Dataset\n",
    "\n",
    "We split the user-item dataset with features into training and test sets using an 80-20 ratio with a seed of 42 to ensure reproducibility.\n",
    "\n",
    "We then trained the ALS model with the following parameters: 10 iterations (maxIter=10), regularization parameter of 0.09 (regParam=0.09), rank of 25 (rank=25), user column as \"UserId\" (userCol=\"UserId\"), item column as \"MovieId\" (itemCol=\"MovieId\"), rating column as \"Rating\" (ratingCol=\"Rating\"), cold start strategy as \"drop\" (coldStartStrategy=\"drop\"), and nonnegative constraints enabled (nonnegative=True)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "(training, test) = sparkdf_ratings.randomSplit([0.8, 0.2], seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "als = ALS(\n",
    "    maxIter=10,\n",
    "    regParam=0.09,\n",
    "    rank=25,\n",
    "    userCol=\"UserId\",\n",
    "    itemCol=\"MovieId\",\n",
    "    ratingCol=\"Rating\",\n",
    "    coldStartStrategy=\"drop\",\n",
    "    nonnegative=True,\n",
    ")\n",
    "\n",
    "als_model = als.fit(training)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the Model\n",
    "\n",
    "To evaluate the performance of our collaborative filtering model, we use the `RegressionEvaluator` class from the `pyspark.ml.evaluation` module. This is appropriate since we are dealing with a regression problem.\n",
    "\n",
    "By specifying the metric name and the label column, we can effectively measure how well our model is performing. For this evaluation, we choose RMSE (Root Mean Squared Error) as our metric.\n",
    "\n",
    ">Lower RMSE values signify better model performance, indicating that the predictions are closer to the actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "evaluator = RegressionEvaluator(\n",
    "    metricName=\"rmse\", labelCol=\"Rating\", predictionCol=\"prediction\"\n",
    ")\n",
    "\n",
    "predictions = als_model.transform(test)\n",
    "rmse = evaluator.evaluate(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE=0.8644971827501335\n",
      "+------+-------+------+----------+\n",
      "|UserId|MovieId|Rating|prediction|\n",
      "+------+-------+------+----------+\n",
      "|   148|     11|     5| 4.1102242|\n",
      "|   148|     17|     4|  3.742461|\n",
      "|   148|    107|     4|  3.472805|\n",
      "|   148|    165|     3| 3.9436054|\n",
      "|   148|    185|     3| 3.5168533|\n",
      "|   148|    258|     3| 3.1584296|\n",
      "|   148|    293|     3| 3.7409556|\n",
      "|   148|    300|     2| 3.5762565|\n",
      "|   148|    329|     3|  3.627264|\n",
      "|   148|    361|     4| 3.9254122|\n",
      "|   148|    368|     5| 3.8833969|\n",
      "|   148|    370|     4| 3.2340643|\n",
      "|   148|    376|     4| 3.6962354|\n",
      "|   148|    379|     5|  3.410802|\n",
      "|   148|    426|     4| 3.1038935|\n",
      "|   148|    455|     4| 3.2562442|\n",
      "|   148|    480|     5|  4.203467|\n",
      "|   148|    502|     3| 2.6015346|\n",
      "|   148|    519|     3| 2.4774315|\n",
      "|   148|    575|     4| 3.2777834|\n",
      "+------+-------+------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"RMSE=\" + str(rmse))\n",
    "predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 189:========================================>            (77 + 16) / 100]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+--------------------+\n",
      "|UserId|     recommendations|\n",
      "+------+--------------------+\n",
      "|    31|[{1198, 4.4969316...|\n",
      "|    34|[{318, 4.8424253}...|\n",
      "|    53|[{557, 5.7679563}...|\n",
      "|    65|[{2562, 5.0581145...|\n",
      "|    78|[{1851, 4.450576}...|\n",
      "|    85|[{3382, 4.251342}...|\n",
      "|   108|[{2700, 4.3242207...|\n",
      "|   133|[{1725, 5.00006},...|\n",
      "|   137|[{2562, 4.4740252...|\n",
      "|   148|[{2562, 4.855299}...|\n",
      "+------+--------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "user_recs = als_model.recommendForAllUsers(25)\n",
    "user_recs.show(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommendation Algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Providing Recommendations - for 2 Users\n",
    "Let's take two random users, such as User 10 and User 20. The goal is to generate recommendations for all users and then filters these to include only the specified users' recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def give_predictions_from_ids(UserId1, UserId2, model, verbose=False):\n",
    "    users = [UserId1, UserId2]\n",
    "    user_recs = model.recommendForAllUsers(25)\n",
    "\n",
    "    recs_two_users = user_recs.filter(user_recs.UserId.isin(users))\n",
    "    recs_two_users.show(truncate=False) if verbose else None\n",
    "    return recs_two_users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|UserId|recommendations                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|5     |[{557, 4.497085}, {2309, 4.412388}, {854, 4.365153}, {1138, 4.262051}, {1423, 4.261824}, {53, 4.1591277}, {2931, 4.1188335}, {2512, 4.079133}, {668, 4.070599}, {2503, 4.0430436}, {1420, 4.0379033}, {3089, 4.030359}, {670, 4.02029}, {2395, 4.0185575}, {1237, 4.0164294}, {989, 4.016105}, {3416, 4.008739}, {3022, 4.0061536}, {1260, 4.001025}, {3775, 3.9882495}, {718, 3.9844015}, {2997, 3.9814823}, {3012, 3.952671}, {1743, 3.949483}, {3134, 3.9493964}]    |\n",
      "|23    |[{598, 3.9513183}, {2905, 3.8954017}, {1198, 3.8931572}, {1240, 3.8721936}, {2571, 3.8711295}, {787, 3.8679273}, {589, 3.8547904}, {1036, 3.844397}, {1214, 3.8300242}, {1200, 3.8190308}, {2309, 3.810412}, {557, 3.7917666}, {260, 3.7825403}, {2762, 3.777908}, {989, 3.771945}, {3092, 3.7442589}, {296, 3.7396724}, {50, 3.739489}, {1136, 3.7332728}, {593, 3.7303207}, {318, 3.7288766}, {858, 3.724982}, {3421, 3.7068706}, {439, 3.6865442}, {2825, 3.6831636}]|\n",
      "+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DataFrame[UserId: int, recommendations: array<struct<MovieId:int,rating:float>>]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rand_users = [5, 23]\n",
    "give_predictions_from_ids(rand_users[0], rand_users[1], als_model, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def give_recommandations_for_two_users(UserId1, UserId2, model):\n",
    "    recs_two_users = give_predictions_from_ids(UserId1, UserId2, model)\n",
    "\n",
    "    # Explode the recommendations and select relevant columns\n",
    "    recs_two_users = recs_two_users.select(\"UserId\", explode(\"recommendations\").alias(\"recommendation\"))\n",
    "    recs_two_users = recs_two_users.select(\"UserId\", col(\"recommendation.MovieId\").alias(\"MovieId\"), col(\"recommendation.Rating\").alias(\"Rating_Prediction\"))\n",
    "\n",
    "    # Group by MovieId and calculate the average rating prediction\n",
    "    combined_recs = recs_two_users.groupBy(\"MovieId\").agg({\"Rating_Prediction\": \"avg\"})\n",
    "    combined_recs = combined_recs.withColumnRenamed(\"avg(Rating_Prediction)\", \"AVG_Rating_Prediction\")\n",
    "\n",
    "    combined_recs.orderBy(col(\"AVG_Rating_Prediction\").desc())\n",
    "    return combined_recs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 337:==============================================>      (88 + 12) / 100]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+---------------------+\n",
      "|MovieId|AVG_Rating_Prediction|\n",
      "+-------+---------------------+\n",
      "|3089   |4.030358791351318    |\n",
      "|53     |4.159127712249756    |\n",
      "|1743   |3.9494829177856445   |\n",
      "|1260   |4.001025199890137    |\n",
      "|3022   |4.006153583526611    |\n",
      "|3012   |3.9526710510253906   |\n",
      "|1420   |4.037903308868408    |\n",
      "|989    |3.8940250873565674   |\n",
      "|854    |4.365152835845947    |\n",
      "|2512   |4.079133033752441    |\n",
      "|2395   |4.018557548522949    |\n",
      "|2309   |4.11139988899231     |\n",
      "|1423   |4.261824131011963    |\n",
      "|3775   |3.9882495403289795   |\n",
      "|1237   |4.016429424285889    |\n",
      "|2503   |4.043043613433838    |\n",
      "|718    |3.9844014644622803   |\n",
      "|668    |4.07059907913208     |\n",
      "|2931   |4.118833541870117    |\n",
      "|557    |4.144425868988037    |\n",
      "+-------+---------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "ranked_recommendations = give_recommandations_for_two_users(rand_users[0], rand_users[1], als_model)\n",
    "ranked_recommendations.show(truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 412:=============>(99 + 1) / 100][Stage 413:>              (0 + 15) / 16]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|MovieId|Title                                      |Date|AVG_Rating_Prediction|cast                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|3089   |The Bicycle Thief                          |1948|4.030358791351318    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|53     |Lamerica                                   |1994|4.159127712249756    |['Gianni Amelio', 'Franco Piersanti', 'Dritan Alia', 'Samantha Antonnicola', 'Esmeralda Ara', 'Ilir Ara', 'Cristina Casati', 'Mario Cecchi Gori', 'Elinor Cekliqi', 'Shkelqim Daja', 'Carmelo Di Mazzarelli', 'Nikolin Elezi', 'Stefano Forcina', 'Cristina Francioni', 'Altin Garupi', 'Gezim Garupi', 'Fatmir Gjyla', 'Elona Hoti', 'Ismail Kukaj', 'Artan Marina', 'Elina Ndreu', 'Ferit Nutai', 'Gezim Nutai', 'Sefer Pema', 'Andrea Porporati', 'Antonio Privitera', 'Klodian Rakaj', 'Marco Sallustri', 'Idajet Sejdia', 'Alessandro Sermoneta', 'Liliana Subashi', 'Marian Tifozi', 'Saimir Tila', 'Nazzareno Ubertini', 'Nuredin Ujkaj', 'Vojo Kola', 'Arbin Trebicka', 'Nadon Kallulli', 'Alfonso Vicori', 'Massimiliano Gigante', 'Renato Spadolini', 'Bardhyl Plumbi', 'Erdit Gega', 'Kujtim Veshi', 'Skender Basi', 'Lulzim Gogja', 'Roland Arani', 'Atnan Duka', 'Fehim Alliu', 'Massimiliano Tentoni', 'Aldo Peroni', 'Alberto Azzetti', 'Alessandro Romano', 'Abedin Bocka', 'Armand Diko', 'Ilir Vreto', 'Ramazan Tare', 'Kujtim Seidi', 'Clirim Fuga', 'Bujar Kamberi', 'Agron Haxhiu', 'Alessandro Dionosio', 'Maria Camilla Crespi', 'Vladimir Lila', 'Luftar Braçe', 'Xhemal Koçi', 'Ylli Muça', 'Kreshnik Hashorva', 'Michele Pellegrino', 'Giorgio Chierchiè', 'Giuseppe Menegatti', 'Vito Gallo', 'Ubaldo Panunzi', 'Violeta Banushi', 'Michele Lancia', 'Sabri Basha', 'Romano Donati', 'Riccardo Luciano', 'Marco Tota', 'Maurizio Lentini']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "|1743   |Arguing the World                          |1996|3.9494829177856445   |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|1260   |M                                          |1931|4.001025199890137    |['Peter Lorre', 'Fritz Lang', 'Theo Lingen', 'Robert Baberske', 'Fritz Arno Wagner', 'Edvard Grieg', 'Gerhard Bienert', 'Rudolf Blümner', 'Paul Falkenberg', 'Georg John', 'Friedrich Gnaß', 'Gustaf Gründgens', 'Emil Hasler', 'Adolf Jansen', 'Paul Kemp', 'Inge Landgut', 'Theodor Loos', 'Fritz Odemar', 'Karl Platen', 'Ernst Stahl-Nachbaur', 'Franz Stein', 'Rosa Valetti', 'Karl Vollbrecht', 'Horst von Harbou', 'Thea von Harbou', 'Hertha von Walther', 'Otto Wernicke', 'Ellen Widmann', 'Ernst Wolff', 'Wilhelm Weber', 'Lucie Rhoden']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3022   |The General                                |1927|4.006153583526611    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3012   |Battling Butler                            |1926|3.9526710510253906   |['Budd Fine', 'Bert Haines', 'Ed Levy', 'Lex Neal', \"Mary O'Brien\", 'Charles Henry Smith']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "|1420   |Message to Love: The Isle of Wight Festival|1996|4.037903308868408    |['Andy Carchrae', 'Ricki Farr', 'Laurie Horsley', 'John Wellard', 'Benjamin Darling', 'Martin Barre', 'Ric Lee', 'Leo Lyons', 'Chick Churchill', 'Gareth Haywood', 'Clive Bunker', 'John Wilson', 'Richard McCracken', 'Julian Harvey', 'Taste', 'Robert H. Montgomery Jr.', \"Judi O'Brian\", \"Morton 'Tim' Fry\", 'Vincent Villandi', 'Jennifer Cox', 'Colin Richards', 'Tic Ticnor']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|989    |Schlafes Bruder                            |1995|3.8940250873565674   |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|854    |The Ballad of Narayama                     |1958|4.365152835845947    |['Shichirô Fukazawa', 'Danko Ichikawa', 'Keisuke Kinoshita', 'Matsunosuke Nozawa', 'Ryuzo Otani', 'Kotohisa Satsukime', 'Heiji Yoshida', 'Katsuyuki Hattori', 'Toshikazu Sugiyama', 'Genzô Furumiya']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "|2512   |The Ballad of Narayama                     |1982|4.079133033752441    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|2395   |Rushmore                                   |1998|4.018557548522949    |['Bill Murray', 'Seymour Cassel', 'Mason Gamble', 'Michael Cedar', 'Cathie Filian', 'Jason Schwartzman', 'Felix Mendelssohn', 'Mark Mothersbaugh', 'Monica Allen', 'Amanda Anderson', 'Eric Chase Anderson', 'Wes Anderson', 'Adebayo Asabi', 'John Boccaccio', 'David M. Bowen', 'Linda Brattain', 'Aristide Bruant', 'Craig Busch', 'Donny Caicedo', 'Cecilia Cardwell', 'Bob Casale', 'Lydia Cedrone', 'Layne Chaney', 'Cheryl Clarson', 'Dennis Clay', 'Barbara Cohen', 'Lin Coleman', 'Jessica Dailey', 'Paul Desmond', 'Stephen Dignan', 'Virginia Díaz', 'George Farish', 'Francis Fernandez', 'Al Fielder', 'Danny Fine', 'Grant Foerster', 'Steve Fontano', 'Christine C. Fransen', 'Ed Geldart', 'Moira Gill', 'Nicole Gorg', 'Bob Harper', 'Sally J. Harper', 'Terri Harper', 'Jeffrey B. Hartmann', 'Jonathan Hennessey', 'Terry Hubbard', 'John Killalea', 'Ali Ktiri', 'Angela Laux', 'Hae-Joon Lee', 'Stephen Light', 'Tammy Lorkovic', 'Chad Loucks', 'Billy MacTavish', 'Michael Maggart', 'Marietta Marich', 'Melissa Matthies', 'Keith McCawley', 'Ronnie McCawley', 'Thayer McClanahan', 'Stephen McCole', 'Doug Mercer', 'Roy Metcalf', 'Vojislav Mikulic', 'David Moritz', 'Manning Mott', 'Kim Naves', 'Daniel R. Padgett', 'Dipak Pallana', 'Christopher S. Parker', 'Jim Pasque', 'Karen Patch', 'Colin Platt', 'Ella Pryor', 'Ron Radvinsky', 'Van Redin', 'Morgan Redmond', 'Lucille Sadikin', 'David Sanders', 'Antoni Scarano', 'Debra Secher', 'Kevin Sims', 'Cat Stevens', 'J.J. Stonebraker', 'Scott Streetman', 'Chad Stuart', 'Sara Tanaka', 'Brian Tenenbaum', 'Doug Terranova', 'Kim Terry', 'Dalton Tomlin', 'Harry Tracosas', 'Ric Trzeciak', 'Kyle Ryan Urquhart', 'Peter J. Verrando', 'Eric Bennett Weems', 'Melissa Wiechmann', 'Olivia Williams', 'Patricia Winkler', 'Michael R. Wohlfield', 'Mark Van Stone', 'Ted Broden', 'Jamelle Flowers', 'Linell Kathleen Townes', 'Fred Davis', 'Carrie Albers', 'Jason Hawkins', 'Victor Lacour', 'Renee Roth', 'Jennifer Simpson', 'Danping Wong', 'Vicki Daniels Johnson', 'Kalen Hoyle', 'McCauley Penderdast', 'Nancy Doyle', 'Maura Tighe', 'William Lau', 'Ronnie Lane', 'Charles Wesley', 'Robbie Lee', 'Tommy Moeller', 'Brian Parker', 'Unit Four Plus Two', 'The Vince Guaraldi Trio', 'Chad and Jeremy', 'The Creation', 'Eddie Phillips']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "|2309   |The Inheritors                             |1998|4.11139988899231     |['Michaela Berner', 'Stephan Biebl', 'Ruth Blankenstein', 'Martin Dvoracek-Riegler', 'Christine Egger', 'Eddie Fischnaller', 'Joe Fischnaller', 'Stefan Gruber', 'Christoph Gusenbauer', 'Christian Heitler', 'Christine Hermann', 'Marid Hofmann', 'Franz Hollaus', 'Joe Joachim', 'Andreas Kahri', 'Karl Kastner', 'Helga Klein', 'Susanne Kolar', 'Michael Kraxner', 'Michael Pogo Kreiner', 'Otger Kunert', 'Gertraud Mayböck', 'Sigi Mayer', 'Toni Mayer', 'Johann Naderer', 'David Neve', 'Dietmar Nigsch', 'Norbert Perchtold', 'Rebecca Rabche', 'Stefan Ruzowitzky', 'Georgie Schmid-Schmidsfelden', 'Bernhard Schober', 'Kirstin Schwab', 'Simon Schwarz', 'Ernestine Schöllbauer', 'Heino Schöllbauer', 'Patricia Schömitz', 'Susanne Silverio', 'Werner Stefansich', 'Ferdinand Stoiber', 'Gabi Traindl', 'Barbara Vögel', 'Harald Wurmbrand', 'Daniela Hoffmann']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |\n",
      "|1423   |Hearts and Minds                           |1996|4.261824131011963    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3775   |Make Mine Music                            |1946|3.9882495403289795   |['Ed Benedict', 'James Bodrero', 'Homer Brightman', 'Al Cameron', 'Robert Cormack', 'Jon Dodson', 'Andy Engman', 'Erwin Graham', 'Eric Gurney', 'T. Hee', 'Sylvia Moberly-Holland', 'Dick Kinney', 'Jack Kinney', 'Turner Leighton', 'Bud Linn', 'Jesse Marsh', 'Tom Massey', 'Mique Nelson', 'Charles Payzant', 'Tatiana Riabouchinska', 'Dick Shaw', 'Jimi Trout', 'Ted Weems', 'Roy Williams', 'Sid Weiss', 'John Niendorff']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |\n",
      "|1237   |The Seventh Seal                           |1957|4.016429424285889    |['Ingmar Bergman', 'Bibi Andersson', 'Max von Sydow', 'Gunnar Fischer', 'Bertil Anderberg', 'Evald Andersson', 'Gunnar Björnstrand', 'Carl-Henry Cagarp', 'Sixten Ehrling', 'Anders Ek', 'Allan Ekelund', 'Bengt Ekerot', 'Katinka Faragó', 'Else Fisher', 'Åke Fridell', 'Inga Gill', 'Maud Hansson', 'Louis Huch', 'Inga Landgré', 'Gunnel Lindblom', 'Manne Lindholm', 'P.A. Lundgren', 'Åke Nilsson', 'Nils Nittel', 'Erik Nordgren', 'Gunnar Olsson', 'Lennart Olsson', 'Nils Poppe', 'Erik Strandmark', 'Lennart Wallén', 'Lennart Wallin', 'Aaby Wedin']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |\n",
      "|2503   |The Apple                                  |1998|4.043043613433838    |['Ghorban Ali Naderi', 'Hossein Amiri', 'Samira Makhmalbaf', 'Akbar Meshkini', 'Azizeh Mohamadi', 'Massoumeh Naderi', 'Zahra Naderi', 'Zahra Saghrisaz', 'Iraj Sarbaz', 'Reza Sheykhi', 'Caroline Phouthavy', 'John Robert Miller']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n",
      "|718    |The Visitors                               |1993|3.9844014644622803   |['Christian Clavier', 'Jean-Laurent Agrinier', 'Fabrice Anut', 'Patrick Archambault', \"François d'Artemare\", 'André Atellian', 'Pierre Aussedat', 'Eric Averlant', 'Paul Bandey', 'Gilles Bannier', 'Frédéric Baptiste', 'Brigitte Barbier', 'Muriel Baurens', 'Benoît Bayle', 'Isabelle Beauchesne', 'Jérôme Berthoud', 'Louis-Fernand Billet', 'Laure Blaess', 'Célina Blanc', 'Dominik Borde', 'Camille Bordes-Resnais', 'Sylviane Bouget', 'Amandine Boyadjian', 'Yohan Boyadjian', 'Christian Bujeau', 'Patrick Burgel', 'Didier Bénureau', 'Bernard Caroff', 'Jean-Luc Caron', 'Agnès Chaulier', 'Marie-Anne Chazel', 'Jean-Pierre Clami', 'Christophe Cojan', 'Laurence Coq', 'Rachel Corlet', 'Alain Cousseau', 'Gilbert Crozet', 'Thierry Crépin', 'Charlotte David', 'Francis de Gueltzl', 'Katia Delagarde', 'Éric Denize', 'Hervé Dieu', 'Frédéric Doniguian', 'Sébastien Drhey', 'Terrance Eugene Fredericks', 'David Gabison', 'Jean-Michel Gageat', 'Tara Gano', 'Leslie Rain', 'Jacques Gastineau', 'Emmanuel Gateau', 'Anna Gaylor', 'Lizi Gelber', 'Michel Grimaud', 'Bela Grushka', 'Michel Guarguir', 'Paul Gueu', 'Frédéric Hamelin', 'Pascal Henin', 'Dominique Hulin', 'Gisèle Jacquin', 'Fanny Jakubowicz', 'Sylvie Jouineau', 'Catherine Kelber', 'Gilles Laboulandine', 'François Lalande', 'Nadia Laurent', 'Jean-Guillaume Le Dantec', 'Jean-Yves Le Mener', 'Patrick Lefebvre', 'Valérie Lemercier', 'Bernard Lependu', 'Bernard Le Roux', 'Catherine Leterrier', 'Eric Levi', 'Thierry Liagre', 'Philippe Lièvre', 'Christophe Legal', 'Claire Magnin', 'Jean-Charles Maratier', 'Madeleine Marie', 'Stéphanie Marie', 'Daniel Mauvignier', 'Jacques Mironneau', 'Suzette Monlouis', 'Philippe Morlier', 'Jean-Marc Mouligne', 'Jean-Paul Muel', 'Isabelle Nanty', 'Roland Neunreuther', 'Eric Nové', 'Didier Pain', 'Chantal Pernecker', 'Michel Peyrelon', 'Pierre Pochy', 'Jean-Marie Poiré', 'Claude Pollan', 'Jérôme Prébois', 'André Raffard', 'Didier Ranz', 'Annick Redon', 'Dorian Rigal-Ansous', 'Dany Roca', 'Philippe Ros', 'Jean-Charles Ruault', 'Gary Saint-Martin', 'Julie Salvador', 'Jean-Pierre Sauvaire', 'Michel Scourneau', 'Bertrand Seitz', 'Théophile Sowié', 'Arielle Sémenoff', 'Gérard Séty', 'Jean-Claude Temporelli', 'Alain Terzian', 'Frank Thevenon', 'Evelyne Tison', 'Jean-Philippe Vallespir', 'Noël Vandendries-Lestienne', 'Nadia Vasil', 'Bobane Veselinovic', 'Sophie Vesoul', 'Pierre Vial', 'Thomas Vichy', 'Lionel Vitrant', 'Katja Weitzenböck', 'Patrick Widdrington', 'Olivier Wojciechowski', 'Adeline Yoyotte', 'Odile Alliet', 'Jean-Noël Andrieux', 'Sophie Antonuccio', 'Frédéric Atellian', 'Jean-Robert Blanche', 'Marie-Françoise Coureur Le Duc', 'Muriel Cravatte', 'Dominique Crepet', 'Badara Faye', 'Olivier Foucher', 'Jean Gouello', 'Bruno Goury', 'Stephan Guillemet', 'Jérôme Haumont', 'Florence Lagrange', 'Serge Margiocchi', 'Vincent Maroutian', 'Céline Migeon', 'Francis Pesquer', 'Jacques Puiseux', 'Gilles Rossire', 'Philippe Savoir', 'Jean-Marie Sonnet', 'Christophe Thiollier', 'Patrick Tombette', 'Manuel Jorge', 'Caroline Gautier', 'Benoît Lemaire', 'Daniel Maltret', 'Pierre Roger', 'Roger Saillard', 'Richard Brodet', 'Nathalie Delest', 'Caroline Emery', 'Jean-Louis Lalet', 'Carmen Lima', 'Anne Marchal', 'Gilles Poulin', 'Joel Matray', 'Bernard Sachsé', 'Dominique Delguste', 'Cédric Houplain', 'Bruno Lefebvre', 'Czech Symphony Orchestra']|\n",
      "|668    |Pather Panchali                            |1955|4.07059907913208     |['Satyajit Ray', 'Bibhutibhushan Bandyopadhyay', 'Haren Banerjee', 'Runki Banerjee', 'Kanu Bannerjee', 'Karuna Bannerjee', 'Subir Banerjee', 'Tulsi Chakraborty', 'Bansi Chandragupta', 'Santi Chatterjee', 'Anil Choudhury', 'Uma Das Gupta', 'Aparna Devi', 'Chunibala Devi', 'Nibhanani Devi', 'Reba Devi', 'Dulal Dutta', 'Rama Gangopadhaya', 'Roma Ganguli', 'Bhupen Ghosh', 'Dinen Gupta', 'Subrata Mitra', 'Binoy Mukherjee', 'Harimohan Nag', 'Kshirod Roy', 'Soumendu Roy', 'Dakshina Mohan Tagore', 'Aloknath Dey', 'Tapeshwar Prasad', 'Sukumar Sengupta', 'Satyen Chatterjee', 'Suresh Chandra Chandra', 'Subir Hajra', 'Harinarayan Mukhopadhyay', 'R.B. Mehta', 'Haridhan Nag', 'Nimai Roy', 'Pratap Narayan Mitra', 'Dibakar Dutta', 'Sitaram Das', 'Suren Sahu', 'Rampada Das', 'Bemananda Brajwasi', 'Ashish Barman', 'Baren Bhattacharya', 'Jagmohan Behrowa', 'Somen Gupta', 'Samir Ghosh', 'Durgacharan Mitra', 'Suren Roy']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n",
      "|2931   |Time of the Gypsies                        |1989|4.118833541870117    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|557    |Mamma Roma                                 |1962|4.144425868988037    |['Maria Bernardini', 'Nino Bionci', 'Franco Casati', 'Marcello Ceccarelli', 'Santino Citti', 'Silvana Corsini', \"Lina D'Amico\", 'Emanuele Di Bari', 'Pasquale Ferrarese', 'Fernando Franchi', 'Mirella Gamacchio', 'Ettore Garofolo', 'Luciano Gonini', 'Joselito', 'Vittorio La Paglia', 'Luisa Loiano', 'Lamberto Maggiorani', 'Sandro Meschino', 'Piero Morgia', 'Leandro Santarelli', 'Gioacchino Sofia', 'Marcello Sorrentino', 'Vincenzo Taito', 'Franco Tovo', 'Nino Venzi', 'Paolo Volponi', 'Giulio De Stephanis', 'Franco Ceccarelli']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |\n",
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "recommendations = ranked_recommendations.join(sparkdf_movies, on=\"MovieId\")\n",
    "recommendations.select(\"MovieId\", \"Title\", \"Date\", \"AVG_Rating_Prediction\", \"cast\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Filtering**\n",
    "\n",
    "To provide personalized recommendations, it is essential to exclude movies that users have already seen. Here is a detailed process:\n",
    "\n",
    "1. **Identifying Watched Movies**: We start by identifying the movies that each user has already watched. This is done by filtering the ratings DataFrame to find the movies seen by User 1 and User 2.\n",
    "\n",
    "2. **Adding Indicator Columns**: We add columns to the final recommendations DataFrame to indicate whether a user has seen a particular movie.\n",
    "\n",
    "3. **Excluding Watched Movies**: Next, we filter out the movies in the recommendations that either user has already seen, ensuring that the recommendations only include movies that neither User 1 nor User 2 has watched.\n",
    "\n",
    "By following this process, we ensure that the suggestions are **fresh** and **relevant** for both users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def give_unseen_recommandations_for_two_users(UserId1, UserId2, model, sparkdf_movies, sparkdf_ratings):\n",
    "    ranked_recommendations = give_recommandations_for_two_users(rand_users[0], rand_users[1], model)\n",
    "    ranked_recommendations = ranked_recommendations.join(sparkdf_movies, on=\"MovieId\")\n",
    "\n",
    "    user_1_seen = sparkdf_ratings.filter(sparkdf_ratings.UserId == UserId1).select(\"MovieID\").withColumn(\"UserId1_watched_it\", F.lit(1))\n",
    "    user_2_seen = sparkdf_ratings.filter(sparkdf_ratings.UserId == UserId2).select(\"MovieID\").withColumn(\"UserId2_watched_it\", F.lit(1))\n",
    "    user_1_seen = user_1_seen.withColumnRenamed(\"MovieId\", \"MovieId_seen_1\")\n",
    "    user_2_seen = user_2_seen.withColumnRenamed(\"MovieId\", \"MovieId_seen_2\")\n",
    "\n",
    "    unseen_recommendations = ranked_recommendations.alias(\"fr\")\n",
    "    unseen_recommendations = unseen_recommendations.join(user_1_seen, unseen_recommendations.MovieId == user_1_seen.MovieId_seen_1, how=\"left\").drop(\"MovieId_seen_1\")\n",
    "    unseen_recommendations = unseen_recommendations.join(user_2_seen, unseen_recommendations.MovieId == user_2_seen.MovieId_seen_2, how=\"left\").drop(\"MovieId_seen_2\")\n",
    "    unseen_recommendations = unseen_recommendations.fillna(0, subset=[\"UserId1_watched_it\", \"UserId2_watched_it\"])\n",
    "\n",
    "    return unseen_recommendations.filter(~((col(\"UserId1_watched_it\") == 1) | (col(\"UserId2_watched_it\") == 1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|MovieId|Title                                      |Date|AVG_Rating_Prediction|cast                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|3089   |The Bicycle Thief                          |1948|4.030358791351318    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|53     |Lamerica                                   |1994|4.159127712249756    |['Gianni Amelio', 'Franco Piersanti', 'Dritan Alia', 'Samantha Antonnicola', 'Esmeralda Ara', 'Ilir Ara', 'Cristina Casati', 'Mario Cecchi Gori', 'Elinor Cekliqi', 'Shkelqim Daja', 'Carmelo Di Mazzarelli', 'Nikolin Elezi', 'Stefano Forcina', 'Cristina Francioni', 'Altin Garupi', 'Gezim Garupi', 'Fatmir Gjyla', 'Elona Hoti', 'Ismail Kukaj', 'Artan Marina', 'Elina Ndreu', 'Ferit Nutai', 'Gezim Nutai', 'Sefer Pema', 'Andrea Porporati', 'Antonio Privitera', 'Klodian Rakaj', 'Marco Sallustri', 'Idajet Sejdia', 'Alessandro Sermoneta', 'Liliana Subashi', 'Marian Tifozi', 'Saimir Tila', 'Nazzareno Ubertini', 'Nuredin Ujkaj', 'Vojo Kola', 'Arbin Trebicka', 'Nadon Kallulli', 'Alfonso Vicori', 'Massimiliano Gigante', 'Renato Spadolini', 'Bardhyl Plumbi', 'Erdit Gega', 'Kujtim Veshi', 'Skender Basi', 'Lulzim Gogja', 'Roland Arani', 'Atnan Duka', 'Fehim Alliu', 'Massimiliano Tentoni', 'Aldo Peroni', 'Alberto Azzetti', 'Alessandro Romano', 'Abedin Bocka', 'Armand Diko', 'Ilir Vreto', 'Ramazan Tare', 'Kujtim Seidi', 'Clirim Fuga', 'Bujar Kamberi', 'Agron Haxhiu', 'Alessandro Dionosio', 'Maria Camilla Crespi', 'Vladimir Lila', 'Luftar Braçe', 'Xhemal Koçi', 'Ylli Muça', 'Kreshnik Hashorva', 'Michele Pellegrino', 'Giorgio Chierchiè', 'Giuseppe Menegatti', 'Vito Gallo', 'Ubaldo Panunzi', 'Violeta Banushi', 'Michele Lancia', 'Sabri Basha', 'Romano Donati', 'Riccardo Luciano', 'Marco Tota', 'Maurizio Lentini']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "|1743   |Arguing the World                          |1996|3.9494829177856445   |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|1260   |M                                          |1931|4.001025199890137    |['Peter Lorre', 'Fritz Lang', 'Theo Lingen', 'Robert Baberske', 'Fritz Arno Wagner', 'Edvard Grieg', 'Gerhard Bienert', 'Rudolf Blümner', 'Paul Falkenberg', 'Georg John', 'Friedrich Gnaß', 'Gustaf Gründgens', 'Emil Hasler', 'Adolf Jansen', 'Paul Kemp', 'Inge Landgut', 'Theodor Loos', 'Fritz Odemar', 'Karl Platen', 'Ernst Stahl-Nachbaur', 'Franz Stein', 'Rosa Valetti', 'Karl Vollbrecht', 'Horst von Harbou', 'Thea von Harbou', 'Hertha von Walther', 'Otto Wernicke', 'Ellen Widmann', 'Ernst Wolff', 'Wilhelm Weber', 'Lucie Rhoden']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3022   |The General                                |1927|4.006153583526611    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3012   |Battling Butler                            |1926|3.9526710510253906   |['Budd Fine', 'Bert Haines', 'Ed Levy', 'Lex Neal', \"Mary O'Brien\", 'Charles Henry Smith']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "|1420   |Message to Love: The Isle of Wight Festival|1996|4.037903308868408    |['Andy Carchrae', 'Ricki Farr', 'Laurie Horsley', 'John Wellard', 'Benjamin Darling', 'Martin Barre', 'Ric Lee', 'Leo Lyons', 'Chick Churchill', 'Gareth Haywood', 'Clive Bunker', 'John Wilson', 'Richard McCracken', 'Julian Harvey', 'Taste', 'Robert H. Montgomery Jr.', \"Judi O'Brian\", \"Morton 'Tim' Fry\", 'Vincent Villandi', 'Jennifer Cox', 'Colin Richards', 'Tic Ticnor']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|989    |Schlafes Bruder                            |1995|3.8940250873565674   |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|854    |The Ballad of Narayama                     |1958|4.365152835845947    |['Shichirô Fukazawa', 'Danko Ichikawa', 'Keisuke Kinoshita', 'Matsunosuke Nozawa', 'Ryuzo Otani', 'Kotohisa Satsukime', 'Heiji Yoshida', 'Katsuyuki Hattori', 'Toshikazu Sugiyama', 'Genzô Furumiya']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "|2512   |The Ballad of Narayama                     |1982|4.079133033752441    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|2309   |The Inheritors                             |1998|4.11139988899231     |['Michaela Berner', 'Stephan Biebl', 'Ruth Blankenstein', 'Martin Dvoracek-Riegler', 'Christine Egger', 'Eddie Fischnaller', 'Joe Fischnaller', 'Stefan Gruber', 'Christoph Gusenbauer', 'Christian Heitler', 'Christine Hermann', 'Marid Hofmann', 'Franz Hollaus', 'Joe Joachim', 'Andreas Kahri', 'Karl Kastner', 'Helga Klein', 'Susanne Kolar', 'Michael Kraxner', 'Michael Pogo Kreiner', 'Otger Kunert', 'Gertraud Mayböck', 'Sigi Mayer', 'Toni Mayer', 'Johann Naderer', 'David Neve', 'Dietmar Nigsch', 'Norbert Perchtold', 'Rebecca Rabche', 'Stefan Ruzowitzky', 'Georgie Schmid-Schmidsfelden', 'Bernhard Schober', 'Kirstin Schwab', 'Simon Schwarz', 'Ernestine Schöllbauer', 'Heino Schöllbauer', 'Patricia Schömitz', 'Susanne Silverio', 'Werner Stefansich', 'Ferdinand Stoiber', 'Gabi Traindl', 'Barbara Vögel', 'Harald Wurmbrand', 'Daniela Hoffmann']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |\n",
      "|1423   |Hearts and Minds                           |1996|4.261824131011963    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3775   |Make Mine Music                            |1946|3.9882495403289795   |['Ed Benedict', 'James Bodrero', 'Homer Brightman', 'Al Cameron', 'Robert Cormack', 'Jon Dodson', 'Andy Engman', 'Erwin Graham', 'Eric Gurney', 'T. Hee', 'Sylvia Moberly-Holland', 'Dick Kinney', 'Jack Kinney', 'Turner Leighton', 'Bud Linn', 'Jesse Marsh', 'Tom Massey', 'Mique Nelson', 'Charles Payzant', 'Tatiana Riabouchinska', 'Dick Shaw', 'Jimi Trout', 'Ted Weems', 'Roy Williams', 'Sid Weiss', 'John Niendorff']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |\n",
      "|1237   |The Seventh Seal                           |1957|4.016429424285889    |['Ingmar Bergman', 'Bibi Andersson', 'Max von Sydow', 'Gunnar Fischer', 'Bertil Anderberg', 'Evald Andersson', 'Gunnar Björnstrand', 'Carl-Henry Cagarp', 'Sixten Ehrling', 'Anders Ek', 'Allan Ekelund', 'Bengt Ekerot', 'Katinka Faragó', 'Else Fisher', 'Åke Fridell', 'Inga Gill', 'Maud Hansson', 'Louis Huch', 'Inga Landgré', 'Gunnel Lindblom', 'Manne Lindholm', 'P.A. Lundgren', 'Åke Nilsson', 'Nils Nittel', 'Erik Nordgren', 'Gunnar Olsson', 'Lennart Olsson', 'Nils Poppe', 'Erik Strandmark', 'Lennart Wallén', 'Lennart Wallin', 'Aaby Wedin']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |\n",
      "|2503   |The Apple                                  |1998|4.043043613433838    |['Ghorban Ali Naderi', 'Hossein Amiri', 'Samira Makhmalbaf', 'Akbar Meshkini', 'Azizeh Mohamadi', 'Massoumeh Naderi', 'Zahra Naderi', 'Zahra Saghrisaz', 'Iraj Sarbaz', 'Reza Sheykhi', 'Caroline Phouthavy', 'John Robert Miller']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n",
      "|718    |The Visitors                               |1993|3.9844014644622803   |['Christian Clavier', 'Jean-Laurent Agrinier', 'Fabrice Anut', 'Patrick Archambault', \"François d'Artemare\", 'André Atellian', 'Pierre Aussedat', 'Eric Averlant', 'Paul Bandey', 'Gilles Bannier', 'Frédéric Baptiste', 'Brigitte Barbier', 'Muriel Baurens', 'Benoît Bayle', 'Isabelle Beauchesne', 'Jérôme Berthoud', 'Louis-Fernand Billet', 'Laure Blaess', 'Célina Blanc', 'Dominik Borde', 'Camille Bordes-Resnais', 'Sylviane Bouget', 'Amandine Boyadjian', 'Yohan Boyadjian', 'Christian Bujeau', 'Patrick Burgel', 'Didier Bénureau', 'Bernard Caroff', 'Jean-Luc Caron', 'Agnès Chaulier', 'Marie-Anne Chazel', 'Jean-Pierre Clami', 'Christophe Cojan', 'Laurence Coq', 'Rachel Corlet', 'Alain Cousseau', 'Gilbert Crozet', 'Thierry Crépin', 'Charlotte David', 'Francis de Gueltzl', 'Katia Delagarde', 'Éric Denize', 'Hervé Dieu', 'Frédéric Doniguian', 'Sébastien Drhey', 'Terrance Eugene Fredericks', 'David Gabison', 'Jean-Michel Gageat', 'Tara Gano', 'Leslie Rain', 'Jacques Gastineau', 'Emmanuel Gateau', 'Anna Gaylor', 'Lizi Gelber', 'Michel Grimaud', 'Bela Grushka', 'Michel Guarguir', 'Paul Gueu', 'Frédéric Hamelin', 'Pascal Henin', 'Dominique Hulin', 'Gisèle Jacquin', 'Fanny Jakubowicz', 'Sylvie Jouineau', 'Catherine Kelber', 'Gilles Laboulandine', 'François Lalande', 'Nadia Laurent', 'Jean-Guillaume Le Dantec', 'Jean-Yves Le Mener', 'Patrick Lefebvre', 'Valérie Lemercier', 'Bernard Lependu', 'Bernard Le Roux', 'Catherine Leterrier', 'Eric Levi', 'Thierry Liagre', 'Philippe Lièvre', 'Christophe Legal', 'Claire Magnin', 'Jean-Charles Maratier', 'Madeleine Marie', 'Stéphanie Marie', 'Daniel Mauvignier', 'Jacques Mironneau', 'Suzette Monlouis', 'Philippe Morlier', 'Jean-Marc Mouligne', 'Jean-Paul Muel', 'Isabelle Nanty', 'Roland Neunreuther', 'Eric Nové', 'Didier Pain', 'Chantal Pernecker', 'Michel Peyrelon', 'Pierre Pochy', 'Jean-Marie Poiré', 'Claude Pollan', 'Jérôme Prébois', 'André Raffard', 'Didier Ranz', 'Annick Redon', 'Dorian Rigal-Ansous', 'Dany Roca', 'Philippe Ros', 'Jean-Charles Ruault', 'Gary Saint-Martin', 'Julie Salvador', 'Jean-Pierre Sauvaire', 'Michel Scourneau', 'Bertrand Seitz', 'Théophile Sowié', 'Arielle Sémenoff', 'Gérard Séty', 'Jean-Claude Temporelli', 'Alain Terzian', 'Frank Thevenon', 'Evelyne Tison', 'Jean-Philippe Vallespir', 'Noël Vandendries-Lestienne', 'Nadia Vasil', 'Bobane Veselinovic', 'Sophie Vesoul', 'Pierre Vial', 'Thomas Vichy', 'Lionel Vitrant', 'Katja Weitzenböck', 'Patrick Widdrington', 'Olivier Wojciechowski', 'Adeline Yoyotte', 'Odile Alliet', 'Jean-Noël Andrieux', 'Sophie Antonuccio', 'Frédéric Atellian', 'Jean-Robert Blanche', 'Marie-Françoise Coureur Le Duc', 'Muriel Cravatte', 'Dominique Crepet', 'Badara Faye', 'Olivier Foucher', 'Jean Gouello', 'Bruno Goury', 'Stephan Guillemet', 'Jérôme Haumont', 'Florence Lagrange', 'Serge Margiocchi', 'Vincent Maroutian', 'Céline Migeon', 'Francis Pesquer', 'Jacques Puiseux', 'Gilles Rossire', 'Philippe Savoir', 'Jean-Marie Sonnet', 'Christophe Thiollier', 'Patrick Tombette', 'Manuel Jorge', 'Caroline Gautier', 'Benoît Lemaire', 'Daniel Maltret', 'Pierre Roger', 'Roger Saillard', 'Richard Brodet', 'Nathalie Delest', 'Caroline Emery', 'Jean-Louis Lalet', 'Carmen Lima', 'Anne Marchal', 'Gilles Poulin', 'Joel Matray', 'Bernard Sachsé', 'Dominique Delguste', 'Cédric Houplain', 'Bruno Lefebvre', 'Czech Symphony Orchestra']|\n",
      "|668    |Pather Panchali                            |1955|4.07059907913208     |['Satyajit Ray', 'Bibhutibhushan Bandyopadhyay', 'Haren Banerjee', 'Runki Banerjee', 'Kanu Bannerjee', 'Karuna Bannerjee', 'Subir Banerjee', 'Tulsi Chakraborty', 'Bansi Chandragupta', 'Santi Chatterjee', 'Anil Choudhury', 'Uma Das Gupta', 'Aparna Devi', 'Chunibala Devi', 'Nibhanani Devi', 'Reba Devi', 'Dulal Dutta', 'Rama Gangopadhaya', 'Roma Ganguli', 'Bhupen Ghosh', 'Dinen Gupta', 'Subrata Mitra', 'Binoy Mukherjee', 'Harimohan Nag', 'Kshirod Roy', 'Soumendu Roy', 'Dakshina Mohan Tagore', 'Aloknath Dey', 'Tapeshwar Prasad', 'Sukumar Sengupta', 'Satyen Chatterjee', 'Suresh Chandra Chandra', 'Subir Hajra', 'Harinarayan Mukhopadhyay', 'R.B. Mehta', 'Haridhan Nag', 'Nimai Roy', 'Pratap Narayan Mitra', 'Dibakar Dutta', 'Sitaram Das', 'Suren Sahu', 'Rampada Das', 'Bemananda Brajwasi', 'Ashish Barman', 'Baren Bhattacharya', 'Jagmohan Behrowa', 'Somen Gupta', 'Samir Ghosh', 'Durgacharan Mitra', 'Suren Roy']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n",
      "|2931   |Time of the Gypsies                        |1989|4.118833541870117    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|557    |Mamma Roma                                 |1962|4.144425868988037    |['Maria Bernardini', 'Nino Bionci', 'Franco Casati', 'Marcello Ceccarelli', 'Santino Citti', 'Silvana Corsini', \"Lina D'Amico\", 'Emanuele Di Bari', 'Pasquale Ferrarese', 'Fernando Franchi', 'Mirella Gamacchio', 'Ettore Garofolo', 'Luciano Gonini', 'Joselito', 'Vittorio La Paglia', 'Luisa Loiano', 'Lamberto Maggiorani', 'Sandro Meschino', 'Piero Morgia', 'Leandro Santarelli', 'Gioacchino Sofia', 'Marcello Sorrentino', 'Vincenzo Taito', 'Franco Tovo', 'Nino Venzi', 'Paolo Volponi', 'Giulio De Stephanis', 'Franco Ceccarelli']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |\n",
      "|3134   |Grand Illusion                             |1937|3.9493963718414307   |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unseen_recommendations = give_unseen_recommandations_for_two_users(rand_users[0], rand_users[1], als_model, sparkdf_movies, sparkdf_ratings)\n",
    "unseen_recommendations.select(\"MovieId\", \"Title\", \"Date\", \"AVG_Rating_Prediction\", \"cast\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another example with 2 differents users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|MovieId|Title                                      |Date|AVG_Rating_Prediction|cast                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|3089   |The Bicycle Thief                          |1948|4.030358791351318    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|53     |Lamerica                                   |1994|4.159127712249756    |['Gianni Amelio', 'Franco Piersanti', 'Dritan Alia', 'Samantha Antonnicola', 'Esmeralda Ara', 'Ilir Ara', 'Cristina Casati', 'Mario Cecchi Gori', 'Elinor Cekliqi', 'Shkelqim Daja', 'Carmelo Di Mazzarelli', 'Nikolin Elezi', 'Stefano Forcina', 'Cristina Francioni', 'Altin Garupi', 'Gezim Garupi', 'Fatmir Gjyla', 'Elona Hoti', 'Ismail Kukaj', 'Artan Marina', 'Elina Ndreu', 'Ferit Nutai', 'Gezim Nutai', 'Sefer Pema', 'Andrea Porporati', 'Antonio Privitera', 'Klodian Rakaj', 'Marco Sallustri', 'Idajet Sejdia', 'Alessandro Sermoneta', 'Liliana Subashi', 'Marian Tifozi', 'Saimir Tila', 'Nazzareno Ubertini', 'Nuredin Ujkaj', 'Vojo Kola', 'Arbin Trebicka', 'Nadon Kallulli', 'Alfonso Vicori', 'Massimiliano Gigante', 'Renato Spadolini', 'Bardhyl Plumbi', 'Erdit Gega', 'Kujtim Veshi', 'Skender Basi', 'Lulzim Gogja', 'Roland Arani', 'Atnan Duka', 'Fehim Alliu', 'Massimiliano Tentoni', 'Aldo Peroni', 'Alberto Azzetti', 'Alessandro Romano', 'Abedin Bocka', 'Armand Diko', 'Ilir Vreto', 'Ramazan Tare', 'Kujtim Seidi', 'Clirim Fuga', 'Bujar Kamberi', 'Agron Haxhiu', 'Alessandro Dionosio', 'Maria Camilla Crespi', 'Vladimir Lila', 'Luftar Braçe', 'Xhemal Koçi', 'Ylli Muça', 'Kreshnik Hashorva', 'Michele Pellegrino', 'Giorgio Chierchiè', 'Giuseppe Menegatti', 'Vito Gallo', 'Ubaldo Panunzi', 'Violeta Banushi', 'Michele Lancia', 'Sabri Basha', 'Romano Donati', 'Riccardo Luciano', 'Marco Tota', 'Maurizio Lentini']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "|1743   |Arguing the World                          |1996|3.9494829177856445   |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|1260   |M                                          |1931|4.001025199890137    |['Peter Lorre', 'Fritz Lang', 'Theo Lingen', 'Robert Baberske', 'Fritz Arno Wagner', 'Edvard Grieg', 'Gerhard Bienert', 'Rudolf Blümner', 'Paul Falkenberg', 'Georg John', 'Friedrich Gnaß', 'Gustaf Gründgens', 'Emil Hasler', 'Adolf Jansen', 'Paul Kemp', 'Inge Landgut', 'Theodor Loos', 'Fritz Odemar', 'Karl Platen', 'Ernst Stahl-Nachbaur', 'Franz Stein', 'Rosa Valetti', 'Karl Vollbrecht', 'Horst von Harbou', 'Thea von Harbou', 'Hertha von Walther', 'Otto Wernicke', 'Ellen Widmann', 'Ernst Wolff', 'Wilhelm Weber', 'Lucie Rhoden']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3022   |The General                                |1927|4.006153583526611    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3012   |Battling Butler                            |1926|3.9526710510253906   |['Budd Fine', 'Bert Haines', 'Ed Levy', 'Lex Neal', \"Mary O'Brien\", 'Charles Henry Smith']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "|1420   |Message to Love: The Isle of Wight Festival|1996|4.037903308868408    |['Andy Carchrae', 'Ricki Farr', 'Laurie Horsley', 'John Wellard', 'Benjamin Darling', 'Martin Barre', 'Ric Lee', 'Leo Lyons', 'Chick Churchill', 'Gareth Haywood', 'Clive Bunker', 'John Wilson', 'Richard McCracken', 'Julian Harvey', 'Taste', 'Robert H. Montgomery Jr.', \"Judi O'Brian\", \"Morton 'Tim' Fry\", 'Vincent Villandi', 'Jennifer Cox', 'Colin Richards', 'Tic Ticnor']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|989    |Schlafes Bruder                            |1995|3.8940250873565674   |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|854    |The Ballad of Narayama                     |1958|4.365152835845947    |['Shichirô Fukazawa', 'Danko Ichikawa', 'Keisuke Kinoshita', 'Matsunosuke Nozawa', 'Ryuzo Otani', 'Kotohisa Satsukime', 'Heiji Yoshida', 'Katsuyuki Hattori', 'Toshikazu Sugiyama', 'Genzô Furumiya']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
      "|2512   |The Ballad of Narayama                     |1982|4.079133033752441    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|2395   |Rushmore                                   |1998|4.018557548522949    |['Bill Murray', 'Seymour Cassel', 'Mason Gamble', 'Michael Cedar', 'Cathie Filian', 'Jason Schwartzman', 'Felix Mendelssohn', 'Mark Mothersbaugh', 'Monica Allen', 'Amanda Anderson', 'Eric Chase Anderson', 'Wes Anderson', 'Adebayo Asabi', 'John Boccaccio', 'David M. Bowen', 'Linda Brattain', 'Aristide Bruant', 'Craig Busch', 'Donny Caicedo', 'Cecilia Cardwell', 'Bob Casale', 'Lydia Cedrone', 'Layne Chaney', 'Cheryl Clarson', 'Dennis Clay', 'Barbara Cohen', 'Lin Coleman', 'Jessica Dailey', 'Paul Desmond', 'Stephen Dignan', 'Virginia Díaz', 'George Farish', 'Francis Fernandez', 'Al Fielder', 'Danny Fine', 'Grant Foerster', 'Steve Fontano', 'Christine C. Fransen', 'Ed Geldart', 'Moira Gill', 'Nicole Gorg', 'Bob Harper', 'Sally J. Harper', 'Terri Harper', 'Jeffrey B. Hartmann', 'Jonathan Hennessey', 'Terry Hubbard', 'John Killalea', 'Ali Ktiri', 'Angela Laux', 'Hae-Joon Lee', 'Stephen Light', 'Tammy Lorkovic', 'Chad Loucks', 'Billy MacTavish', 'Michael Maggart', 'Marietta Marich', 'Melissa Matthies', 'Keith McCawley', 'Ronnie McCawley', 'Thayer McClanahan', 'Stephen McCole', 'Doug Mercer', 'Roy Metcalf', 'Vojislav Mikulic', 'David Moritz', 'Manning Mott', 'Kim Naves', 'Daniel R. Padgett', 'Dipak Pallana', 'Christopher S. Parker', 'Jim Pasque', 'Karen Patch', 'Colin Platt', 'Ella Pryor', 'Ron Radvinsky', 'Van Redin', 'Morgan Redmond', 'Lucille Sadikin', 'David Sanders', 'Antoni Scarano', 'Debra Secher', 'Kevin Sims', 'Cat Stevens', 'J.J. Stonebraker', 'Scott Streetman', 'Chad Stuart', 'Sara Tanaka', 'Brian Tenenbaum', 'Doug Terranova', 'Kim Terry', 'Dalton Tomlin', 'Harry Tracosas', 'Ric Trzeciak', 'Kyle Ryan Urquhart', 'Peter J. Verrando', 'Eric Bennett Weems', 'Melissa Wiechmann', 'Olivia Williams', 'Patricia Winkler', 'Michael R. Wohlfield', 'Mark Van Stone', 'Ted Broden', 'Jamelle Flowers', 'Linell Kathleen Townes', 'Fred Davis', 'Carrie Albers', 'Jason Hawkins', 'Victor Lacour', 'Renee Roth', 'Jennifer Simpson', 'Danping Wong', 'Vicki Daniels Johnson', 'Kalen Hoyle', 'McCauley Penderdast', 'Nancy Doyle', 'Maura Tighe', 'William Lau', 'Ronnie Lane', 'Charles Wesley', 'Robbie Lee', 'Tommy Moeller', 'Brian Parker', 'Unit Four Plus Two', 'The Vince Guaraldi Trio', 'Chad and Jeremy', 'The Creation', 'Eddie Phillips']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |\n",
      "|2309   |The Inheritors                             |1998|4.11139988899231     |['Michaela Berner', 'Stephan Biebl', 'Ruth Blankenstein', 'Martin Dvoracek-Riegler', 'Christine Egger', 'Eddie Fischnaller', 'Joe Fischnaller', 'Stefan Gruber', 'Christoph Gusenbauer', 'Christian Heitler', 'Christine Hermann', 'Marid Hofmann', 'Franz Hollaus', 'Joe Joachim', 'Andreas Kahri', 'Karl Kastner', 'Helga Klein', 'Susanne Kolar', 'Michael Kraxner', 'Michael Pogo Kreiner', 'Otger Kunert', 'Gertraud Mayböck', 'Sigi Mayer', 'Toni Mayer', 'Johann Naderer', 'David Neve', 'Dietmar Nigsch', 'Norbert Perchtold', 'Rebecca Rabche', 'Stefan Ruzowitzky', 'Georgie Schmid-Schmidsfelden', 'Bernhard Schober', 'Kirstin Schwab', 'Simon Schwarz', 'Ernestine Schöllbauer', 'Heino Schöllbauer', 'Patricia Schömitz', 'Susanne Silverio', 'Werner Stefansich', 'Ferdinand Stoiber', 'Gabi Traindl', 'Barbara Vögel', 'Harald Wurmbrand', 'Daniela Hoffmann']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |\n",
      "|1423   |Hearts and Minds                           |1996|4.261824131011963    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|3775   |Make Mine Music                            |1946|3.9882495403289795   |['Ed Benedict', 'James Bodrero', 'Homer Brightman', 'Al Cameron', 'Robert Cormack', 'Jon Dodson', 'Andy Engman', 'Erwin Graham', 'Eric Gurney', 'T. Hee', 'Sylvia Moberly-Holland', 'Dick Kinney', 'Jack Kinney', 'Turner Leighton', 'Bud Linn', 'Jesse Marsh', 'Tom Massey', 'Mique Nelson', 'Charles Payzant', 'Tatiana Riabouchinska', 'Dick Shaw', 'Jimi Trout', 'Ted Weems', 'Roy Williams', 'Sid Weiss', 'John Niendorff']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |\n",
      "|1237   |The Seventh Seal                           |1957|4.016429424285889    |['Ingmar Bergman', 'Bibi Andersson', 'Max von Sydow', 'Gunnar Fischer', 'Bertil Anderberg', 'Evald Andersson', 'Gunnar Björnstrand', 'Carl-Henry Cagarp', 'Sixten Ehrling', 'Anders Ek', 'Allan Ekelund', 'Bengt Ekerot', 'Katinka Faragó', 'Else Fisher', 'Åke Fridell', 'Inga Gill', 'Maud Hansson', 'Louis Huch', 'Inga Landgré', 'Gunnel Lindblom', 'Manne Lindholm', 'P.A. Lundgren', 'Åke Nilsson', 'Nils Nittel', 'Erik Nordgren', 'Gunnar Olsson', 'Lennart Olsson', 'Nils Poppe', 'Erik Strandmark', 'Lennart Wallén', 'Lennart Wallin', 'Aaby Wedin']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |\n",
      "|2503   |The Apple                                  |1998|4.043043613433838    |['Ghorban Ali Naderi', 'Hossein Amiri', 'Samira Makhmalbaf', 'Akbar Meshkini', 'Azizeh Mohamadi', 'Massoumeh Naderi', 'Zahra Naderi', 'Zahra Saghrisaz', 'Iraj Sarbaz', 'Reza Sheykhi', 'Caroline Phouthavy', 'John Robert Miller']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n",
      "|718    |The Visitors                               |1993|3.9844014644622803   |['Christian Clavier', 'Jean-Laurent Agrinier', 'Fabrice Anut', 'Patrick Archambault', \"François d'Artemare\", 'André Atellian', 'Pierre Aussedat', 'Eric Averlant', 'Paul Bandey', 'Gilles Bannier', 'Frédéric Baptiste', 'Brigitte Barbier', 'Muriel Baurens', 'Benoît Bayle', 'Isabelle Beauchesne', 'Jérôme Berthoud', 'Louis-Fernand Billet', 'Laure Blaess', 'Célina Blanc', 'Dominik Borde', 'Camille Bordes-Resnais', 'Sylviane Bouget', 'Amandine Boyadjian', 'Yohan Boyadjian', 'Christian Bujeau', 'Patrick Burgel', 'Didier Bénureau', 'Bernard Caroff', 'Jean-Luc Caron', 'Agnès Chaulier', 'Marie-Anne Chazel', 'Jean-Pierre Clami', 'Christophe Cojan', 'Laurence Coq', 'Rachel Corlet', 'Alain Cousseau', 'Gilbert Crozet', 'Thierry Crépin', 'Charlotte David', 'Francis de Gueltzl', 'Katia Delagarde', 'Éric Denize', 'Hervé Dieu', 'Frédéric Doniguian', 'Sébastien Drhey', 'Terrance Eugene Fredericks', 'David Gabison', 'Jean-Michel Gageat', 'Tara Gano', 'Leslie Rain', 'Jacques Gastineau', 'Emmanuel Gateau', 'Anna Gaylor', 'Lizi Gelber', 'Michel Grimaud', 'Bela Grushka', 'Michel Guarguir', 'Paul Gueu', 'Frédéric Hamelin', 'Pascal Henin', 'Dominique Hulin', 'Gisèle Jacquin', 'Fanny Jakubowicz', 'Sylvie Jouineau', 'Catherine Kelber', 'Gilles Laboulandine', 'François Lalande', 'Nadia Laurent', 'Jean-Guillaume Le Dantec', 'Jean-Yves Le Mener', 'Patrick Lefebvre', 'Valérie Lemercier', 'Bernard Lependu', 'Bernard Le Roux', 'Catherine Leterrier', 'Eric Levi', 'Thierry Liagre', 'Philippe Lièvre', 'Christophe Legal', 'Claire Magnin', 'Jean-Charles Maratier', 'Madeleine Marie', 'Stéphanie Marie', 'Daniel Mauvignier', 'Jacques Mironneau', 'Suzette Monlouis', 'Philippe Morlier', 'Jean-Marc Mouligne', 'Jean-Paul Muel', 'Isabelle Nanty', 'Roland Neunreuther', 'Eric Nové', 'Didier Pain', 'Chantal Pernecker', 'Michel Peyrelon', 'Pierre Pochy', 'Jean-Marie Poiré', 'Claude Pollan', 'Jérôme Prébois', 'André Raffard', 'Didier Ranz', 'Annick Redon', 'Dorian Rigal-Ansous', 'Dany Roca', 'Philippe Ros', 'Jean-Charles Ruault', 'Gary Saint-Martin', 'Julie Salvador', 'Jean-Pierre Sauvaire', 'Michel Scourneau', 'Bertrand Seitz', 'Théophile Sowié', 'Arielle Sémenoff', 'Gérard Séty', 'Jean-Claude Temporelli', 'Alain Terzian', 'Frank Thevenon', 'Evelyne Tison', 'Jean-Philippe Vallespir', 'Noël Vandendries-Lestienne', 'Nadia Vasil', 'Bobane Veselinovic', 'Sophie Vesoul', 'Pierre Vial', 'Thomas Vichy', 'Lionel Vitrant', 'Katja Weitzenböck', 'Patrick Widdrington', 'Olivier Wojciechowski', 'Adeline Yoyotte', 'Odile Alliet', 'Jean-Noël Andrieux', 'Sophie Antonuccio', 'Frédéric Atellian', 'Jean-Robert Blanche', 'Marie-Françoise Coureur Le Duc', 'Muriel Cravatte', 'Dominique Crepet', 'Badara Faye', 'Olivier Foucher', 'Jean Gouello', 'Bruno Goury', 'Stephan Guillemet', 'Jérôme Haumont', 'Florence Lagrange', 'Serge Margiocchi', 'Vincent Maroutian', 'Céline Migeon', 'Francis Pesquer', 'Jacques Puiseux', 'Gilles Rossire', 'Philippe Savoir', 'Jean-Marie Sonnet', 'Christophe Thiollier', 'Patrick Tombette', 'Manuel Jorge', 'Caroline Gautier', 'Benoît Lemaire', 'Daniel Maltret', 'Pierre Roger', 'Roger Saillard', 'Richard Brodet', 'Nathalie Delest', 'Caroline Emery', 'Jean-Louis Lalet', 'Carmen Lima', 'Anne Marchal', 'Gilles Poulin', 'Joel Matray', 'Bernard Sachsé', 'Dominique Delguste', 'Cédric Houplain', 'Bruno Lefebvre', 'Czech Symphony Orchestra']|\n",
      "|668    |Pather Panchali                            |1955|4.07059907913208     |['Satyajit Ray', 'Bibhutibhushan Bandyopadhyay', 'Haren Banerjee', 'Runki Banerjee', 'Kanu Bannerjee', 'Karuna Bannerjee', 'Subir Banerjee', 'Tulsi Chakraborty', 'Bansi Chandragupta', 'Santi Chatterjee', 'Anil Choudhury', 'Uma Das Gupta', 'Aparna Devi', 'Chunibala Devi', 'Nibhanani Devi', 'Reba Devi', 'Dulal Dutta', 'Rama Gangopadhaya', 'Roma Ganguli', 'Bhupen Ghosh', 'Dinen Gupta', 'Subrata Mitra', 'Binoy Mukherjee', 'Harimohan Nag', 'Kshirod Roy', 'Soumendu Roy', 'Dakshina Mohan Tagore', 'Aloknath Dey', 'Tapeshwar Prasad', 'Sukumar Sengupta', 'Satyen Chatterjee', 'Suresh Chandra Chandra', 'Subir Hajra', 'Harinarayan Mukhopadhyay', 'R.B. Mehta', 'Haridhan Nag', 'Nimai Roy', 'Pratap Narayan Mitra', 'Dibakar Dutta', 'Sitaram Das', 'Suren Sahu', 'Rampada Das', 'Bemananda Brajwasi', 'Ashish Barman', 'Baren Bhattacharya', 'Jagmohan Behrowa', 'Somen Gupta', 'Samir Ghosh', 'Durgacharan Mitra', 'Suren Roy']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n",
      "|2931   |Time of the Gypsies                        |1989|4.118833541870117    |['']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
      "|557    |Mamma Roma                                 |1962|4.144425868988037    |['Maria Bernardini', 'Nino Bionci', 'Franco Casati', 'Marcello Ceccarelli', 'Santino Citti', 'Silvana Corsini', \"Lina D'Amico\", 'Emanuele Di Bari', 'Pasquale Ferrarese', 'Fernando Franchi', 'Mirella Gamacchio', 'Ettore Garofolo', 'Luciano Gonini', 'Joselito', 'Vittorio La Paglia', 'Luisa Loiano', 'Lamberto Maggiorani', 'Sandro Meschino', 'Piero Morgia', 'Leandro Santarelli', 'Gioacchino Sofia', 'Marcello Sorrentino', 'Vincenzo Taito', 'Franco Tovo', 'Nino Venzi', 'Paolo Volponi', 'Giulio De Stephanis', 'Franco Ceccarelli']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |\n",
      "+-------+-------------------------------------------+----+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unseen_recommendations = give_unseen_recommandations_for_two_users(17, 312, als_model, sparkdf_movies, sparkdf_ratings)\n",
    "unseen_recommendations.select(\"MovieId\", \"Title\", \"Date\", \"AVG_Rating_Prediction\", \"cast\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**I don't know why but .desc() is not working well, so the results are not properly sorted**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We used the ALS algorithm to make recommendations to users. We evaluated our model using RMSE.\n",
    "\n",
    "We provided 2 types of recommendations:\n",
    "- One on all the movies availables in the list\n",
    "- The other only on new movies (the unseen movies)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "Overall, this project demonstrated the effectiveness of the ALS algorithm for movie recommendations, highlighting the importance of comprehensive data preprocessing and careful model evaluation. The resulting system provides a solid foundation for further enhancements and more sophisticated recommendation techniques like combining collaborative filtering with content-based filtering or deep learning models to address cold start and sparsity issues **(which was the primary goal)**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}