Fine Tune Sentence Transformer Embeddings without Model Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5bc0c0fe",
   "metadata": {},
   "source": [
    "<h2> Fine Tune Sentence Transformer Embeddings without Model Training </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d4bee683",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:40:41.043889Z",
     "start_time": "2022-10-02T07:40:26.844490Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, TFAutoModel\n",
    "from scipy.spatial.distance import cosine\n",
    "from sklearn.preprocessing import normalize\n",
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac54408e",
   "metadata": {},
   "source": [
    "<b> Sentence Transformers:</b> <p>Sentence Transformers is a Python framework for state-of-the-art sentence, text embeddings. It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like semantic search , paraphrase mining.</p>\n",
    "\n",
    "<b> Paraphrase-MiniLM-L6-v2:</b> <p> is based on BERT with 6 Transformer Encoder Layers,it can handle 512 tokens and return dense vector representation with 384 features</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f3d203f6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:40:41.074444Z",
     "start_time": "2022-10-02T07:40:41.045778Z"
    }
   },
   "outputs": [],
   "source": [
    "#Mean Pooling - Take attention mask into account for correct averaging\n",
    "def mean_pooling(model_output, attention_mask):\n",
    "    token_embeddings = model_output.last_hidden_state\n",
    "    input_mask_expanded = tf.cast(tf.tile(tf.expand_dims(attention_mask, -1), [1, 1, token_embeddings.shape[-1]]), tf.float32)\n",
    "    return tf.math.reduce_sum(token_embeddings * input_mask_expanded, 1) / tf.math.maximum(tf.math.reduce_sum(input_mask_expanded, 1), 1e-9)\n",
    "\n",
    "\n",
    "#Encode text\n",
    "def encode(tokenizer,texts):\n",
    "    # Tokenize sentences\n",
    "    encoded_input = tokenizer(texts, padding=True, truncation=True, return_tensors='tf')\n",
    "    \n",
    "    # Compute token embeddings\n",
    "    model_output = model(**encoded_input, return_dict=True)\n",
    "\n",
    "    # Perform pooling\n",
    "    embeddings = mean_pooling(model_output, encoded_input['attention_mask'])\n",
    "\n",
    "    # Normalize embeddings\n",
    "    embeddings = tf.math.l2_normalize(embeddings, axis=1)\n",
    "\n",
    "    return embeddings\n",
    "\n",
    "#cosine similarity Function\n",
    "def cosine_similarity(vector1,vector2):\n",
    "    return (1- cosine(vector1,vector2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7e09898a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:40:48.264506Z",
     "start_time": "2022-10-02T07:40:41.075721Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All model checkpoint layers were used when initializing TFBertModel.\n",
      "\n",
      "All the layers of TFBertModel were initialized from the model checkpoint at ./sentence-transformer-paraphrase-MiniLM-L6-v2.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained('./sentence-transformer-paraphrase-MiniLM-L6-v2')\n",
    "model = TFAutoModel.from_pretrained('./sentence-transformer-paraphrase-MiniLM-L6-v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a7f4b7",
   "metadata": {},
   "source": [
    "<b> Model Architecture</b>\n",
    "\n",
    "<div style=\"overflow-y: scroll; height:400px;\">\n",
    "<pre>\n",
    "BertModel(\n",
    "  (embeddings): BertEmbeddings(\n",
    "    (word_embeddings): Embedding(30522, 384, padding_idx=0)\n",
    "    (position_embeddings): Embedding(512, 384)\n",
    "    (token_type_embeddings): Embedding(2, 384)\n",
    "    (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "    (dropout): Dropout(p=0.1, inplace=False)\n",
    "  )\n",
    "  (encoder): BertEncoder(\n",
    "    (layer): ModuleList(\n",
    "      (0): BertLayer(\n",
    "        (attention): BertAttention(\n",
    "          (self): BertSelfAttention(\n",
    "            (query): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (key): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (value): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "          (output): BertSelfOutput(\n",
    "            (dense): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "        )\n",
    "        (intermediate): BertIntermediate(\n",
    "          (dense): Linear(in_features=384, out_features=1536, bias=True)\n",
    "          (intermediate_act_fn): GELUActivation()\n",
    "        )\n",
    "        (output): BertOutput(\n",
    "          (dense): Linear(in_features=1536, out_features=384, bias=True)\n",
    "          (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "          (dropout): Dropout(p=0.1, inplace=False)\n",
    "        )\n",
    "      )\n",
    "      (1): BertLayer(\n",
    "        (attention): BertAttention(\n",
    "          (self): BertSelfAttention(\n",
    "            (query): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (key): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (value): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "          (output): BertSelfOutput(\n",
    "            (dense): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "        )\n",
    "        (intermediate): BertIntermediate(\n",
    "          (dense): Linear(in_features=384, out_features=1536, bias=True)\n",
    "          (intermediate_act_fn): GELUActivation()\n",
    "        )\n",
    "        (output): BertOutput(\n",
    "          (dense): Linear(in_features=1536, out_features=384, bias=True)\n",
    "          (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "          (dropout): Dropout(p=0.1, inplace=False)\n",
    "        )\n",
    "      )\n",
    "      (2): BertLayer(\n",
    "        (attention): BertAttention(\n",
    "          (self): BertSelfAttention(\n",
    "            (query): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (key): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (value): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "          (output): BertSelfOutput(\n",
    "            (dense): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "        )\n",
    "        (intermediate): BertIntermediate(\n",
    "          (dense): Linear(in_features=384, out_features=1536, bias=True)\n",
    "          (intermediate_act_fn): GELUActivation()\n",
    "        )\n",
    "        (output): BertOutput(\n",
    "          (dense): Linear(in_features=1536, out_features=384, bias=True)\n",
    "          (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "          (dropout): Dropout(p=0.1, inplace=False)\n",
    "        )\n",
    "      )\n",
    "      (3): BertLayer(\n",
    "        (attention): BertAttention(\n",
    "          (self): BertSelfAttention(\n",
    "            (query): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (key): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (value): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "          (output): BertSelfOutput(\n",
    "            (dense): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "        )\n",
    "        (intermediate): BertIntermediate(\n",
    "          (dense): Linear(in_features=384, out_features=1536, bias=True)\n",
    "          (intermediate_act_fn): GELUActivation()\n",
    "        )\n",
    "        (output): BertOutput(\n",
    "          (dense): Linear(in_features=1536, out_features=384, bias=True)\n",
    "          (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "          (dropout): Dropout(p=0.1, inplace=False)\n",
    "        )\n",
    "      )\n",
    "      (4): BertLayer(\n",
    "        (attention): BertAttention(\n",
    "          (self): BertSelfAttention(\n",
    "            (query): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (key): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (value): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "          (output): BertSelfOutput(\n",
    "            (dense): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "        )\n",
    "        (intermediate): BertIntermediate(\n",
    "          (dense): Linear(in_features=384, out_features=1536, bias=True)\n",
    "          (intermediate_act_fn): GELUActivation()\n",
    "        )\n",
    "        (output): BertOutput(\n",
    "          (dense): Linear(in_features=1536, out_features=384, bias=True)\n",
    "          (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "          (dropout): Dropout(p=0.1, inplace=False)\n",
    "        )\n",
    "      )\n",
    "      (5): BertLayer(\n",
    "        (attention): BertAttention(\n",
    "          (self): BertSelfAttention(\n",
    "            (query): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (key): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (value): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "          (output): BertSelfOutput(\n",
    "            (dense): Linear(in_features=384, out_features=384, bias=True)\n",
    "            (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "            (dropout): Dropout(p=0.1, inplace=False)\n",
    "          )\n",
    "        )\n",
    "        (intermediate): BertIntermediate(\n",
    "          (dense): Linear(in_features=384, out_features=1536, bias=True)\n",
    "          (intermediate_act_fn): GELUActivation()\n",
    "        )\n",
    "        (output): BertOutput(\n",
    "          (dense): Linear(in_features=1536, out_features=384, bias=True)\n",
    "          (LayerNorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)\n",
    "          (dropout): Dropout(p=0.1, inplace=False)\n",
    "        )\n",
    "      )\n",
    "    )\n",
    "  )\n",
    "  (pooler): BertPooler(\n",
    "    (dense): Linear(in_features=384, out_features=384, bias=True)\n",
    "    (activation): Tanh()\n",
    "  )\n",
    ")\n",
    "</pre>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3815f99",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T06:44:51.868897Z",
     "start_time": "2022-10-02T06:44:51.856250Z"
    }
   },
   "source": [
    "<h3> Objective: </h3>\n",
    "<ol>\n",
    "    <li> Modify Existing word embeddings to adapt it for our use case</li>\n",
    "    <li> Add New Words to Vocabulary </li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6838c84b",
   "metadata": {},
   "source": [
    "<h4> Vocabulary Exploration </h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "df68805f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:40:50.933573Z",
     "start_time": "2022-10-02T07:40:50.725786Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30522\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('[PAD]', 0),\n",
       " ('[unused0]', 1),\n",
       " ('[unused1]', 2),\n",
       " ('[unused2]', 3),\n",
       " ('[unused3]', 4),\n",
       " ('[unused4]', 5),\n",
       " ('[unused5]', 6),\n",
       " ('[unused6]', 7),\n",
       " ('[unused7]', 8),\n",
       " ('[unused8]', 9)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_wrd2idx = tokenizer.vocab\n",
    "vocab_idx2wrd = {v:k for k,v in vocab_wrd2idx.items()}\n",
    "\n",
    "print(len(vocab_wrd2idx))\n",
    "\n",
    "sorted(vocab_wrd2idx.items(),key=lambda x:x[1])[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aeed051",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T06:47:47.531027Z",
     "start_time": "2022-10-02T06:47:47.515756Z"
    }
   },
   "source": [
    "<p> We have 30,522 tokens in our vocab for paraphrase-MiniLM-l6-v2 model.\n",
    "It has 993 tokens which are not used, represented by [unusedXXX], \n",
    "These are the token which we can replace to add new tokens to our vocab.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8323e27d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:40:54.643514Z",
     "start_time": "2022-10-02T07:40:54.255107Z"
    }
   },
   "outputs": [],
   "source": [
    "#extracting model weights\n",
    "model_weights = model.get_weights()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8190305c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:40:55.093668Z",
     "start_time": "2022-10-02T07:40:55.077079Z"
    }
   },
   "outputs": [],
   "source": [
    "#getting vocab weights\n",
    "vocab_weights = model_weights[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eef93795",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T06:49:03.051389Z",
     "start_time": "2022-10-02T06:49:03.032783Z"
    }
   },
   "source": [
    "<h3>1. Modify Existing word embeddings to adapt it for our use case </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d7dbde3",
   "metadata": {},
   "source": [
    "Based on your use case same word can have different meaning in different context although transformers are capable of handling such scenarios but a lot of places:\n",
    "\n",
    "<ul>\n",
    "    <li>It wouldn't have that much context to work with</li>\n",
    "    <li>The degree to which it's considering the closeness is not satisfactory</li>\n",
    "</ul>\n",
    "\n",
    "That's why fine-tuning is required on the data you are working with, but if you are not able to do it due to any reason , the following can help.\n",
    "\n",
    "<p> Eg: In Finance dataset, statements and bills are used synonymously, but since you are using a pretrained model\n",
    "    there is high possibility that your model is not considering it to a degree which you would it to</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66504027",
   "metadata": {},
   "source": [
    "Let's understand the distribution of words close to statement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "29ef410c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:42:52.441285Z",
     "start_time": "2022-10-02T07:42:52.422353Z"
    }
   },
   "outputs": [],
   "source": [
    "#Function to find top_k similar words\n",
    "def most_similar(search_word,top_k=5):\n",
    "    \"\"\"This function takes a word and compute cosine similarity between the given word\n",
    "       and all other words and return top_k most similar words\"\"\"\n",
    "    search_idx = vocab_wrd2idx[search_word]\n",
    "    similarity_ls=[]\n",
    "    for word_idx,word_embed in enumerate(vocab_weights):\n",
    "        similarity_score = cosine_similarity(vocab_weights[search_idx],word_embed)\n",
    "        similarity_ls.append((word_idx,vocab_idx2wrd[word_idx],similarity_score))\n",
    "    \n",
    "    return pd.DataFrame(similarity_ls,columns=['word_index','word','score']).sort_values(by='score',ascending=False)[0:top_k]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3ecc2a80",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:42:57.966846Z",
     "start_time": "2022-10-02T07:42:52.782316Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_index</th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4861</th>\n",
       "      <td>4861</td>\n",
       "      <td>statement</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8635</th>\n",
       "      <td>8635</td>\n",
       "      <td>statements</td>\n",
       "      <td>0.635414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15974</th>\n",
       "      <td>15974</td>\n",
       "      <td>spokesperson</td>\n",
       "      <td>0.359564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7615</th>\n",
       "      <td>7615</td>\n",
       "      <td>comment</td>\n",
       "      <td>0.358950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8874</th>\n",
       "      <td>8874</td>\n",
       "      <td>announcement</td>\n",
       "      <td>0.354991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14056</th>\n",
       "      <td>14056</td>\n",
       "      <td>spokesman</td>\n",
       "      <td>0.353516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3661</th>\n",
       "      <td>3661</td>\n",
       "      <td>letter</td>\n",
       "      <td>0.352468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8170</th>\n",
       "      <td>8170</td>\n",
       "      <td>declaration</td>\n",
       "      <td>0.337585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12629</th>\n",
       "      <td>12629</td>\n",
       "      <td>remarks</td>\n",
       "      <td>0.329412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23617</th>\n",
       "      <td>23617</td>\n",
       "      <td>assertion</td>\n",
       "      <td>0.319534</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       word_index          word     score\n",
       "4861         4861     statement  1.000000\n",
       "8635         8635    statements  0.635414\n",
       "15974       15974  spokesperson  0.359564\n",
       "7615         7615       comment  0.358950\n",
       "8874         8874  announcement  0.354991\n",
       "14056       14056     spokesman  0.353516\n",
       "3661         3661        letter  0.352468\n",
       "8170         8170   declaration  0.337585\n",
       "12629       12629       remarks  0.329412\n",
       "23617       23617     assertion  0.319534"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('statement',top_k=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e09cfdc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T06:50:32.151705Z",
     "start_time": "2022-10-02T06:50:32.140059Z"
    }
   },
   "source": [
    "From the above output we see that the general context for the word \"statement\" is considered as the data distribution of the corpus used broad, so a general model of language is attained."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91e44bfe",
   "metadata": {},
   "source": [
    "Let's understand the vector closeness between statement and bill"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0b8f1a5d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:43:48.773709Z",
     "start_time": "2022-10-02T07:43:48.759747Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1263842135667801"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_similarity(vocab_weights[vocab_wrd2idx['statement']],vocab_weights[vocab_wrd2idx['bill']])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c7d8e15",
   "metadata": {},
   "source": [
    "From the above result, we can see that the similarity between the word \"statement\" and \"bill\" is very less than we would like it to be,\n",
    "now with transformer model this comparison technique is not completely accurate, as we are just taking the embeddings from the first layer where no context modelling is done.\n",
    "\n",
    "Let's look at the similarity between encoded sentences with the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "207187a6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:45:30.857086Z",
     "start_time": "2022-10-02T07:45:30.041621Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7034520506858826"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'please send me the statement'\n",
    "text2 = 'please transfer the statement'\n",
    "\n",
    "vector1 = encode(tokenizer,text1)\n",
    "vector2 = encode(tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af8571f2",
   "metadata": {},
   "source": [
    "Both sentences are similar, and we are getting a similarity score of 0.70 which is somewhat reasonable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a4460e72",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:45:49.383517Z",
     "start_time": "2022-10-02T07:45:48.451900Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3917175233364105"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'please send me the bill'\n",
    "text2 = 'please transfer the statement'\n",
    "\n",
    "vector1 = encode(tokenizer,text1)\n",
    "vector2 = encode(tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d83f92ba",
   "metadata": {},
   "source": [
    "Now with statement replaced in one sentence the score is dropping by a lot, ideally, we would like it to be greater than 50% at least."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74992a7f",
   "metadata": {},
   "source": [
    "<h3> Let's understand the role of context here</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2821c594",
   "metadata": {},
   "source": [
    "<b>Example 1:</b> Without proper context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bc827dd7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:47:10.101838Z",
     "start_time": "2022-10-02T07:47:09.300933Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3873918354511261"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'i did not recieve any bill' #from the bank\n",
    "text2 = \"i did'nt got any statement\" #from the bank\n",
    "\n",
    "vector1 = encode(tokenizer,text1)\n",
    "vector2 = encode(tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "150ad1af",
   "metadata": {},
   "source": [
    "<b> Result: </b> Similarity Score is very low as general meaning of bill is considered here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "860bc8cb",
   "metadata": {},
   "source": [
    "<b>Example 2:</b> With Added Context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "12efd854",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:47:12.333366Z",
     "start_time": "2022-10-02T07:47:11.921918Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6437106728553772"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'i did not recieve any bill from the bank'\n",
    "text2 = \"i did'nt got any statement from the bank\"\n",
    "\n",
    "vector1 = encode(tokenizer,text1)\n",
    "vector2 = encode(tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36ff6f07",
   "metadata": {},
   "source": [
    "<b>Result:</b> just by adding \"from the bank\", we can observe a significant improvement from 0.38 to 0.64 in the similarity score."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1928608e",
   "metadata": {},
   "source": [
    "But while working with the data it's not necessary that a proper context will be always present, let's go back to our previous example and try to modify the embeddings to get reasonable similarity score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3ee7ccee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:50:36.855597Z",
     "start_time": "2022-10-02T07:50:36.835338Z"
    }
   },
   "outputs": [],
   "source": [
    "def weighted_average_of_vectors(word_wghts):\n",
    "    \"\"\"This Function takes words and thier weights and creates a new vector taking weighted avg of the vectors\"\"\"\n",
    "    \n",
    "    vector = np.zeros(384, dtype=np.float64)\n",
    "    \n",
    "    for key,value in word_wghts.items():\n",
    "        vector += (1 - value) * vocab_weights[vocab_wrd2idx[key]]\n",
    "    \n",
    "    vector = normalize(vector.reshape(1,-1))[0]\n",
    "    \n",
    "    return vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4019ab77",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:50:37.965178Z",
     "start_time": "2022-10-02T07:50:37.940033Z"
    }
   },
   "outputs": [],
   "source": [
    "#modifying the embedding of statement to incorporate meaning of bill as well in the vector\n",
    "\n",
    "stmt_wghts = {\n",
    "    'statement':0.5,\n",
    "    'bill':0.5\n",
    "}\n",
    "\n",
    "vocab_weights[vocab_wrd2idx['statement']] = weighted_average_of_vectors(stmt_wghts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7e99f9ef",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:50:58.153925Z",
     "start_time": "2022-10-02T07:50:57.593027Z"
    }
   },
   "outputs": [],
   "source": [
    "#settings updated weights\n",
    "model_weights[0] = vocab_weights\n",
    "model.set_weights(model_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9597b574",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:51:02.270536Z",
     "start_time": "2022-10-02T07:50:58.160217Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_index</th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4861</th>\n",
       "      <td>4861</td>\n",
       "      <td>statement</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3021</th>\n",
       "      <td>3021</td>\n",
       "      <td>bill</td>\n",
       "      <td>0.761461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8236</th>\n",
       "      <td>8236</td>\n",
       "      <td>bills</td>\n",
       "      <td>0.520205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8635</th>\n",
       "      <td>8635</td>\n",
       "      <td>statements</td>\n",
       "      <td>0.465498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3661</th>\n",
       "      <td>3661</td>\n",
       "      <td>letter</td>\n",
       "      <td>0.393868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6094</th>\n",
       "      <td>6094</td>\n",
       "      <td>legislation</td>\n",
       "      <td>0.375259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2552</th>\n",
       "      <td>2552</td>\n",
       "      <td>act</td>\n",
       "      <td>0.331694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2928</th>\n",
       "      <td>2928</td>\n",
       "      <td>mark</td>\n",
       "      <td>0.313210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3189</th>\n",
       "      <td>3189</td>\n",
       "      <td>report</td>\n",
       "      <td>0.308153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3820</th>\n",
       "      <td>3820</td>\n",
       "      <td>agreement</td>\n",
       "      <td>0.294916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      word_index         word     score\n",
       "4861        4861    statement  1.000000\n",
       "3021        3021         bill  0.761461\n",
       "8236        8236        bills  0.520205\n",
       "8635        8635   statements  0.465498\n",
       "3661        3661       letter  0.393868\n",
       "6094        6094  legislation  0.375259\n",
       "2552        2552          act  0.331694\n",
       "2928        2928         mark  0.313210\n",
       "3189        3189       report  0.308153\n",
       "3820        3820    agreement  0.294916"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('statement',top_k=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "630a2e07",
   "metadata": {},
   "source": [
    "<b>Result</b>: From the above result we can observe that now bill and statement vectors are nearby in the vector space"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d864f297",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:14:29.350327Z",
     "start_time": "2022-10-02T07:14:29.337759Z"
    }
   },
   "source": [
    "<b> Example 3:</b> After Updating the Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "21e4b8df",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:51:07.739831Z",
     "start_time": "2022-10-02T07:51:07.293186Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5900149941444397"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'please send me the bill'\n",
    "text2 = 'please transfer the statement'\n",
    "\n",
    "vector1 = encode(tokenizer,text1)\n",
    "vector2 = encode(tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc97b76",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:12:17.708063Z",
     "start_time": "2022-10-02T07:12:17.691801Z"
    }
   },
   "source": [
    "<b>Result:</b> Now with the embeddings modified we see a significant jump in similarity score from 0.38 (Example 1) to 0.59"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3315d81",
   "metadata": {},
   "source": [
    "<h2>2. Add New Words to Vocabulary </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "96d697d8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:51:14.089845Z",
     "start_time": "2022-10-02T07:51:14.077065Z"
    }
   },
   "outputs": [],
   "source": [
    "def add_word(word_to_replace,word_to_add,word_wghts):\n",
    "    '''This Function takes the token to replace , word to add and word weights,\n",
    "       and replaces the word and it's embedding.\n",
    "    '''\n",
    "    #editing the dictionary \n",
    "    vocab_index = vocab_wrd2idx[word_to_replace]\n",
    "    vocab_wrd2idx[word_to_add] = vocab_index\n",
    "    del vocab_wrd2idx[vocab_idx2wrd[vocab_index]]\n",
    "    vocab_idx2wrd[vocab_index] = word_to_add\n",
    "    \n",
    "    #add the vector\n",
    "    vocab_weights[vocab_index] = weighted_average_of_vectors(word_wghts)\n",
    "    \n",
    "    #settings weights\n",
    "    model_weights[0] = vocab_weights\n",
    "    model.set_weights(model_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "43e660af",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:51:15.289165Z",
     "start_time": "2022-10-02T07:51:15.152720Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[False, False, True]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking if words are in vocab\n",
    "\n",
    "words_to_add = [\"stmt\",\"ftp\",'http']\n",
    "\n",
    "[wrd in tokenizer.vocab for wrd in words_to_add]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d31849dd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:24:58.708556Z",
     "start_time": "2022-10-02T07:24:58.683572Z"
    }
   },
   "source": [
    "<b>Result:</b> only http is present in the vocab. ftp and stmt (short for stament) is not present (you can always replace the word at preprocessing,this is just for demonstration)\n",
    "\n",
    "Let's try to add the 'ftp' and 'stmt' to the vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ea4dc8f5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:51:45.750622Z",
     "start_time": "2022-10-02T07:51:45.033147Z"
    }
   },
   "outputs": [],
   "source": [
    "#defining weights for new words\n",
    "\n",
    "ftp_wghts = {\n",
    "    'file':0.3,\n",
    "    'transfer':0.3,\n",
    "    'protocol':0.4,\n",
    "}\n",
    "\n",
    "stmt_wghts = {\n",
    "    'statement':0.5,\n",
    "    'bill':0.5\n",
    "}\n",
    "\n",
    "#adding words\n",
    "add_word('[unused900]','ftp',ftp_wghts)\n",
    "add_word('[unused902]','stmt',stmt_wghts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "44d80caa",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:51:50.483469Z",
     "start_time": "2022-10-02T07:51:46.575172Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_index</th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>907</th>\n",
       "      <td>907</td>\n",
       "      <td>stmt</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3021</th>\n",
       "      <td>3021</td>\n",
       "      <td>bill</td>\n",
       "      <td>0.955079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4861</th>\n",
       "      <td>4861</td>\n",
       "      <td>statement</td>\n",
       "      <td>0.919354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8236</th>\n",
       "      <td>8236</td>\n",
       "      <td>bills</td>\n",
       "      <td>0.658201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6094</th>\n",
       "      <td>6094</td>\n",
       "      <td>legislation</td>\n",
       "      <td>0.446626</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      word_index         word     score\n",
       "907          907         stmt  1.000000\n",
       "3021        3021         bill  0.955079\n",
       "4861        4861    statement  0.919354\n",
       "8236        8236        bills  0.658201\n",
       "6094        6094  legislation  0.446626"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('stmt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "770e58dc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:51:57.930164Z",
     "start_time": "2022-10-02T07:51:53.835439Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_index</th>\n",
       "      <th>word</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>905</th>\n",
       "      <td>905</td>\n",
       "      <td>ftp</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5371</th>\n",
       "      <td>5371</td>\n",
       "      <td>file</td>\n",
       "      <td>0.691518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4651</th>\n",
       "      <td>4651</td>\n",
       "      <td>transfer</td>\n",
       "      <td>0.664068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8778</th>\n",
       "      <td>8778</td>\n",
       "      <td>protocol</td>\n",
       "      <td>0.572106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6764</th>\n",
       "      <td>6764</td>\n",
       "      <td>files</td>\n",
       "      <td>0.536686</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      word_index      word     score\n",
       "905          905       ftp  1.000000\n",
       "5371        5371      file  0.691518\n",
       "4651        4651  transfer  0.664068\n",
       "8778        8778  protocol  0.572106\n",
       "6764        6764     files  0.536686"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('ftp')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "885bc77b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:28:07.489060Z",
     "start_time": "2022-10-02T07:28:07.469720Z"
    }
   },
   "source": [
    "<b> Inference: </b> From the above table we can infer that the new words added have a proper vector representation and are close to similar meaning words."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e53afb4",
   "metadata": {},
   "source": [
    "We have successfully added the new word and the embeddings, but the tokenizer vocab is not updated, it cannot be directly updated,\n",
    "so, we will be write our vocab dictionary to the tokenizer.json and then load the updated tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "0b285f70",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:52:54.076159Z",
     "start_time": "2022-10-02T07:52:53.694146Z"
    }
   },
   "outputs": [],
   "source": [
    "#Updating the tokenizer\n",
    "with open('./sentence-transformer-paraphrase-MiniLM-L6-v2/tokenizer.json','r',encoding='utf-8') as f:\n",
    "    tokenizer_json = json.load(f)\n",
    "    \n",
    "tokenizer_json['model']['vocab']=dict(sorted(vocab_wrd2idx.items(),key=lambda x:x[1]))\n",
    "\n",
    "#renaming vocab file \n",
    "\n",
    "os.chdir('./sentence-transformer-paraphrase-MiniLM-L6-v2')\n",
    "\n",
    "#renaming old vocab file\n",
    "!ren tokenizer.json tokenizer_old.json\n",
    "\n",
    "os.chdir('../')\n",
    "\n",
    "with open('./sentence-transformer-paraphrase-MiniLM-L6-v2/tokenizer.json','w',encoding='utf-8') as f:\n",
    "    json.dump(tokenizer_json,f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "657779c6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:52:56.724704Z",
     "start_time": "2022-10-02T07:52:56.636890Z"
    }
   },
   "outputs": [],
   "source": [
    "updated_tokenizer = AutoTokenizer.from_pretrained('./sentence-transformer-paraphrase-MiniLM-L6-v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cdaef195",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:52:57.362708Z",
     "start_time": "2022-10-02T07:52:57.269760Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[True, True, True]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking if words are in vocab\n",
    "\n",
    "words_to_add = [\"stmt\",\"ftp\",'http']\n",
    "\n",
    "[wrd in updated_tokenizer.vocab for wrd in words_to_add]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d05603b9",
   "metadata": {},
   "source": [
    "<b> Inference: </b> The New Tokenizer has all the updated words."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbae8959",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:33:51.185939Z",
     "start_time": "2022-10-02T07:33:51.173175Z"
    }
   },
   "source": [
    "<h3> Sample 1: </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf581bc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:34:07.399456Z",
     "start_time": "2022-10-02T07:34:07.375769Z"
    }
   },
   "source": [
    "<h6> With Old Tokenizer without the word stmt added </h6>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "27e0755f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:53:03.522492Z",
     "start_time": "2022-10-02T07:53:02.715064Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3759283423423767"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'the statement is incorrect'\n",
    "text2 = 'the stmt is wrong'\n",
    "\n",
    "vector1 = encode(tokenizer,text1)\n",
    "vector2 = encode(tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3a0573",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:34:48.748635Z",
     "start_time": "2022-10-02T07:34:48.729265Z"
    }
   },
   "source": [
    "<h6> With Updated Tokenizer containing the word stmt</h6>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3b7970be",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:53:05.321980Z",
     "start_time": "2022-10-02T07:53:04.873397Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8123685717582703"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'the statement is incorrect'\n",
    "text2 = 'the stmt is wrong'\n",
    "\n",
    "vector1 = encode(updated_tokenizer,text1)\n",
    "vector2 = encode(updated_tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3b23a4",
   "metadata": {},
   "source": [
    "<b> Result:</b> From the above Example we observe a significant improvement on Similarity Score, as the model now has an understanding of the word stmt, Earlier the old Tokenizer was breaking the word <i>\"stmt\"</i> into <i>'st'</i> and <i>'##mt'</i> as it couldn't find stmt in the vocab."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5581c970",
   "metadata": {},
   "source": [
    "<h3> Sample 2: </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f6ad0a",
   "metadata": {},
   "source": [
    "<h6> With Old Tokenizer without the word ftp added </h6>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ecabcaaa",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:54:14.067107Z",
     "start_time": "2022-10-02T07:54:13.246760Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6022258400917053"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'unable to upload the data through ftp'\n",
    "text2 = 'file transfer protocol upload is not working'\n",
    "\n",
    "vector1 = encode(tokenizer,text1)\n",
    "vector2 = encode(tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c2ea6b6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:37:28.500372Z",
     "start_time": "2022-10-02T07:37:28.476674Z"
    }
   },
   "source": [
    "<h6> With Updated Tokenizer containing the word ftp</h6>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "4711f563",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:53:14.760859Z",
     "start_time": "2022-10-02T07:53:14.321679Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.766219437122345"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'unable to upload the data through ftp'\n",
    "text2 = 'file transfer protocol upload is not working'\n",
    "\n",
    "vector1 = encode(updated_tokenizer,text1)\n",
    "vector2 = encode(updated_tokenizer,text2)\n",
    "\n",
    "cosine_similarity(vector1,vector2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "723f16db",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:37:55.898664Z",
     "start_time": "2022-10-02T07:37:55.886614Z"
    }
   },
   "source": [
    "<b> Result:</b> In this case as well, we can observe a good increase in Similarity Score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba7205b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-02T07:38:30.003674Z",
     "start_time": "2022-10-02T07:38:29.977837Z"
    }
   },
   "source": [
    "<p>\n",
    "    <b> Conclusion: From the above experiments </b>\n",
    "    <ul>\n",
    "        <li>We learnt how we can add and modify the vectors manually to Increase the model Performance</li>\n",
    "        <li>We understood the model Architecture, and understood why there are so many [unused0] tokens available in the vocabulary</li>\n",
    "        <li>Built Functions like:\n",
    "            <ul>\n",
    "                <li><i>most_similar</i> to understand the words similarity in vector space </li>\n",
    "                <li><i>add_words,weighted_average_of_vectors</i> to add new words to our vocab</li>\n",
    "            </ul>\n",
    "        <li> Understood the importance of context for Embeddings</li>\n",
    "</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}