Linear Regression Consulting Mini Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Consulting Mini Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "[Hyundai Heavy Industries](http://www.hyundai.eu/en) is one of the world's largest ship manufacturing companies and builds cruise liners.\n",
    "\n",
    "You've been flown to their headquarters in Ulsan, South Korea to help them give accurate estimates of how many crew members a ship will require.\n",
    "\n",
    "They are currently building new ships for some customers and want you to create a model and use it to predict how many crew members the ships will need.\n",
    "\n",
    "Here is what the data looks like so far:\n",
    "\n",
    "    Description: Measurements of ship size, capacity, crew, and age for 158 cruise\n",
    "    ships.\n",
    "\n",
    "\n",
    "    Variables/Columns\n",
    "    Ship Name     1-20\n",
    "    Cruise Line   21-40\n",
    "    Age (as of 2013)   46-48\n",
    "    Tonnage (1000s of tons)   50-56\n",
    "    passengers (100s)   58-64\n",
    "    Length (100s of feet)  66-72\n",
    "    Cabins  (100s)   74-80\n",
    "    Passenger Density   82-88\n",
    "    Crew  (100s)   90-96\n",
    "    \n",
    "It is saved in a csv file for you called \"cruise_ship_info.csv\". Your job is to create a regression model that will help predict how many crew members will be needed for future ships. The client also mentioned that they have found that particular cruise lines will differ in acceptable crew counts, so it is most likely an important feature to include in your analysis! \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName('cruise').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.csv('cruise_ship_info.csv',inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Ship_name: string (nullable = true)\n",
      " |-- Cruise_line: string (nullable = true)\n",
      " |-- Age: integer (nullable = true)\n",
      " |-- Tonnage: double (nullable = true)\n",
      " |-- passengers: double (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- cabins: double (nullable = true)\n",
      " |-- passenger_density: double (nullable = true)\n",
      " |-- crew: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|  Ship_name|Cruise_line|Age|           Tonnage|passengers|length|cabins|passenger_density|crew|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "|    Journey|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|      Quest|    Azamara|  6|30.276999999999997|      6.94|  5.94|  3.55|            42.64|3.55|\n",
      "|Celebration|   Carnival| 26|            47.262|     14.86|  7.22|  7.43|             31.8| 6.7|\n",
      "|   Conquest|   Carnival| 11|             110.0|     29.74|  9.53| 14.88|            36.99|19.1|\n",
      "|    Destiny|   Carnival| 17|           101.353|     26.42|  8.92| 13.21|            38.36|10.0|\n",
      "|    Ecstasy|   Carnival| 22|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Elation|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Fantasy|   Carnival| 23|            70.367|     20.56|  8.55| 10.22|            34.23| 9.2|\n",
      "|Fascination|   Carnival| 19|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|    Freedom|   Carnival|  6|110.23899999999999|      37.0|  9.51| 14.87|            29.79|11.5|\n",
      "|      Glory|   Carnival| 10|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Holiday|   Carnival| 28|            46.052|     14.52|  7.27|  7.26|            31.72| 6.6|\n",
      "|Imagination|   Carnival| 18|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|Inspiration|   Carnival| 17|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|     Legend|   Carnival| 11|              86.0|     21.24|  9.63| 10.62|            40.49| 9.3|\n",
      "|   Liberty*|   Carnival|  8|             110.0|     29.74|  9.51| 14.87|            36.99|11.6|\n",
      "|    Miracle|   Carnival|  9|              88.5|     21.24|  9.63| 10.62|            41.67|10.3|\n",
      "|   Paradise|   Carnival| 15|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "|      Pride|   Carnival| 12|              88.5|     21.24|  9.63| 11.62|            41.67| 9.3|\n",
      "|  Sensation|   Carnival| 20|            70.367|     20.52|  8.55|  10.2|            34.29| 9.2|\n",
      "+-----------+-----------+---+------------------+----------+------+------+-----------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|summary|Ship_name|Cruise_line|               Age|           Tonnage|       passengers|           length|            cabins|passenger_density|             crew|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "|  count|      158|        158|               158|               158|              158|              158|               158|              158|              158|\n",
      "|   mean| Infinity|       null|15.689873417721518| 71.28467088607599|18.45740506329114|8.130632911392404| 8.830000000000005|39.90094936708861|7.794177215189873|\n",
      "| stddev|      NaN|       null| 7.615691058751413|37.229540025907866|9.677094775143416|1.793473548054825|4.4714172221480615| 8.63921711391542|3.503486564627034|\n",
      "|    min|Adventure|    Azamara|                 4|             2.329|             0.66|             2.79|              0.33|             17.7|             0.59|\n",
      "|    max|Zuiderdam|   Windstar|                48|             220.0|             54.0|            11.82|              27.0|            71.43|             21.0|\n",
      "+-------+---------+-----------+------------------+------------------+-----------------+-----------------+------------------+-----------------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.describe().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with the Cruise_line categorical variable\n",
    "Ship Name is a useless arbitrary string, but the cruise_line itself may be useful. Let's make it into a categorical variable!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------+-----+\n",
      "|      Cruise_line|count|\n",
      "+-----------------+-----+\n",
      "|            Costa|   11|\n",
      "|              P&O|    6|\n",
      "|           Cunard|    3|\n",
      "|Regent_Seven_Seas|    5|\n",
      "|              MSC|    8|\n",
      "|         Carnival|   22|\n",
      "|          Crystal|    2|\n",
      "|           Orient|    1|\n",
      "|         Princess|   17|\n",
      "|        Silversea|    4|\n",
      "|         Seabourn|    3|\n",
      "| Holland_American|   14|\n",
      "|         Windstar|    3|\n",
      "|           Disney|    2|\n",
      "|        Norwegian|   13|\n",
      "|          Oceania|    3|\n",
      "|          Azamara|    2|\n",
      "|        Celebrity|   10|\n",
      "|             Star|    6|\n",
      "|  Royal_Caribbean|   23|\n",
      "+-----------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.groupBy('Cruise_line').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Ship_name='Journey', Cruise_line='Azamara', Age=6, Tonnage=30.276999999999997, passengers=6.94, length=5.94, cabins=3.55, passenger_density=42.64, crew=3.55, cruise_cat=16.0),\n",
       " Row(Ship_name='Quest', Cruise_line='Azamara', Age=6, Tonnage=30.276999999999997, passengers=6.94, length=5.94, cabins=3.55, passenger_density=42.64, crew=3.55, cruise_cat=16.0),\n",
       " Row(Ship_name='Celebration', Cruise_line='Carnival', Age=26, Tonnage=47.262, passengers=14.86, length=7.22, cabins=7.43, passenger_density=31.8, crew=6.7, cruise_cat=1.0),\n",
       " Row(Ship_name='Conquest', Cruise_line='Carnival', Age=11, Tonnage=110.0, passengers=29.74, length=9.53, cabins=14.88, passenger_density=36.99, crew=19.1, cruise_cat=1.0),\n",
       " Row(Ship_name='Destiny', Cruise_line='Carnival', Age=17, Tonnage=101.353, passengers=26.42, length=8.92, cabins=13.21, passenger_density=38.36, crew=10.0, cruise_cat=1.0)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.ml.feature import StringIndexer\n",
    "indexer = StringIndexer(inputCol=\"Cruise_line\", outputCol=\"cruise_cat\")\n",
    "indexed = indexer.fit(df).transform(df)\n",
    "indexed.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Ship_name',\n",
       " 'Cruise_line',\n",
       " 'Age',\n",
       " 'Tonnage',\n",
       " 'passengers',\n",
       " 'length',\n",
       " 'cabins',\n",
       " 'passenger_density',\n",
       " 'crew',\n",
       " 'cruise_cat']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexed.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(\n",
    "  inputCols=['Age',\n",
    "             'Tonnage',\n",
    "             'passengers',\n",
    "             'length',\n",
    "             'cabins',\n",
    "             'passenger_density',\n",
    "             'cruise_cat'],\n",
    "    outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = assembler.transform(indexed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----+\n",
      "|            features|crew|\n",
      "+--------------------+----+\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[6.0,30.276999999...|3.55|\n",
      "|[26.0,47.262,14.8...| 6.7|\n",
      "|[11.0,110.0,29.74...|19.1|\n",
      "|[17.0,101.353,26....|10.0|\n",
      "|[22.0,70.367,20.5...| 9.2|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[23.0,70.367,20.5...| 9.2|\n",
      "|[19.0,70.367,20.5...| 9.2|\n",
      "|[6.0,110.23899999...|11.5|\n",
      "|[10.0,110.0,29.74...|11.6|\n",
      "|[28.0,46.052,14.5...| 6.6|\n",
      "|[18.0,70.367,20.5...| 9.2|\n",
      "|[17.0,70.367,20.5...| 9.2|\n",
      "|[11.0,86.0,21.24,...| 9.3|\n",
      "|[8.0,110.0,29.74,...|11.6|\n",
      "|[9.0,88.5,21.24,9...|10.3|\n",
      "|[15.0,70.367,20.5...| 9.2|\n",
      "|[12.0,88.5,21.24,...| 9.3|\n",
      "|[20.0,70.367,20.5...| 9.2|\n",
      "+--------------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.select(\"features\", \"crew\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_data = output.select(\"features\", \"crew\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data,test_data = final_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.regression import LinearRegression\n",
    "# Create a Linear Regression Model object\n",
    "lr = LinearRegression(labelCol='crew')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model to the data and call this model lrModel\n",
    "lrModel = lr.fit(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficients: [-0.011708645221028638,0.013997830480544982,-0.1358927421629388,0.41992691070705523,0.7690336769628063,-0.00649467639968532,0.051438607416677634] Intercept: -0.7145771274521302\n"
     ]
    }
   ],
   "source": [
    "# Print the coefficients and intercept for linear regression\n",
    "print(\"Coefficients: {} Intercept: {}\".format(lrModel.coefficients,lrModel.intercept))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results = lrModel.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.2165589917056279\n",
      "MSE: 1.480015780299814\n",
      "R2: 0.8939244717316173\n"
     ]
    }
   ],
   "source": [
    "print(\"RMSE: {}\".format(test_results.rootMeanSquaredError))\n",
    "print(\"MSE: {}\".format(test_results.meanSquaredError))\n",
    "print(\"R2: {}\".format(test_results.r2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# R2 of 0.86 is pretty good, let's check the data a little closer\n",
    "from pyspark.sql.functions import corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------+\n",
      "|corr(crew, passengers)|\n",
      "+----------------------+\n",
      "|    0.9152341306065384|\n",
      "+----------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr('crew','passengers')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+\n",
      "|corr(crew, cabins)|\n",
      "+------------------+\n",
      "|0.9508226063578497|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(corr('crew','cabins')).show()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}