Parte 1. S03. Instalacion de las herramientas para el Taller de SparkR.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "370bfa4e-8166-45db-a647-fafb435f326a"
    }
   },
   "source": [
    "<span style='font-size:2.4em'>Procesamiento masivo de datos con SparkR</span>\n",
    "\n",
    "<span style='font-size:1.5em'>VIII Jornadas de usuarios de R. Albacete, Castilla-La Mancha, 17 y 18 de noviembre de 2016</span>\n",
    "\n",
    "Taller impartido por: <span style='font-size:1.2em'>Manuel Jesús Parra Royón</span>\n",
    "\n",
    "\n",
    "![Alt](https://sites.google.com/site/manuparra/home/logoparty.png)\n",
    "\n",
    "<HR>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "c698a2da-cc7f-4b9b-809d-0761b5113cc6"
    }
   },
   "source": [
    "# Instalación y despliegue de la infraestructura del Taller."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "9e2fe0c4-601e-4d73-a965-669b2bea7448"
    }
   },
   "source": [
    "![Spark+R](https://sites.google.com/site/manuparra/home/SparkRlogo.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Para el taller se ha creado una Máquina Virtual  en VirtualBox que contiene"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Spark versión 2.01."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Scala versión 2.16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* R versión 3.3.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Jupyter Notebooks ( y Conda )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Zeppelin 0.62"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se ha tratado de incluir todos los componentes en las versiones más actualizados posibles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requisitos necesarios para trabajar con la Máquina Virtual:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Tener instalado VIRTUALBOX, disponible en: https://www.virtualbox.org/wiki/Downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "* Disponer de al menos 2GB de RAM para la Máquina Virtual"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "* El PC debe ser de 64bits y contar con al menos 4GB de RAM (2GB para la MVirtual y otros 2GB para el PC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "* Compatible con Windows, Mac OSX y Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Descarga de la Máquina Virtual: All-in-one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La máquina virtual que proveemos para el Taller completo contiene todas las herramientas necesitas para el desarrollo del trabajo, por lo tanto es la opción más sencilla para poder empezar el taller.\n",
    "\n",
    "* Descargar la máquina virtual del taller: https://drive.google.com/file/d/0ByPBMv-S_GMEakRCVVRTejZKVm8/view?usp=sharing  (aprox: 4GB).\n",
    "* Una vez descargda la Máquina Virtual, ejecutar el fichero y se importará a VirtualBox.\n",
    "* Hecho esto, ejecutar la nueva máquina instalada.\n",
    "\n",
    "Los datos de acceso a la Máquina Virtual son:\n",
    "\n",
    "* usuario: **``root``**\n",
    "* clave: **``sparkR``**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Descarga e instalación manual de las herramientas de taller:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se recomenda usar la Máquina virtual antes que instalar todo manualmente. \n",
    "\n",
    "El tutorial para la instalación de todas las herramientas para el taller es el siguiente:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actualizamos:\n",
    "\n",
    "```\n",
    "yum update\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instalamos  R:\n",
    "\n",
    "```\n",
    "yum install R\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Descargamos Spark 2.0.1\n",
    "\n",
    "```\n",
    "curl -O http://apache.rediris.es/spark/spark-2.0.1/spark-2.0.1-bin-hadoop2.7.tgz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Descomprimimos el fichero de Spark:\n",
    "\n",
    "```\n",
    "tar xfvz spark-2.0.1-bin-hadoop2.7.tgz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copiamos Spark a su caperta:\n",
    "\n",
    "```\n",
    "mv spark-2.0.1-bin-hadoop2.7 /usr/local/spark\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Descargamos Scala\n",
    "\n",
    "```\n",
    "curl -O http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Descomprimimos Scala y lo movemos a su caperta\n",
    "\n",
    "```\n",
    "mv scala-2.11.8 /usr/local/scala\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Establecemos los PATH para scala y spark:\n",
    "\n",
    "```\n",
    "echo 'export PATH=$PATH:/usr/local/spark/bin/' >> $HOME/.bash_profile \n",
    "echo 'export PATH=$PATH:/usr/local/scala/bin/' >> $HOME/.bash_profile \n",
    "echo 'export SPARK_HOME=/usr/local/spark/' >> $HOME/.bash_profile\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ejecutamos:\n",
    "\n",
    "```\n",
    "source .bash_profile\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decargamos CONDA (para JupyterNotebooks)\n",
    "\n",
    "```\n",
    "curl -O https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ejecutamos CONDA y lo instalamos (seguir todos los pasos por defecto):\n",
    "\n",
    "```\n",
    "sh Miniconda2-latest-Linux-x86_64.sh \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instalamos Jupyter:\n",
    "\n",
    "```\n",
    "conda install jupyter\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inicio del entorno de trabajo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tanto si te descagas la Máquina virtual, como si instalas todo manualmente, es necesario ejecutar los siguientes comandos antes de trabajar en el taller:\n",
    "\n",
    "**Arrancamos el entorno de Spark:**\n",
    "\n",
    "```\n",
    "/usr/local/spark/sbin/start-all.sh\n",
    "```\n",
    "\n",
    "**Arrancamos JupyterNotebook:**\n",
    "\n",
    "```\n",
    "jupyter notebook --notebook-dir=/root/TallerSparkR --no-browser --port=8888 --ip=0.0.0.0\n",
    "```\n",
    "\n",
    "Esto hará que funcione JupyterNotebooks en el puerto 8888, por lo tenemos que acceder desde nuestro navegador web a la dirección: http://localhost:8888\n",
    "\n",
    "\n",
    "Para cerrar todo el entorno de trabajo, cerramos JupyterNotebook y ejecutamos:\n",
    "\n",
    "```\n",
    "/usr/local/spark/sbin/stop-all.sh \n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Descarga del material del taller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Toda la información del Taller está disponible en GITHUB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accedemos la Máquina Virtual, escribimos:\n",
    "\n",
    "```\n",
    "cd /root/TallerSparkR/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clonamos el repositorio o bien descargamos el zip donde está toda la información:\n",
    "\n",
    "**Con GIT:**\n",
    "```\n",
    "    git clone https://github.com/manuparra/taller_SparkR NOTEBOOKS\n",
    "```\n",
    "\n",
    "**Descarga el paquete completo de guiones y descomprime:**\n",
    "```\n",
    " curl -O https://github.com/manuparra/taller_SparkR/archive/master.zip\n",
    " \n",
    " unzip master.zip\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.1"
  },
  "nbpresent": {
   "slides": {},
   "themes": {
    "default": "e0e51597-5ca5-40be-84b2-d2bb41981c19",
    "theme": {
     "563b9d05-fb33-4f5d-88f7-bda098be8308": {
      "backgrounds": {
       "backgroundColor": {
        "background-color": "backgroundColor",
        "id": "backgroundColor"
       }
      },
      "id": "563b9d05-fb33-4f5d-88f7-bda098be8308",
      "palette": {
       "backgroundColor": {
        "id": "backgroundColor",
        "rgb": [
         34,
         34,
         34
        ]
       },
       "headingColor": {
        "id": "headingColor",
        "rgb": [
         238,
         238,
         238
        ]
       },
       "linkColor": {
        "id": "linkColor",
        "rgb": [
         170,
         34,
         51
        ]
       },
       "mainColor": {
        "id": "mainColor",
        "rgb": [
         238,
         238,
         238
        ]
       }
      },
      "rules": {
       "a": {
        "color": "linkColor"
       },
       "h1": {
        "color": "headingColor",
        "font-family": "Ubuntu",
        "font-size": 7
       },
       "h2": {
        "color": "headingColor",
        "font-family": "Ubuntu",
        "font-size": 5
       },
       "h3": {
        "color": "headingColor",
        "font-family": "Ubuntu",
        "font-size": 3.75
       },
       "h4": {
        "color": "headingColor",
        "font-family": "Ubuntu",
        "font-size": 3
       },
       "h5": {
        "color": "headingColor",
        "font-family": "Ubuntu"
       },
       "h6": {
        "color": "headingColor",
        "font-family": "Ubuntu"
       },
       "h7": {
        "color": "headingColor",
        "font-family": "Ubuntu"
       },
       "li": {
        "color": "mainColor",
        "font-family": "Ubuntu",
        "font-size": 5
       },
       "p": {
        "color": "mainColor",
        "font-family": "Ubuntu",
        "font-size": 5
       }
      },
      "text-base": {
       "color": "mainColor",
       "font-family": "Ubuntu",
       "font-size": 5
      }
     },
     "e0e51597-5ca5-40be-84b2-d2bb41981c19": {
      "backgrounds": {
       "backgroundColor": {
        "background-color": "backgroundColor",
        "id": "backgroundColor"
       }
      },
      "id": "e0e51597-5ca5-40be-84b2-d2bb41981c19",
      "palette": {
       "backgroundColor": {
        "id": "backgroundColor",
        "rgb": [
         253,
         246,
         227
        ]
       },
       "headingColor": {
        "id": "headingColor",
        "rgb": [
         88,
         110,
         117
        ]
       },
       "linkColor": {
        "id": "linkColor",
        "rgb": [
         38,
         139,
         210
        ]
       },
       "mainColor": {
        "id": "mainColor",
        "rgb": [
         101,
         123,
         131
        ]
       }
      },
      "rules": {
       "a": {
        "color": "linkColor"
       },
       "h1": {
        "color": "headingColor",
        "font-family": "Oswald",
        "font-size": 7
       },
       "h2": {
        "color": "headingColor",
        "font-family": "Oswald",
        "font-size": 5
       },
       "h3": {
        "color": "headingColor",
        "font-family": "Oswald",
        "font-size": 3.75
       },
       "h4": {
        "color": "headingColor",
        "font-family": "Oswald",
        "font-size": 3
       },
       "h5": {
        "color": "headingColor",
        "font-family": "Oswald"
       },
       "h6": {
        "color": "headingColor",
        "font-family": "Oswald"
       },
       "h7": {
        "color": "headingColor",
        "font-family": "Oswald"
       },
       "li": {
        "color": "mainColor",
        "font-family": "Lato",
        "font-size": 5
       },
       "p": {
        "color": "mainColor",
        "font-family": "Lato",
        "font-size": 5
       }
      },
      "text-base": {
       "color": "mainColor",
       "font-family": "Lato",
       "font-size": 5
      }
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}