Creating a Json Database HDRdata, IMF and WDI jsonv GitHub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center\">\n",
    "# <span style=\"color:#285c8c\"><big>&#9733; Data Extractor v2.0 ! &#9733;</span><big><span style=\"color:#00acee;\"></i>\n",
    "</big></big></span><br>\n",
    "![image](http://stats4dev.com/img/Extractor.png)\n",
    "</div>\n",
    "<br>\n",
    "### &#9733;<i>Thomas Roca, PhD - AFD Research Direction &#9733;</i>\n",
    "*Version march 2016*\n",
    "\n",
    "#### This notebook gather 3 scripts that download and reshape data from three different sources:\n",
    "\n",
    "The data received from the different APIs having different format (different json shapes) we designed our own way to store the information. Our output is Json files that best fit our purpose, providing country information. We design them this way:\n",
    "- Yearly;\n",
    "- By country;\n",
    "- With new meta: sources, YearMin/Max & label for each series, country information, By group averages (Income and Regions)\n",
    "\n",
    "##### What remain on the to-do-list:\n",
    "- clean the code further, improve comments\n",
    "- create a script to create json from any formatted csv to incorparte whatever data\n",
    "\n",
    "#### Example:\n",
    "```javascript\n",
    "{\n",
    "  \"SP.POP.TOTL\": {\n",
    "    \"label\": \"Total Population (in number of people)\",\n",
    "    \"info\": \" \",\n",
    "    \"USA\": {\n",
    "      \"1980\": 227225000,\n",
    "       ...\n",
    "      \"2014\": 318857056,\n",
    "      \"2015\": null,\n",
    "      \"YearMin\": 1980,\n",
    "      \"YearMax\": 2014\n",
    "    },\n",
    "   ...\n",
    "    \"WLD\": {\n",
    "      \"1980\": 4436590356,\n",
    "       ...\n",
    "      \"2014\": 7260710677,\n",
    "      \"2015\": null,\n",
    "      \"YearMin\": 1980,\n",
    "      \"YearMax\": 2014\n",
    "    }\n",
    "  },\n",
    "  \"NE.EXP.GNFS.CD\": {\n",
    "    \"label\": \"Exports of goods and services (current US$)\",\n",
    "    \"info\": \" \",\n",
    "    \"USA\": {\n",
    "      \"1980\": 280773000000,\n",
    "       ...\n",
    "```\n",
    "\n",
    "#### This allows an easy automation - see below or [link](http://www.jsfiddle.net/ThomasRoca/peegp981)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"100%\" height=\"300\" src=\"http://www.jsfiddle.net/ThomasRoca/peegp981/embedded/js,result/ \"frameborder=\"0\"></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<iframe width=\"100%\" height=\"300\" src=\"http://www.jsfiddle.net/ThomasRoca/peegp981/embedded/js,result/ \"frameborder=\"0\"></iframe>''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "![image](http://stats4dev.com/img/IMFs.png)\n",
    "\n",
    "## <span style=\"color:#285c8c\">IMF WEO</span> \n",
    "\n",
    "### What it does\n",
    "This script logs to [Quandl API](http://www.quandl.com) to get IMF/WEO data. In order to proceed, we will follow these steps:\n",
    "+ **Download** all available WEO indicators for each countries (211 countries = 211 csv files - 1 per country)\n",
    "+ **Compute world, regional and income group averages**, this way:\n",
    "    + merge all CSVs into a unique dataset\n",
    "    + compute group averages for each years\n",
    "+ **Write json files which contain the data and meta we want** (begining and end of the serie for each indicator/ countries) and other meta information such as: indicator label, source, download date, Country meta, etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Download the data (a csv by country)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#!pip install Quandl\n",
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "import numpy as np\n",
    "import Quandl\n",
    "from datetime import datetime\n",
    "import sys,os,os.path\n",
    "\n",
    "#===========================>  Set up proxy environnement <==================================\n",
    "os.environ['HTTP_PROXY']=\"xxxxxxxxx\"\n",
    "os.environ['HTTPS_PROXY']=\"xxxxxxxxxxx\"\n",
    "#============================================================================================\n",
    "\n",
    "#Set begining and end of the serie\n",
    "start=1980\n",
    "end=2020\n",
    "\n",
    "#https://www.quandl.com/data/ODA/documentation/documentation\n",
    "APIKey=\"DWys3k7zkj4oyByx6Z48\" # optional, to allow more than 50 request per day\n",
    "\n",
    "#Set weo indicator list\n",
    "indicator_list=['NID_NGDP','PPPGDP','BCA_NGDPD','BCA','LE','TX_RPCH','TXG_RPCH','NGDP_FY','NGDP_D','NGDP_RPCH','NGDP_R','NGDP','NGDPD','NGDPRPC','NGDPPC','LUR','NGDPDPC','GGXWDG_NGDP','GGXWDG','GGXWDN_NGDP','GGXWDN','GGXCNL_NGDP','GGXCNL','GGXONLB_NGDP','GGXONLB','GGR_NGDP','GGR','GGSB_NPGDP','GGSB','GGX_NGDP','GGX','NGSD_NGDP','PPPEX','TM_RPCH','TMG_RPCH','PCPIPCH','PCPIEPCH','PCPI','PCPIE','NGAP_NPGDP','PPPPC','LP','PPPSH']\n",
    "#set country iso list\n",
    "isolist=['ABW', 'AFG', 'AGO', 'ALB', 'AND', 'ARE', 'ARG', 'ARM', 'ASM', 'ATG', 'AUS', 'AUT', 'AZE', 'BDI', 'BEL', 'BEN', 'BFA', 'BGD', 'BGR', 'BHR', 'BHS', 'BIH', 'BLR', 'BLZ', 'BMU', 'BOL', 'BRA', 'BRB', 'BRN', 'BTN', 'BWA', 'CAF', 'CAN', 'CHE', 'CHL', 'CHN', 'CIV', 'CMR', 'COD', 'COG', 'COL', 'COM', 'CPV', 'CRI', 'CUB', 'CUW', 'CYM', 'CYP', 'CZE', 'DEU', 'DJI', 'DMA', 'DNK', 'DOM', 'DZA', 'ECU', 'EGY', 'ERI', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FRA', 'FRO', 'FSM', 'GAB', 'GBR', 'GEO', 'GHA', 'GIN', 'GMB', 'GNB', 'GNQ', 'GRC', 'GRD', 'GRL', 'GTM', 'GUM', 'GUY', 'HKG', 'HND', 'HRV', 'HTI', 'HUN', 'IDN', 'IMN', 'IND', 'IRL', 'IRN', 'IRQ', 'ISL', 'ITA', 'ISR', 'JAM', 'JOR', 'JPN', 'KAZ', 'KEN', 'KGZ', 'KHM', 'KNA', 'KOR', 'KWT', 'LAO', 'LBN', 'LBR', 'LBY', 'LCA', 'LIE', 'LKA', 'LSO', 'LTU', 'LUX', 'LVA', 'MAC', 'MAF', 'MAR', 'MCO', 'MDA', 'MDG', 'MDV', 'MEX', 'MHL', 'MKD', 'MLI', 'MLT', 'MMR', 'MNE', 'MNG', 'MNP', 'MOZ', 'MRT', 'MUS', 'MWI', 'MYS', 'NAM', 'NCL', 'NER', 'NGA', 'NIC', 'NLD', 'NOR', 'NPL', 'NZL', 'OMN', 'PAK', 'PAN', 'PER', 'PHL', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'PSE', 'PYF', 'QAT', 'ROU', 'RUS', 'RWA', 'SAU', 'SDN', 'SEN', 'SGP', 'SLB', 'SLE', 'SLV', 'SMR', 'SOM', 'SRB', 'SSD', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SWZ', 'SXM', 'SYC', 'SYR', 'TCA', 'TCD', 'TGO', 'THA', 'TJK', 'TKM', 'TLS', 'TON', 'TTO', 'TUN', 'TUR', 'TUV', 'TWN', 'TZA', 'UGA', 'UKR', 'URY', 'USA', 'UZB', 'VCT', 'VEN', 'VIR', 'VNM', 'VUT', 'WSM', 'YEM', 'ZAF', 'ZMB', 'ZWE']\n",
    "\n",
    "#Start looping over countries\n",
    "for ISO in isolist:\n",
    "    #empty list to start with\n",
    "    isolistPlus=[] \n",
    "\n",
    "    #Add the meta for countries after the last list\n",
    "    with open (\"data/json meta country.json.txt\", \"r\") as meta:\n",
    "        for line in meta :\n",
    "            if line.startswith('\"'+ISO+'\"') :\n",
    "                metacountry=line\n",
    "\n",
    "    #This dictionnary associate a country name to an iso3 code \n",
    "    country_dict = {\"Arab World\":\"ARB\", \"Central Europe and the Baltics\":\"CEB\", \"Caribbean small states\":\"CSS\", \"East Asia & Pacific\":\"EAP\", \"East Asia & Pacific (all income levels)\":\"EAS\", \"Europe & Central Asia\":\"ECA\", \"Europe & Central Asia (all income levels)\":\"ECS\", \"Euro area\":\"EMU\", \"European Union\":\"EUU\", \"Fragile and conflict affected situations\":\"FCS\", \"High income\":\"HIC\", \"Heavily indebted poor countries (HIPC)\":\"HPC\", \"Latin America & Caribbean\":\"LAC\", \"Latin America & Caribbean (all income levels)\":\"LCN\", \"Least developed countries: UN classification\":\"LDC\", \"Low income\":\"LIC\", \"Lower middle income\":\"LMC\", \"Low & middle income\":\"LMY\", \"Middle East & North Africa (all income levels)\":\"MEA\", \"Middle income\":\"MIC\", \"Middle East & North Africa\":\"MNA\", \"North America\":\"NAC\", \"High income: nonOECD\":\"NOC\", \"High income: OECD\":\"OEC\", \"OECD members\":\"OED\", \"Other small states\":\"OSS\", \"Pacific island small states\":\"PSS\", \"South Asia\":\"SAS\", \"Sub-Saharan Africa\":\"SSA\", \"Sub-Saharan Africa (all income levels)\":\"SSF\", \"Small states\":\"SST\", \"Upper middle income\":\"UMC\", \"World\":\"WLD\",\"Cote d'Ivoire\":\"CIV\",\"Curacao\":\"CUW\",\"Sao Tome and Principe\":\"STP\",\"American Samoa\":\"ASM\",\"Australia\":\"AUS\",\"Brunei Darussalam\":\"BRN\",\"China\":\"CHN\",\"Fiji\":\"FJI\",\"Micronesia, Fed. Sts.\":\"FSM\",\"Guam\":\"GUM\",\"Hong Kong SAR, China\":\"HKG\",\"Indonesia\":\"IDN\",\"Japan\":\"JPN\",\"Cambodia\":\"KHM\",\"Kiribati\":\"KIR\",\"Korea, Rep.\":\"KOR\",\"Lao PDR\":\"LAO\",\"Macao SAR, China\":\"MAC\",\"Marshall Islands\":\"MHL\",\"Myanmar\":\"MMR\",\"Mongolia\":\"MNG\",\"Northern Mariana Islands\":\"MNP\",\"Malaysia\":\"MYS\",\"New Caledonia\":\"NCL\",\"New Zealand\":\"NZL\",\"Philippines\":\"PHL\",\"Palau\":\"PLW\",\"Papua New Guinea\":\"PNG\",\"Korea, Dem. Rep.\":\"PRK\",\"French Polynesia\":\"PYF\",\"Singapore\":\"SGP\",\"Solomon Islands\":\"SLB\",\"Thailand\":\"THA\",\"Timor-Leste\":\"TLS\",\"Tonga\":\"TON\",\"Tuvalu\":\"TUV\",\"Taiwan, China\":\"TWN\",\"Vietnam\":\"VNM\",\"Vanuatu\":\"VUT\",\"Samoa\":\"WSM\",\"Albania\":\"ALB\",\"Andorra\":\"AND\",\"Armenia\":\"ARM\",\"Austria\":\"AUT\",\"Azerbaijan\":\"AZE\",\"Belgium\":\"BEL\",\"Bulgaria\":\"BGR\",\"Bosnia and Herzegovina\":\"BIH\",\"Belarus\":\"BLR\",\"Switzerland\":\"CHE\",\"Channel Islands\":\"CHI\",\"Cyprus\":\"CYP\",\"Czech Republic\":\"CZE\",\"Germany\":\"DEU\",\"Denmark\":\"DNK\",\"Spain\":\"ESP\",\"Estonia\":\"EST\",\"Finland\":\"FIN\",\"France\":\"FRA\",\"Faeroe Islands\":\"FRO\",\"United Kingdom\":\"GBR\",\"Georgia\":\"GEO\",\"Greece\":\"GRC\",\"Greenland\":\"GRL\",\"Croatia\":\"HRV\",\"Hungary\":\"HUN\",\"Isle of Man\":\"IMN\",\"Ireland\":\"IRL\",\"Iceland\":\"ISL\",\"Italy\":\"ITA\",\"Kazakhstan\":\"KAZ\",\"Kyrgyz Republic\":\"KGZ\",\"Liechtenstein\":\"LIE\",\"Lithuania\":\"LTU\",\"Luxembourg\":\"LUX\",\"Latvia\":\"LVA\",\"Monaco\":\"MCO\",\"Moldova\":\"MDA\",\"Macedonia, FYR\":\"MKD\",\"Montenegro\":\"MNE\",\"Netherlands\":\"NLD\",\"Norway\":\"NOR\",\"Poland\":\"POL\",\"Portugal\":\"PRT\",\"Romania\":\"ROU\",\"Russian Federation\":\"RUS\",\"San Marino\":\"SMR\",\"Serbia\":\"SRB\",\"Slovak Republic\":\"SVK\",\"Slovenia\":\"SVN\",\"Sweden\":\"SWE\",\"Tajikistan\":\"TJK\",\"Turkmenistan\":\"TKM\",\"Turkey\":\"TUR\",\"Ukraine\":\"UKR\",\"Uzbekistan\":\"UZB\",\"Aruba\":\"ABW\",\"Argentina\":\"ARG\",\"Antigua and Barbuda\":\"ATG\",\"Bahamas, The\":\"BHS\",\"Belize\":\"BLZ\",\"Bolivia\":\"BOL\",\"Brazil\":\"BRA\",\"Barbados\":\"BRB\",\"Chile\":\"CHL\",\"Colombia\":\"COL\",\"Costa Rica\":\"CRI\",\"Cuba\":\"CUB\",\"Curaçao\":\"CUW\",\"Cayman Islands\":\"CYM\",\"Dominica\":\"DMA\",\"Dominican Republic\":\"DOM\",\"Ecuador\":\"ECU\",\"Grenada\":\"GRD\",\"Guatemala\":\"GTM\",\"Guyana\":\"GUY\",\"Honduras\":\"HND\",\"Haiti\":\"HTI\",\"Jamaica\":\"JAM\",\"St. Kitts and Nevis\":\"KNA\",\"St. Lucia\":\"LCA\",\"St. Martin (French part)\":\"MAF\",\"Mexico\":\"MEX\",\"Nicaragua\":\"NIC\",\"Panama\":\"PAN\",\"Peru\":\"PER\",\"Puerto Rico\":\"PRI\",\"Paraguay\":\"PRY\",\"El Salvador\":\"SLV\",\"Suriname\":\"SUR\",\"Sint Maarten (Dutch part)\":\"SXM\",\"Turks and Caicos Islands\":\"TCA\",\"Trinidad and Tobago\":\"TTO\",\"Uruguay\":\"URY\",\"St. Vincent and the Grenadines\":\"VCT\",\"Venezuela, RB\":\"VEN\",\"Virgin Islands (U.S.)\":\"VIR\",\"United Arab Emirates\":\"ARE\",\"Bahrain\":\"BHR\",\"Djibouti\":\"DJI\",\"Algeria\":\"DZA\",\"Egypt, Arab Rep.\":\"EGY\",\"Iran, Islamic Rep.\":\"IRN\",\"Iraq\":\"IRQ\",\"Israel\":\"ISR\",\"Jordan\":\"JOR\",\"Kuwait\":\"KWT\",\"Lebanon\":\"LBN\",\"Libya\":\"LBY\",\"Morocco\":\"MAR\",\"Malta\":\"MLT\",\"Oman\":\"OMN\",\"West Bank and Gaza\":\"PSE\",\"Qatar\":\"QAT\",\"Saudi Arabia\":\"SAU\",\"Syrian Arab Republic\":\"SYR\",\"Tunisia\":\"TUN\",\"Yemen, Rep.\":\"YEM\",\"Bermuda\":\"BMU\",\"Canada\":\"CAN\",\"United States\":\"USA\",\"Afghanistan\":\"AFG\",\"Bangladesh\":\"BGD\",\"Bhutan\":\"BTN\",\"India\":\"IND\",\"Sri Lanka\":\"LKA\",\"Maldives\":\"MDV\",\"Nepal\":\"NPL\",\"Pakistan\":\"PAK\",\"Angola\":\"AGO\",\"Burundi\":\"BDI\",\"Benin\":\"BEN\",\"Burkina Faso\":\"BFA\",\"Botswana\":\"BWA\",\"Central African Republic\":\"CAF\",\"Cote d'Ivoire\":\"CIV\",\"Cameroon\":\"CMR\",\"Congo, Dem. Rep.\":\"COD\",\"Congo, Rep.\":\"COG\",\"Comoros\":\"COM\",\"Cabo Verde\":\"CPV\",\"Eritrea\":\"ERI\",\"Ethiopia\":\"ETH\",\"Gabon\":\"GAB\",\"Ghana\":\"GHA\",\"Guinea\":\"GIN\",\"Gambia, The\":\"GMB\",\"Guinea-Bissau\":\"GNB\",\"Equatorial Guinea\":\"GNQ\",\"Kenya\":\"KEN\",\"Liberia\":\"LBR\",\"Lesotho\":\"LSO\",\"Madagascar\":\"MDG\",\"Mali\":\"MLI\",\"Mozambique\":\"MOZ\",\"Mauritania\":\"MRT\",\"Mauritius\":\"MUS\",\"Malawi\":\"MWI\",\"Namibia\":\"NAM\",\"Niger\":\"NER\",\"Nigeria\":\"NGA\",\"Rwanda\":\"RWA\",\"Sudan\":\"SDN\",\"Senegal\":\"SEN\",\"Sierra Leone\":\"SLE\",\"Somalia\":\"SOM\",\"South Sudan\":\"SSD\",\"São Tomé and Principe\":\"STP\",\"Swaziland\":\"SWZ\",\"Seychelles\":\"SYC\",\"Chad\":\"TCD\",\"Togo\":\"TGO\",\"Tanzania\":\"TZA\",\"Uganda\":\"UGA\",\"South Africa\":\"ZAF\",\"Zambia\":\"ZMB\",\"Zimbabwe\":\"ZWE\"}\n",
    "\n",
    "    dataset_Final=DataFrame()\n",
    "    #open en empty list to store indicator that were not available for download\n",
    "    failed_list=[]\n",
    "    \n",
    "    #loop over indicators\n",
    "    for ind in indicator_list:\n",
    "        \n",
    "        try:\n",
    "            print('Attempt downloading....'+ISO+'.. Indicator: '+ind)\n",
    "            #=========================================< Quandl API Query >===========================================================\n",
    "            data = Quandl.get(\"ODA/\"+ISO+\"_\"+ind, authtoken=APIKey, trim_start=\"1980-12-31\", trim_end=\"2050-12-31\")\n",
    "            #===============================================< Query >=================================================================\n",
    "\n",
    "            iso3=[]\n",
    "            year=[] \n",
    "            data.columns=[ind]\n",
    "            year=data.index.get_level_values('Date').year\n",
    "            for item in year:\n",
    "                iso3.append(ISO)\n",
    "\n",
    "            data = data.set_index([iso3, year])\n",
    "            #identify data that are not empty\n",
    "            Yrange=np.where(data[ind].loc[iso3].notnull())[0]\n",
    "            #merge indicators\n",
    "            dataset_Final=pd.concat([data,dataset_Final], axis=1, join='outer')\n",
    "            #Set index name\n",
    "            dataset_Final.index.name = ['iso3', 'year']\n",
    "\n",
    "        except:\n",
    "            # This exception deals with indicators not available for dwnd\n",
    "            print('Attempt downloading....'+ISO+'.. Indicator: '+ind+'... failed')\n",
    "            failed_list.append(ind)\n",
    "            continue\n",
    "\n",
    "#==================< Show the indicators that were not available for dwnd and add empty columns for them >============================\n",
    "    print('***These indicators were not available for '+ISO+' :')\n",
    "    print(failed_list)       \n",
    "    \n",
    "    #Add empty columns for the indicators that were not available\n",
    "    for fail_ind in failed_list:\n",
    "        dataset_Final[fail_ind]= np.nan  \n",
    "    \n",
    "    dataset_Final.to_csv('data/imf/csv'+ISO+'_WEO.csv')\n",
    "\n",
    "#========================================================< END >=======================================================================\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "winsound.PlaySound(\"SystemHand\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Compute group averages\n",
    "\n",
    "NB. This script as evolve, it is functional but it needs to be cleaned to avoid the extra steps of saving as csv (to erase multi index) I shouldn't have use multi indexing to begining with. Check if i can do otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "*******************# Job Well done ! #********************\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "import numpy as np\n",
    "import csv\n",
    "import winsound\n",
    "\n",
    "#working directory\n",
    "folder=\"data/imf/csv/\"\n",
    "\n",
    "isolist=['ABW', 'AFG', 'AGO', 'ALB', 'AND', 'ARE', 'ARG', 'ARM', 'ASM', 'ATG', 'AUS', 'AUT', 'AZE', 'BDI', 'BEL', 'BEN', 'BFA', 'BGD', 'BGR', 'BHR', 'BHS', 'BIH', 'BLR', 'BLZ', 'BMU', 'BOL', 'BRA', 'BRB', 'BRN', 'BTN', 'BWA', 'CAF', 'CAN', 'CHE', 'CHL', 'CHN', 'CIV', 'CMR', 'COD', 'COG', 'COL', 'COM', 'CPV', 'CRI', 'CUB', 'CUW', 'CYM', 'CYP', 'CZE', 'DEU', 'DJI', 'DMA', 'DNK', 'DOM', 'DZA', 'ECU', 'EGY', 'ERI', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FRA', 'FRO', 'FSM', 'GAB', 'GBR', 'GEO', 'GHA', 'GIN', 'GMB', 'GNB', 'GNQ', 'GRC', 'GRD', 'GRL', 'GTM', 'GUM', 'GUY', 'HKG', 'HND', 'HRV', 'HTI', 'HUN', 'IDN', 'IMN', 'IND', 'IRL', 'IRN', 'IRQ', 'ISL', 'ISR', 'ITA', 'JAM', 'JOR', 'JPN', 'KAZ', 'KEN', 'KGZ', 'KHM', 'KNA', 'KOR', 'KWT', 'LAO', 'LBN', 'LBR', 'LBY', 'LCA', 'LIE', 'LKA', 'LSO', 'LTU', 'LUX', 'LVA', 'MAC', 'MAF', 'MAR', 'MCO', 'MDA', 'MDG', 'MDV', 'MEX', 'MHL', 'MKD', 'MLI', 'MLT', 'MMR', 'MNE', 'MNG', 'MNP', 'MOZ', 'MRT', 'MUS', 'MWI', 'MYS', 'NAM', 'NCL', 'NER', 'NGA', 'NIC', 'NLD', 'NOR', 'NPL', 'NZL', 'OMN', 'PAK', 'PAN', 'PER', 'PHL', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'PSE', 'PYF', 'QAT', 'ROU', 'RUS', 'RWA', 'SAU', 'SDN', 'SEN', 'SGP', 'SLB', 'SLE', 'SLV', 'SMR', 'SOM', 'SRB', 'SSD', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SWZ', 'SXM', 'SYC', 'SYR', 'TCA', 'TCD', 'TGO', 'THA', 'TJK', 'TKM', 'TLS', 'TON', 'TTO', 'TUN', 'TUR', 'TUV', 'TWN', 'TZA', 'UGA', 'UKR', 'URY', 'USA', 'UZB', 'VCT', 'VEN', 'VIR', 'VNM', 'VUT', 'WSM', 'YEM', 'ZAF', 'ZMB', 'ZWE']\n",
    "\n",
    "#Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "\n",
    "#====================== Generate IMF WEO complete dataset =====================\n",
    "iso_len=len(isolist)\n",
    "frame = dict()\n",
    "#this concatenates all WEO-IMF csv files\n",
    "i=1\n",
    "for iso in isolist:\n",
    "    isolistPlus=[]\n",
    "    frame[i]=pd.read_csv(folder+iso+'_WEO.csv')\n",
    "    \n",
    "    # Add income class and Region \n",
    "    #browse the region dictionnary\n",
    "    for iso3,region_iso in region_dict.items(): \n",
    "            if iso==iso3:\n",
    "                isolistPlus.append(iso)\n",
    "                for item in region_iso.split(','):\n",
    "                    isolistPlus.append(str(item))\n",
    "                    \n",
    "    #save income class et Region in a separate variable\n",
    "    frame[i]['Income_class']= isolistPlus[1]\n",
    "    frame[i]['Region']= isolistPlus[2]\n",
    "    #Add name columns names for col 1 and 2:  iso3 and year\n",
    "    frame[i]=frame[i].rename(columns = {'Unnamed: 0':'iso3', 'Unnamed: 1':'year'})\n",
    "    i+=1\n",
    "    \n",
    "#Concat all country files frame[1]...frame[n] \n",
    "final=DataFrame()\n",
    "j=1\n",
    "\n",
    "#concatenate all files\n",
    "for iso in isolist:\n",
    "    final = concat([final, frame[j]])\n",
    "    j+=1\n",
    "final.to_csv(folder+'final_WEO.csv')\n",
    "\n",
    "#========================>   Computing averages    <=========================\n",
    "#Compute average by income for all indicators in the DataFrame 'final' \n",
    "income= final.groupby(['Income_class', 'year']).agg([np.mean])\n",
    "#Compute average by region for all indicators in the DataFrame 'final' \n",
    "region= final.groupby(['Region', 'year']).agg([np.mean])\n",
    "#compute world average\n",
    "world= final.groupby(['year']).agg([np.mean])\n",
    "\n",
    "#!! world is not multiindexed here, it only has year, we thus need to create a multi index containing iso3= WLD\n",
    "yearRange=[]\n",
    "isolistW=[]\n",
    "\n",
    "#Save as list dates ex 1980, 1981, ... 2020\n",
    "yearRange=world.index.values\n",
    "#Save as list  'WLD' time the number of values in world.index.values\n",
    "for item in world.index.values:\n",
    "    isolistW.append('WLD')\n",
    "\n",
    "#Redindex World as (WLD,year) eg (WLD, 1980)...(WLD,2020)\n",
    "world.index= list(zip(isolistW, yearRange))\n",
    "#world.index.name=['iso3', 'year']\n",
    "\n",
    "#in a new DataFrame, concatenate income, region averages and world average\n",
    "final_mean = concat([income,region,world])\n",
    "final_mean.index.name = ['iso3', 'year']\n",
    "#save it as a csv\n",
    "final_mean.to_csv(folder+'WEO_mean.csv')\n",
    "\n",
    "#==============================================< Append dataset and means >============================================\n",
    "dataset = pd.read_csv(folder+'WEO_mean.csv')\n",
    "dataset=dataset.rename(columns = {'Unnamed: 0':'iso3', 'Unnamed: 1':'year'})\n",
    "dataset=dataset.drop(dataset.index[[0,1]])\n",
    "dataset_weo=pd.read_csv(folder+'final_WEO.csv')\n",
    "dataset=dataset.append(dataset_weo)\n",
    "dataset = dataset.drop('Unnamed: 0', 1)\n",
    "dataset.to_csv(folder+'final_WEO_with_avg.csv')\n",
    "\n",
    "#====================================================< END >============================================================\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "winsound.PlaySound(\"SystemHand\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Write the jsons from CSVs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "from datetime import datetime\n",
    "import numpy as np\n",
    "import winsound\n",
    "import os\n",
    "\n",
    "#Working directory\n",
    "folder=\"data/imf/csv/\"\n",
    "folder_final=\"data/imf/\"\n",
    "\n",
    "#Set begining and end of the serie\n",
    "start=1980\n",
    "end=2020\n",
    "\n",
    "#=================================================================================================\n",
    "indicator_list=['NID_NGDP','PPPGDP','BCA_NGDPD','BCA','LE','TX_RPCH','TXG_RPCH','NGDP_FY','NGDP_D','NGDP_RPCH','NGDP_R','NGDP','NGDPD','NGDPRPC','NGDPPC','LUR','NGDPDPC','GGXWDG_NGDP','GGXWDG','GGXWDN_NGDP','GGXWDN','GGXCNL_NGDP','GGXCNL','GGXONLB_NGDP','GGXONLB','GGR_NGDP','GGR','GGSB_NPGDP','GGSB','GGX_NGDP','GGX','NGSD_NGDP','PPPEX','TM_RPCH','TMG_RPCH','PCPIPCH','PCPIEPCH','PCPI','PCPIE','NGAP_NPGDP','PPPPC','LP','PPPSH']\n",
    "iso3list=['ABW', 'AFG', 'AGO', 'ALB', 'AND', 'ARE', 'ARG', 'ARM', 'ASM', 'ATG', 'AUS', 'AUT', 'AZE', 'BDI', 'BEL', 'BEN', 'BFA', 'BGD', 'BGR', 'BHR', 'BHS', 'BIH', 'BLR', 'BLZ', 'BMU', 'BOL', 'BRA', 'BRB', 'BRN', 'BTN', 'BWA', 'CAF', 'CAN', 'CHE', 'CHL', 'CHN', 'CIV', 'CMR', 'COD', 'COG', 'COL', 'COM', 'CPV', 'CRI', 'CUB', 'CUW', 'CYM', 'CYP', 'CZE', 'DEU', 'DJI', 'DMA', 'DNK', 'DOM', 'DZA', 'ECU', 'EGY', 'ERI', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FRA', 'FRO', 'FSM', 'GAB', 'GBR', 'GEO', 'GHA', 'GIN', 'GMB', 'GNB', 'GNQ', 'GRC', 'GRD', 'GRL', 'GTM', 'GUM', 'GUY', 'HKG', 'HND', 'HRV', 'HTI', 'HUN', 'IDN', 'IMN', 'IND', 'IRL', 'IRN', 'IRQ', 'ISL', 'ISR', 'ITA', 'JAM', 'JOR', 'JPN', 'KAZ', 'KEN', 'KGZ', 'KHM', 'KNA', 'KOR', 'KWT', 'LAO', 'LBN', 'LBR', 'LBY', 'LCA', 'LIE', 'LKA', 'LSO', 'LTU', 'LUX', 'LVA', 'MAC', 'MAF', 'MAR', 'MCO', 'MDA', 'MDG', 'MDV', 'MEX', 'MHL', 'MKD', 'MLI', 'MLT', 'MMR', 'MNE', 'MNG', 'MNP', 'MOZ', 'MRT', 'MUS', 'MWI', 'MYS', 'NAM', 'NCL', 'NER', 'NGA', 'NIC', 'NLD', 'NOR', 'NPL', 'NZL', 'OMN', 'PAK', 'PAN', 'PER', 'PHL', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'PSE', 'PYF', 'QAT', 'ROU', 'RUS', 'RWA', 'SAU', 'SDN', 'SEN', 'SGP', 'SLB', 'SLE', 'SLV', 'SMR', 'SOM', 'SRB', 'SSD', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SWZ', 'SXM', 'SYC', 'SYR', 'TCA', 'TCD', 'TGO', 'THA', 'TJK', 'TKM', 'TLS', 'TON', 'TTO', 'TUN', 'TUR', 'TUV', 'TWN', 'TZA', 'UGA', 'UKR', 'URY', 'USA', 'UZB', 'VCT', 'VEN', 'VIR', 'VNM', 'VUT', 'WSM', 'YEM', 'ZAF', 'ZMB', 'ZWE']\n",
    "\n",
    "#This dictionnary associate a country name to an iso3 code \n",
    "#Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "#=================================================================================================\n",
    "\n",
    "dataset = pd.read_csv(folder+'final_WEO_with_avg.csv')\n",
    "\n",
    "for ISO in iso3list:\n",
    "    print(\"Proceed: \",ISO)\n",
    "    if dataset.empty: continue\n",
    "    isolistPlus=[] \n",
    "    #browse the region dictionnary\n",
    "    for iso3,region_iso in region_dict.items(): \n",
    "            if ISO==iso3:\n",
    "                isolistPlus.append(ISO)\n",
    "                for item in region_iso.split(','):\n",
    "                    isolistPlus.append(str(item))\n",
    "                    \n",
    "    print(isolistPlus)\n",
    "    #================== store the meta information ====================================================\n",
    "    source='\"data_source\":{\"source\":\"IMF, WEO\", \"provider\":\"Quandl\", \"Download_date\":\"'+datetime.now().strftime('%Y-%m-%d')+'\" }'\n",
    "    #read country meta stored in a text file\n",
    "    with open (folder_final+\"json meta country.json.txt\", \"r\") as meta:\n",
    "        for line in meta :\n",
    "            if line.startswith('\"'+ISO+'\"') :\n",
    "                metafinal=\",\"+line+\",\"+source\n",
    "    #=================================================================================================\n",
    "\n",
    "\n",
    "    #store name of the last ind in the list\n",
    "    last=indicator_list[len(indicator_list)-1]\n",
    "\n",
    "    file = open(folder_final+\"temp.json\", \"w\")\n",
    "    #Start writing the json file:\n",
    "    file.write('{')\n",
    "\n",
    "    #Loop over indicator list\n",
    "    for ind in indicator_list:\n",
    "        \n",
    "        #Get indicator meta from external file\n",
    "        filemeta=open(folder+\"meta indicator IMF.txt\", \"r\")\n",
    "        for line in filemeta :\n",
    "            if line.startswith('\"'+ind+'\"') : meta=line\n",
    "        filemeta.close()\n",
    "        #write the meta\n",
    "        file.write(meta)\n",
    "\n",
    "        #loop over the isolistPlus which contain [countryiso,incomegrp, regionalgrp, world] \n",
    "        for iso in isolistPlus:\n",
    "            #write iso name\n",
    "            file.write('\"'+iso+'\":{')  \n",
    "            #set min and max year for each ind and iso\n",
    "            year_range=dataset[\"year\"][(dataset[\"iso3\"]==iso)&(dataset[ind].notnull())]\n",
    "            if len(year_range)!=0:\n",
    "                Ymin=min(year_range)\n",
    "                Ymax=max(year_range)\n",
    "            else:\n",
    "                Ymin=Ymax=\"null\"\n",
    "            #write this information within the serie        \n",
    "            file.write('\"YearMin\":'+str(Ymin)+',')\n",
    "            file.write('\"YearMax\":'+str(Ymax)+',')\n",
    "            for year in range(start,end+1):\n",
    "                x= dataset[ind][(dataset[\"year\"] == year) & (dataset[\"iso3\"]==iso)]\n",
    "                if x.empty :  \n",
    "                    x=\"null\"\n",
    "                else :\n",
    "                    x=str('{:0.3f}'.format(float(x)))\n",
    "                if ind!=last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}},')\n",
    "                if ind==last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}}'+metafinal+'}')\n",
    "\n",
    "    file.close()\n",
    "\n",
    "    file_in = open(folder_final+\"temp.json\", \"r\")\n",
    "    file_out = open(folder_final+\"IMF_WEO_\"+ISO+\".json\", \"wt\")\n",
    "    for line in file_in: file_out.write( line.replace(':nan', \":null\") )\n",
    "    file_in.close()\n",
    "    file_out.close()\n",
    "\n",
    "    #os.remove(folder+'dataset'+ISO+'.csv')\n",
    "    print(\"done:IMF_WEO_\"+ISO+\".json\")\n",
    "\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "#os.remove(folder_final+\"temp.json\")\n",
    "winsound.PlaySound(\"SystemHand\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image](http://stats4dev.com/img/UNDP.png)\n",
    "## <span style=\"color:#285c8c\">HDRdata </span>\n",
    "\n",
    "Data from the Human Development Report Office\n",
    "http://hdr.undp.org/en/data\n",
    "\n",
    "### <span style=\"color:#285c8c\">What it does</span>\n",
    "This programme logs to HDRdata API to get Human Development data. Information: http://hdr.undp.org/en/content/developers-data-api\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Download the data and create csv files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "import os, sys\n",
    "from datetime import datetime\n",
    "import numpy as np\n",
    "import sys,os,os.path\n",
    "\n",
    "#Working directory\n",
    "folder=\"data/HDRdata/csv\"\n",
    "\n",
    "#===========================>  Set up proxy environnement <==================================\n",
    "os.environ['HTTP_PROXY']=\"xxxxxxxxxxxxxx\"\n",
    "os.environ['HTTPS_PROXY']=\"xxxxxxxxxxxxxxx\"\n",
    "#============================================================================================\n",
    "\n",
    "#Country available in HDR database\n",
    "HDRcountry=['AFG','ALB','DZA','AND','AGO','ATG','ARG','ARM','AUS','AUT','AZE','BHS','BHR','BGD','BRB','BLR','BEL',\\\n",
    "            'BLZ','BEN','BTN','BOL','BIH','BWA','BRA','BRN','BGR','BFA','BDI','KHM','CMR','CAN','CPV','CAF','TCD',\\\n",
    "            'CHL','CHN','COL','COM','COG','COD','CRI','CIV','HRV','CUB','CYP','CZE','DNK','DJI','DMA','DOM','ECU',\\\n",
    "            'EGY','SLV','GNQ','ERI','EST','ETH','FJI','FIN','FRA','GAB','GMB','GEO','DEU','GHA','GRC','GRD','GTM',\\\n",
    "            'GIN','GNB','GUY','HTI','HND','HKG','HUN','ISL','IND','IDN','IRN','IRQ','IRL','ISR','ITA','JAM','JPN',\\\n",
    "            'JOR','KAZ','KEN','KIR','KOR','KWT','KGZ','LAO','LVA','LBN','LSO','LBR','LBY','LIE','LTU','LUX','MKD',\\\n",
    "            'MDG','MWI','MYS','MDV','MLI','MLT','MHL','MRT','MUS','MEX','FSM','MDA','MCO','MNG','MNE','MAR','MOZ',\\\n",
    "            'MMR','NAM', 'NPL','NLD','NZL','NIC','NER','NGA','NOR','OMN','PAK','PLW','PAN','PNG','PRY','PER','PHL',\\\n",
    "            'POL','PRT','QAT','ROU','RUS','RWA','KNA','LCA','VCT','WSM','SMR','STP','SAU','SEN','SRB','SYC','SLE',\\\n",
    "            'SGP','SVK','SVN','SLB','SOM','ZAF','ESP','LKA','SDN','SUR','SWZ','SWE','CHE','SYR','TJK','TZA','THA',\\\n",
    "            'TLS','TGO','TON','TTO','TUN','TUR','TKM','TUV','UGA','UKR','ARE','GBR','USA','URY','UZB','VUT','VEN',\\\n",
    "            'VNM','YEM','ZMB','ZWE','SSD'] #Non available iso : 'NRU','PSE', PRK'\n",
    "\n",
    "#Year available\n",
    "ymin, ymax = 1980, 2015\n",
    "YearList=[]\n",
    "\n",
    "#Set YearList\n",
    "for y in range (ymin, ymax+1):\n",
    "    YearList.append(y)\n",
    "YearList=','.join(map(str,YearList))\n",
    "\n",
    "#========================================> Starting loop <============================================================\n",
    "\n",
    "for iso in HDRcountry: # loop over country\n",
    "    \n",
    "    data=DataFrame()\n",
    "    dataset=pd.DataFrame()\n",
    "    dataset_Final=DataFrame()\n",
    "    \n",
    "    #the request cannot handle the 40 indicators requested in one time. I created two lists to avoid the problem\n",
    "    HDRindlistA=['306','24106','24206','24806','27706','38406','38506','38606','44706','46106','68006','68106','68606','69706','71106','71206','71306','71406','71506','71606']\n",
    "    HDRindlistB=['72206','98106','101006','103006','103606','103706','120606','121106','121206','123306','123406','123506','123606','135206','136906','137006','137506','137906','138806','142506']\n",
    "    #trick to switch list \n",
    "    HDRindlist0= HDRindlist00= HDRindlistA\n",
    "    \n",
    "    #list as one string to fit in the API request\n",
    "    HDRindlist1='306,24106,24206,24806,27706,38406,38506,38606,44706,46106,68006,68106,68606,69706,71106,71206,71306,71406,71506,71606'\n",
    "    HDRindlist2='72206,98106,101006,103006,103606,103706,120606,121106,121206,123306,123406,123506,123606,135206,136906,137006,137506,137906,138806,142506'\n",
    "    HDR=[HDRindlist1,HDRindlist2] \n",
    "    \n",
    "    #empty list to start with\n",
    "    isolistPlus=[] \n",
    "\n",
    "    #Add the meta for countries after the last list\n",
    "    with open (\"data/json meta country.json.txt\", \"r\") as meta:\n",
    "        for line in meta :\n",
    "            if line.startswith('\"'+iso+'\"') :\n",
    "                metacountry=line\n",
    "\n",
    "    #This dictionnary associate a country name to an iso3 code \n",
    "    country_dict = {\"Arab World\":\"ARB\", \"Central Europe and the Baltics\":\"CEB\", \"Caribbean small states\":\"CSS\", \"East Asia & Pacific\":\"EAP\", \"East Asia & Pacific (all income levels)\":\"EAS\", \"Europe & Central Asia\":\"ECA\", \"Europe & Central Asia (all income levels)\":\"ECS\", \"Euro area\":\"EMU\", \"European Union\":\"EUU\", \"Fragile and conflict affected situations\":\"FCS\", \"High income\":\"HIC\", \"Heavily indebted poor countries (HIPC)\":\"HPC\", \"Latin America & Caribbean\":\"LAC\", \"Latin America & Caribbean (all income levels)\":\"LCN\", \"Least developed countries: UN classification\":\"LDC\", \"Low income\":\"LIC\", \"Lower middle income\":\"LMC\", \"Low & middle income\":\"LMY\", \"Middle East & North Africa (all income levels)\":\"MEA\", \"Middle income\":\"MIC\", \"Middle East & North Africa\":\"MNA\", \"North America\":\"NAC\", \"High income: nonOECD\":\"NOC\", \"High income: OECD\":\"OEC\", \"OECD members\":\"OED\", \"Other small states\":\"OSS\", \"Pacific island small states\":\"PSS\", \"South Asia\":\"SAS\", \"Sub-Saharan Africa\":\"SSA\", \"Sub-Saharan Africa (all income levels)\":\"SSF\", \"Small states\":\"SST\", \"Upper middle income\":\"UMC\", \"World\":\"WLD\",\"Cote d'Ivoire\":\"CIV\",\"Curacao\":\"CUW\",\"Sao Tome and Principe\":\"STP\",\"American Samoa\":\"ASM\",\"Australia\":\"AUS\",\"Brunei Darussalam\":\"BRN\",\"China\":\"CHN\",\"Fiji\":\"FJI\",\"Micronesia, Fed. Sts.\":\"FSM\",\"Guam\":\"GUM\",\"Hong Kong SAR, China\":\"HKG\",\"Indonesia\":\"IDN\",\"Japan\":\"JPN\",\"Cambodia\":\"KHM\",\"Kiribati\":\"KIR\",\"Korea, Rep.\":\"KOR\",\"Lao PDR\":\"LAO\",\"Macao SAR, China\":\"MAC\",\"Marshall Islands\":\"MHL\",\"Myanmar\":\"MMR\",\"Mongolia\":\"MNG\",\"Northern Mariana Islands\":\"MNP\",\"Malaysia\":\"MYS\",\"New Caledonia\":\"NCL\",\"New Zealand\":\"NZL\",\"Philippines\":\"PHL\",\"Palau\":\"PLW\",\"Papua New Guinea\":\"PNG\",\"Korea, Dem. Rep.\":\"PRK\",\"French Polynesia\":\"PYF\",\"Singapore\":\"SGP\",\"Solomon Islands\":\"SLB\",\"Thailand\":\"THA\",\"Timor-Leste\":\"TLS\",\"Tonga\":\"TON\",\"Tuvalu\":\"TUV\",\"Taiwan, China\":\"TWN\",\"Vietnam\":\"VNM\",\"Vanuatu\":\"VUT\",\"Samoa\":\"WSM\",\"Albania\":\"ALB\",\"Andorra\":\"AND\",\"Armenia\":\"ARM\",\"Austria\":\"AUT\",\"Azerbaijan\":\"AZE\",\"Belgium\":\"BEL\",\"Bulgaria\":\"BGR\",\"Bosnia and Herzegovina\":\"BIH\",\"Belarus\":\"BLR\",\"Switzerland\":\"CHE\",\"Channel Islands\":\"CHI\",\"Cyprus\":\"CYP\",\"Czech Republic\":\"CZE\",\"Germany\":\"DEU\",\"Denmark\":\"DNK\",\"Spain\":\"ESP\",\"Estonia\":\"EST\",\"Finland\":\"FIN\",\"France\":\"FRA\",\"Faeroe Islands\":\"FRO\",\"United Kingdom\":\"GBR\",\"Georgia\":\"GEO\",\"Greece\":\"GRC\",\"Greenland\":\"GRL\",\"Croatia\":\"HRV\",\"Hungary\":\"HUN\",\"Isle of Man\":\"IMN\",\"Ireland\":\"IRL\",\"Iceland\":\"ISL\",\"Italy\":\"ITA\",\"Kazakhstan\":\"KAZ\",\"Kyrgyz Republic\":\"KGZ\",\"Liechtenstein\":\"LIE\",\"Lithuania\":\"LTU\",\"Luxembourg\":\"LUX\",\"Latvia\":\"LVA\",\"Monaco\":\"MCO\",\"Moldova\":\"MDA\",\"Macedonia, FYR\":\"MKD\",\"Montenegro\":\"MNE\",\"Netherlands\":\"NLD\",\"Norway\":\"NOR\",\"Poland\":\"POL\",\"Portugal\":\"PRT\",\"Romania\":\"ROU\",\"Russian Federation\":\"RUS\",\"San Marino\":\"SMR\",\"Serbia\":\"SRB\",\"Slovak Republic\":\"SVK\",\"Slovenia\":\"SVN\",\"Sweden\":\"SWE\",\"Tajikistan\":\"TJK\",\"Turkmenistan\":\"TKM\",\"Turkey\":\"TUR\",\"Ukraine\":\"UKR\",\"Uzbekistan\":\"UZB\",\"Aruba\":\"ABW\",\"Argentina\":\"ARG\",\"Antigua and Barbuda\":\"ATG\",\"Bahamas, The\":\"BHS\",\"Belize\":\"BLZ\",\"Bolivia\":\"BOL\",\"Brazil\":\"BRA\",\"Barbados\":\"BRB\",\"Chile\":\"CHL\",\"Colombia\":\"COL\",\"Costa Rica\":\"CRI\",\"Cuba\":\"CUB\",\"Curaçao\":\"CUW\",\"Cayman Islands\":\"CYM\",\"Dominica\":\"DMA\",\"Dominican Republic\":\"DOM\",\"Ecuador\":\"ECU\",\"Grenada\":\"GRD\",\"Guatemala\":\"GTM\",\"Guyana\":\"GUY\",\"Honduras\":\"HND\",\"Haiti\":\"HTI\",\"Jamaica\":\"JAM\",\"St. Kitts and Nevis\":\"KNA\",\"St. Lucia\":\"LCA\",\"St. Martin (French part)\":\"MAF\",\"Mexico\":\"MEX\",\"Nicaragua\":\"NIC\",\"Panama\":\"PAN\",\"Peru\":\"PER\",\"Puerto Rico\":\"PRI\",\"Paraguay\":\"PRY\",\"El Salvador\":\"SLV\",\"Suriname\":\"SUR\",\"Sint Maarten (Dutch part)\":\"SXM\",\"Turks and Caicos Islands\":\"TCA\",\"Trinidad and Tobago\":\"TTO\",\"Uruguay\":\"URY\",\"St. Vincent and the Grenadines\":\"VCT\",\"Venezuela, RB\":\"VEN\",\"Virgin Islands (U.S.)\":\"VIR\",\"United Arab Emirates\":\"ARE\",\"Bahrain\":\"BHR\",\"Djibouti\":\"DJI\",\"Algeria\":\"DZA\",\"Egypt, Arab Rep.\":\"EGY\",\"Iran, Islamic Rep.\":\"IRN\",\"Iraq\":\"IRQ\",\"Israel\":\"ISR\",\"Jordan\":\"JOR\",\"Kuwait\":\"KWT\",\"Lebanon\":\"LBN\",\"Libya\":\"LBY\",\"Morocco\":\"MAR\",\"Malta\":\"MLT\",\"Oman\":\"OMN\",\"West Bank and Gaza\":\"PSE\",\"Qatar\":\"QAT\",\"Saudi Arabia\":\"SAU\",\"Syrian Arab Republic\":\"SYR\",\"Tunisia\":\"TUN\",\"Yemen, Rep.\":\"YEM\",\"Bermuda\":\"BMU\",\"Canada\":\"CAN\",\"United States\":\"USA\",\"Afghanistan\":\"AFG\",\"Bangladesh\":\"BGD\",\"Bhutan\":\"BTN\",\"India\":\"IND\",\"Sri Lanka\":\"LKA\",\"Maldives\":\"MDV\",\"Nepal\":\"NPL\",\"Pakistan\":\"PAK\",\"Angola\":\"AGO\",\"Burundi\":\"BDI\",\"Benin\":\"BEN\",\"Burkina Faso\":\"BFA\",\"Botswana\":\"BWA\",\"Central African Republic\":\"CAF\",\"Cote d'Ivoire\":\"CIV\",\"Cameroon\":\"CMR\",\"Congo, Dem. Rep.\":\"COD\",\"Congo, Rep.\":\"COG\",\"Comoros\":\"COM\",\"Cabo Verde\":\"CPV\",\"Eritrea\":\"ERI\",\"Ethiopia\":\"ETH\",\"Gabon\":\"GAB\",\"Ghana\":\"GHA\",\"Guinea\":\"GIN\",\"Gambia, The\":\"GMB\",\"Guinea-Bissau\":\"GNB\",\"Equatorial Guinea\":\"GNQ\",\"Kenya\":\"KEN\",\"Liberia\":\"LBR\",\"Lesotho\":\"LSO\",\"Madagascar\":\"MDG\",\"Mali\":\"MLI\",\"Mozambique\":\"MOZ\",\"Mauritania\":\"MRT\",\"Mauritius\":\"MUS\",\"Malawi\":\"MWI\",\"Namibia\":\"NAM\",\"Niger\":\"NER\",\"Nigeria\":\"NGA\",\"Rwanda\":\"RWA\",\"Sudan\":\"SDN\",\"Senegal\":\"SEN\",\"Sierra Leone\":\"SLE\",\"Somalia\":\"SOM\",\"South Sudan\":\"SSD\",\"São Tomé and Principe\":\"STP\",\"Swaziland\":\"SWZ\",\"Seychelles\":\"SYC\",\"Chad\":\"TCD\",\"Togo\":\"TGO\",\"Tanzania\":\"TZA\",\"Uganda\":\"UGA\",\"South Africa\":\"ZAF\",\"Zambia\":\"ZMB\",\"Zimbabwe\":\"ZWE\"}\n",
    "    #Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "    region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "\n",
    "    for item in HDR:  #loop over indicator list (the number of indicator is to big to allow request in 1 time 2 request are maid)\n",
    "        url=\"http://ec2-52-1-168-42.compute-1.amazonaws.com/version/1/country_code/\"+str(iso)+\"/indicator_id/\"+str(item)+\"/year/\"+YearList+\"?structure=icy\"\n",
    "        print(url)\n",
    "        df=pd.read_json(url)\n",
    "        # for indicateur - for each request - 3 lists are created: one containig all years between 1980 and 2013, one with t time current iso, the last one will host the data\n",
    "        for ind in HDRindlist0:\n",
    "            yearList=[] #list of years to create the index\n",
    "            isoList=[] #list of current iso to create the index\n",
    "            vars()[\"v_\"+str(ind)]=[]  #to host the values\n",
    "            \n",
    "            for year in range(1980, 2014):  #for each years between the range\n",
    "                yearList.append(year)\n",
    "                isoList.append(iso)\n",
    "                try:\n",
    "                    value=df['indicator_value'][str(ind)][str(iso)][str(year)]\n",
    "                    vars()[\"v_\"+str(ind)].append(float(value))\n",
    "                except:\n",
    "                    value=float('NaN')\n",
    "                    vars()[\"v_\"+str(ind)].append(value)\n",
    "                    continue\n",
    "        \n",
    "            dataset = pd.DataFrame(vars()[\"v_\"+str(ind)], columns=[\"v_\"+str(ind)]) #add the data\n",
    "            dataset = dataset.set_index([isoList, yearList]) #create the index\n",
    "            dataset_Final = pd.concat([dataset,dataset_Final], axis=1, join='outer') #concact\n",
    "\n",
    "            #change the indicator list for the 2nd sequence \n",
    "            HDRindlist0=HDRindlistB\n",
    "                                            \n",
    "    #================================================================================================\n",
    "    #browse the region dictionnary\n",
    "    for iso3,region_iso in region_dict.items(): \n",
    "            if iso==iso3:\n",
    "                isolistPlus.append(iso)\n",
    "                for i in region_iso.split(','):\n",
    "                    isolistPlus.append(str(i))\n",
    "\n",
    "    #save income class et Region in a separate variable\n",
    "    dataset_Final['Income_class']= isolistPlus[1]\n",
    "    dataset_Final['Region']= isolistPlus[2]\n",
    "    #print(dataset_Final)\n",
    "    dataset_Final.to_csv(folder+'HDRdata_'+iso+'.csv', orient='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Compute group averages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "*******************# Job Well done ! #********************\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "import numpy as np\n",
    "import csv\n",
    "import winsound\n",
    "\n",
    "#working directory\n",
    "folder=\"data/HDRdata/csv/\"\n",
    "\n",
    "isolist=['AFG','ALB','DZA','AND','AGO','ATG','ARG','ARM','AUS','AUT','AZE','BHS','BHR','BGD','BRB','BLR','BEL',\\\n",
    "        'BLZ','BEN','BTN','BOL','BIH','BWA','BRA','BRN','BGR','BFA','BDI','KHM','CMR','CAN','CPV','CAF','TCD',\\\n",
    "        'CHL','CHN','COL','COM','COG','COD','CRI','CIV','HRV','CUB','CYP','CZE','DNK','DJI','DMA','DOM','ECU',\\\n",
    "        'EGY','SLV','GNQ','ERI','EST','ETH','FJI','FIN','FRA','GAB','GMB','GEO','DEU','GHA','GRC','GRD','GTM',\\\n",
    "        'GIN','GNB','GUY','HTI','HND','HKG','HUN','ISL','IND','IDN','IRN','IRQ','IRL','ISR','ITA','JAM','JPN',\\\n",
    "        'JOR','KAZ','KEN','KIR','KOR','KWT','KGZ','LAO','LVA','LBN','LSO','LBR','LBY','LIE','LTU','LUX','MKD',\\\n",
    "        'MDG','MWI','MYS','MDV','MLI','MLT','MHL','MRT','MUS','MEX','FSM','MDA','MCO','MNG','MNE','MAR','MOZ',\\\n",
    "        'MMR','NAM', 'NPL','NLD','NZL','NIC','NER','NGA','NOR','OMN','PAK','PLW','PAN','PNG','PRY','PER','PHL',\\\n",
    "        'POL','PRT','QAT','ROU','RUS','RWA','KNA','LCA','VCT','WSM','SMR','STP','SAU','SEN','SRB','SYC','SLE',\\\n",
    "        'SGP','SVK','SVN','SLB','SOM','ZAF','ESP','LKA','SDN','SUR','SWZ','SWE','CHE','SYR','TJK','TZA','THA',\\\n",
    "        'TLS','TGO','TON','TTO','TUN','TUR','TKM','TUV','UGA','UKR','ARE','GBR','USA','URY','UZB','VUT','VEN',\\\n",
    "        'VNM','YEM','ZMB','ZWE','SSD']\n",
    "\n",
    "#Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "\n",
    "#====================== Generate IMF WEO complete dataset =====================\n",
    "iso_len=len(isolist)\n",
    "frame = dict()\n",
    "#this concatenates all WEO-IMF csv files\n",
    "i=1\n",
    "for iso in isolist:\n",
    "    isolistPlus=[]\n",
    "    frame[i]=pd.read_csv(folder+'HDRdata_'+iso+'.csv')\n",
    "    \n",
    "    # Add income class and Region \n",
    "    #browse the region dictionnary\n",
    "    for iso3,region_iso in region_dict.items(): \n",
    "            if iso==iso3:\n",
    "                isolistPlus.append(iso)\n",
    "                for item in region_iso.split(','):\n",
    "                    isolistPlus.append(str(item))\n",
    "                    \n",
    "    #save income class et Region in a separate variable\n",
    "    frame[i]['Income_class']= isolistPlus[1]\n",
    "    frame[i]['Region']= isolistPlus[2]\n",
    "    #Add name columns names for col 1 and 2:  iso3 and year\n",
    "    frame[i]=frame[i].rename(columns = {'Unnamed: 0':'iso3', 'Unnamed: 1':'year'})\n",
    "    i+=1\n",
    "    \n",
    "#Concat all country files frame[1]...frame[n] \n",
    "final=DataFrame()\n",
    "j=1\n",
    "#concatenate all files\n",
    "for iso in isolist:\n",
    "    final = concat([final, frame[j]])\n",
    "    j+=1\n",
    "\n",
    "final.to_csv(folder+'final_HDR.csv')\n",
    "\n",
    "#========================>   Computing averages    <=========================\n",
    "\n",
    "#Compute average by income for all indicators in the DataFrame 'final' \n",
    "income= final.groupby(['Income_class', 'year']).agg([np.mean])\n",
    "#Compute average by region for all indicators in the DataFrame 'final' \n",
    "region= final.groupby(['Region', 'year']).agg([np.mean])\n",
    "#compute world average\n",
    "world= final.groupby(['year']).agg([np.mean])\n",
    "\n",
    "#!! world is not multiindexed here, it only has year, we thus need to create a multi index containing iso3= WLD\n",
    "yearRange=[]\n",
    "isolistW=[]\n",
    "\n",
    "#Save as list dates ex 1980, 1981, ... 2020\n",
    "yearRange=world.index.values\n",
    "#Save as list  'WLD' time the number of values in world.index.values\n",
    "for item in world.index.values:\n",
    "    isolistW.append('WLD')\n",
    "\n",
    "#Redindex World as (WLD,year) eg (WLD, 1980)...(WLD,2020)\n",
    "world.index= list(zip(isolistW, yearRange))\n",
    "#world.index.name=['iso3', 'year']\n",
    "\n",
    "#in a new DataFrame, concatenate income, region averages and world average\n",
    "final_mean = concat([income,region,world])\n",
    "final_mean.index.name = ['iso3', 'year']\n",
    "#save it as a csv\n",
    "final_mean.to_csv(folder+'HDR_mean.csv')\n",
    "\n",
    "#==============================================< Append dataset and means >============================================\n",
    "dataset = pd.read_csv(folder+'HDR_mean.csv')\n",
    "dataset=dataset.rename(columns = {'Unnamed: 0':'iso3', 'Unnamed: 1':'year'})\n",
    "dataset=dataset.drop(dataset.index[[0,1]])\n",
    "dataset_weo=pd.read_csv(folder+'final_HDR.csv')\n",
    "dataset=dataset.append(dataset_weo)\n",
    "dataset = dataset.drop('Unnamed: 0', 1)\n",
    "dataset.to_csv(folder+'final_HDR_with_avg.csv')\n",
    "\n",
    "#====================================================< END >============================================================\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "winsound.PlaySound(\"SystemHand\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the Json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "from datetime import datetime\n",
    "import numpy as np\n",
    "import winsound\n",
    "import os\n",
    "\n",
    "#Working directory\n",
    "folder=\"data/HDRdata/csv/\"\n",
    "folder_final=\"data/HDRdata/\"\n",
    "\n",
    "#Set begining and end of the serie\n",
    "start=1980\n",
    "end=2015\n",
    "\n",
    "#=================================================================================================\n",
    "#working directory\n",
    "iso3list=['AFG','ALB','DZA','AND','AGO','ATG','ARG','ARM','AUS','AUT','AZE','BHS','BHR','BGD','BRB','BLR','BEL',\\\n",
    "        'BLZ','BEN','BTN','BOL','BIH','BWA','BRA','BRN','BGR','BFA','BDI','KHM','CMR','CAN','CPV','CAF','TCD',\\\n",
    "        'CHL','CHN','COL','COM','COG','COD','CRI','CIV','HRV','CUB','CYP','CZE','DNK','DJI','DMA','DOM','ECU',\\\n",
    "        'EGY','SLV','GNQ','ERI','EST','ETH','FJI','FIN','FRA','GAB','GMB','GEO','DEU','GHA','GRC','GRD','GTM',\\\n",
    "        'GIN','GNB','GUY','HTI','HND','HKG','HUN','ISL','IND','IDN','IRN','IRQ','IRL','ISR','ITA','JAM','JPN',\\\n",
    "        'JOR','KAZ','KEN','KIR','KOR','KWT','KGZ','LAO','LVA','LBN','LSO','LBR','LBY','LIE','LTU','LUX','MKD',\\\n",
    "        'MDG','MWI','MYS','MDV','MLI','MLT','MHL','MRT','MUS','MEX','FSM','MDA','MCO','MNG','MNE','MAR','MOZ',\\\n",
    "        'MMR','NAM', 'NPL','NLD','NZL','NIC','NER','NGA','NOR','OMN','PAK','PLW','PAN','PNG','PRY','PER','PHL',\\\n",
    "        'POL','PRT','QAT','ROU','RUS','RWA','KNA','LCA','VCT','WSM','SMR','STP','SAU','SEN','SRB','SYC','SLE',\\\n",
    "        'SGP','SVK','SVN','SLB','SOM','ZAF','ESP','LKA','SDN','SUR','SWZ','SWE','CHE','SYR','TJK','TZA','THA',\\\n",
    "        'TLS','TGO','TON','TTO','TUN','TUR','TKM','TUV','UGA','UKR','ARE','GBR','USA','URY','UZB','VUT','VEN',\\\n",
    "        'VNM','YEM','ZMB','ZWE','SSD']\n",
    "\n",
    "#Indicator_list\n",
    "HDRindlistA=['306','24106','24206','24806','27706','38406','38506','38606','44706','46106','68006','68106','68606','69706','71106','71206','71306','71406','71506','71606']\n",
    "HDRindlistB=['72206','98106','101006','103006','103606','103706','120606','121106','121206','123306','123406','123506','123606','135206','136906','137006','137506','137906','138806','142506']\n",
    "\n",
    "indicator_list= HDRindlistA + HDRindlistB\n",
    "    \n",
    "#This dictionnary associate a country name to an iso3 code \n",
    "#Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "#=================================================================================================\n",
    "\n",
    "#Load the dataset\n",
    "dataset = pd.read_csv(folder+'final_HDR_with_avg.csv')\n",
    "\n",
    "for ISO in iso3list:\n",
    "    print(\"Proceed: \",ISO)\n",
    "    if dataset.empty: continue\n",
    "    isolistPlus=[] \n",
    "    #browse the region dictionnary\n",
    "    for iso3,region_iso in region_dict.items(): \n",
    "            if ISO==iso3:\n",
    "                isolistPlus.append(ISO)\n",
    "                for item in region_iso.split(','):\n",
    "                    isolistPlus.append(str(item))\n",
    "                    \n",
    "    print(isolistPlus)\n",
    "    #================== store the meta information ====================================================\n",
    "    #Set source meta\n",
    "    source='\"data_source\":{\"source\":\"UNDP, Human Development Report Office\", \"provider\":\"HDR data API\", \"Download_date\":\"'+datetime.now().strftime('%Y-%m-%d')+'\" }'\n",
    "\n",
    "    #read country meta stored in a text file\n",
    "    with open (folder_final+\"json meta country.json.txt\", \"r\") as meta:\n",
    "        for line in meta :\n",
    "            if line.startswith('\"'+ISO+'\"') :\n",
    "                metafinal=\",\"+line+\",\"+source\n",
    "    #=================================================================================================\n",
    "\n",
    "    #store name of the last ind in the list\n",
    "    last=indicator_list[len(indicator_list)-1]\n",
    "\n",
    "    file = open(folder_final+\"temp.json\", \"w\")\n",
    "    #Start writing the json file:\n",
    "    file.write('{')\n",
    "\n",
    "    #Loop over indicator list\n",
    "    for ind in indicator_list:\n",
    "        \n",
    "        #Get indicator meta from external file\n",
    "        filemeta=open(folder+\"meta HDR.txt\", \"r\")\n",
    "        for line in filemeta :\n",
    "            if line.startswith('\"v_'+ind+'\"') : meta=line\n",
    "        filemeta.close()\n",
    "        #write the meta\n",
    "        file.write(meta)\n",
    "\n",
    "        #loop over the isolistPlus which contain [countryiso,incomegrp, regionalgrp, world] \n",
    "        for iso in isolistPlus:\n",
    "            #write iso name\n",
    "            file.write('\"'+iso+'\":{')  \n",
    "            #set min and max year for each ind and iso\n",
    "            year_range=dataset[\"year\"][(dataset[\"iso3\"]==iso)&(dataset['v_'+ind].notnull())]\n",
    "            if len(year_range)!=0:\n",
    "                Ymin=min(year_range)\n",
    "                Ymax=max(year_range)\n",
    "            else:\n",
    "                Ymin=Ymax=\"null\"\n",
    "            #write this information within the serie        \n",
    "            file.write('\"YearMin\":'+str(Ymin)+',')\n",
    "            file.write('\"YearMax\":'+str(Ymax)+',')\n",
    "            for year in range(start,end+1):\n",
    "                x= dataset['v_'+ind][(dataset[\"year\"] == year) & (dataset[\"iso3\"]==iso)]\n",
    "                if x.empty :  \n",
    "                    x=\"null\"\n",
    "                else :\n",
    "                    x=str('{:0.3f}'.format(float(x)))\n",
    "                if ind!=last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}},')\n",
    "                if ind==last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}}'+metafinal+'}')\n",
    "\n",
    "    file.close()\n",
    "\n",
    "    file_in = open(folder_final+\"temp.json\", \"r\")\n",
    "    file_out = open(folder_final+\"HDR_\"+ISO+\".json\", \"wt\")\n",
    "    for line in file_in: file_out.write( line.replace(':nan', \":null\") )\n",
    "    file_in.close()\n",
    "    file_out.close()\n",
    "\n",
    "    #os.remove(folder+'dataset'+ISO+'.csv')\n",
    "    print(\"done:HDR_\"+ISO+\".json\")\n",
    "\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "os.remove(folder+\"temp.json\")\n",
    "winsound.PlaySound(\"SystemHand\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "![image](http://stats4dev.com/img/WBG.jpg)\n",
    "\n",
    "## World Bank data downloading script\n",
    "\n",
    "### What it does:\n",
    "This programme log to World Bank Open data API and create CSVs by countries to be transformed into JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas.io import wb\n",
    "from pandas import Series, DataFrame, concat\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "import sys,os,os.path\n",
    "import winsound\n",
    "\n",
    "#===========================>  Set up proxy environnement <==================================\n",
    "os.environ['HTTP_PROXY']=\"xxxxxxxxxxxxx\"\n",
    "os.environ['HTTPS_PROXY']=\"xxxxxxxxxxxxx\"\n",
    "#============================================================================================\n",
    "\n",
    "#---------------------------------------------------------------------------------------------------------------------\n",
    "print(\"Started... Please wait...\")\n",
    "#-------------------------------------------< Set Indicator list >----------------------------------------------------\n",
    "\n",
    "#Specifies the extremum of the time series\n",
    "start=1980\n",
    "end=2015\n",
    "Serie_range=end-start+1\n",
    "\n",
    "#working directory\n",
    "folder=\"data/WDI/csv/\"\n",
    "\n",
    "#=============================== ALL WDI (we wont' use those) ========================================================\n",
    "# Missing / no longer available indicators: 'IS.ROD.DNST.K2', 'IS.ROD.DESL.KT',  'IS.ROD.DESL.PC', 'IS.ROD.ENGY.KT', 'IS.ROD.ENGY.PC','IS.ROD.ENGY.ZS', 'IS.ROD.GOOD.MT.K6', 'IS.ROD.PAVE.ZS','IS.ROD.PSGR.K6', 'IS.ROD.SGAS.KT', 'IS.ROD.SGAS.PC','IS.ROD.TOTL.KM', 'IS.SHP.GOOD.TU', 'IS.VEH.NVEH.P3','IS.VEH.PCAR.P3', 'IS.VEH.ROAD.K1'\n",
    "#Indicators available in the API (for the others, see missing/ no longer available list)\n",
    "indicator_listALL1=['AG.AGR.TRAC.NO','AG.CON.FERT.PT.ZS','AG.CON.FERT.ZS','AG.LND.AGRI.K2','AG.LND.AGRI.ZS','AG.LND.ARBL.HA','AG.LND.ARBL.HA.PC','AG.LND.ARBL.ZS','AG.LND.CREL.HA','AG.LND.CROP.ZS','AG.LND.EL5M.ZS','AG.LND.FRST.K2','AG.LND.FRST.ZS','AG.LND.IRIG.AG.ZS','AG.LND.PRCP.MM','AG.LND.TOTL.K2','AG.LND.TRAC.ZS','AG.PRD.CREL.MT','AG.PRD.CROP.XD','AG.PRD.FOOD.XD','AG.PRD.LVSK.XD','AG.SRF.TOTL.K2','AG.YLD.CREL.KG','BG.GSR.NFSV.GD.ZS','BM.GSR.CMCP.ZS','BM.GSR.FCTY.CD','BM.GSR.GNFS.CD','BM.GSR.INSF.ZS','BM.GSR.MRCH.CD','BM.GSR.NFSV.CD','BM.GSR.ROYL.CD','BM.GSR.TOTL.CD','BM.GSR.TRAN.ZS','BM.GSR.TRVL.ZS','BM.KLT.DINV.GD.ZS','BM.TRF.PRVT.CD','BM.TRF.PWKR.CD.DT','BN.CAB.XOKA.CD','BN.CAB.XOKA.GD.ZS','BN.FIN.TOTL.CD','BN.GSR.FCTY.CD','BN.GSR.GNFS.CD','BN.GSR.MRCH.CD','BN.KAC.EOMS.CD','BN.KLT.DINV.CD','BN.KLT.PTXL.CD','BN.RES.INCL.CD','BN.TRF.CURR.CD','BN.TRF.KOGT.CD','BX.GRT.EXTA.CD.WD','BX.GRT.TECH.CD.WD','BX.GSR.CCIS.CD','BX.GSR.CCIS.ZS','BX.GSR.CMCP.ZS','BX.GSR.FCTY.CD','BX.GSR.GNFS.CD','BX.GSR.INSF.ZS','BX.GSR.MRCH.CD','BX.GSR.NFSV.CD','BX.GSR.ROYL.CD','BX.GSR.TOTL.CD','BX.GSR.TRAN.ZS','BX.GSR.TRVL.ZS','BX.KLT.DINV.CD.WD','BX.KLT.DINV.WD.GD.ZS','BX.KLT.DREM.CD.DT','BX.PEF.TOTL.CD.WD','BX.TRF.CURR.CD','BX.TRF.PWKR.CD','BX.TRF.PWKR.CD.DT','BX.TRF.PWKR.DT.GD.ZS','CM.MKT.INDX.ZG','CM.MKT.LCAP.CD','CM.MKT.LCAP.GD.ZS','CM.MKT.LDOM.NO','CM.MKT.TRAD.CD','CM.MKT.TRAD.GD.ZS','CM.MKT.TRNR','DC.DAC.AUSL.CD','DC.DAC.AUTL.CD','DC.DAC.BELL.CD','DC.DAC.CANL.CD','DC.DAC.CECL.CD','DC.DAC.CHEL.CD','DC.DAC.CZEL.CD','DC.DAC.DEUL.CD','DC.DAC.DNKL.CD','DC.DAC.ESPL.CD','DC.DAC.FINL.CD','DC.DAC.FRAL.CD','DC.DAC.GBRL.CD','DC.DAC.GRCL.CD','DC.DAC.IRLL.CD','DC.DAC.ISLL.CD','DC.DAC.ITAL.CD','DC.DAC.JPNL.CD','DC.DAC.LUXL.CD','DC.DAC.NLDL.CD','DC.DAC.NORL.CD','DC.DAC.NZLL.CD','DC.DAC.POLL.CD','DC.DAC.PRTL.CD','DC.DAC.SVKL.CD','DC.DAC.SVNL.CD','DC.DAC.SWEL.CD','DC.DAC.TOTL.CD','DC.DAC.USAL.CD','DT.AMT.BLAT.CD','DT.AMT.BLTC.CD','DT.AMT.DIMF.CD','DT.AMT.DLTF.CD','DT.AMT.DLXF.CD','DT.AMT.DPNG.CD','DT.AMT.DPPG.CD','DT.AMT.MIBR.CD','DT.AMT.MIDA.CD','DT.AMT.MLAT.CD','DT.AMT.MLTC.CD','DT.AMT.OFFT.CD','DT.AMT.PBND.CD','DT.AMT.PCBK.CD','DT.AMT.PNGB.CD','DT.AMT.PNGC.CD','DT.AMT.PROP.CD','DT.AMT.PRVT.CD','DT.AXA.DPPG.CD','DT.AXA.OFFT.CD','DT.AXA.PRVT.CD','DT.AXF.DPPG.CD','DT.AXR.DPPG.CD','DT.AXR.OFFT.CD','DT.AXR.PRVT.CD','DT.COM.DPPG.CD','DT.COM.MIBR.CD','DT.COM.MIDA.CD','DT.COM.OFFT.CD','DT.COM.PRVT.CD','DT.CUR.DMAK.ZS','DT.CUR.EURO.ZS','DT.CUR.FFRC.ZS','DT.CUR.JYEN.ZS','DT.CUR.MULC.ZS','DT.CUR.OTHC.ZS','DT.CUR.SDRW.ZS','DT.CUR.SWFR.ZS','DT.CUR.UKPS.ZS','DT.CUR.USDL.ZS','DT.DFR.DPPG.CD','DT.DIS.BLAT.CD','DT.DIS.BLTC.CD','DT.DIS.DIMF.CD','DT.DIS.DLTF.CD','DT.DIS.DLXF.CD','DT.DIS.DPNG.CD','DT.DIS.DPPG.CD','DT.DIS.IDAG.CD','DT.DIS.MIBR.CD','DT.DIS.MIDA.CD','DT.DIS.MLAT.CD','DT.DIS.MLTC.CD','DT.DIS.OFFT.CD','DT.DIS.PBND.CD','DT.DIS.PCBK.CD','DT.DIS.PNGB.CD','DT.DIS.PNGC.CD','DT.DIS.PROP.CD','DT.DIS.PRVT.CD','DT.DOD.ALLC.CD','DT.DOD.ALLC.ZS','DT.DOD.BLAT.CD','DT.DOD.BLTC.CD','DT.DOD.DECT.CD','DT.DOD.DECT.CD.CG','DT.DOD.DECT.EX.ZS','DT.DOD.DECT.GN.ZS','DT.DOD.DIMF.CD','DT.DOD.DLXF.CD','DT.DOD.DPNG.CD','DT.DOD.DPPG.CD','DT.DOD.DSTC.CD','DT.DOD.DSTC.IR.ZS','DT.DOD.DSTC.XP.ZS','DT.DOD.DSTC.ZS','DT.DOD.MDRI.CD','DT.DOD.MIBR.CD','DT.DOD.MIDA.CD','DT.DOD.MLAT.CD','DT.DOD.MLAT.ZS','DT.DOD.MLTC.CD','DT.DOD.MWBG.CD','DT.DOD.OFFT.CD','DT.DOD.PBND.CD','DT.DOD.PCBK.CD','DT.DOD.PNGB.CD','DT.DOD.PNGC.CD','DT.DOD.PROP.CD','DT.DOD.PRVS.CD','DT.DOD.PRVT.CD','DT.DOD.PUBS.CD','DT.DOD.PVLX.CD','DT.DOD.PVLX.EX.ZS','DT.DOD.PVLX.GN.ZS','DT.DOD.RSDL.CD','DT.DOD.VTOT.CD','DT.DSB.DPPG.CD','DT.DSF.DPPG.CD','DT.DXR.DPPG.CD','DT.GPA.DPPG','DT.GPA.OFFT','DT.GPA.PRVT','DT.GRE.DPPG','DT.GRE.OFFT','DT.GRE.PRVT','DT.INR.DPPG','DT.INR.OFFT','DT.INR.PRVT','DT.INT.BLAT.CD','DT.INT.BLTC.CD','DT.INT.DECT.CD','DT.INT.DECT.EX.ZS','DT.INT.DECT.GN.ZS','DT.INT.DIMF.CD','DT.INT.DLXF.CD','DT.INT.DPNG.CD','DT.INT.DPPG.CD','DT.INT.DSTC.CD','DT.INT.MIBR.CD','DT.INT.MIDA.CD','DT.INT.MLAT.CD','DT.INT.MLTC.CD','DT.INT.OFFT.CD','DT.INT.PBND.CD','DT.INT.PCBK.CD','DT.INT.PNGB.CD','DT.INT.PNGC.CD','DT.INT.PROP.CD','DT.INT.PRVT.CD','DT.IXA.DPPG.CD','DT.IXA.DPPG.CD.CG','DT.IXA.OFFT.CD','DT.IXA.PRVT.CD','DT.IXF.DPPG.CD','DT.IXR.DPPG.CD','DT.IXR.OFFT.CD','DT.IXR.PRVT.CD','DT.MAT.DPPG','DT.MAT.OFFT','DT.MAT.PRVT','DT.NFL.BLAT.CD','DT.NFL.BLTC.CD','DT.NFL.BOND.CD','DT.NFL.DECT.CD','DT.NFL.DLXF.CD','DT.NFL.DPNG.CD','DT.NFL.DPPG.CD','DT.NFL.DSTC.CD','DT.NFL.IAEA.CD','DT.NFL.IFAD.CD','DT.NFL.IMFC.CD','DT.NFL.IMFN.CD','DT.NFL.MIBR.CD','DT.NFL.MIDA.CD','DT.NFL.MLAT.CD','DT.NFL.MLTC.CD','DT.NFL.MOTH.CD','DT.NFL.NEBR.CD','DT.NFL.NIFC.CD','DT.NFL.OFFT.CD','DT.NFL.PBND.CD','DT.NFL.PCBK.CD','DT.NFL.PCBO.CD','DT.NFL.PNGB.CD','DT.NFL.PNGC.CD','DT.NFL.PROP.CD','DT.NFL.PRVT.CD','DT.NFL.RDBC.CD','DT.NFL.RDBN.CD','DT.NFL.UNAI.CD','DT.NFL.UNCF.CD','DT.NFL.UNCR.CD','DT.NFL.UNDP.CD','DT.NFL.UNEC.CD','DT.NFL.UNFP.CD','DT.NFL.UNPB.CD','DT.NFL.UNRW.CD','DT.NFL.UNTA.CD','DT.NFL.WFPG.CD','DT.NFL.WHOL.CD','DT.NTR.BLAT.CD','DT.NTR.BLTC.CD','DT.NTR.DECT.CD','DT.NTR.DLXF.CD','DT.NTR.DPNG.CD','DT.NTR.DPPG.CD','DT.NTR.MIBR.CD','DT.NTR.MIDA.CD','DT.NTR.MLAT.CD','DT.NTR.MLTC.CD','DT.NTR.OFFT.CD','DT.NTR.PBND.CD','DT.NTR.PCBK.CD','DT.NTR.PNGB.CD','DT.NTR.PNGC.CD','DT.NTR.PROP.CD','DT.NTR.PRVT.CD','DT.ODA.ALLD.CD','DT.ODA.ALLD.KD','DT.ODA.OATL.CD','DT.ODA.OATL.KD','DT.ODA.ODAT.CD','DT.ODA.ODAT.GI.ZS','DT.ODA.ODAT.GN.ZS','DT.ODA.ODAT.KD','DT.ODA.ODAT.MP.ZS','DT.ODA.ODAT.PC.ZS','DT.ODA.ODAT.XP.ZS','DT.TDS.BLAT.CD','DT.TDS.BLTC.CD','DT.TDS.DECT.CD','DT.TDS.DECT.EX.ZS','DT.TDS.DECT.GN.ZS','DT.TDS.DIMF.CD','DT.TDS.DLXF.CD','DT.TDS.DPNG.CD','DT.TDS.DPPF.XP.ZS','DT.TDS.DPPG.CD','DT.TDS.DPPG.GN.ZS','DT.TDS.DPPG.XP.ZS','DT.TDS.MIBR.CD','DT.TDS.MIDA.CD','DT.TDS.MLAT.CD','DT.TDS.MLAT.PG.ZS','DT.TDS.MLTC.CD','DT.TDS.OFFT.CD','DT.TDS.PBND.CD','DT.TDS.PCBK.CD','DT.TDS.PNGB.CD','DT.TDS.PNGC.CD','DT.TDS.PROP.CD','DT.TDS.PRVT.CD','DT.TXR.DPPG.CD','DT.UND.DPPG.CD','DT.UND.OFFT.CD','DT.UND.PRVT.CD','EA.PRD.AGRI.KD','EG.ELC.ACCS.RU.ZS','EG.ELC.ACCS.UR.ZS','EG.ELC.ACCS.ZS','EG.ELC.COAL.ZS','EG.ELC.FOSL.ZS','EG.ELC.HYRO.ZS','EG.ELC.LOSS.ZS','EG.ELC.NGAS.ZS','EG.ELC.NUCL.ZS','EG.ELC.PETR.ZS','EG.ELC.RNWX.KH','EG.ELC.RNWX.ZS','EG.GDP.PUSE.KO.PP','EG.GDP.PUSE.KO.PP.KD','EG.IMP.CONS.ZS','EG.NSF.ACCS.RU.ZS','EG.NSF.ACCS.UR.ZS','EG.NSF.ACCS.ZS','EG.USE.COMM.CL.ZS','EG.USE.COMM.FO.ZS','EG.USE.COMM.GD.PP.KD','EG.USE.CRNW.ZS','EG.USE.ELEC.KH.PC','EG.USE.PCAP.KG.OE','EN.ATM.CO2E.EG.ZS','EN.ATM.CO2E.GF.KT','EN.ATM.CO2E.GF.ZS','EN.ATM.CO2E.KD.GD','EN.ATM.CO2E.KT','EN.ATM.CO2E.LF.KT','EN.ATM.CO2E.LF.ZS','EN.ATM.CO2E.PC','EN.ATM.CO2E.PP.GD','EN.ATM.CO2E.PP.GD.KD','EN.ATM.CO2E.SF.KT','EN.ATM.CO2E.SF.ZS','EN.ATM.GHGO.KT.CE','EN.ATM.HFCG.KT.CE','EN.ATM.METH.AG.KT.CE','EN.ATM.METH.AG.ZS','EN.ATM.METH.EG.KT.CE','EN.ATM.METH.EG.ZS','EN.ATM.METH.KT.CE','EN.ATM.NOXE.AG.KT.CE','EN.ATM.NOXE.AG.ZS','EN.ATM.NOXE.EG.KT.CE','EN.ATM.NOXE.EI.ZS','EN.ATM.NOXE.IN.KT.CE','EN.ATM.NOXE.KT.CE','EN.ATM.PFCG.KT.CE','EN.ATM.PM25.MC.M3','EN.ATM.PM25.MC.ZS','EN.ATM.SF6G.KT.CE','EN.BIR.THRD.NO','EN.CLC.DRSK.XQ','EN.CLC.GHGR.MT.CE','EN.CLC.MDAT.ZS','EN.CO2.BLDG.ZS','EN.CO2.ETOT.ZS','EN.CO2.MANF.ZS','EN.CO2.OTHX.ZS','EN.CO2.TRAN.ZS','EN.FSH.THRD.NO','EN.HPT.THRD.NO','EN.MAM.THRD.NO','EN.POP.EL5M.ZS','EN.URB.MCTY.TL.ZS','EP.PMP.DESL.CD','EP.PMP.SGAS.CD','ER.BDV.TOTL.XQ','ER.GDP.FWTL.M3.KD','ER.H2O.FWAG.ZS','ER.H2O.FWDM.ZS','ER.H2O.FWIN.ZS','ER.H2O.FWTL.K3','ER.H2O.FWTL.ZS','ER.H2O.INTR.K3','ER.H2O.INTR.PC','ER.LND.PTLD.ZS','ER.MRN.PTMR.ZS','ER.PTD.TOTL.ZS','FB.AST.NPER.ZS','FB.ATM.TOTL.P5','FB.BNK.CAPA.ZS','FB.CBK.BRCH.P5','FB.CBK.BRWR.P3','FB.CBK.DPTR.P3','FB.POS.TOTL.P5','FD.AST.PRVT.GD.ZS','FD.RES.LIQU.AS.ZS','FI.RES.TOTL.CD','FI.RES.TOTL.DT.ZS','FI.RES.TOTL.MO','FI.RES.XGLD.CD','FM.AST.CGOV.ZG.M3','FM.AST.DOMO.ZG.M3','FM.AST.DOMS.CN','FM.AST.NFRG.CN','FM.AST.PRVT.ZG.M3','FM.LBL.BMNY.CN','FM.LBL.BMNY.GD.ZS','FM.LBL.BMNY.IR.ZS','FM.LBL.BMNY.ZG','FM.LBL.MONY.CN','FM.LBL.MQMY.CN','FM.LBL.MQMY.GD.ZS','FM.LBL.MQMY.IR.ZS','FM.LBL.MQMY.ZG','FM.LBL.QMNY.CN','FP.CPI.TOTL','FP.CPI.TOTL.ZG','FP.WPI.TOTL','FR.INR.DPST','FR.INR.LEND','FR.INR.LNDP','FR.INR.RINR','FR.INR.RISK','FS.AST.CGOV.GD.ZS','FS.AST.DOMO.GD.ZS','FS.AST.DOMS.GD.ZS','FS.AST.PRVT.GD.ZS','FS.LBL.LIQU.GD.ZS','FS.LBL.QLIQ.GD.ZS','GB.XPD.RSDV.GD.ZS','GC.BAL.CASH.CN','GC.BAL.CASH.GD.ZS','GC.DOD.TOTL.CN','GC.DOD.TOTL.GD.ZS','GC.FIN.DOMS.CN','GC.FIN.DOMS.GD.ZS','GC.FIN.FRGN.CN','GC.FIN.FRGN.GD.ZS','GC.REV.GOTR.CN','GC.REV.GOTR.ZS','GC.REV.SOCL.CN','GC.REV.SOCL.ZS','GC.REV.XGRT.CN','GC.REV.XGRT.GD.ZS','GC.TAX.EXPT.CN','GC.TAX.EXPT.ZS','GC.TAX.GSRV.CN','GC.TAX.GSRV.RV.ZS','GC.TAX.GSRV.VA.ZS','GC.TAX.IMPT.CN','GC.TAX.IMPT.ZS','GC.TAX.INTT.CN','GC.TAX.INTT.RV.ZS','GC.TAX.OTHR.CN','GC.TAX.OTHR.RV.ZS','GC.TAX.TOTL.CN','GC.TAX.TOTL.GD.ZS','GC.TAX.YPKG.CN','GC.TAX.YPKG.RV.ZS','GC.TAX.YPKG.ZS','GC.XPN.COMP.CN','GC.XPN.COMP.ZS','GC.XPN.GSRV.CN','GC.XPN.GSRV.ZS','GC.XPN.INTP.CN','GC.XPN.INTP.RV.ZS','GC.XPN.INTP.ZS','GC.XPN.OTHR.CN','GC.XPN.OTHR.ZS','GC.XPN.TOTL.CN','GC.XPN.TOTL.GD.ZS','GC.XPN.TRFT.CN','GC.XPN.TRFT.ZS','IC.BUS.DFRN.XQ','IC.BUS.DISC.XQ','IC.BUS.EASE.XQ','IC.BUS.NDNS.ZS','IC.BUS.NREG','IC.CRD.INFO.XQ','IC.CRD.PRVT.ZS','IC.CRD.PUBL.ZS','IC.CUS.DURS.EX','IC.ELC.DURS','IC.ELC.OUTG','IC.ELC.TIME','IC.EXP.COST.CD','IC.EXP.DOCS','IC.EXP.DURS','IC.FRM.BKWC.ZS','IC.FRM.BNKS.ZS','IC.FRM.BRIB.ZS','IC.FRM.CMPU.ZS','IC.FRM.CORR.ZS','IC.FRM.CRIM.ZS','IC.FRM.DURS','IC.FRM.FEMO.ZS','IC.FRM.FREG.ZS','IC.FRM.INFM.ZS','IC.FRM.ISOC.ZS','IC.FRM.OUTG.ZS']\n",
    "indicator_listALL2=['IC.FRM.TRNG.ZS','IC.GOV.DURS.ZS','IC.IMP.COST.CD','IC.IMP.DOCS','IC.IMP.DURS','IC.ISV.DURS','IC.LGL.CRED.XQ','IC.LGL.DURS','IC.LGL.PROC','IC.PRP.DURS','IC.PRP.PROC','IC.REG.COST.PC.ZS','IC.REG.DURS','IC.REG.PROC','IC.TAX.DURS','IC.TAX.GIFT.ZS','IC.TAX.LABR.CP.ZS','IC.TAX.METG','IC.TAX.OTHR.CP.ZS','IC.TAX.PAYM','IC.TAX.PRFT.CP.ZS','IC.TAX.TOTL.CP.ZS','IC.WRH.DURS','IC.WRH.PROC','IE.PPI.ENGY.CD','IE.PPI.TELE.CD','IE.PPI.TRAN.CD','IE.PPI.WATR.CD','IP.JRN.ARTC.SC','IP.PAT.NRES','IP.PAT.RESD','IP.TMK.NRES','IP.TMK.RESD','IP.TMK.TOTL','IQ.CPA.BREG.XQ','IQ.CPA.DEBT.XQ','IQ.CPA.ECON.XQ','IQ.CPA.ENVR.XQ','IQ.CPA.FINQ.XQ','IQ.CPA.FINS.XQ','IQ.CPA.FISP.XQ','IQ.CPA.GNDR.XQ','IQ.CPA.HRES.XQ','IQ.CPA.IRAI.XQ','IQ.CPA.MACR.XQ','IQ.CPA.PADM.XQ','IQ.CPA.PRES.XQ','IQ.CPA.PROP.XQ','IQ.CPA.PROT.XQ','IQ.CPA.PUBS.XQ','IQ.CPA.REVN.XQ','IQ.CPA.SOCI.XQ','IQ.CPA.STRC.XQ','IQ.CPA.TRAD.XQ','IQ.CPA.TRAN.XQ','IQ.SCI.MTHD','IQ.SCI.OVRL','IQ.SCI.PRDC','IQ.SCI.SRCE','IQ.WEF.CUST.XQ','IQ.WEF.PORT.XQ','IS.AIR.DPRT','IS.AIR.GOOD.MT.K1','IS.AIR.PSGR','IS.RRS.GOOD.MT.K6','IS.RRS.PASG.KM','IS.RRS.TOTL.KM','IS.SHP.GCNW.XQ','IT.CEL.SETS','IT.CEL.SETS.P2','IT.MLT.MAIN','IT.MLT.MAIN.P2','IT.NET.BBND','IT.NET.BBND.P2','IT.NET.SECR','IT.NET.SECR.P6','IT.NET.USER.P2','LP.EXP.DURS.MD','LP.IMP.DURS.MD','LP.LPI.CUST.XQ','LP.LPI.INFR.XQ','LP.LPI.ITRN.XQ','LP.LPI.LOGS.XQ','LP.LPI.OVRL.XQ','LP.LPI.TIME.XQ','LP.LPI.TRAC.XQ','MS.MIL.MPRT.KD','MS.MIL.TOTL.P1','MS.MIL.TOTL.TF.ZS','MS.MIL.XPND.CN','MS.MIL.XPND.GD.ZS','MS.MIL.XPND.ZS','MS.MIL.XPRT.KD','NE.CON.GOVT.CD','NE.CON.GOVT.CN','NE.CON.GOVT.KD','NE.CON.GOVT.KD.ZG','NE.CON.GOVT.KN','NE.CON.GOVT.ZS','NE.CON.PETC.CD','NE.CON.PETC.CN','NE.CON.PETC.KD','NE.CON.PETC.KD.ZG','NE.CON.PETC.KN','NE.CON.PETC.ZS','NE.CON.PRVT.CD','NE.CON.PRVT.CN','NE.CON.PRVT.KD','NE.CON.PRVT.KD.ZG','NE.CON.PRVT.KN','NE.CON.PRVT.PC.KD','NE.CON.PRVT.PC.KD.ZG','NE.CON.PRVT.PP.CD','NE.CON.PRVT.PP.KD','NE.CON.TETC.CD','NE.CON.TETC.CN','NE.CON.TETC.KD','NE.CON.TETC.KD.ZG','NE.CON.TETC.KN','NE.CON.TETC.ZS','NE.CON.TOTL.CD','NE.CON.TOTL.CN','NE.CON.TOTL.KD','NE.CON.TOTL.KN','NE.DAB.DEFL.ZS','NE.DAB.TOTL.CD','NE.DAB.TOTL.CN','NE.DAB.TOTL.KD','NE.DAB.TOTL.KN','NE.DAB.TOTL.ZS','NE.EXP.GNFS.CD','NE.EXP.GNFS.CN','NE.EXP.GNFS.KD','NE.EXP.GNFS.KD.ZG','NE.EXP.GNFS.KN','NE.EXP.GNFS.ZS','NE.GDI.FPRV.CN','NE.GDI.FPRV.ZS','NE.GDI.FTOT.CD','NE.GDI.FTOT.CN','NE.GDI.FTOT.KD','NE.GDI.FTOT.KD.ZG','NE.GDI.FTOT.KN','NE.GDI.FTOT.ZS','NE.GDI.STKB.CD','NE.GDI.STKB.CN','NE.GDI.STKB.KN','NE.GDI.TOTL.CD','NE.GDI.TOTL.CN','NE.GDI.TOTL.KD','NE.GDI.TOTL.KD.ZG','NE.GDI.TOTL.KN','NE.GDI.TOTL.ZS','NE.IMP.GNFS.CD','NE.IMP.GNFS.CN','NE.IMP.GNFS.KD','NE.IMP.GNFS.KD.ZG','NE.IMP.GNFS.KN','NE.IMP.GNFS.ZS','NE.RSB.GNFS.CD','NE.RSB.GNFS.CN','NE.RSB.GNFS.KN','NE.RSB.GNFS.ZS','NE.TRD.GNFS.ZS','NV.AGR.TOTL.CD','NV.AGR.TOTL.CN','NV.AGR.TOTL.KD','NV.AGR.TOTL.KD.ZG','NV.AGR.TOTL.KN','NV.AGR.TOTL.ZS','NV.IND.MANF.CD','NV.IND.MANF.CN','NV.IND.MANF.KD','NV.IND.MANF.KD.ZG','NV.IND.MANF.KN','NV.IND.MANF.ZS','NV.IND.TOTL.CD','NV.IND.TOTL.CN','NV.IND.TOTL.KD','NV.IND.TOTL.KD.ZG','NV.IND.TOTL.KN','NV.IND.TOTL.ZS','NV.MNF.CHEM.ZS.UN','NV.MNF.FBTO.ZS.UN','NV.MNF.MTRN.ZS.UN','NV.MNF.OTHR.ZS.UN','NV.MNF.TXTL.ZS.UN','NV.SRV.TETC.CD','NV.SRV.TETC.CN','NV.SRV.TETC.KD','NV.SRV.TETC.KD.ZG','NV.SRV.TETC.KN','NV.SRV.TETC.ZS','NY.ADJ.AEDU.CD','NY.ADJ.AEDU.GN.ZS','NY.ADJ.DCO2.CD','NY.ADJ.DCO2.GN.ZS','NY.ADJ.DFOR.CD','NY.ADJ.DFOR.GN.ZS','NY.ADJ.DKAP.CD','NY.ADJ.DKAP.GN.ZS','NY.ADJ.DMIN.CD','NY.ADJ.DMIN.GN.ZS','NY.ADJ.DNGY.CD','NY.ADJ.DNGY.GN.ZS','NY.ADJ.DPEM.CD','NY.ADJ.DPEM.GN.ZS','NY.ADJ.DRES.GN.ZS','NY.ADJ.ICTR.GN.ZS','NY.ADJ.NNAT.CD','NY.ADJ.NNAT.GN.ZS','NY.ADJ.NNTY.CD','NY.ADJ.NNTY.KD','NY.ADJ.NNTY.KD.ZG','NY.ADJ.NNTY.PC.CD','NY.ADJ.SVNG.CD','NY.ADJ.SVNG.GN.ZS','NY.ADJ.SVNX.CD','NY.ADJ.SVNX.GN.ZS','NY.EXP.CAPM.KN','NY.GDP.COAL.RT.ZS','NY.GDP.DEFL.KD.ZG','NY.GDP.DEFL.ZS','NY.GDP.DISC.CN','NY.GDP.DISC.KN','NY.GDP.FCST.CD','NY.GDP.FCST.CN','NY.GDP.FCST.KD','NY.GDP.FCST.KN','NY.GDP.FRST.RT.ZS','NY.GDP.MINR.RT.ZS','NY.GDP.MKTP.CD','NY.GDP.MKTP.CN','NY.GDP.MKTP.KD','NY.GDP.MKTP.KD.ZG','NY.GDP.MKTP.KN','NY.GDP.MKTP.PP.CD','NY.GDP.MKTP.PP.KD','NY.GDP.NGAS.RT.ZS','NY.GDP.PCAP.CD','NY.GDP.PCAP.CN','NY.GDP.PCAP.KD','NY.GDP.PCAP.KD.ZG','NY.GDP.PCAP.KN','NY.GDP.PCAP.PP.CD','NY.GDP.PCAP.PP.KD','NY.GDP.PETR.RT.ZS','NY.GDP.TOTL.RT.ZS','NY.GDS.TOTL.CD','NY.GDS.TOTL.CN','NY.GDS.TOTL.KN','NY.GDS.TOTL.ZS','NY.GDY.TOTL.KD','NY.GDY.TOTL.KN','NY.GNP.ATLS.CD','NY.GNP.MKTP.CD','NY.GNP.MKTP.CN','NY.GNP.MKTP.KD','NY.GNP.MKTP.KD.ZG','NY.GNP.MKTP.KN','NY.GNP.MKTP.PP.CD','NY.GNP.MKTP.PP.KD','NY.GNP.PCAP.CD','NY.GNP.PCAP.CN','NY.GNP.PCAP.KD','NY.GNP.PCAP.KD.ZG','NY.GNP.PCAP.KN','NY.GNP.PCAP.PP.CD','NY.GNP.PCAP.PP.KD','NY.GNS.ICTR.CD','NY.GNS.ICTR.CN','NY.GNS.ICTR.GN.ZS','NY.GNS.ICTR.ZS','NY.GSR.NFCY.CD','NY.GSR.NFCY.CN','NY.GSR.NFCY.KN','NY.TAX.NIND.CD','NY.TAX.NIND.CN','NY.TAX.NIND.KN','NY.TRF.NCTR.CD','NY.TRF.NCTR.CN','NY.TRF.NCTR.KN','NY.TTF.GNFS.KN','PA.NUS.ATLS','PA.NUS.FCRF','PA.NUS.PPP','PA.NUS.PPP.05','PA.NUS.PPPC.RF','PA.NUS.PRVT.PP','PA.NUS.PRVT.PP.05','per_allsp.adq_pop_tot','per_allsp.ben_q1_tot','per_allsp.cov_pop_tot','per_lm_alllm.adq_pop_tot','per_lm_alllm.ben_q1_tot','per_lm_alllm.cov_pop_tot','per_sa_allsa.adq_pop_tot','per_sa_allsa.ben_q1_tot','per_sa_allsa.cov_pop_tot','per_si_allsi.adq_pop_tot','per_si_allsi.ben_q1_tot','per_si_allsi.cov_pop_tot','PX.REX.REER','SE.ADT.1524.LT.FE.ZS','SE.ADT.1524.LT.FM.ZS','SE.ADT.1524.LT.MA.ZS','SE.ADT.1524.LT.ZS','SE.ADT.LITR.FE.ZS','SE.ADT.LITR.MA.ZS','SE.ADT.LITR.ZS','SE.ENR.PRIM.FM.ZS','SE.ENR.PRSC.FM.ZS','SE.ENR.SECO.FM.ZS','SE.ENR.TERT.FM.ZS','SE.PRE.ENRR','SE.PRE.ENRR.FE','SE.PRE.ENRR.MA','SE.PRM.AGES','SE.PRM.CMPT.FE.ZS','SE.PRM.CMPT.MA.ZS','SE.PRM.CMPT.ZS','SE.PRM.DURS','SE.PRM.ENRL','SE.PRM.ENRL.FE.ZS','SE.PRM.ENRL.TC.ZS','SE.PRM.ENRR','SE.PRM.ENRR.FE','SE.PRM.ENRR.MA','SE.PRM.GINT.FE.ZS','SE.PRM.GINT.MA.ZS','SE.PRM.GINT.ZS','SE.PRM.NENR','SE.PRM.NENR.FE','SE.PRM.NENR.MA','SE.PRM.NINT.FE.ZS','SE.PRM.NINT.MA.ZS','SE.PRM.NINT.ZS','SE.PRM.PRIV.ZS','SE.PRM.PRS5.FE.ZS','SE.PRM.PRS5.MA.ZS','SE.PRM.PRS5.ZS','SE.PRM.PRSL.FE.ZS','SE.PRM.PRSL.MA.ZS','SE.PRM.PRSL.ZS','SE.PRM.REPT.FE.ZS','SE.PRM.REPT.MA.ZS','SE.PRM.REPT.ZS','SE.PRM.TCAQ.FE.ZS','SE.PRM.TCAQ.MA.ZS','SE.PRM.TCAQ.ZS','SE.PRM.TCHR','SE.PRM.TCHR.FE.ZS','SE.PRM.TENR','SE.PRM.TENR.FE','SE.PRM.TENR.MA','SE.PRM.UNER','SE.PRM.UNER.FE','SE.PRM.UNER.MA','SE.SEC.AGES','SE.SEC.CMPT.LO.FE.ZS','SE.SEC.CMPT.LO.MA.ZS','SE.SEC.CMPT.LO.ZS','SE.SEC.DURS','SE.SEC.ENRL','SE.SEC.ENRL.FE.ZS','SE.SEC.ENRL.GC','SE.SEC.ENRL.GC.FE.ZS','SE.SEC.ENRL.TC.ZS','SE.SEC.ENRL.VO','SE.SEC.ENRL.VO.FE.ZS','SE.SEC.ENRR','SE.SEC.ENRR.FE','SE.SEC.ENRR.MA','SE.SEC.NENR','SE.SEC.NENR.FE','SE.SEC.NENR.MA','SE.SEC.PRIV.ZS','SE.SEC.PROG.FE.ZS','SE.SEC.PROG.MA.ZS','SE.SEC.PROG.ZS','SE.SEC.REPT.FE.ZS','SE.SEC.REPT.MA.ZS','SE.SEC.REPT.ZS','SE.SEC.TCHR','SE.SEC.TCHR.FE','SE.SEC.TCHR.FE.ZS','SE.TER.ENRR','SE.TER.ENRR.FE','SE.TER.ENRR.MA','SE.TER.TCHR.FE.ZS','SE.XPD.CPRM.ZS','SE.XPD.CSEC.ZS','SE.XPD.CTER.ZS','SE.XPD.CTOT.ZS','SE.XPD.MPRM.ZS','SE.XPD.MSEC.ZS','SE.XPD.MTER.ZS','SE.XPD.MTOT.ZS','SE.XPD.PRIM.PC.ZS','SE.XPD.PRIM.ZS','SE.XPD.SECO.PC.ZS','SE.XPD.SECO.ZS','SE.XPD.TERT.PC.ZS','SE.XPD.TERT.ZS','SE.XPD.TOTL.GB.ZS','SE.XPD.TOTL.GD.ZS','SG.GEN.LSOM.ZS','SG.GEN.PARL.ZS','SG.VAW.ARGU.ZS','SG.VAW.BURN.ZS','SG.VAW.GOES.ZS','SG.VAW.NEGL.ZS','SG.VAW.REAS.ZS','SG.VAW.REFU.ZS','SH.ANM.NPRG.ZS','SH.CON.1524.FE.ZS','SH.CON.1524.MA.ZS','SH.DTH.IMRT','SH.DTH.MORT','SH.DTH.NMRT','SH.DYN.AIDS.FE.ZS','SH.DYN.AIDS.ZS','SH.DYN.MORT','SH.DYN.MORT.FE','SH.DYN.MORT.MA','SH.DYN.NMRT','SH.H2O.SAFE.RU.ZS','SH.H2O.SAFE.UR.ZS','SH.H2O.SAFE.ZS','SH.HIV.0014','SH.HIV.1524.FE.ZS','SH.HIV.1524.MA.ZS','SH.HIV.ARTC.ZS','SH.IMM.IDPT','SH.IMM.MEAS','SH.MED.BEDS.ZS','SH.MED.CMHW.P3','SH.MED.NUMW.P3','SH.MED.PHYS.ZS','SH.MLR.NETS.ZS','SH.MLR.TRET.ZS','SH.MMR.DTHS','SH.MMR.RISK','SH.MMR.RISK.ZS','SH.PRG.ANEM','SH.PRV.SMOK.FE','SH.PRV.SMOK.MA','SH.STA.ACSN','SH.STA.ACSN.RU','SH.STA.ACSN.UR','SH.STA.ANVC.ZS','SH.STA.ARIC.ZS','SH.STA.BFED.ZS','SH.STA.BRTC.ZS','SH.STA.BRTW.ZS','SH.STA.MALN.FE.ZS','SH.STA.MALN.MA.ZS','SH.STA.MALN.ZS','SH.STA.MMRT','SH.STA.MMRT.NE','SH.STA.ORCF.ZS','SH.STA.ORTH','SH.STA.OWGH.FE.ZS','SH.STA.OWGH.MA.ZS','SH.STA.OWGH.ZS','SH.STA.STNT.FE.ZS','SH.STA.STNT.MA.ZS','SH.STA.STNT.ZS','SH.STA.WAST.FE.ZS','SH.STA.WAST.MA.ZS','SH.STA.WAST.ZS','SH.SVR.WAST.FE.ZS','SH.SVR.WAST.MA.ZS','SH.SVR.WAST.ZS','SH.TBS.CURE.ZS','SH.TBS.DTEC.ZS','SH.TBS.INCD','SH.VAC.TTNS.ZS','SH.XPD.EXTR.ZS','SH.XPD.OOPC.TO.ZS','SH.XPD.OOPC.ZS','SH.XPD.PCAP','SH.XPD.PCAP.PP.KD','SH.XPD.PRIV.ZS','SH.XPD.PUBL','SH.XPD.PUBL.GX.ZS','SH.XPD.PUBL.ZS','SH.XPD.TOTL.ZS','SI.DST.02ND.20','SI.DST.03RD.20','SI.DST.04TH.20','SI.DST.05TH.20','SI.DST.10TH.10','SI.DST.FRST.10','SI.DST.FRST.20','SI.POV.2DAY','SI.POV.DDAY','SI.POV.GAP2','SI.POV.GAPS','SI.POV.GINI','SI.POV.NAGP','SI.POV.NAHC','SI.POV.RUGP','SI.POV.RUHC','SI.POV.URGP','SI.POV.URHC','SI.SPR.PC40','SI.SPR.PC40.ZG','SI.SPR.PCAP','SI.SPR.PCAP.ZG','SL.AGR.0714.FE.ZS','SL.AGR.0714.MA.ZS','SL.AGR.0714.ZS','SL.AGR.EMPL.FE.ZS','SL.AGR.EMPL.MA.ZS','SL.AGR.EMPL.ZS','SL.EMP.1524.SP.FE.NE.ZS','SL.EMP.1524.SP.FE.ZS','SL.EMP.1524.SP.MA.NE.ZS','SL.EMP.1524.SP.MA.ZS','SL.EMP.1524.SP.NE.ZS','SL.EMP.1524.SP.ZS','SL.EMP.INSV.FE.ZS','SL.EMP.MPYR.FE.ZS','SL.EMP.MPYR.MA.ZS','SL.EMP.MPYR.ZS','SL.EMP.SELF.FE.ZS','SL.EMP.SELF.MA.ZS','SL.EMP.SELF.ZS','SL.EMP.TOTL.SP.FE.NE.ZS','SL.EMP.TOTL.SP.FE.ZS','SL.EMP.TOTL.SP.MA.NE.ZS']\n",
    "indicator_listALL3=['SL.EMP.TOTL.SP.MA.ZS','SL.EMP.TOTL.SP.NE.ZS','SL.EMP.TOTL.SP.ZS','SL.EMP.VULN.FE.ZS','SL.EMP.VULN.MA.ZS','SL.EMP.VULN.ZS','SL.EMP.WORK.FE.ZS','SL.EMP.WORK.MA.ZS','SL.EMP.WORK.ZS','SL.FAM.0714.FE.ZS','SL.FAM.0714.MA.ZS','SL.FAM.0714.ZS','SL.FAM.WORK.FE.ZS','SL.FAM.WORK.MA.ZS','SL.FAM.WORK.ZS','SL.GDP.PCAP.EM.KD','SL.IND.EMPL.FE.ZS','SL.IND.EMPL.MA.ZS','SL.IND.EMPL.ZS','SL.MNF.0714.FE.ZS','SL.MNF.0714.MA.ZS','SL.MNF.0714.ZS','SL.SLF.0714.FE.ZS','SL.SLF.0714.MA.ZS','SL.SLF.0714.ZS','SL.SRV.0714.FE.ZS','SL.SRV.0714.MA.ZS','SL.SRV.0714.ZS','SL.SRV.EMPL.FE.ZS','SL.SRV.EMPL.MA.ZS','SL.SRV.EMPL.ZS','SL.TLF.0714.FE.ZS','SL.TLF.0714.MA.ZS','SL.TLF.0714.SW.FE.TM','SL.TLF.0714.SW.FE.ZS','SL.TLF.0714.SW.MA.TM','SL.TLF.0714.SW.MA.ZS','SL.TLF.0714.SW.TM','SL.TLF.0714.SW.ZS','SL.TLF.0714.WK.FE.TM','SL.TLF.0714.WK.FE.ZS','SL.TLF.0714.WK.MA.TM','SL.TLF.0714.WK.MA.ZS','SL.TLF.0714.WK.TM','SL.TLF.0714.WK.ZS','SL.TLF.0714.ZS','SL.TLF.ACTI.1524.FE.NE.ZS','SL.TLF.ACTI.1524.MA.NE.ZS','SL.TLF.ACTI.1524.NE.ZS','SL.TLF.CACT.FE.NE.ZS','SL.TLF.CACT.FE.ZS','SL.TLF.CACT.FM.NE.ZS','SL.TLF.CACT.FM.ZS','SL.TLF.CACT.MA.NE.ZS','SL.TLF.CACT.MA.ZS','SL.TLF.CACT.NE.ZS','SL.TLF.CACT.ZS','SL.TLF.PART.FE.ZS','SL.TLF.PART.MA.ZS','SL.TLF.PART.TL.FE.ZS','SL.TLF.PART.ZS','SL.TLF.PRIM.FE.ZS','SL.TLF.PRIM.MA.ZS','SL.TLF.PRIM.ZS','SL.TLF.SECO.FE.ZS','SL.TLF.SECO.MA.ZS','SL.TLF.SECO.ZS','SL.TLF.TERT.FE.ZS','SL.TLF.TERT.MA.ZS','SL.TLF.TERT.ZS','SL.TLF.TOTL.FE.ZS','SL.TLF.TOTL.IN','SL.UEM.1524.FE.NE.ZS','SL.UEM.1524.FE.ZS','SL.UEM.1524.MA.NE.ZS','SL.UEM.1524.MA.ZS','SL.UEM.1524.NE.ZS','SL.UEM.1524.ZS','SL.UEM.LTRM.FE.ZS','SL.UEM.LTRM.MA.ZS','SL.UEM.LTRM.ZS','SL.UEM.NEET.FE.ZS','SL.UEM.NEET.MA.ZS','SL.UEM.NEET.ZS','SL.UEM.PRIM.FE.ZS','SL.UEM.PRIM.MA.ZS','SL.UEM.PRIM.ZS','SL.UEM.SECO.FE.ZS','SL.UEM.SECO.MA.ZS','SL.UEM.SECO.ZS','SL.UEM.TERT.FE.ZS','SL.UEM.TERT.MA.ZS','SL.UEM.TERT.ZS','SL.UEM.TOTL.FE.NE.ZS','SL.UEM.TOTL.FE.ZS','SL.UEM.TOTL.MA.NE.ZS','SL.UEM.TOTL.MA.ZS','SL.UEM.TOTL.NE.ZS','SL.UEM.TOTL.ZS','SL.WAG.0714.FE.ZS','SL.WAG.0714.MA.ZS','SL.WAG.0714.ZS','SM.EMI.TERT.ZS','SM.POP.NETM','SM.POP.REFG','SM.POP.REFG.OR','SM.POP.TOTL','SM.POP.TOTL.ZS','SN.ITK.DEFC.ZS','SN.ITK.DFCT','SN.ITK.SALT.ZS','SN.ITK.VITA.ZS','SP.ADO.TFRT','SP.DTH.INFR.ZS','SP.DTH.REPT.ZS','SP.DYN.AMRT.FE','SP.DYN.AMRT.MA','SP.DYN.CBRT.IN','SP.DYN.CDRT.IN','SP.DYN.CONU.ZS','SP.DYN.IMRT.FE.IN','SP.DYN.IMRT.IN','SP.DYN.IMRT.MA.IN','SP.DYN.LE00.FE.IN','SP.DYN.LE00.IN','SP.DYN.LE00.MA.IN','SP.DYN.TFRT.IN','SP.DYN.TO65.FE.ZS','SP.DYN.TO65.MA.ZS','SP.DYN.WFRT','SP.HOU.FEMA.ZS','SP.MTR.1519.ZS','SP.POP.0014.TO.ZS','SP.POP.1564.TO.ZS','SP.POP.65UP.TO.ZS','SP.POP.DPND','SP.POP.DPND.OL','SP.POP.DPND.YG','SP.POP.GROW','SP.POP.SCIE.RD.P6','SP.POP.TECH.RD.P6','SP.POP.TOTL','SP.POP.TOTL.FE.ZS','SP.REG.BRTH.RU.ZS','SP.REG.BRTH.UR.ZS','SP.REG.BRTH.ZS','SP.RUR.TOTL','SP.RUR.TOTL.ZG','SP.RUR.TOTL.ZS','SP.URB.GROW','SP.URB.TOTL','SP.URB.TOTL.IN.ZS','SP.UWT.TFRT','ST.INT.ARVL','ST.INT.DPRT','ST.INT.RCPT.CD','ST.INT.RCPT.XP.ZS','ST.INT.TRNR.CD','ST.INT.TRNX.CD','ST.INT.TVLR.CD','ST.INT.TVLX.CD','ST.INT.XPND.CD','ST.INT.XPND.MP.ZS','TG.VAL.TOTL.GD.ZS','TM.QTY.MRCH.XD.WD','TM.TAX.MANF.BC.ZS','TM.TAX.MANF.BR.ZS','TM.TAX.MANF.IP.ZS','TM.TAX.MANF.SM.AR.ZS','TM.TAX.MANF.SM.FN.ZS','TM.TAX.MANF.SR.ZS','TM.TAX.MANF.WM.AR.ZS','TM.TAX.MANF.WM.FN.ZS','TM.TAX.MRCH.BC.ZS','TM.TAX.MRCH.BR.ZS','TM.TAX.MRCH.IP.ZS','TM.TAX.MRCH.SM.AR.ZS','TM.TAX.MRCH.SM.FN.ZS','TM.TAX.MRCH.SR.ZS','TM.TAX.MRCH.WM.AR.ZS','TM.TAX.MRCH.WM.FN.ZS','TM.TAX.TCOM.BC.ZS','TM.TAX.TCOM.BR.ZS','TM.TAX.TCOM.IP.ZS','TM.TAX.TCOM.SM.AR.ZS','TM.TAX.TCOM.SM.FN.ZS','TM.TAX.TCOM.SR.ZS','TM.TAX.TCOM.WM.AR.ZS','TM.TAX.TCOM.WM.FN.ZS','TM.VAL.AGRI.ZS.UN','TM.VAL.FOOD.ZS.UN','TM.VAL.FUEL.ZS.UN','TM.VAL.ICTG.ZS.UN','TM.VAL.INSF.ZS.WT','TM.VAL.MANF.ZS.UN','TM.VAL.MMTL.ZS.UN','TM.VAL.MRCH.AL.ZS','TM.VAL.MRCH.CD.WT','TM.VAL.MRCH.HI.ZS','TM.VAL.MRCH.OR.ZS','TM.VAL.MRCH.R1.ZS','TM.VAL.MRCH.R2.ZS','TM.VAL.MRCH.R3.ZS','TM.VAL.MRCH.R4.ZS','TM.VAL.MRCH.R5.ZS','TM.VAL.MRCH.R6.ZS','TM.VAL.MRCH.RS.ZS','TM.VAL.MRCH.WL.CD','TM.VAL.MRCH.WR.ZS','TM.VAL.MRCH.XD.WD','TM.VAL.OTHR.ZS.WT','TM.VAL.SERV.CD.WT','TM.VAL.TRAN.ZS.WT','TM.VAL.TRVL.ZS.WT','TT.PRI.MRCH.XD.WD','TX.QTY.MRCH.XD.WD','TX.VAL.AGRI.ZS.UN','TX.VAL.FOOD.ZS.UN','TX.VAL.FUEL.ZS.UN','TX.VAL.ICTG.ZS.UN','TX.VAL.INSF.ZS.WT','TX.VAL.MANF.ZS.UN','TX.VAL.MMTL.ZS.UN','TX.VAL.MRCH.AL.ZS','TX.VAL.MRCH.CD.WT','TX.VAL.MRCH.HI.ZS','TX.VAL.MRCH.OR.ZS','TX.VAL.MRCH.R1.ZS','TX.VAL.MRCH.R2.ZS','TX.VAL.MRCH.R3.ZS','TX.VAL.MRCH.R4.ZS','TX.VAL.MRCH.R5.ZS','TX.VAL.MRCH.R6.ZS','TX.VAL.MRCH.RS.ZS','TX.VAL.MRCH.WL.CD','TX.VAL.MRCH.WR.ZS','TX.VAL.MRCH.XD.WD','TX.VAL.OTHR.ZS.WT','TX.VAL.SERV.CD.WT','TX.VAL.TECH.CD','TX.VAL.TECH.MF.ZS','TX.VAL.TRAN.ZS.WT','TX.VAL.TRVL.ZS.WT','VC.BTL.DETH','VC.IDP.TOTL.HE','VC.IDP.TOTL.LE','VC.IHR.PSRC.P5','VC.PKP.TOTL.UN']\n",
    "#Add list 1 to 3\n",
    "indicator_listALL=indicator_listALL1+indicator_listALL2+indicator_listALL3\n",
    "\n",
    "#Indicator not to include: the ones that we are not interested in (and that also belong to indicator_listALL)\n",
    "indicator_listSUPPR=['AG.CON.FERT.PT.ZS', 'DT.NFL.UNAI.CD', 'DT.NFL.UNDP.CD', 'DT.NFL.UNCR.CD', 'DT.NFL.UNCF.CD', 'DT.NFL.UNRW.CD', 'DT.NFL.UNTA.CD', 'DT.NFL.WFPG.CD', 'NY.GDP.DISC.KN', 'NY.GDP.DISC.CN', 'DC.DAC.AUSL.CD', 'DC.DAC.AUTL.CD', 'DC.DAC.BELL.CD', 'DC.DAC.CANL.CD', 'DC.DAC.CZEL.CD', 'DC.DAC.FINL.CD', 'DC.DAC.GRCL.CD', 'DC.DAC.ISLL.CD', 'DC.DAC.IRLL.CD', 'DC.DAC.ITAL.CD', 'DC.DAC.LUXL.CD', 'DC.DAC.NLDL.CD', 'DC.DAC.NZLL.CD', 'DC.DAC.NORL.CD', 'DC.DAC.POLL.CD', 'DC.DAC.PRTL.CD', 'DC.DAC.SVKL.CD', 'DC.DAC.SVNL.CD', 'DC.DAC.ESPL.CD', 'DC.DAC.SWEL.CD', 'DC.DAC.CHEL.CD', 'SE.PRM.NINT.FE.ZS', 'SE.PRM.NINT.MA.ZS', 'SE.PRM.NINT.ZS', 'SE.PRM.PRS5.FE.ZS', 'SE.PRM.PRS5.MA.ZS', 'SE.PRM.PRS5.ZS', 'SE.PRM.PRSL.FE.ZS', 'SE.PRM.PRSL.MA.ZS', 'SE.PRM.PRSL.ZS', 'EN.FSH.THRD.NO', 'NY.ADJ.DPEM.CD', 'EN.MAM.THRD.NO', 'NY.ADJ.DPEM.GN.ZS', 'BX.PEF.TOTL.CD.WD', 'DT.COM.MIBR.CD', 'DT.DOD.BLTC.CD', 'DT.INT.BLTC.CD', 'DT.NFL.BLTC.CD', 'DT.DOD.ALLC.ZS', 'DT.TDS.BLTC.CD', 'DT.CUR.OTHC.ZS', 'DT.DIS.BLAT.CD', 'DT.INT.BLAT.CD', 'DT.NTR.BLAT.CD', 'DT.TDS.BLAT.CD', 'DT.AMT.PBND.CD', 'DT.DIS.PBND.CD', 'DT.DOD.PBND.CD', 'DT.CUR.SWFR.ZS', 'DT.INT.PBND.CD', 'DT.CUR.USDL.ZS', 'DT.NFL.PBND.CD', 'DT.NTR.PBND.CD', 'DT.TDS.PBND.CD', 'DT.DFR.DPPG.CD', 'DT.DIS.PCBK.CD', 'DT.DOD.PCBK.CD', 'DT.INT.PCBK.CD', 'DT.NFL.PCBK.CD', 'DT.TDS.DECT.CD', 'DT.NTR.PCBK.CD', 'DT.TDS.PCBK.CD', 'DT.AMT.MIBR.CD', 'DT.DXR.DPPG.CD', 'DT.DIS.MIBR.CD', 'DT.DOD.MIBR.CD', 'DT.INT.MIBR.CD', 'DT.NTR.MIBR.CD', 'DT.TDS.MIBR.CD', 'DT.NFL.NEBR.CD', 'DT.AMT.MIDA.CD', 'DT.DIS.MIDA.CD', 'DT.DOD.MIDA.CD', 'DT.INT.MIDA.CD', 'DT.DOD.PRVS.CD', 'DT.NTR.MIDA.CD', 'DT.TDS.MIDA.CD', 'DT.AMT.MLAT.CD', 'DT.AMT.MLTC.CD', 'DT.DOD.MLTC.CD', 'DT.TDS.MLTC.CD', 'DT.DIS.MLAT.CD', 'DT.DOD.MLAT.CD', 'DT.DOD.MWBG.CD', 'DT.INT.MLAT.CD', 'DT.NTR.MLAT.CD', 'DT.NFL.NIFC.CD', 'DT.AMT.OFFT.CD', 'DT.INT.DIMF.CD', 'DT.DIS.OFFT.CD', 'DT.DIS.DIMF.CD', 'DT.DOD.OFFT.CD', 'DT.AMT.DIMF.CD', 'DT.INT.OFFT.CD', 'DT.TDS.DIMF.CD', 'DT.NFL.OFFT.CD', 'DT.NTR.OFFT.CD', 'DT.TDS.OFFT.CD', 'DT.AMT.PROP.CD', 'DT.DIS.PROP.CD', 'DT.DOD.PROP.CD', 'DT.INT.PROP.CD', 'DT.NFL.PROP.CD', 'DT.NTR.PROP.CD', 'DT.TDS.PROP.CD', 'DT.AMT.PRVT.CD', 'DT.DIS.PRVT.CD', 'DT.DOD.PRVT.CD', 'DT.INT.PRVT.CD', 'DT.NFL.PRVT.CD', 'DT.NTR.PRVT.CD', 'DT.TDS.PRVT.CD', 'DT.DOD.PVLX.CD', 'DT.NFL.MIBR.CD', 'DT.NFL.MIDA.CD', 'DT.AXF.DPPG.CD', 'DT.AMT.DLXF.CD', 'DT.AMT.DLTF.CD', 'DT.AMT.DPNG.CD', 'DT.AMT.DPPG.CD', 'DT.AXR.DPPG.CD', 'DT.AXR.OFFT.CD', 'DT.AXR.PRVT.CD', 'DT.ODA.ALLD.CD', 'DT.ODA.ODAT.CD', 'DT.NFL.IAEA.CD', 'DT.NFL.UNEC.CD', 'DT.TXR.DPPG.CD', 'DT.NFL.UNPB.CD', 'DT.NFL.WHOL.CD', 'DT.NTR.DLXF.CD', 'DT.NTR.DECT.CD', 'DT.AMT.PNGB.CD', 'DT.UND.PRVT.CD', 'DT.DIS.PNGB.CD', 'DT.UND.DPPG.CD', 'DT.DOD.PNGB.CD', 'DT.DOD.DIMF.CD', 'DT.INT.PNGB.CD', 'CM.MKT.INDX.ZG', 'FP.WPI.TOTL', 'SE.PRM.PRS5.FE.ZS', 'SE.PRM.PRS5.MA.ZS', 'SE.PRM.PRSL.FE.ZS', 'SE.PRM.PRSL.MA.ZS', 'SE.PRM.NINT.FE.ZS']\n",
    "\n",
    "#withdraw the indicator we are not interested in\n",
    "indicator_list= list(set(indicator_listALL) - set(indicator_listSUPPR))\n",
    "#========================================================================================================================\n",
    "\n",
    "#List At a glance\n",
    "indicator_listAtGlance=['SP.POP.TOTL', 'NE.EXP.GNFS.CD', 'SP.POP.0014.TO.ZS', 'SP.POP.1564.TO.ZS', 'SP.POP.65UP.TO.ZS', 'SL.TLF.ACTI.1524.NE.ZS', 'SP.URB.TOTL.IN.ZS', \\\n",
    "                        'NY.GDP.PCAP.CD', 'NY.GDP.PCAP.PP.KD', 'SI.POV.NAHC', 'SP.DYN.LE00.IN', 'SP.DYN.IMRT.IN', 'SH.H2O.SAFE.ZS', 'SE.ADT.LITR.ZS', 'SE.PRM.ENRR', \\\n",
    "                        'SL.UEM.TOTL.ZS', 'NV.AGR.TOTL.ZS', 'NV.IND.TOTL.ZS', 'NV.SRV.TETC.ZS', 'NY.GDP.MKTP.CN', 'NY.GDP.MKTP.CD', 'NE.IMP.GNFS.CD', 'NY.EXP.CAPM.KN', \\\n",
    "                        'GC.DOD.TOTL.GD.ZS', 'DT.DOD.DECT.GN.ZS', 'NY.GDP.MKTP.KD.ZG', 'NY.GDP.PCAP.KD.ZG', 'NE.GDI.TOTL.ZS', 'NY.GNS.ICTR.ZS', 'FP.CPI.TOTL','NE.EXP.GNFS.KD.ZG',\\\n",
    "                        'BN.CAB.XOKA.GD.ZS', 'FI.RES.TOTL.MO', 'PA.NUS.FCRF', 'PX.REX.REER', 'DT.TDS.DECT.GN.ZS', 'GC.XPN.INTP.RV.ZS', 'GC.REV.XGRT.GD.ZS',\\\n",
    "                        'FR.INR.RINR', 'FS.AST.PRVT.GD.ZS', 'FM.LBL.MQMY.GD.ZS']\n",
    "\n",
    "#list country dashboard\n",
    "indicator_listCD=['SL.UEM.TOTL.FE.ZS','SL.UEM.TOTL.MA.ZS','SL.EMP.INSV.FE.ZS','SL.TLF.CACT.FE.ZS','FM.AST.DOMS.CN','FS.AST.PRVT.GD.ZS','SI.POV.DDAY',\\\n",
    "                'SI.POV.NAHC','SI.POV.URHC','SI.POV.RUHC','PA.NUS.FCRF','GC.TAX.TOTL.GD.ZS','EG.IMP.CONS.ZS','SP.DYN.LE00.FE.IN','SP.DYN.LE00.MA.IN',\\\n",
    "                'SH.DYN.MORT','SH.DYN.AIDS.ZS','SE.ADT.1524.LT.FE.ZS','SE.ADT.1524.LT.MA.ZS','SE.PRM.ENRL.TC.ZS','SE.PRM.CMPT.MA.ZS','SE.PRM.CMPT.FE.ZS',\\\n",
    "                'SE.PRM.ENRR','SE.SEC.ENRR','SE.TER.ENRR','SM.POP.REFG','SG.GEN.PARL.ZS','IC.TAX.DURS','IC.REG.PROC','SE.PRM.UNER.MA','SE.PRM.CMPT.ZS',\\\n",
    "                'EN.ATM.PM25.MC.M3','EN.ATM.PM25.MC.ZS','EN.POP.EL5M.ZS','EN.HPT.THRD.NO','ER.MRN.PTMR.ZS','EN.FSH.THRD.NO','EN.MAM.THRD.NO',\\\n",
    "                'EN.ATM.GHGO.KT.CE','EN.ATM.METH.KT.CE','EN.ATM.CO2E.PC','EG.USE.COMM.FO.ZS','EG.FEC.RNEW.ZS','EG.ELC.RNEW.ZS','EG.USE.ELEC.KH.PC',\\\n",
    "                'EG.USE.CRNW.ZS','EG.USE.COMM.CL.ZS', 'IT.CEL.SETS.P2', 'IT.NET.USER.P2', 'EG.ELC.ACCS.ZS', 'SH.H2O.SAFE.ZS', 'SP.POP.GROW',\\\n",
    "                'SP.DYN.TFRT.IN']\n",
    "\n",
    "\n",
    "#-------------\n",
    "indicator_list= indicator_listCD #indicator_listAtGlance\n",
    "#--------------\n",
    "isolist=['ABW', 'AFG', 'AGO', 'ALB', 'AND', 'ARE', 'ARG', 'ARM', 'ASM', 'ATG', 'AUS', 'AUT', 'AZE', 'BDI', 'BEL', 'BEN', 'BFA', 'BGD', \\\n",
    "         'BGR', 'BHR', 'BHS', 'BIH', 'BLR', 'BLZ', 'BMU', 'BOL', 'BRA', 'BRB', 'BRN', 'BTN', 'BWA', 'CAF', 'CAN', 'CHE', 'CHL', 'CHN', \\\n",
    "         'CIV', 'CMR', 'COD', 'COG', 'COL', 'COM', 'CPV', 'CRI', 'CUB', 'CUW', 'CYM', 'CYP', 'CZE', 'DEU', 'DJI', 'DMA', 'DNK', 'DOM', \\\n",
    "         'DZA', 'ECU', 'EGY', 'ERI', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FRA', 'FSM', 'GAB', 'GBR', 'GEO', 'GHA', 'GIN', 'GMB', \\\n",
    "         'GNB', 'GNQ', 'GRC', 'GRD', 'GRL', 'GTM', 'GUM', 'GUY', 'HKG', 'HND', 'HRV', 'HTI', 'HUN', 'IDN', 'IMN', 'IND', 'IRL', 'IRN', \\\n",
    "         'IRQ', 'ISL', 'ISR', 'ITA', 'JAM', 'JOR', 'JPN', 'KAZ', 'KEN', 'KGZ', 'KHM', 'KNA', 'KOR', 'KWT', 'LAO', 'LBN', 'LBR', 'LBY', \\\n",
    "         'LCA', 'LIE', 'LKA', 'LSO', 'LTU', 'LUX', 'LVA', 'MAC', 'MAF', 'MAR', 'MCO', 'MDA', 'MDG', 'MDV', 'MEX', 'MHL', 'MKD', 'MLI', \\\n",
    "         'MLT', 'MMR', 'MNE', 'MNG', 'MNP', 'MOZ', 'MRT', 'MUS', 'MWI', 'MYS', 'NAM', 'NCL', 'NER', 'NGA', 'NIC', 'NLD', 'NOR', 'NPL', \\\n",
    "         'NZL', 'OMN', 'PAK', 'PAN', 'PER', 'PHL', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'PSE', 'PYF', 'QAT', 'ROU', 'RUS', 'RWA', \\\n",
    "         'SAU', 'SDN', 'SEN', 'SGP', 'SLB', 'SLE', 'SLV', 'SMR', 'SOM', 'SRB', 'SSD', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SWZ', 'SXM',\\\n",
    "         'SYC', 'SYR', 'TCA', 'TCD', 'TGO', 'THA', 'TJK', 'TKM', 'TLS', 'TON', 'TTO', 'TUN', 'TUR', 'TUV', 'TWN', 'TZA', 'UGA', 'UKR', \\\n",
    "         'URY', 'USA', 'UZB', 'VCT', 'VEN', 'VIR', 'VNM', 'VUT', 'WSM', 'YEM', 'ZAF', 'ZMB', 'ZWE']\n",
    "\n",
    "#create a list for countries not to process\n",
    "reject=['PRK', 'FRO']\n",
    "\n",
    "#non standard recognized by the API -> regioniso=['EAS','ECS','LCN','LIC','LMC','MEA','MIC','NAC','NOC','OEC','SAS','SSF','UMC','WLD']\n",
    "#non standard ISO= ['ARB', 'CEB', 'CSS', 'EAP', 'EAS', 'ECA', 'ECS', 'EMU', 'EUU', 'FCS', 'HIC', 'HPC', 'LAC', 'LCN', 'LDC', 'LIC', 'LMC', 'LMY', 'MEA', 'MIC', 'MNA', 'NAC', 'NOC', 'OEC', 'OED', 'OSS', 'PSS', 'SAS', 'SSA', 'SSF', 'SST', 'UMC', 'WLD']\n",
    "\n",
    "    \n",
    "#create an empty list called done\n",
    "done=list()\n",
    "dirs = os.listdir(folder)\n",
    "# This will store all the file names in the folder and store iso3 code in the 'done' list\n",
    "for file in dirs:\n",
    "    #take a look at file name excluse .txt and temp_wb.jon\n",
    "    if not \"temp_WB.json\" in file and not '.txt' in file and not '.json' in file: \n",
    "        #keep the other file letter 7 to 10 (iso3)\n",
    "        #print(file[7:10])\n",
    "        done.append(file[7:10])\n",
    "\n",
    "# The final list:\n",
    "iso3list=list(set(isolist) - set(done) - set(reject))\n",
    "lenght_iso=len(iso3list)\n",
    "print('Remaining countries:' + str(lenght_iso)+' || : '+ str(iso3list))\n",
    "\n",
    "while lenght_iso > 0:\n",
    "    dirs = os.listdir(folder)\n",
    "    # This will store all the file names in the folder and store iso3 code in the 'done' list\n",
    "    for file in dirs:\n",
    "        #take a look at file name excluse .txt and temp_wb.jon\n",
    "        if not \"temp_WB.json\" in file and not '.txt' in file and not '.json' in file: \n",
    "            #keep the other file letter 7 to 10 (iso3)\n",
    "            #print(file[7:10])\n",
    "            done.append(file[7:10])\n",
    "\n",
    "    # The final list:\n",
    "    iso3list=list(set(isolist) - set(done))\n",
    "    lenght_iso=len(iso3list)\n",
    "        \n",
    "    #This dictionnary associate a country name to an iso3 code \n",
    "    country_dict = {\"Arab World\":\"ARB\", \"Central Europe and the Baltics\":\"CEB\", \"Caribbean small states\":\"CSS\", \"East Asia & Pacific\":\"EAP\", \"East Asia & Pacific (all income levels)\":\"EAS\", \"Europe & Central Asia\":\"ECA\", \"Europe & Central Asia (all income levels)\":\"ECS\", \"Euro area\":\"EMU\", \"European Union\":\"EUU\", \"Fragile and conflict affected situations\":\"FCS\", \"High income\":\"HIC\", \"Heavily indebted poor countries (HIPC)\":\"HPC\", \"Latin America & Caribbean\":\"LAC\", \"Latin America & Caribbean (all income levels)\":\"LCN\", \"Least developed countries: UN classification\":\"LDC\", \"Low income\":\"LIC\", \"Lower middle income\":\"LMC\", \"Low & middle income\":\"LMY\", \"Middle East & North Africa (all income levels)\":\"MEA\", \"Middle income\":\"MIC\", \"Middle East & North Africa\":\"MNA\", \"North America\":\"NAC\", \"High income: nonOECD\":\"NOC\", \"High income: OECD\":\"OEC\", \"OECD members\":\"OED\", \"Other small states\":\"OSS\", \"Pacific island small states\":\"PSS\", \"South Asia\":\"SAS\", \"Sub-Saharan Africa\":\"SSA\", \"Sub-Saharan Africa (all income levels)\":\"SSF\", \"Small states\":\"SST\", \"Upper middle income\":\"UMC\", \"World\":\"WLD\",\"Cote d'Ivoire\":\"CIV\",\"Curacao\":\"CUW\",\"Sao Tome and Principe\":\"STP\",\"American Samoa\":\"ASM\",\"Australia\":\"AUS\",\"Brunei Darussalam\":\"BRN\",\"China\":\"CHN\",\"Fiji\":\"FJI\",\"Micronesia, Fed. Sts.\":\"FSM\",\"Guam\":\"GUM\",\"Hong Kong SAR, China\":\"HKG\",\"Indonesia\":\"IDN\",\"Japan\":\"JPN\",\"Cambodia\":\"KHM\",\"Kiribati\":\"KIR\",\"Korea, Rep.\":\"KOR\",\"Lao PDR\":\"LAO\",\"Macao SAR, China\":\"MAC\",\"Marshall Islands\":\"MHL\",\"Myanmar\":\"MMR\",\"Mongolia\":\"MNG\",\"Northern Mariana Islands\":\"MNP\",\"Malaysia\":\"MYS\",\"New Caledonia\":\"NCL\",\"New Zealand\":\"NZL\",\"Philippines\":\"PHL\",\"Palau\":\"PLW\",\"Papua New Guinea\":\"PNG\",\"Korea, Dem. Rep.\":\"PRK\",\"French Polynesia\":\"PYF\",\"Singapore\":\"SGP\",\"Solomon Islands\":\"SLB\",\"Thailand\":\"THA\",\"Timor-Leste\":\"TLS\",\"Tonga\":\"TON\",\"Tuvalu\":\"TUV\",\"Taiwan, China\":\"TWN\",\"Vietnam\":\"VNM\",\"Vanuatu\":\"VUT\",\"Samoa\":\"WSM\",\"Albania\":\"ALB\",\"Andorra\":\"AND\",\"Armenia\":\"ARM\",\"Austria\":\"AUT\",\"Azerbaijan\":\"AZE\",\"Belgium\":\"BEL\",\"Bulgaria\":\"BGR\",\"Bosnia and Herzegovina\":\"BIH\",\"Belarus\":\"BLR\",\"Switzerland\":\"CHE\",\"Channel Islands\":\"CHI\",\"Cyprus\":\"CYP\",\"Czech Republic\":\"CZE\",\"Germany\":\"DEU\",\"Denmark\":\"DNK\",\"Spain\":\"ESP\",\"Estonia\":\"EST\",\"Finland\":\"FIN\",\"France\":\"FRA\",\"Faeroe Islands\":\"FRO\",\"United Kingdom\":\"GBR\",\"Georgia\":\"GEO\",\"Greece\":\"GRC\",\"Greenland\":\"GRL\",\"Croatia\":\"HRV\",\"Hungary\":\"HUN\",\"Isle of Man\":\"IMN\",\"Ireland\":\"IRL\",\"Iceland\":\"ISL\",\"Italy\":\"ITA\",\"Kazakhstan\":\"KAZ\",\"Kyrgyz Republic\":\"KGZ\",\"Liechtenstein\":\"LIE\",\"Lithuania\":\"LTU\",\"Luxembourg\":\"LUX\",\"Latvia\":\"LVA\",\"Monaco\":\"MCO\",\"Moldova\":\"MDA\",\"Macedonia, FYR\":\"MKD\",\"Montenegro\":\"MNE\",\"Netherlands\":\"NLD\",\"Norway\":\"NOR\",\"Poland\":\"POL\",\"Portugal\":\"PRT\",\"Romania\":\"ROU\",\"Russian Federation\":\"RUS\",\"San Marino\":\"SMR\",\"Serbia\":\"SRB\",\"Slovak Republic\":\"SVK\",\"Slovenia\":\"SVN\",\"Sweden\":\"SWE\",\"Tajikistan\":\"TJK\",\"Turkmenistan\":\"TKM\",\"Turkey\":\"TUR\",\"Ukraine\":\"UKR\",\"Uzbekistan\":\"UZB\",\"Aruba\":\"ABW\",\"Argentina\":\"ARG\",\"Antigua and Barbuda\":\"ATG\",\"Bahamas, The\":\"BHS\",\"Belize\":\"BLZ\",\"Bolivia\":\"BOL\",\"Brazil\":\"BRA\",\"Barbados\":\"BRB\",\"Chile\":\"CHL\",\"Colombia\":\"COL\",\"Costa Rica\":\"CRI\",\"Cuba\":\"CUB\",\"Curaçao\":\"CUW\",\"Cayman Islands\":\"CYM\",\"Dominica\":\"DMA\",\"Dominican Republic\":\"DOM\",\"Ecuador\":\"ECU\",\"Grenada\":\"GRD\",\"Guatemala\":\"GTM\",\"Guyana\":\"GUY\",\"Honduras\":\"HND\",\"Haiti\":\"HTI\",\"Jamaica\":\"JAM\",\"St. Kitts and Nevis\":\"KNA\",\"St. Lucia\":\"LCA\",\"St. Martin (French part)\":\"MAF\",\"Mexico\":\"MEX\",\"Nicaragua\":\"NIC\",\"Panama\":\"PAN\",\"Peru\":\"PER\",\"Puerto Rico\":\"PRI\",\"Paraguay\":\"PRY\",\"El Salvador\":\"SLV\",\"Suriname\":\"SUR\",\"Sint Maarten (Dutch part)\":\"SXM\",\"Turks and Caicos Islands\":\"TCA\",\"Trinidad and Tobago\":\"TTO\",\"Uruguay\":\"URY\",\"St. Vincent and the Grenadines\":\"VCT\",\"Venezuela, RB\":\"VEN\",\"Virgin Islands (U.S.)\":\"VIR\",\"United Arab Emirates\":\"ARE\",\"Bahrain\":\"BHR\",\"Djibouti\":\"DJI\",\"Algeria\":\"DZA\",\"Egypt, Arab Rep.\":\"EGY\",\"Iran, Islamic Rep.\":\"IRN\",\"Iraq\":\"IRQ\",\"Israel\":\"ISR\",\"Jordan\":\"JOR\",\"Kuwait\":\"KWT\",\"Lebanon\":\"LBN\",\"Libya\":\"LBY\",\"Morocco\":\"MAR\",\"Malta\":\"MLT\",\"Oman\":\"OMN\",\"West Bank and Gaza\":\"PSE\",\"Qatar\":\"QAT\",\"Saudi Arabia\":\"SAU\",\"Syrian Arab Republic\":\"SYR\",\"Tunisia\":\"TUN\",\"Yemen, Rep.\":\"YEM\",\"Bermuda\":\"BMU\",\"Canada\":\"CAN\",\"United States\":\"USA\",\"Afghanistan\":\"AFG\",\"Bangladesh\":\"BGD\",\"Bhutan\":\"BTN\",\"India\":\"IND\",\"Sri Lanka\":\"LKA\",\"Maldives\":\"MDV\",\"Nepal\":\"NPL\",\"Pakistan\":\"PAK\",\"Angola\":\"AGO\",\"Burundi\":\"BDI\",\"Benin\":\"BEN\",\"Burkina Faso\":\"BFA\",\"Botswana\":\"BWA\",\"Central African Republic\":\"CAF\",\"Cote d'Ivoire\":\"CIV\",\"Cameroon\":\"CMR\",\"Congo, Dem. Rep.\":\"COD\",\"Congo, Rep.\":\"COG\",\"Comoros\":\"COM\",\"Cabo Verde\":\"CPV\",\"Eritrea\":\"ERI\",\"Ethiopia\":\"ETH\",\"Gabon\":\"GAB\",\"Ghana\":\"GHA\",\"Guinea\":\"GIN\",\"Gambia, The\":\"GMB\",\"Guinea-Bissau\":\"GNB\",\"Equatorial Guinea\":\"GNQ\",\"Kenya\":\"KEN\",\"Liberia\":\"LBR\",\"Lesotho\":\"LSO\",\"Madagascar\":\"MDG\",\"Mali\":\"MLI\",\"Mozambique\":\"MOZ\",\"Mauritania\":\"MRT\",\"Mauritius\":\"MUS\",\"Malawi\":\"MWI\",\"Namibia\":\"NAM\",\"Niger\":\"NER\",\"Nigeria\":\"NGA\",\"Rwanda\":\"RWA\",\"Sudan\":\"SDN\",\"Senegal\":\"SEN\",\"Sierra Leone\":\"SLE\",\"Somalia\":\"SOM\",\"South Sudan\":\"SSD\",\"São Tomé and Principe\":\"STP\",\"Swaziland\":\"SWZ\",\"Seychelles\":\"SYC\",\"Chad\":\"TCD\",\"Togo\":\"TGO\",\"Tanzania\":\"TZA\",\"Uganda\":\"UGA\",\"South Africa\":\"ZAF\",\"Zambia\":\"ZMB\",\"Zimbabwe\":\"ZWE\"}\n",
    "    #Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "    region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "    try:\n",
    "        #-------------------------------------<  Loop over iso3list   >-------------------------------------------------------\n",
    "        #Loop over iso3list\n",
    "        for ISO in iso3list:\n",
    "            #Affect world, income and region group code to corresponding iso3, using region_dict dictionnary, stored in isolistPlus     \n",
    "            #empty list to start with\n",
    "            isolistPlus=[] \n",
    "\n",
    "            #browse the region dictionnary\n",
    "            for iso3,region_iso in region_dict.items(): \n",
    "                    if ISO==iso3:\n",
    "                        isolistPlus.append(ISO)\n",
    "                        for item in region_iso.split(','):\n",
    "                            isolistPlus.append(str(item))\n",
    "\n",
    "            print(\"Proceed: \",isolistPlus)\n",
    "\n",
    "            #-------------------------------------> Request to WB API<-------------------------------------------------------\n",
    "            dataset=wb.download(indicator=indicator_list, country=isolistPlus, start=start, end=end)\n",
    "            #----------------------------------------------------------------------------------------------------------------\n",
    "            iso_array=list()\n",
    "            country_array=list()\n",
    "\n",
    "            country_array=dataset.index.get_level_values('country')\n",
    "            year_array=dataset.index.get_level_values('year')\n",
    "\n",
    "            #set iso array\n",
    "            dataset['country'] = pd.Series(country_array, index=dataset.index)\n",
    "            for item in dataset['country']:\n",
    "                for cname,iso in country_dict.items(): \n",
    "                    if item==cname:\n",
    "                        iso_array.append(iso)\n",
    "            #-----------------------------------------------------------------\n",
    "            #change index and drop var country\n",
    "            dataset = dataset.set_index([iso_array, year_array]).drop(['country'], axis=1)\n",
    "            dataset.index.names = [\"iso3\", \"year\"]\n",
    "            dataset.to_csv(folder+'dataset'+ISO+'.csv')\n",
    "            print(\"Done: \" + ISO +' || Remaining countries:' + str(lenght_iso)+'||\\n')\n",
    "            #print(dataset.head())\n",
    "\n",
    "    except: \n",
    "        winsound.PlaySound(\"SystemExit\", winsound.SND_ALIAS)\n",
    "        print(\"\\n *********** HTTP error detected... Pass and continue ******************\\n\")\n",
    "        continue\n",
    "\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "winsound.PlaySound(\"SystemExit\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a json from CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "from datetime import datetime\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "\n",
    "#Specifies the extremum of the time series\n",
    "start=1980\n",
    "end=2015\n",
    "Serie_range=end-start+1\n",
    "\n",
    "#=================================================================================================\n",
    "#Folder to use\n",
    "folder=\"data/WDI/csv/\"\n",
    "folder_final=\"data/WDI/\"\n",
    "#=================================================================================================\n",
    "#list of json already created\n",
    "#create an empty list called done\n",
    "done=list()\n",
    "dirs = os.listdir(folder)\n",
    "# This will store all the file names in the folder and store iso3 code in the 'done' list\n",
    "for file in dirs:\n",
    "    #take a look at file name excluse .txt and temp_wb.jon\n",
    "    if not \"temp.json\" in file and not '.txt' in file and not '.csv' in file: \n",
    "        #keep the other file letter 7 to 10 (iso3)\n",
    "        #print(file[7:10])\n",
    "        done.append(file[7:10])\n",
    "#=================================================================================================\n",
    "\n",
    "\n",
    "indicator_listAtGlance=['SP.POP.TOTL', 'NE.EXP.GNFS.CD', 'SP.POP.0014.TO.ZS', 'SP.POP.1564.TO.ZS', 'SP.POP.65UP.TO.ZS', 'SL.TLF.ACTI.1524.NE.ZS', 'SP.URB.TOTL.IN.ZS', 'NY.GDP.PCAP.CD', 'NY.GDP.PCAP.PP.KD', 'SI.POV.NAHC', 'SP.DYN.LE00.IN', 'SP.DYN.IMRT.IN', 'SH.H2O.SAFE.ZS', 'SE.ADT.LITR.ZS', 'SE.PRM.ENRR', 'SL.UEM.TOTL.ZS', 'NV.AGR.TOTL.ZS', 'NV.IND.TOTL.ZS', 'NV.SRV.TETC.ZS', 'NY.GDP.MKTP.CN', 'NY.GDP.MKTP.CD', 'NE.IMP.GNFS.CD', 'NY.EXP.CAPM.KN', 'GC.DOD.TOTL.GD.ZS', 'DT.DOD.DECT.GN.ZS', 'NY.GDP.MKTP.KD.ZG', 'NY.GDP.PCAP.KD.ZG', 'NE.GDI.TOTL.ZS', 'NY.GNS.ICTR.ZS', 'FP.CPI.TOTL', 'NE.EXP.GNFS.KD.ZG', 'BN.CAB.XOKA.GD.ZS', 'FI.RES.TOTL.MO', 'PA.NUS.FCRF', 'PX.REX.REER', 'DT.TDS.DECT.GN.ZS', 'GC.XPN.INTP.RV.ZS', 'GC.REV.XGRT.GD.ZS', 'FR.INR.RINR', 'FS.AST.PRVT.GD.ZS', 'FM.LBL.MQMY.GD.ZS']\n",
    "\n",
    "\n",
    "isolist=['ABW', 'AFG', 'AGO', 'ALB', 'AND', 'ARE', 'ARG', 'ARM', 'ASM', 'ATG', 'AUS', 'AUT', 'AZE', 'BDI', 'BEL', 'BEN', 'BFA', 'BGD', \\\n",
    "         'BGR', 'BHR', 'BHS', 'BIH', 'BLR', 'BLZ', 'BMU', 'BOL', 'BRA', 'BRB', 'BRN', 'BTN', 'BWA', 'CAF', 'CAN', 'CHE', 'CHL', 'CHN', \\\n",
    "         'CIV', 'CMR', 'COD', 'COG', 'COL', 'COM', 'CPV', 'CRI', 'CUB', 'CUW', 'CYM', 'CYP', 'CZE', 'DEU', 'DJI', 'DMA', 'DNK', 'DOM', \\\n",
    "         'DZA', 'ECU', 'EGY', 'ERI', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FRA', 'FSM', 'GAB', 'GBR', 'GEO', 'GHA', 'GIN', 'GMB', \\\n",
    "         'GNB', 'GNQ', 'GRC', 'GRD', 'GRL', 'GTM', 'GUM', 'GUY', 'HKG', 'HND', 'HRV', 'HTI', 'HUN', 'IDN', 'IMN', 'IND', 'IRL', 'IRN', \\\n",
    "         'IRQ', 'ISL', 'ISR', 'ITA', 'JAM', 'JOR', 'JPN', 'KAZ', 'KEN', 'KGZ', 'KHM', 'KNA', 'KOR', 'KWT', 'LAO', 'LBN', 'LBR', 'LBY', \\\n",
    "         'LCA', 'LIE', 'LKA', 'LSO', 'LTU', 'LUX', 'LVA', 'MAC', 'MAF', 'MAR', 'MCO', 'MDA', 'MDG', 'MDV', 'MEX', 'MHL', 'MKD', 'MLI', \\\n",
    "         'MLT', 'MMR', 'MNE', 'MNG', 'MNP', 'MOZ', 'MRT', 'MUS', 'MWI', 'MYS', 'NAM', 'NCL', 'NER', 'NGA', 'NIC', 'NLD', 'NOR', 'NPL', \\\n",
    "         'NZL', 'OMN', 'PAK', 'PAN', 'PER', 'PHL', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'PSE', 'PYF', 'QAT', 'ROU', 'RUS', 'RWA', \\\n",
    "         'SAU', 'SDN', 'SEN', 'SGP', 'SLB', 'SLE', 'SLV', 'SMR', 'SOM', 'SRB', 'SSD', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SWZ', 'SXM',\\\n",
    "         'SYC', 'SYR', 'TCA', 'TCD', 'TGO', 'THA', 'TJK', 'TKM', 'TLS', 'TON', 'TTO', 'TUN', 'TUR', 'TUV', 'TWN', 'TZA', 'UGA', 'UKR', \\\n",
    "         'URY', 'USA', 'UZB', 'VCT', 'VEN', 'VIR', 'VNM', 'VUT', 'WSM', 'YEM', 'ZAF', 'ZMB', 'ZWE']\n",
    "#non standard recognized by the API -> regioniso=['EAS','ECS','LCN','LIC','LMC','MEA','MIC','NAC','NOC','OEC','SAS','SSF','UMC','WLD']\n",
    "#non standard ISO= ['ARB', 'CEB', 'CSS', 'EAP', 'EAS', 'ECA', 'ECS', 'EMU', 'EUU', 'FCS', 'HIC', 'HPC', 'LAC', 'LCN', 'LDC', 'LIC', 'LMC', 'LMY', 'MEA', 'MIC', 'MNA', 'NAC', 'NOC', 'OEC', 'OED', 'OSS', 'PSS', 'SAS', 'SSA', 'SSF', 'SST', 'UMC', 'WLD']\n",
    "\n",
    "#create a list for countries not to process\n",
    "reject=['PRK', 'FRO']\n",
    "#This dictionnary associate a country name to an iso3 code \n",
    "#Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "\n",
    "indicator_list=indicator_listAtGlance\n",
    "iso3list=list(set(isolist) -  set(reject) - set(done))\n",
    "#=================================================================================================\n",
    "\n",
    "for ISO in iso3list:\n",
    "    isolistPlus=[] \n",
    "    #browse the region dictionnary\n",
    "    for iso3,region_iso in region_dict.items(): \n",
    "            if ISO==iso3:\n",
    "                isolistPlus.append(ISO)\n",
    "                for item in region_iso.split(','):\n",
    "                    isolistPlus.append(str(item))\n",
    "                    \n",
    "    print(isolistPlus)\n",
    "    #================== store the meta information ====================================================\n",
    "    source='\"data_source\":{\"source\":\"World Bank, WDI\", \"provider\":\"World Bank, API\", \"Download_date\":\"'+datetime.now().strftime('%Y-%m-%d')+'\" }'\n",
    "\n",
    "    with open (folder+\"json meta country.json.txt\", \"r\") as meta:\n",
    "        for line in meta :\n",
    "            if line.startswith('\"'+ISO+'\"') :\n",
    "                metafinal=\",\"+line+\",\"+source\n",
    "    #================== store the meta information ====================================================\n",
    "\n",
    "    dataset = pd.read_csv(folder+'dataset'+ISO+'.csv')\n",
    "\n",
    "    #store name of the last ind in the list\n",
    "    last=indicator_list[len(indicator_list)-1]\n",
    "\n",
    "    file = open(folder_final+\"temp.json\", \"w\")\n",
    "    #Start writing the json file:\n",
    "    file.write('{')\n",
    "\n",
    "    for ind in indicator_list:\n",
    "\n",
    "        #Get indicator meta from external file\n",
    "        filemeta=open(folder_final+\"meta indicator.txt\", \"r\")\n",
    "        for line in filemeta :\n",
    "            if line.startswith('\"'+ind+'\"') : meta=line\n",
    "        filemeta.close()\n",
    "        #write the meta\n",
    "        file.write(meta)\n",
    "\n",
    "        #loop over the isolistPlus which contain [countryiso,incomegrp, regionalgrp, world] \n",
    "        for iso in isolistPlus:\n",
    "            #write iso name\n",
    "            file.write('\"'+iso+'\":{')  \n",
    "            #set min and max year for each ind and iso\n",
    "            year_range=dataset[\"year\"][(dataset[\"iso3\"]==iso)&(dataset[ind].notnull())]\n",
    "            if len(year_range)!=0:\n",
    "                Ymin=min(year_range)\n",
    "                Ymax=max(year_range)\n",
    "            else:\n",
    "                Ymin=Ymax=\"null\"\n",
    "            #write this information within the serie        \n",
    "            file.write('\"YearMin\":'+str(Ymin)+',')\n",
    "            file.write('\"YearMax\":'+str(Ymax)+',')\n",
    "            for year in range(start,end+1):\n",
    "                x= dataset[ind][(dataset[\"year\"] == year) & (dataset[\"iso3\"]==iso)]\n",
    "                if x.empty :  \n",
    "                    x=\"null\"\n",
    "                else :\n",
    "                    x=str('{:0.3f}'.format(float(x)))\n",
    "                if ind!=last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}},')\n",
    "                if ind==last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}}'+metafinal+'}')\n",
    "\n",
    "    file.close()\n",
    "\n",
    "    file_in = open(folder_final+\"temp.json\", \"r\")\n",
    "    file_out = open(folder_final+\"WB_WDI_\"+ISO+\".json\", \"wt\")\n",
    "    #for line in file_in: file_out.write( line.replace('nan', \"null\") ) => replaced by...\n",
    "    #add ':' before to avoid changing all 'nan' in the text, and label \n",
    "    for line in file_in: file_out.write( line.replace(':nan', \":null\") )\n",
    "    file_in.close()\n",
    "    file_out.close()\n",
    "\n",
    "    #os.remove(folder+'dataset'+ISO+'.csv')\n",
    "    print(\"done:WB_WDI_\"+ISO+\".json\")\n",
    "\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "#os.remove(folder+\"temp.json\")\n",
    "winsound.PlaySound(\"SystemHand\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a json from CSV serie2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame, concat\n",
    "from datetime import datetime\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "\n",
    "#Specifies the extremum of the time series\n",
    "start=1980\n",
    "end=2015\n",
    "Serie_range=end-start+1\n",
    "\n",
    "#=================================================================================================\n",
    "#Folder to use\n",
    "folder=\"data/WDI/csv/\"\n",
    "folder_final=\"data/WDI/\"\n",
    "#=================================================================================================\n",
    "#list of json already created\n",
    "#create an empty list called done\n",
    "done=list()\n",
    "dirs = os.listdir(folder)\n",
    "# This will store all the file names in the folder and store iso3 code in the 'done' list\n",
    "for file in dirs:\n",
    "    #take a look at file name excluse .txt and temp_wb.jon\n",
    "    if not \"temp.json\" in file and not '.txt' in file and not '.csv' in file: \n",
    "        #keep the other file letter 7 to 10 (iso3)\n",
    "        #print(file[7:10])\n",
    "        done.append(file[7:10])\n",
    "#=================================================================================================\n",
    "indicator_listCD=['SL.UEM.TOTL.FE.ZS','SL.UEM.TOTL.MA.ZS','SL.EMP.INSV.FE.ZS','SL.TLF.CACT.FE.ZS','FM.AST.DOMS.CN','FS.AST.PRVT.GD.ZS','SI.POV.DDAY',\\\n",
    "                'SI.POV.NAHC','SI.POV.URHC','SI.POV.RUHC','PA.NUS.FCRF','GC.TAX.TOTL.GD.ZS','EG.IMP.CONS.ZS','SP.DYN.LE00.FE.IN','SP.DYN.LE00.MA.IN',\\\n",
    "                'SH.DYN.MORT','SH.DYN.AIDS.ZS','SE.ADT.1524.LT.FE.ZS','SE.ADT.1524.LT.MA.ZS','SE.PRM.ENRL.TC.ZS','SE.PRM.CMPT.MA.ZS','SE.PRM.CMPT.FE.ZS',\\\n",
    "                'SE.PRM.ENRR','SE.SEC.ENRR','SE.TER.ENRR','SM.POP.REFG','SG.GEN.PARL.ZS','IC.TAX.DURS','IC.REG.PROC','SE.PRM.UNER.MA','SE.PRM.CMPT.ZS',\\\n",
    "                'EN.ATM.PM25.MC.M3','EN.ATM.PM25.MC.ZS','EN.POP.EL5M.ZS','EN.HPT.THRD.NO','ER.MRN.PTMR.ZS','EN.FSH.THRD.NO','EN.MAM.THRD.NO',\\\n",
    "                'EN.ATM.GHGO.KT.CE','EN.ATM.METH.KT.CE','EN.ATM.CO2E.PC','EG.USE.COMM.FO.ZS','EG.FEC.RNEW.ZS','EG.ELC.RNEW.ZS','EG.USE.ELEC.KH.PC',\\\n",
    "                'EG.USE.CRNW.ZS','EG.USE.COMM.CL.ZS', 'IT.CEL.SETS.P2', 'IT.NET.USER.P2', 'EG.ELC.ACCS.ZS', 'SH.H2O.SAFE.ZS']\n",
    "\n",
    "isolist=['ABW', 'AFG', 'AGO', 'ALB', 'AND', 'ARE', 'ARG', 'ARM', 'ASM', 'ATG', 'AUS', 'AUT', 'AZE', 'BDI', 'BEL', 'BEN', 'BFA', 'BGD', \\\n",
    "         'BGR', 'BHR', 'BHS', 'BIH', 'BLR', 'BLZ', 'BMU', 'BOL', 'BRA', 'BRB', 'BRN', 'BTN', 'BWA', 'CAF', 'CAN', 'CHE', 'CHL', 'CHN', \\\n",
    "         'CIV', 'CMR', 'COD', 'COG', 'COL', 'COM', 'CPV', 'CRI', 'CUB', 'CUW', 'CYM', 'CYP', 'CZE', 'DEU', 'DJI', 'DMA', 'DNK', 'DOM', \\\n",
    "         'DZA', 'ECU', 'EGY', 'ERI', 'ESP', 'EST', 'ETH', 'FIN', 'FJI', 'FRA', 'FSM', 'GAB', 'GBR', 'GEO', 'GHA', 'GIN', 'GMB', \\\n",
    "         'GNB', 'GNQ', 'GRC', 'GRD', 'GRL', 'GTM', 'GUM', 'GUY', 'HKG', 'HND', 'HRV', 'HTI', 'HUN', 'IDN', 'IMN', 'IND', 'IRL', 'IRN', \\\n",
    "         'IRQ', 'ISL', 'ISR', 'ITA', 'JAM', 'JOR', 'JPN', 'KAZ', 'KEN', 'KGZ', 'KHM', 'KNA', 'KOR', 'KWT', 'LAO', 'LBN', 'LBR', 'LBY', \\\n",
    "         'LCA', 'LIE', 'LKA', 'LSO', 'LTU', 'LUX', 'LVA', 'MAC', 'MAF', 'MAR', 'MCO', 'MDA', 'MDG', 'MDV', 'MEX', 'MHL', 'MKD', 'MLI', \\\n",
    "         'MLT', 'MMR', 'MNE', 'MNG', 'MNP', 'MOZ', 'MRT', 'MUS', 'MWI', 'MYS', 'NAM', 'NCL', 'NER', 'NGA', 'NIC', 'NLD', 'NOR', 'NPL', \\\n",
    "         'NZL', 'OMN', 'PAK', 'PAN', 'PER', 'PHL', 'PLW', 'PNG', 'POL', 'PRI', 'PRT', 'PRY', 'PSE', 'PYF', 'QAT', 'ROU', 'RUS', 'RWA', \\\n",
    "         'SAU', 'SDN', 'SEN', 'SGP', 'SLB', 'SLE', 'SLV', 'SMR', 'SOM', 'SRB', 'SSD', 'STP', 'SUR', 'SVK', 'SVN', 'SWE', 'SWZ', 'SXM',\\\n",
    "         'SYC', 'SYR', 'TCA', 'TCD', 'TGO', 'THA', 'TJK', 'TKM', 'TLS', 'TON', 'TTO', 'TUN', 'TUR', 'TUV', 'TWN', 'TZA', 'UGA', 'UKR', \\\n",
    "         'URY', 'USA', 'UZB', 'VCT', 'VEN', 'VIR', 'VNM', 'VUT', 'WSM', 'YEM', 'ZAF', 'ZMB', 'ZWE']\n",
    "#non standard recognized by the API -> regioniso=['EAS','ECS','LCN','LIC','LMC','MEA','MIC','NAC','NOC','OEC','SAS','SSF','UMC','WLD']\n",
    "#non standard ISO= ['ARB', 'CEB', 'CSS', 'EAP', 'EAS', 'ECA', 'ECS', 'EMU', 'EUU', 'FCS', 'HIC', 'HPC', 'LAC', 'LCN', 'LDC', 'LIC', 'LMC', 'LMY', 'MEA', 'MIC', 'MNA', 'NAC', 'NOC', 'OEC', 'OED', 'OSS', 'PSS', 'SAS', 'SSA', 'SSF', 'SST', 'UMC', 'WLD']\n",
    "\n",
    "#create a list for countries not to process\n",
    "reject=['PRK', 'FRO']\n",
    "#This dictionnary associate a country name to an iso3 code \n",
    "#Region dictionnary that affect to a country iso3 its income group, region group, and worl:\n",
    "region_dict= {\"ASM\":\"UMC,EAS,WLD\",\"AUS\":\"OEC,EAS,WLD\",\"BRN\":\"NOC,EAS,WLD\",\"CHN\":\"UMC,EAS,WLD\",\"FJI\":\"UMC,EAS,WLD\",\"FSM\":\"LMC,EAS,WLD\",\"GUM\":\"NOC,EAS,WLD\",\"HKG\":\"NOC,EAS,WLD\",\"IDN\":\"LMC,EAS,WLD\",\"JPN\":\"OEC,EAS,WLD\",\"KHM\":\"LIC,EAS,WLD\",\"KIR\":\"LMC,EAS,WLD\",\"KOR\":\"OEC,EAS,WLD\",\"LAO\":\"LMC,EAS,WLD\",\"MAC\":\"NOC,EAS,WLD\",\"MHL\":\"UMC,EAS,WLD\",\"MMR\":\"LMC,EAS,WLD\",\"MNG\":\"UMC,EAS,WLD\",\"MNP\":\"NOC,EAS,WLD\",\"MYS\":\"UMC,EAS,WLD\",\"NCL\":\"NOC,EAS,WLD\",\"NZL\":\"OEC,EAS,WLD\",\"PHL\":\"LMC,EAS,WLD\",\"PLW\":\"UMC,EAS,WLD\",\"PNG\":\"LMC,EAS,WLD\",\"PRK\":\"LIC,EAS,WLD\",\"PYF\":\"NOC,EAS,WLD\",\"SGP\":\"NOC,EAS,WLD\",\"SLB\":\"LMC,EAS,WLD\",\"THA\":\"UMC,EAS,WLD\",\"TLS\":\"LMC,EAS,WLD\",\"TON\":\"UMC,EAS,WLD\",\"TUV\":\"UMC,EAS,WLD\",\"TWN\":\"NOC,EAS,WLD\",\"VNM\":\"LMC,EAS,WLD\",\"VUT\":\"LMC,EAS,WLD\",\"WSM\":\"LMC,EAS,WLD\",\"\":\"LMC,ECS,WLD\",\"ALB\":\"UMC,ECS,WLD\",\"AND\":\"NOC,ECS,WLD\",\"ARM\":\"LMC,ECS,WLD\",\"AUT\":\"OEC,ECS,WLD\",\"AZE\":\"UMC,ECS,WLD\",\"BEL\":\"OEC,ECS,WLD\",\"BGR\":\"UMC,ECS,WLD\",\"BIH\":\"UMC,ECS,WLD\",\"BLR\":\"UMC,ECS,WLD\",\"CHE\":\"OEC,ECS,WLD\",\"CHI\":\"NOC,ECS,WLD\",\"CYP\":\"NOC,ECS,WLD\",\"CZE\":\"OEC,ECS,WLD\",\"DEU\":\"OEC,ECS,WLD\",\"DNK\":\"OEC,ECS,WLD\",\"ESP\":\"OEC,ECS,WLD\",\"EST\":\"OEC,ECS,WLD\",\"FIN\":\"OEC,ECS,WLD\",\"FRA\":\"OEC,ECS,WLD\",\"FRO\":\"NOC,ECS,WLD\",\"GBR\":\"OEC,ECS,WLD\",\"GEO\":\"LMC,ECS,WLD\",\"GRC\":\"OEC,ECS,WLD\",\"GRL\":\"NOC,ECS,WLD\",\"HRV\":\"NOC,ECS,WLD\",\"HUN\":\"OEC,ECS,WLD\",\"IMN\":\"NOC,ECS,WLD\",\"IRL\":\"OEC,ECS,WLD\",\"ISL\":\"OEC,ECS,WLD\",\"ITA\":\"OEC,ECS,WLD\",\"KAZ\":\"UMC,ECS,WLD\",\"KGZ\":\"LMC,ECS,WLD\",\"LIE\":\"NOC,ECS,WLD\",\"LTU\":\"NOC,ECS,WLD\",\"LUX\":\"OEC,ECS,WLD\",\"LVA\":\"NOC,ECS,WLD\",\"MCO\":\"NOC,ECS,WLD\",\"MDA\":\"LMC,ECS,WLD\",\"MKD\":\"UMC,ECS,WLD\",\"MNE\":\"UMC,ECS,WLD\",\"NLD\":\"OEC,ECS,WLD\",\"NOR\":\"OEC,ECS,WLD\",\"POL\":\"OEC,ECS,WLD\",\"PRT\":\"OEC,ECS,WLD\",\"ROU\":\"UMC,ECS,WLD\",\"RUS\":\"NOC,ECS,WLD\",\"SMR\":\"NOC,ECS,WLD\",\"SRB\":\"UMC,ECS,WLD\",\"SVK\":\"OEC,ECS,WLD\",\"SVN\":\"OEC,ECS,WLD\",\"SWE\":\"OEC,ECS,WLD\",\"TJK\":\"LMC,ECS,WLD\",\"TKM\":\"UMC,ECS,WLD\",\"TUR\":\"UMC,ECS,WLD\",\"UKR\":\"LMC,ECS,WLD\",\"UZB\":\"LMC,ECS,WLD\",\"ABW\":\"NOC,LCN,WLD\",\"ARG\":\"NOC,LCN,WLD\",\"ATG\":\"NOC,LCN,WLD\",\"BHS\":\"NOC,LCN,WLD\",\"BLZ\":\"UMC,LCN,WLD\",\"BOL\":\"LMC,LCN,WLD\",\"BRA\":\"UMC,LCN,WLD\",\"BRB\":\"NOC,LCN,WLD\",\"CHL\":\"OEC,LCN,WLD\",\"COL\":\"UMC,LCN,WLD\",\"CRI\":\"UMC,LCN,WLD\",\"CUB\":\"UMC,LCN,WLD\",\"CUW\":\"NOC,LCN,WLD\",\"CYM\":\"NOC,LCN,WLD\",\"DMA\":\"UMC,LCN,WLD\",\"DOM\":\"UMC,LCN,WLD\",\"ECU\":\"UMC,LCN,WLD\",\"GRD\":\"UMC,LCN,WLD\",\"GTM\":\"LMC,LCN,WLD\",\"GUY\":\"LMC,LCN,WLD\",\"HND\":\"LMC,LCN,WLD\",\"HTI\":\"LIC,LCN,WLD\",\"JAM\":\"UMC,LCN,WLD\",\"KNA\":\"NOC,LCN,WLD\",\"LCA\":\"UMC,LCN,WLD\",\"MAF\":\"NOC,LCN,WLD\",\"MEX\":\"UMC,LCN,WLD\",\"NIC\":\"LMC,LCN,WLD\",\"PAN\":\"UMC,LCN,WLD\",\"PER\":\"UMC,LCN,WLD\",\"PRI\":\"NOC,LCN,WLD\",\"PRY\":\"UMC,LCN,WLD\",\"SLV\":\"LMC,LCN,WLD\",\"SUR\":\"UMC,LCN,WLD\",\"SXM\":\"NOC,LCN,WLD\",\"TCA\":\"NOC,LCN,WLD\",\"TTO\":\"NOC,LCN,WLD\",\"URY\":\"NOC,LCN,WLD\",\"VCT\":\"UMC,LCN,WLD\",\"VEN\":\"NOC,LCN,WLD\",\"VIR\":\"NOC,LCN,WLD\",\"ARE\":\"NOC,MEA,WLD\",\"BHR\":\"NOC,MEA,WLD\",\"DJI\":\"LMC,MEA,WLD\",\"DZA\":\"UMC,MEA,WLD\",\"EGY\":\"LMC,MEA,WLD\",\"IRN\":\"UMC,MEA,WLD\",\"IRQ\":\"UMC,MEA,WLD\",\"ISR\":\"OEC,MEA,WLD\",\"JOR\":\"UMC,MEA,WLD\",\"KWT\":\"NOC,MEA,WLD\",\"LBN\":\"UMC,MEA,WLD\",\"LBY\":\"UMC,MEA,WLD\",\"MAR\":\"LMC,MEA,WLD\",\"MLT\":\"NOC,MEA,WLD\",\"OMN\":\"NOC,MEA,WLD\",\"PSE\":\"LMC,MEA,WLD\",\"QAT\":\"NOC,MEA,WLD\",\"SAU\":\"NOC,MEA,WLD\",\"SYR\":\"LMC,MEA,WLD\",\"TUN\":\"UMC,MEA,WLD\",\"YEM\":\"LMC,MEA,WLD\",\"BMU\":\"NOC,NAC,WLD\",\"CAN\":\"OEC,NAC,WLD\",\"USA\":\"OEC,NAC,WLD\",\"AFG\":\"LIC,SAS,WLD\",\"BGD\":\"LMC,SAS,WLD\",\"BTN\":\"LMC,SAS,WLD\",\"IND\":\"LMC,SAS,WLD\",\"LKA\":\"LMC,SAS,WLD\",\"MDV\":\"UMC,SAS,WLD\",\"NPL\":\"LIC,SAS,WLD\",\"PAK\":\"LMC,SAS,WLD\",\"AGO\":\"UMC,SSF,WLD\",\"BDI\":\"LIC,SSF,WLD\",\"BEN\":\"LIC,SSF,WLD\",\"BFA\":\"LIC,SSF,WLD\",\"BWA\":\"UMC,SSF,WLD\",\"CAF\":\"LIC,SSF,WLD\",\"CIV\":\"LMC,SSF,WLD\",\"CMR\":\"LMC,SSF,WLD\",\"COD\":\"LIC,SSF,WLD\",\"COG\":\"LMC,SSF,WLD\",\"COM\":\"LIC,SSF,WLD\",\"CPV\":\"LMC,SSF,WLD\",\"ERI\":\"LIC,SSF,WLD\",\"ETH\":\"LIC,SSF,WLD\",\"GAB\":\"UMC,SSF,WLD\",\"GHA\":\"LMC,SSF,WLD\",\"GIN\":\"LIC,SSF,WLD\",\"GMB\":\"LIC,SSF,WLD\",\"GNB\":\"LIC,SSF,WLD\",\"GNQ\":\"NOC,SSF,WLD\",\"KEN\":\"LMC,SSF,WLD\",\"LBR\":\"LIC,SSF,WLD\",\"LSO\":\"LMC,SSF,WLD\",\"MDG\":\"LIC,SSF,WLD\",\"MLI\":\"LIC,SSF,WLD\",\"MOZ\":\"LIC,SSF,WLD\",\"MRT\":\"LMC,SSF,WLD\",\"MUS\":\"UMC,SSF,WLD\",\"MWI\":\"LIC,SSF,WLD\",\"NAM\":\"UMC,SSF,WLD\",\"NER\":\"LIC,SSF,WLD\",\"NGA\":\"LMC,SSF,WLD\",\"RWA\":\"LIC,SSF,WLD\",\"SDN\":\"LMC,SSF,WLD\",\"SEN\":\"LMC,SSF,WLD\",\"SLE\":\"LIC,SSF,WLD\",\"SOM\":\"LIC,SSF,WLD\",\"SSD\":\"LIC,SSF,WLD\",\"STP\":\"LMC,SSF,WLD\",\"SWZ\":\"LMC,SSF,WLD\",\"SYC\":\"NOC,SSF,WLD\",\"TCD\":\"LIC,SSF,WLD\",\"TGO\":\"LIC,SSF,WLD\",\"TZA\":\"LIC,SSF,WLD\",\"UGA\":\"LIC,SSF,WLD\",\"ZAF\":\"UMC,SSF,WLD\",\"ZMB\":\"LMC,SSF,WLD\",\"ZWE\":\"LIC,SSF,WLD\"}\n",
    "\n",
    "indicator_list=indicator_listCD\n",
    "iso3list=list(set(isolist) -  set(reject) - set(done))\n",
    "#=================================================================================================\n",
    "\n",
    "for ISO in iso3list:\n",
    "    isolistPlus=[] \n",
    "    #browse the region dictionnary\n",
    "    for iso3,region_iso in region_dict.items(): \n",
    "            if ISO==iso3:\n",
    "                isolistPlus.append(ISO)\n",
    "                for item in region_iso.split(','):\n",
    "                    isolistPlus.append(str(item))\n",
    "                    \n",
    "    print(isolistPlus)\n",
    "    #================== store the meta information ====================================================\n",
    "    source='\"data_source\":{\"source\":\"World Bank, WDI\", \"provider\":\"World Bank, API\", \"Download_date\":\"'+datetime.now().strftime('%Y-%m-%d')+'\" }'\n",
    "\n",
    "    with open (folder+\"json meta country.json.txt\", \"r\") as meta:\n",
    "        for line in meta :\n",
    "            if line.startswith('\"'+ISO+'\"') :\n",
    "                metafinal=\",\"+line+\",\"+source\n",
    "    #================== store the meta information ====================================================\n",
    "\n",
    "    dataset = pd.read_csv(folder+'dataset'+ISO+'.csv')\n",
    "\n",
    "    #store name of the last ind in the list\n",
    "    last=indicator_list[len(indicator_list)-1]\n",
    "\n",
    "    file = open(folder_final+\"temp.json\", \"w\")\n",
    "    #Start writing the json file:\n",
    "    file.write('{')\n",
    "\n",
    "    for ind in indicator_list:\n",
    "\n",
    "        #Get indicator meta from external file\n",
    "        filemeta=open(folder_final+\"meta indicator.txt\", \"r\")\n",
    "        for line in filemeta :\n",
    "            if line.startswith('\"'+ind+'\"') : meta=line\n",
    "        filemeta.close()\n",
    "        #write the meta\n",
    "        file.write(meta)\n",
    "\n",
    "        #loop over the isolistPlus which contain [countryiso,incomegrp, regionalgrp, world] \n",
    "        for iso in isolistPlus:\n",
    "            #write iso name\n",
    "            file.write('\"'+iso+'\":{')  \n",
    "            #set min and max year for each ind and iso\n",
    "            year_range=dataset[\"year\"][(dataset[\"iso3\"]==iso)&(dataset[ind].notnull())]\n",
    "            if len(year_range)!=0:\n",
    "                Ymin=min(year_range)\n",
    "                Ymax=max(year_range)\n",
    "            else:\n",
    "                Ymin=Ymax=\"null\"\n",
    "            #write this information within the serie        \n",
    "            file.write('\"YearMin\":'+str(Ymin)+',')\n",
    "            file.write('\"YearMax\":'+str(Ymax)+',')\n",
    "            for year in range(start,end+1):\n",
    "                x= dataset[ind][(dataset[\"year\"] == year) & (dataset[\"iso3\"]==iso)]\n",
    "                if x.empty :  \n",
    "                    x=\"null\"\n",
    "                else :\n",
    "                    x=str('{:0.3f}'.format(float(x)))\n",
    "                if ind!=last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}},')\n",
    "                if ind==last:\n",
    "                    if year!=int(end) : file.write('\"'+str(year)+'\":'+str(x)+',')\n",
    "                    if (iso!=\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'},')\n",
    "                    if (iso==\"WLD\") & (year==int(end)): file.write('\"'+str(year)+'\":'+str(x)+'}}'+metafinal+'}')\n",
    "\n",
    "    file.close()\n",
    "\n",
    "    file_in = open(folder_final+\"temp.json\", \"r\")\n",
    "    file_out = open(folder_final+\"WB_WDI2_\"+ISO+\".json\", \"wt\")\n",
    "    #for line in file_in: file_out.write( line.replace('nan', \"null\") ) => replaced by...\n",
    "    #add ':' before to avoid changing all 'nan' in the text, and label \n",
    "    for line in file_in: file_out.write( line.replace(':nan', \":null\") )\n",
    "    file_in.close()\n",
    "    file_out.close()\n",
    "\n",
    "    #os.remove(folder+'dataset'+ISO+'.csv')\n",
    "    print(\"done:WB_WDI2_\"+ISO+\".json\")\n",
    "\n",
    "print(\"\\n*******************# Job Well done ! #********************\\n\")\n",
    "#os.remove(folder+\"temp.json\")\n",
    "winsound.PlaySound(\"SystemHand\", winsound.SND_ALIAS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}