Homework 4_POS_concise_Olanipekun.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f893ad14",
   "metadata": {},
   "source": [
    "# Part of Speech tagging\n",
    "- Taggers:\n",
    "    - NLTK\n",
    "    - Spacy\n",
    "    - Build Your Own Tagger\n",
    "    - n-gram tagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "400b6aaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import numpy as np\n",
    "from string import punctuation\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.stem import SnowballStemmer\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.corpus import wordnet\n",
    "import pandas as pd\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "from pprint import pprint\n",
    "\n",
    "pd.options.display.max_colwidth = 200\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "735b156b",
   "metadata": {},
   "source": [
    "## Question 1:\n",
    "### Run one of the part-of-speech (POS) taggers available in Python. \n",
    "\n",
    "1.\tFind the longest sentence you can, longer than 10 words, that the POS tagger tags correctly. Show the input and output.\n",
    "2.\tFind the shortest sentence you can, shorter than 10 words, that the POS tagger fails to tag 100 percent correctly. \n",
    "    - Show the input and output. Explain your conjecture as to why the tagger might have been less than perfect with this sentence.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86fba48f",
   "metadata": {},
   "source": [
    "As a geologist I decided to find the required long and  short sentences in the description of sedimentary rocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "05b4a082",
   "metadata": {},
   "outputs": [],
   "source": [
    "import wikipedia as wk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a2c39571",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load the description of sedimentary rocks from Wikipedia\n",
    "sed_rk = wk.page(\"Sedimentary rock\")\n",
    "sed_rk_txt = sed_rk.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bf768a57",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Sedimentary rocks are types of rock that are formed by the accumulation or deposition of mineral or organic particles at the Earth\\'s surface, followed by cementation. Sedimentation is the collective name for processes that cause these particles to settle in place. The particles that form a sedimentary rock are called sediment, and may be composed of geological detritus (minerals) or biological detritus (organic matter). The geological detritus originated from weathering and erosion of existing rocks, or from the solidification of molten lava blobs erupted by volcanoes. The geological detritus is transported to the place of deposition by water, wind, ice or mass movement, which are called agents of denudation. Biological detritus was formed by bodies and parts (mainly shells) of dead aquatic organisms, as well as their fecal mass, suspended in water and slowly piling up on the floor of water bodies (marine snow). Sedimentation may also occur as dissolved minerals precipitate from water solution.\\nThe sedimentary rock cover of the continents of the Earth\\'s crust is extensive (73% of the Earth\\'s current land surface), but sedimentary rock is estimated to be only 8% of the volume of the crust. Sedimentary rocks are only a thin veneer over a crust consisting mainly of igneous and metamorphic rocks. Sedimentary rocks are deposited in layers as strata, forming a structure called bedding. Sedimentary rocks are often deposited in large structures called sedimentary basins. Sedimentary rocks have also been found on Mars.\\nThe study of sedimentary rocks and rock strata provides information about the subsurface that is useful for civil engineering, for example in the construction of roads, houses, tunnels, canals or other structures. Sedimentary rocks are also important sources of natural resources including coal, fossil fuels, drinking water and ores.\\nThe study of the sequence of sedimentary rock strata is the main source for an understanding of the Earth\\'s history, including palaeogeography, paleoclimatology and the history of life. The scientific discipline that studies the properties and origin of sedimentary rocks is called sedimentology. Sedimentology is part of both geology and physical geography and overlaps partly with other disciplines in the Earth sciences, such as pedology, geomorphology, geochemistry and structural geology.\\n\\n\\n== Classification based on origin ==\\n\\nSedimentary rocks can be subdivided into four groups based on the processes responsible for their formation: clastic sedimentary rocks, biochemical (biogenic) sedimentary rocks, chemical sedimentary rocks, and a fourth category for \"other\" sedimentary rocks formed by impacts, volcanism, and other minor processes.\\n\\n\\n=== Clastic sedimentary rocks ===\\n\\nClastic sedimentary rocks are composed of rock fragments (clasts) that have been cemented together. The clasts are commonly individual grains of quartz, feldspar, clay minerals, or mica. However, any type of mineral may be present. Clasts may also be lithic fragments composed of more than one mineral.\\nClastic sedimentary rocks are subdivided according to the dominant particle size. Most geologists use the Udden-Wentworth grain size scale and divide unconsolidated sediment into three fractions: gravel (>2 mm diameter), sand (1/16 to 2 mm diameter), and mud (<1/256 mm diameter). Mud is further divided into silt (1/16 to 1/256 mm diameter) and clay (<1/256 mm diameter). The classification of clastic sedimentary rocks parallels this scheme; conglomerates and breccias are made mostly of gravel, sandstones are made mostly of sand, and mudrocks are made mostly of mud. This tripartite subdivision is mirrored by the broad categories of rudites, arenites, and lutites, respectively, in older literature.\\nThe subdivision of these three broad categories is based on differences in clast shape (conglomerates and breccias), composition (sandstones), or grain size or texture (mudrocks).\\n\\n\\n==== Conglomerates and breccias ====\\n\\nConglomerates are dominantly composed of rounded gravel, while breccias are composed of dominantly angular gravel.\\n\\n\\n==== Sandstones ====\\n\\nSandstone classification schemes vary widely, but most geologists have adopted the Dott scheme, which uses the relative abundance of quartz, feldspar, and lithic framework grains and the abundance of a muddy matrix between the larger grains.\\n\\nComposition of framework grains\\nThe relative abundance of sand-sized framework grains determines the first word in a sandstone name. Naming depends on the dominance of the three most abundant components quartz, feldspar, or the lithic fragments that originated from other rocks. All other minerals are considered accessories and not used in the naming of the rock, regardless of abundance.Quartz sandstones have >90% quartz grains\\nFeldspathic sandstones have <90% quartz grains and more feldspar grains than lithic grains\\nLithic sandstones have <90% quartz grains and more lithic grains than feldspar grainsAbundance of muddy matrix material between sand grains\\nWhen sand-sized particles are deposited, the space between the grains either remains open or is filled with mud (silt and/or clay sized particle).\\n\"Clean\" sandstones with open pore space (that may later be filled with matrix material) are called arenites.\\nMuddy sandstones with abundant (>10%) muddy matrix are called wackes.Six sandstone names are possible using the descriptors for grain composition (quartz-, feldspathic-, and lithic-) and the amount of matrix (wacke or arenite). For example, a quartz arenite would be composed of mostly (>90%) quartz grains and have little or no clayey matrix between the grains, a lithic wacke would have abundant lithic grains and abundant muddy matrix, etc.\\nAlthough the Dott classification scheme is widely used by sedimentologists, common names like greywacke, arkose, and quartz sandstone are still widely used by non-specialists and in popular literature.\\n\\n\\n==== Mudrocks ====\\n\\nMudrocks are sedimentary rocks composed of at least 50% silt- and clay-sized particles. These relatively fine-grained particles are commonly transported by turbulent flow in water or air, and deposited as the flow calms and the particles settle out of suspension.\\nMost authors presently use the term \"mudrock\" to refer to all rocks composed dominantly of mud. Mudrocks can be divided into siltstones, composed dominantly of silt-sized particles; mudstones with subequal mixture of silt- and clay-sized particles; and claystones, composed mostly of clay-sized particles. Most authors use \"shale\" as a term for a fissile mudrock (regardless of grain size) although some older literature uses the term \"shale\" as a synonym for mudrock.\\n\\n\\n=== Biochemical sedimentary rocks ===\\n\\nBiochemical sedimentary rocks are created when organisms use materials dissolved in air or water to build their tissue. Examples include:\\n\\nMost types of limestone are formed from the calcareous skeletons of organisms such as corals, mollusks, and foraminifera.\\nCoal, formed from plants that have removed carbon from the atmosphere and combined it with other elements to build their tissue.\\nDeposits of chert formed from the accumulation of siliceous skeletons of microscopic organisms such as radiolaria and diatoms.\\n\\n\\n=== Chemical sedimentary rocks ===\\nChemical sedimentary rock forms when mineral constituents in solution become supersaturated and inorganically precipitate. Common chemical sedimentary rocks include oolitic limestone and rocks composed of evaporite minerals, such as halite (rock salt), sylvite, baryte and gypsum.\\n\\n\\n=== Other sedimentary rocks ===\\nThis fourth miscellaneous category includes volcanic tuff and volcanic breccias formed by deposition and later cementation of lava fragments erupted by volcanoes, and impact breccias formed after impact events.\\n\\n\\t\\t\\n\\t\\t\\n\\n\\n== Classification based on composition ==\\n\\nAlternatively, sedimentary rocks can be subdivided into compositional groups based on their mineralogy:\\n\\nSiliciclastic sedimentary rocks, are dominantly composed of silicate minerals. The sediment that makes up these rocks was transported as bed load, suspended load, or by sediment gravity flows. Siliciclastic sedimentary rocks are subdivided into conglomerates and breccias, sandstone, and mudrocks.\\nCarbonate sedimentary rocks are composed of calcite (rhombohedral CaCO3), aragonite (orthorhombic CaCO3), dolomite (CaMg(CO3)2), and other carbonate minerals based on the CO2−3 ion. Common examples include limestone and the rock dolomite.\\nEvaporite sedimentary rocks are composed of minerals formed from the evaporation of water. The most common evaporite minerals are carbonates (calcite and others based on CO2−3), chlorides (halite and others built on Cl−), and sulfates (gypsum and others built on SO2−4). Evaporite rocks commonly include abundant halite (rock salt), gypsum, and anhydrite.\\nOrganic-rich sedimentary rocks have significant amounts of organic material, generally in excess of 3% total organic carbon. Common examples include coal, oil shale as well as source rocks for oil and natural gas.\\nSiliceous sedimentary rocks are almost entirely composed of silica (SiO2), typically as chert, opal, chalcedony or other microcrystalline forms.\\nIron-rich sedimentary rocks are composed of >15% iron; the most common forms are banded iron formations and ironstones.\\nPhosphatic sedimentary rocks are composed of phosphate minerals and contain more than 6.5% phosphorus; examples include deposits of phosphate nodules, bone beds, and phosphatic mudrocks.\\n\\n\\n== Deposition and transformation ==\\n\\n\\n=== Sediment transport and deposition ===\\n\\nSedimentary rocks are formed when sediment is deposited out of air, ice, wind, gravity, or water flows carrying the particles in suspension. This sediment is often formed when weathering and erosion break down a rock into loose material in a source area. The material is then transported from the source area to the deposition area. The type of sediment transported depends on the geology of the hinterland (the source area of the sediment). However, some sedimentary rocks, such as evaporites, are composed of material that form at the place of deposition. The nature of a sedimentary rock, therefore, not only depends on the sediment supply, but also on the sedimentary depositional environment in which it formed.\\n\\n\\n=== Transformation (Diagenesis) ===\\n\\nAs sediments accumulate in a depositional environment, older sediments are buried by younger sediments, and they undergo diagenesis. Diagenesis includes all the chemical, physical, and biological changes, exclusive of surface weathering, undergone by a sediment after its initial deposition. This includes compaction and lithification of the sediments. Early stages of diagenesis, described as eogenesis, take place at shallow depths (a few tens of meters) and is characterized by bioturbation and mineralogical changes in the sediments, with only slight compaction. The red hematite that gives red bed sandstones their color is likely formed during eogenesis. Some biochemical processes, like the activity of bacteria, can affect minerals in a rock and are therefore seen as part of diagenesis.Deeper burial is accompanied by mesogenesis, during which most of the compaction and lithification takes place. Compaction takes place as the sediments come under increasing overburden (lithostatic) pressure from overlying sediments. Sediment grains move into more compact arrangements, grains of ductile minerals (such as mica) are deformed, and pore space is reduced. Sediments are typically saturated with groundwater or seawater when originally deposited, and as pore space is reduced, much of these connate fluids are expelled. In addition to this physical compaction, chemical compaction may take place via pressure solution. Points of contact between grains are under the greatest strain, and the strained mineral is more soluble than the rest of the grain. As a result, the contact points are dissolved away, allowing the grains to come into closer contact. The increased pressure and temperature stimulate further chemical reactions, such as the reactions by which organic material becomes lignite or coal.Lithification follows closely on compaction, as increased temperatures at depth hasten the precipitation of cement that binds the grains together. Pressure solution contributes to this process of cementation, as the mineral dissolved from strained contact points is redeposited in the unstrained pore spaces. This further reduces porosity and makes the rock more compact and competent.Unroofing of buried sedimentary rock is accompanied by telogenesis, the third and final stage of diagenesis. As erosion reduces the depth of burial, renewed exposure to meteoric water produces additional changes to the sedimentary rock, such as leaching of some of the cement to produce secondary porosity.At sufficiently high temperature and pressure, the realm of diagenesis makes way for metamorphism, the process that forms metamorphic rock.\\n\\n\\n== Properties ==\\n\\n\\n=== Color ===\\nThe color of a sedimentary rock is often mostly determined by iron, an element with two major oxides: iron(II) oxide and iron(III) oxide. Iron(II) oxide (FeO) only forms under low oxygen (anoxic) circumstances and gives the rock a grey or greenish colour. Iron(III) oxide (Fe2O3) in a richer oxygen environment is often found in the form of the mineral hematite and gives the rock a reddish to brownish colour. In arid continental climates rocks are in direct contact with the atmosphere, and oxidation is an important process, giving the rock a red or orange colour. Thick sequences of red sedimentary rocks formed in arid climates are called red beds. However, a red colour does not necessarily mean the rock formed in a continental environment or arid climate.The presence of organic material can colour a rock black or grey. Organic material is formed from dead organisms, mostly plants. Normally, such material eventually decays by oxidation or bacterial activity. Under anoxic circumstances, however, organic material cannot decay and leaves a dark sediment, rich in organic material. This can, for example, occur at the bottom of deep seas and lakes. There is little water mixing in such environments; as a result, oxygen from surface water is not brought down, and the deposited sediment is normally a fine dark clay. Dark rocks, rich in organic material, are therefore often shales.\\n\\n\\n=== Texture ===\\n\\nThe size, form and orientation of clasts (the original pieces of rock) in a sediment is called its texture. The texture is a small-scale property of a rock, but determines many of its large-scale properties, such as the density, porosity or permeability.The 3D orientation of the clasts is called the fabric of the rock. The size and form of clasts can be used to determine the velocity and direction of current in the sedimentary environment that moved the clasts from their origin; fine, calcareous mud only settles in quiet water while gravel and larger clasts are moved only by rapidly moving water. The grain size of a rock is usually expressed with the Wentworth scale, though alternative scales are sometimes used. The grain size can be expressed as a diameter or a volume, and is always an average value, since a rock is composed of clasts with different sizes. The statistical distribution of grain sizes is different for different rock types and is described in a property called the sorting of the rock. When all clasts are more or less of the same size, the rock is called \\'well-sorted\\', and when there is a large spread in grain size, the rock is called \\'poorly sorted\\'.\\n\\nThe form of the clasts can reflect the origin of the rock. For example, coquina, a rock composed of clasts of broken shells, can only form in energetic water. The form of a clast can be described by using four parameters:\\nSurface texture describes the amount of small-scale relief of the surface of a grain that is too small to influence the general shape. For example, frosted grains, which are covered with small-scale fractures, are characteristic of eolian sandstones.\\nRounding describes the general smoothness of the shape of a grain.\\nSphericity describes the degree to which the grain approaches a sphere.\\nGrain form describes the three-dimensional shape of the grain.Chemical sedimentary rocks have a non-clastic texture, consisting entirely of crystals. To describe such a texture, only the average size of the crystals and the fabric are necessary.\\n\\n\\n=== Mineralogy ===\\n\\nMost sedimentary rocks contain either quartz (siliciclastic rocks) or calcite (carbonate rocks). In contrast to igneous and metamorphic rocks, a sedimentary rock usually contains very few different major minerals. However, the origin of the minerals in a sedimentary rock is often more complex than in an igneous rock. Minerals in a sedimentary rock may have been present in the original sediments or may formed by precipitation during diagenesis. In the second case, a mineral precipitate may have grown over an older generation of cement. A complex diagenetic history can be established by optical mineralogy, using a petrographic microscope.\\nCarbonate rocks predominantly consist of carbonate minerals such as calcite, aragonite or dolomite. Both the cement and the clasts (including fossils and ooids) of a carbonate sedimentary rock usually consist of carbonate minerals. The mineralogy of a clastic rock is determined by the material supplied by the source area, the manner of its transport to the place of deposition and the stability of that particular mineral.\\nThe resistance of rock-forming minerals to weathering is expressed by the Goldich dissolution series. In this series, quartz is the most stable, followed by feldspar, micas, and finally other less stable minerals that are only present when little weathering has occurred. The amount of weathering depends mainly on the distance to the source area, the local climate and the time it took for the sediment to be transported to the point where it is deposited. In most sedimentary rocks, mica, feldspar and less stable minerals have been weathered to clay minerals like kaolinite, illite or smectite.\\n\\n\\n=== Fossils ===\\n\\nAmong the three major types of rock, fossils are most commonly found in sedimentary rock. Unlike most igneous and metamorphic rocks, sedimentary rocks form at temperatures and pressures that do not destroy fossil remnants. Often these fossils may only be visible under magnification.\\nDead organisms in nature are usually quickly removed by scavengers, bacteria, rotting and erosion, but under exceptional circumstances, these natural processes are unable to take place, leading to fossilisation. The chance of fossilisation is higher when the sedimentation rate is high (so that a carcass is quickly buried), in anoxic environments (where little bacterial activity occurs) or when the organism had a particularly hard skeleton. Larger, well-preserved fossils are relatively rare.\\n\\nFossils can be both the direct remains or imprints of organisms and their skeletons. Most commonly preserved are the harder parts of organisms such as bones, shells, and the woody tissue of plants. Soft tissue has a much smaller chance of being fossilized, and the preservation of soft tissue of animals older than 40 million years is very rare. Imprints of organisms made while they were still alive are called trace fossils, examples of which are burrows, footprints, etc.\\nAs a part of a sedimentary rock, fossils undergo the same diagenetic processes as does the host rock. For example, a shell consisting of calcite can dissolve while a cement of silica then fills the cavity. In the same way, precipitating minerals can fill cavities formerly occupied by blood vessels, vascular tissue or other soft tissues. This preserves the form of the organism but changes the chemical composition, a process called permineralization. The most common minerals involved in permineralization are various forms of amorphous silica (chalcedony, flint, chert), carbonates (especially calcite), and pyrite.\\nAt high pressure and temperature, the organic material of a dead organism undergoes chemical reactions in which volatiles such as water and carbon dioxide are expulsed. The fossil, in the end, consists of a thin layer of pure carbon or its mineralized form, graphite. This form of fossilisation is called carbonisation. It is particularly important for plant fossils. The same process is responsible for the formation of fossil fuels like lignite or coal.\\n\\n\\n=== Primary sedimentary structures ===\\n\\nStructures in sedimentary rocks can be divided into primary structures (formed during deposition) and secondary structures (formed after deposition). Unlike textures, structures are always large-scale features that can easily be studied in the field. Sedimentary structures can indicate something about the sedimentary environment or can serve to tell which side originally faced up where tectonics have tilted or overturned sedimentary layers.\\nSedimentary rocks are laid down in layers called beds or strata. A bed is defined as a layer of rock that has a uniform lithology and texture. Beds form by the deposition of layers of sediment on top of each other. The sequence of beds that characterizes sedimentary rocks is called bedding. Single beds can be a couple of centimetres to several meters thick. Finer, less pronounced layers are called laminae, and the structure a lamina forms in a rock is called lamination. Laminae are usually less than a few centimetres thick. Though bedding and lamination are often originally horizontal in nature, this is not always the case. In some environments, beds are deposited at a (usually small) angle. Sometimes multiple sets of layers with different orientations exist in the same rock, a structure called cross-bedding. Cross-bedding is characteristic of deposition by a flowing medium (wind or water).\\nThe opposite of cross-bedding is parallel lamination, where all sedimentary layering is parallel. Differences in laminations are generally caused by cyclic changes in the sediment supply, caused, for example, by seasonal changes in rainfall, temperature or biochemical activity. Laminae that represent seasonal changes (similar to tree rings) are called varves. Any sedimentary rock composed of millimeter or finer scale layers can be named with the general term laminite. When sedimentary rocks have no lamination at all, their structural character is called massive bedding.\\nGraded bedding is a structure where beds with a smaller grain size occur on top of beds with larger grains. This structure forms when fast flowing water stops flowing. Larger, heavier clasts in suspension settle first, then smaller clasts. Although graded bedding can form in many different environments, it is a characteristic of turbidity currents.The surface of a particular bed, called the bedform, can also be indicative of a particular sedimentary environment. Examples of bed forms include dunes and ripple marks. Sole markings, such as tool marks and flute casts, are grooves eroded on a surface that are preserved by renewed sedimentation. These are often elongated structures and can be used to establish the direction of the flow during deposition.Ripple marks also form in flowing water. There can be symmetric or asymmetric. Asymmetric ripples form in environments where the current is in one direction, such as rivers. The longer flank of such ripples is on the upstream side of the current. Symmetric wave ripples occur in environments where currents reverse directions, such as tidal flats.\\nMudcracks are a bed form caused by the dehydration of sediment that occasionally comes above the water surface. Such structures are commonly found at tidal flats or point bars along rivers.\\n\\n\\n=== Secondary sedimentary structures ===\\n\\nSecondary sedimentary structures are those which formed after deposition. Such structures form by chemical, physical and biological processes within the sediment. They can be indicators of circumstances after deposition. Some can be used as way up criteria.\\nOrganic materials in a sediment can leave more traces than just fossils. Preserved tracks and burrows are examples of trace fossils (also called ichnofossils). Such traces are relatively rare. Most trace fossils are burrows of molluscs or arthropods. This burrowing is called bioturbation by sedimentologists. It can be a valuable indicator of the biological and ecological environment that existed after the sediment was deposited. On the other hand, the burrowing activity of organisms can destroy other (primary) structures in the sediment, making a reconstruction more difficult.\\n\\nSecondary structures can also form by diagenesis or the formation of a soil (pedogenesis) when a sediment is exposed above the water level. An example of a diagenetic structure common in carbonate rocks is a stylolite. Stylolites are irregular planes where material was dissolved into the pore fluids in the rock. This can result in the precipitation of a certain chemical species producing colouring and staining of the rock, or the formation of concretions. Concretions are roughly concentric bodies with a different composition from the host rock. Their formation can be the result of localized precipitation due to small differences in composition or porosity of the host rock, such as around fossils, inside burrows or around plant roots. In carbonate rocks such as limestone or chalk, chert or flint concretions are common, while terrestrial sandstones sometimes contain iron concretions. Calcite concretions in clay containing angular cavities or cracks are called septarian concretions.\\nAfter deposition, physical processes can deform the sediment, producing a third class of secondary structures. Density contrasts between different sedimentary layers, such as between sand and clay, can result in flame structures or load casts, formed by inverted diapirism. While the clastic bed is still fluid, diapirism can cause a denser upper layer to sink into a lower layer. Sometimes, density contrasts occur or are enhanced when one of the lithologies dehydrates. Clay can be easily compressed as a result of dehydration, while sand retains the same volume and becomes relatively less dense. On the other hand, when the pore fluid pressure in a sand layer surpasses a critical point, the sand can break through overlying clay layers and flow through, forming discordant bodies of sedimentary rock called sedimentary dykes. The same process can form mud volcanoes on the surface where they broke through upper layers.\\nSedimentary dykes can also be formed in a cold climate where the soil is permanently frozen during a large part of the year. Frost weathering can form cracks in the soil that fill with rubble from above. Such structures can be used as climate indicators as well as way up structures.Density contrasts can also cause small-scale faulting, even while sedimentation progresses (synchronous-sedimentary faulting). Such faulting can also occur when large masses of non-lithified sediment are deposited on a slope, such as at the front side of a delta or the continental slope. Instabilities in such sediments can result in the deposited material to slump, producing fissures and folding. The resulting structures in the rock are syn-sedimentary folds and faults, which can be difficult to distinguish from folds and faults formed by tectonic forces acting on lithified rocks.\\n\\n\\n== Depositional environments ==\\n\\nThe setting in which a sedimentary rock forms is called the depositional environment. Every environment has a characteristic combination of geologic processes, and circumstances. The type of sediment that is deposited is not only dependent on the sediment that is transported to a place (provenance), but also on the environment itself.A marine environment means that the rock was formed in a sea or ocean. Often, a distinction is made between deep and shallow marine environments. Deep marine usually refers to environments more than 200 m below the water surface (including the abyssal plain). Shallow marine environments exist adjacent to coastlines and can extend to the boundaries of the continental shelf. The water movements in such environments have a generally higher energy than that in deep environments, as wave activity diminishes with depth. This means that coarser sediment particles can be transported and the deposited sediment can be coarser than in deeper environments. When the sediment is transported from the continent, an alternation of sand, clay and silt is deposited. When the continent is far away, the amount of such sediment deposited may be small, and biochemical processes dominate the type of rock that forms. Especially in warm climates, shallow marine environments far offshore mainly see deposition of carbonate rocks. The shallow, warm water is an ideal habitat for many small organisms that build carbonate skeletons. When these organisms die, their skeletons sink to the bottom, forming a thick layer of calcareous mud that may lithify into limestone. Warm shallow marine environments also are ideal environments for coral reefs, where the sediment consists mainly of the calcareous skeletons of larger organisms.In deep marine environments, the water current working the sea bottom is small. Only fine particles can be transported to such places. Typically sediments depositing on the ocean floor are fine clay or small skeletons of micro-organisms. At 4 km depth, the solubility of carbonates increases dramatically (the depth zone where this happens is called the lysocline). Calcareous sediment that sinks below the lysocline dissolves; as a result, no limestone can be formed below this depth. Skeletons of micro-organisms formed of silica (such as radiolarians) are not as soluble and are still deposited. An example of a rock formed of silica skeletons is radiolarite. When the bottom of the sea has a small inclination, for example, at the continental slopes, the sedimentary cover can become unstable, causing turbidity currents. Turbidity currents are sudden disturbances of the normally quiet deep marine environment and can cause the near-instantaneous deposition of large amounts of sediment, such as sand and silt. The rock sequence formed by a turbidity current is called a turbidite.The coast is an environment dominated by wave action. At a beach, dominantly denser sediment such as sand or gravel, often mingled with shell fragments, is deposited, while the silt and clay sized material is kept in mechanical suspension. Tidal flats and shoals are places that sometimes dry because of the tide. They are often cross-cut by gullies, where the current is strong and the grain size of the deposited sediment is larger. Where rivers enter the body of water, either on a sea or lake coast, deltas can form. These are large accumulations of sediment transported from the continent to places in front of the mouth of the river. Deltas are dominantly composed of clastic (rather than chemical) sediment.\\nA continental sedimentary environment is an environment in the interior of a continent. Examples of continental environments are lagoons, lakes, swamps, floodplains and alluvial fans. In the quiet water of swamps, lakes and lagoons, fine sediment is deposited, mingled with organic material from dead plants and animals. In rivers, the energy of the water is much greater and can transport heavier clastic material. Besides transport by water, sediment can be transported by wind or glaciers. Sediment transported by wind is called aeolian and is almost always very well sorted, while sediment transported by a glacier is called glacial till and is characterized by very poor sorting.Aeolian deposits can be quite striking. The depositional environment of the Touchet Formation, located in the Northwestern United States, had intervening periods of aridity which resulted in a series of rhythmite layers. Erosional cracks were later infilled with layers of soil material, especially from aeolian processes. The infilled sections formed vertical inclusions in the horizontally deposited layers, and thus provided evidence of the sequence of events during deposition of the forty-one layers of the formation.\\n\\n\\n=== Sedimentary facies ===\\nThe kind of rock formed in a particular depositional environment is called its sedimentary facies. Sedimentary environments usually exist alongside each other in certain natural successions. A beach, where sand and gravel is deposited, is usually bounded by a deeper marine environment a little offshore, where finer sediments are deposited at the same time. Behind the beach, there can be dunes (where the dominant deposition is well sorted sand) or a lagoon (where fine clay and organic material is deposited). Every sedimentary environment has its own characteristic deposits. When sedimentary strata accumulate through time, the environment can shift, forming a change in facies in the subsurface at one location. On the other hand, when a rock layer with a certain age is followed laterally, the lithology (the type of rock) and facies eventually change.\\n\\nFacies can be distinguished in a number of ways: the most common are by the lithology (for example: limestone, siltstone or sandstone) or by fossil content. Coral, for example, only lives in warm and shallow marine environments and fossils of coral are thus typical for shallow marine facies. Facies determined by lithology are called lithofacies; facies determined by fossils are biofacies.Sedimentary environments can shift their geographical positions through time. Coastlines can shift in the direction of the sea when the sea level drops (regression), when the surface rises (transgression) due to tectonic forces in the Earth\\'s crust or when a river forms a large delta. In the subsurface, such geographic shifts of sedimentary environments of the past are recorded in shifts in sedimentary facies. This means that sedimentary facies can change either parallel or perpendicular to an imaginary layer of rock with a fixed age, a phenomenon described by Walther\\'s Law.The situation in which coastlines move in the direction of the continent is called transgression. In the case of transgression, deeper marine facies are deposited over shallower facies, a succession called onlap. Regression is the situation in which a coastline moves in the direction of the sea. With regression, shallower facies are deposited on top of deeper facies, a situation called offlap.The facies of all rocks of a certain age can be plotted on a map to give an overview of the palaeogeography. A sequence of maps for different ages can give an insight in the development of the regional geography.\\n\\n\\n==== Gallery of sedimentary facies ====\\n\\n\\n== Sedimentary basins ==\\n\\nPlaces where large-scale sedimentation takes place are called sedimentary basins. The amount of sediment that can be deposited in a basin depends on the depth of the basin, the so-called accommodation space. The depth, shape and size of a basin depend on tectonics, movements within the Earth\\'s lithosphere. Where the lithosphere moves upward (tectonic uplift), land eventually rises above sea level and the area becomes a source for new sediment as erosion removes material. Where the lithosphere moves downward (tectonic subsidence), a basin forms and sediments are deposited.\\nA type of basin formed by the moving apart of two pieces of a continent is called a rift basin. Rift basins are elongated, narrow and deep basins. Due to divergent movement, the lithosphere is stretched and thinned, so that the hot asthenosphere rises and heats the overlying rift basin. Apart from continental sediments, rift basins normally also have part of their infill consisting of volcanic deposits. When the basin grows due to continued stretching of the lithosphere, the rift grows and the sea can enter, forming marine deposits.\\nWhen a piece of lithosphere that was heated and stretched cools again, its density rises, causing isostatic subsidence. If this subsidence continues long enough, the basin is called a sag basin. Examples of sag basins are the regions along passive continental margins, but sag basins can also be found in the interior of continents. In sag basins, the extra weight of the newly deposited sediments is enough to keep the subsidence going in a vicious circle. The total thickness of the sedimentary infill in a sag basin can thus exceed 10 km.\\nA third type of basin exists along convergent plate boundaries – places where one tectonic plate moves under another into the asthenosphere. The subducting plate bends and forms a fore-arc basin in front of the overriding plate – an elongated, deep asymmetric basin. Fore-arc basins are filled with deep marine deposits and thick sequences of turbidites. Such infill is called flysch. When the convergent movement of the two plates results in continental collision, the basin becomes shallower and develops into a foreland basin. At the same time, tectonic uplift forms a mountain belt in the overriding plate, from which large amounts of material are eroded and transported to the basin. Such erosional material of a growing mountain chain is called molasse and has either a shallow marine or a continental facies.\\nAt the same time, the growing weight of the mountain belt can cause isostatic subsidence in the area of the overriding plate on the other side to the mountain belt. The basin type resulting from this subsidence is called a back-arc basin and is usually filled by shallow marine deposits and molasse.\\n\\n\\n=== Influence of astronomical cycles ===\\nIn many cases facies changes and other lithological features in sequences of sedimentary rock have a cyclic nature. This cyclic nature was caused by cyclic changes in sediment supply and the sedimentary environment. Most of these cyclic changes are caused by astronomic cycles. Short astronomic cycles can be the difference between the tides or the spring tide every two weeks. On a larger time-scale, cyclic changes in climate and sea level are caused by Milankovitch cycles: cyclic changes in the orientation and/or position of the Earth\\'s rotational axis and orbit around the Sun. There are a number of Milankovitch cycles known, lasting between 10,000 and 200,000 years.Relatively small changes in the orientation of the Earth\\'s axis or length of the seasons can be a major influence on the Earth\\'s climate. An example are the ice ages of the past 2.6 million years (the Quaternary period), which are assumed to have been caused by astronomic cycles. Climate change can influence the global sea level (and thus the amount of accommodation space in sedimentary basins) and sediment supply from a certain region. Eventually, small changes in astronomic parameters can cause large changes in sedimentary environment and sedimentation.\\n\\n\\n== Sedimentation rates ==\\nThe rate at which sediment is deposited differs depending on the location. A channel in a tidal flat can see the deposition of a few metres of sediment in one day, while on the deep ocean floor each year only a few millimetres of sediment accumulate. A distinction can be made between normal sedimentation and sedimentation caused by catastrophic processes. The latter category includes all kinds of sudden exceptional processes like mass movements, rock slides or flooding. Catastrophic processes can see the sudden deposition of a large amount of sediment at once. In some sedimentary environments, most of the total column of sedimentary rock was formed by catastrophic processes, even though the environment is usually a quiet place. Other sedimentary environments are dominated by normal, ongoing sedimentation.In many cases, sedimentation occurs slowly. In a desert, for example, the wind deposits siliciclastic material (sand or silt) in some spots, or catastrophic flooding of a wadi may cause sudden deposits of large quantities of detrital material, but in most places eolian erosion dominates. The amount of sedimentary rock that forms is not only dependent on the amount of supplied material, but also on how well the material consolidates. Erosion removes most deposited sediment shortly after deposition.\\n\\n\\n== Stratigraphy ==\\n\\nThat new rock layers are above older rock layers is stated in the principle of superposition. There are usually some gaps in the sequence called unconformities. These represent periods where no new sediments were laid down, or when earlier sedimentary layers were raised above sea level and eroded away.\\nSedimentary rocks contain important information about the history of the Earth. They contain fossils, the preserved remains of ancient plants and animals. Coal is considered a type of sedimentary rock. The composition of sediments provides us with clues as to the original rock. Differences between successive layers indicate changes to the environment over time. Sedimentary rocks can contain fossils because, unlike most igneous and metamorphic rocks, they form at temperatures and pressures that do not destroy fossil remains.\\n\\n\\n== Provenance ==\\n\\nProvenance is the reconstruction of the origin of sediments. All rock exposed at the Earth\\'s surface is subjected to physical or chemical weathering and broken down into finer grained sediment. All three types of rocks (igneous, sedimentary and metamorphic rocks) can be the source of sedimentary detritus. The purpose of sedimentary provenance studies is to reconstruct and interpret the history of sediment from the initial parent rocks at a source area to final detritus at a burial place.\\n\\n\\n== See also ==\\nBack-stripping\\nDeposition (geology) – Geological process in which sediments, soil and rocks are added to a landform or landmass\\nDunham classification\\nGrowth fault\\nList of minerals – A list of minerals for which there are articles on Wikipedia\\nList of rock types – A list of rock types recognized by geologists\\nSediment transport – The movement of solid particles, typically by gravity and fluid entrainment\\nShelly limestone\\nVolcaniclastics\\n\\n\\n== References ==\\nNotes\\n\\nBibliography\\n\\n\\n== External links ==\\nBasic Sedimentary Rock Classification, by Lynn S. Fichter, James Madison University, Harrisonburg.VI;\\nSedimentary Rocks Tour, introduction to sedimentary rocks, by Bruce Perry, Department of Geological Sciences, California State University at Long Beach .'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let us see the text\n",
    "sed_rk_txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8359925",
   "metadata": {},
   "source": [
    "##### Text preprocesing and normalization\n",
    "- The text contains \\n as newline characters as well as other non alphanumeric characters (e.g. ==)\n",
    "      - that are irrelevent to the actual text.\n",
    "      - We need to remove these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5c8f6951",
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove special new line characters\n",
    "def remove_irrelevant_characters(text, remove_digits=False):\n",
    "    text = re.sub(r'\\n|\\r', ' ', text)\n",
    "    text = re.sub(r'\\t|\\r', ' ', text)\n",
    "    text = re.sub(r'\\n==|\\r', ' ', text)\n",
    "    text = re.sub(r'\\==|\\r', ' ', text)\n",
    "    text = re.sub(r' +', ' ', text)\n",
    "    text = text.strip()\n",
    "    return text\n",
    "\n",
    "sed_3 = remove_irrelevant_characters(sed_rk_txt, \n",
    "                          remove_digits=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5c6691ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Sedimentary rocks are types of rock that are formed by the accumulation or deposition of mineral or organic particles at the Earth\\'s surface, followed by cementation. Sedimentation is the collective name for processes that cause these particles to settle in place. The particles that form a sedimentary rock are called sediment, and may be composed of geological detritus (minerals) or biological detritus (organic matter). The geological detritus originated from weathering and erosion of existing rocks, or from the solidification of molten lava blobs erupted by volcanoes. The geological detritus is transported to the place of deposition by water, wind, ice or mass movement, which are called agents of denudation. Biological detritus was formed by bodies and parts (mainly shells) of dead aquatic organisms, as well as their fecal mass, suspended in water and slowly piling up on the floor of water bodies (marine snow). Sedimentation may also occur as dissolved minerals precipitate from water solution. The sedimentary rock cover of the continents of the Earth\\'s crust is extensive (73% of the Earth\\'s current land surface), but sedimentary rock is estimated to be only 8% of the volume of the crust. Sedimentary rocks are only a thin veneer over a crust consisting mainly of igneous and metamorphic rocks. Sedimentary rocks are deposited in layers as strata, forming a structure called bedding. Sedimentary rocks are often deposited in large structures called sedimentary basins. Sedimentary rocks have also been found on Mars. The study of sedimentary rocks and rock strata provides information about the subsurface that is useful for civil engineering, for example in the construction of roads, houses, tunnels, canals or other structures. Sedimentary rocks are also important sources of natural resources including coal, fossil fuels, drinking water and ores. The study of the sequence of sedimentary rock strata is the main source for an understanding of the Earth\\'s history, including palaeogeography, paleoclimatology and the history of life. The scientific discipline that studies the properties and origin of sedimentary rocks is called sedimentology. Sedimentology is part of both geology and physical geography and overlaps partly with other disciplines in the Earth sciences, such as pedology, geomorphology, geochemistry and structural geology. Classification based on origin Sedimentary rocks can be subdivided into four groups based on the processes responsible for their formation: clastic sedimentary rocks, biochemical (biogenic) sedimentary rocks, chemical sedimentary rocks, and a fourth category for \"other\" sedimentary rocks formed by impacts, volcanism, and other minor processes. = Clastic sedimentary rocks = Clastic sedimentary rocks are composed of rock fragments (clasts) that have been cemented together. The clasts are commonly individual grains of quartz, feldspar, clay minerals, or mica. However, any type of mineral may be present. Clasts may also be lithic fragments composed of more than one mineral. Clastic sedimentary rocks are subdivided according to the dominant particle size. Most geologists use the Udden-Wentworth grain size scale and divide unconsolidated sediment into three fractions: gravel (>2 mm diameter), sand (1/16 to 2 mm diameter), and mud (<1/256 mm diameter). Mud is further divided into silt (1/16 to 1/256 mm diameter) and clay (<1/256 mm diameter). The classification of clastic sedimentary rocks parallels this scheme; conglomerates and breccias are made mostly of gravel, sandstones are made mostly of sand, and mudrocks are made mostly of mud. This tripartite subdivision is mirrored by the broad categories of rudites, arenites, and lutites, respectively, in older literature. The subdivision of these three broad categories is based on differences in clast shape (conglomerates and breccias), composition (sandstones), or grain size or texture (mudrocks). Conglomerates and breccias Conglomerates are dominantly composed of rounded gravel, while breccias are composed of dominantly angular gravel. Sandstones Sandstone classification schemes vary widely, but most geologists have adopted the Dott scheme, which uses the relative abundance of quartz, feldspar, and lithic framework grains and the abundance of a muddy matrix between the larger grains. Composition of framework grains The relative abundance of sand-sized framework grains determines the first word in a sandstone name. Naming depends on the dominance of the three most abundant components quartz, feldspar, or the lithic fragments that originated from other rocks. All other minerals are considered accessories and not used in the naming of the rock, regardless of abundance.Quartz sandstones have >90% quartz grains Feldspathic sandstones have <90% quartz grains and more feldspar grains than lithic grains Lithic sandstones have <90% quartz grains and more lithic grains than feldspar grainsAbundance of muddy matrix material between sand grains When sand-sized particles are deposited, the space between the grains either remains open or is filled with mud (silt and/or clay sized particle). \"Clean\" sandstones with open pore space (that may later be filled with matrix material) are called arenites. Muddy sandstones with abundant (>10%) muddy matrix are called wackes.Six sandstone names are possible using the descriptors for grain composition (quartz-, feldspathic-, and lithic-) and the amount of matrix (wacke or arenite). For example, a quartz arenite would be composed of mostly (>90%) quartz grains and have little or no clayey matrix between the grains, a lithic wacke would have abundant lithic grains and abundant muddy matrix, etc. Although the Dott classification scheme is widely used by sedimentologists, common names like greywacke, arkose, and quartz sandstone are still widely used by non-specialists and in popular literature. Mudrocks Mudrocks are sedimentary rocks composed of at least 50% silt- and clay-sized particles. These relatively fine-grained particles are commonly transported by turbulent flow in water or air, and deposited as the flow calms and the particles settle out of suspension. Most authors presently use the term \"mudrock\" to refer to all rocks composed dominantly of mud. Mudrocks can be divided into siltstones, composed dominantly of silt-sized particles; mudstones with subequal mixture of silt- and clay-sized particles; and claystones, composed mostly of clay-sized particles. Most authors use \"shale\" as a term for a fissile mudrock (regardless of grain size) although some older literature uses the term \"shale\" as a synonym for mudrock. = Biochemical sedimentary rocks = Biochemical sedimentary rocks are created when organisms use materials dissolved in air or water to build their tissue. Examples include: Most types of limestone are formed from the calcareous skeletons of organisms such as corals, mollusks, and foraminifera. Coal, formed from plants that have removed carbon from the atmosphere and combined it with other elements to build their tissue. Deposits of chert formed from the accumulation of siliceous skeletons of microscopic organisms such as radiolaria and diatoms. = Chemical sedimentary rocks = Chemical sedimentary rock forms when mineral constituents in solution become supersaturated and inorganically precipitate. Common chemical sedimentary rocks include oolitic limestone and rocks composed of evaporite minerals, such as halite (rock salt), sylvite, baryte and gypsum. = Other sedimentary rocks = This fourth miscellaneous category includes volcanic tuff and volcanic breccias formed by deposition and later cementation of lava fragments erupted by volcanoes, and impact breccias formed after impact events. Classification based on composition Alternatively, sedimentary rocks can be subdivided into compositional groups based on their mineralogy: Siliciclastic sedimentary rocks, are dominantly composed of silicate minerals. The sediment that makes up these rocks was transported as bed load, suspended load, or by sediment gravity flows. Siliciclastic sedimentary rocks are subdivided into conglomerates and breccias, sandstone, and mudrocks. Carbonate sedimentary rocks are composed of calcite (rhombohedral CaCO3), aragonite (orthorhombic CaCO3), dolomite (CaMg(CO3)2), and other carbonate minerals based on the CO2−3 ion. Common examples include limestone and the rock dolomite. Evaporite sedimentary rocks are composed of minerals formed from the evaporation of water. The most common evaporite minerals are carbonates (calcite and others based on CO2−3), chlorides (halite and others built on Cl−), and sulfates (gypsum and others built on SO2−4). Evaporite rocks commonly include abundant halite (rock salt), gypsum, and anhydrite. Organic-rich sedimentary rocks have significant amounts of organic material, generally in excess of 3% total organic carbon. Common examples include coal, oil shale as well as source rocks for oil and natural gas. Siliceous sedimentary rocks are almost entirely composed of silica (SiO2), typically as chert, opal, chalcedony or other microcrystalline forms. Iron-rich sedimentary rocks are composed of >15% iron; the most common forms are banded iron formations and ironstones. Phosphatic sedimentary rocks are composed of phosphate minerals and contain more than 6.5% phosphorus; examples include deposits of phosphate nodules, bone beds, and phosphatic mudrocks. Deposition and transformation = Sediment transport and deposition = Sedimentary rocks are formed when sediment is deposited out of air, ice, wind, gravity, or water flows carrying the particles in suspension. This sediment is often formed when weathering and erosion break down a rock into loose material in a source area. The material is then transported from the source area to the deposition area. The type of sediment transported depends on the geology of the hinterland (the source area of the sediment). However, some sedimentary rocks, such as evaporites, are composed of material that form at the place of deposition. The nature of a sedimentary rock, therefore, not only depends on the sediment supply, but also on the sedimentary depositional environment in which it formed. = Transformation (Diagenesis) = As sediments accumulate in a depositional environment, older sediments are buried by younger sediments, and they undergo diagenesis. Diagenesis includes all the chemical, physical, and biological changes, exclusive of surface weathering, undergone by a sediment after its initial deposition. This includes compaction and lithification of the sediments. Early stages of diagenesis, described as eogenesis, take place at shallow depths (a few tens of meters) and is characterized by bioturbation and mineralogical changes in the sediments, with only slight compaction. The red hematite that gives red bed sandstones their color is likely formed during eogenesis. Some biochemical processes, like the activity of bacteria, can affect minerals in a rock and are therefore seen as part of diagenesis.Deeper burial is accompanied by mesogenesis, during which most of the compaction and lithification takes place. Compaction takes place as the sediments come under increasing overburden (lithostatic) pressure from overlying sediments. Sediment grains move into more compact arrangements, grains of ductile minerals (such as mica) are deformed, and pore space is reduced. Sediments are typically saturated with groundwater or seawater when originally deposited, and as pore space is reduced, much of these connate fluids are expelled. In addition to this physical compaction, chemical compaction may take place via pressure solution. Points of contact between grains are under the greatest strain, and the strained mineral is more soluble than the rest of the grain. As a result, the contact points are dissolved away, allowing the grains to come into closer contact. The increased pressure and temperature stimulate further chemical reactions, such as the reactions by which organic material becomes lignite or coal.Lithification follows closely on compaction, as increased temperatures at depth hasten the precipitation of cement that binds the grains together. Pressure solution contributes to this process of cementation, as the mineral dissolved from strained contact points is redeposited in the unstrained pore spaces. This further reduces porosity and makes the rock more compact and competent.Unroofing of buried sedimentary rock is accompanied by telogenesis, the third and final stage of diagenesis. As erosion reduces the depth of burial, renewed exposure to meteoric water produces additional changes to the sedimentary rock, such as leaching of some of the cement to produce secondary porosity.At sufficiently high temperature and pressure, the realm of diagenesis makes way for metamorphism, the process that forms metamorphic rock. Properties = Color = The color of a sedimentary rock is often mostly determined by iron, an element with two major oxides: iron(II) oxide and iron(III) oxide. Iron(II) oxide (FeO) only forms under low oxygen (anoxic) circumstances and gives the rock a grey or greenish colour. Iron(III) oxide (Fe2O3) in a richer oxygen environment is often found in the form of the mineral hematite and gives the rock a reddish to brownish colour. In arid continental climates rocks are in direct contact with the atmosphere, and oxidation is an important process, giving the rock a red or orange colour. Thick sequences of red sedimentary rocks formed in arid climates are called red beds. However, a red colour does not necessarily mean the rock formed in a continental environment or arid climate.The presence of organic material can colour a rock black or grey. Organic material is formed from dead organisms, mostly plants. Normally, such material eventually decays by oxidation or bacterial activity. Under anoxic circumstances, however, organic material cannot decay and leaves a dark sediment, rich in organic material. This can, for example, occur at the bottom of deep seas and lakes. There is little water mixing in such environments; as a result, oxygen from surface water is not brought down, and the deposited sediment is normally a fine dark clay. Dark rocks, rich in organic material, are therefore often shales. = Texture = The size, form and orientation of clasts (the original pieces of rock) in a sediment is called its texture. The texture is a small-scale property of a rock, but determines many of its large-scale properties, such as the density, porosity or permeability.The 3D orientation of the clasts is called the fabric of the rock. The size and form of clasts can be used to determine the velocity and direction of current in the sedimentary environment that moved the clasts from their origin; fine, calcareous mud only settles in quiet water while gravel and larger clasts are moved only by rapidly moving water. The grain size of a rock is usually expressed with the Wentworth scale, though alternative scales are sometimes used. The grain size can be expressed as a diameter or a volume, and is always an average value, since a rock is composed of clasts with different sizes. The statistical distribution of grain sizes is different for different rock types and is described in a property called the sorting of the rock. When all clasts are more or less of the same size, the rock is called \\'well-sorted\\', and when there is a large spread in grain size, the rock is called \\'poorly sorted\\'. The form of the clasts can reflect the origin of the rock. For example, coquina, a rock composed of clasts of broken shells, can only form in energetic water. The form of a clast can be described by using four parameters: Surface texture describes the amount of small-scale relief of the surface of a grain that is too small to influence the general shape. For example, frosted grains, which are covered with small-scale fractures, are characteristic of eolian sandstones. Rounding describes the general smoothness of the shape of a grain. Sphericity describes the degree to which the grain approaches a sphere. Grain form describes the three-dimensional shape of the grain.Chemical sedimentary rocks have a non-clastic texture, consisting entirely of crystals. To describe such a texture, only the average size of the crystals and the fabric are necessary. = Mineralogy = Most sedimentary rocks contain either quartz (siliciclastic rocks) or calcite (carbonate rocks). In contrast to igneous and metamorphic rocks, a sedimentary rock usually contains very few different major minerals. However, the origin of the minerals in a sedimentary rock is often more complex than in an igneous rock. Minerals in a sedimentary rock may have been present in the original sediments or may formed by precipitation during diagenesis. In the second case, a mineral precipitate may have grown over an older generation of cement. A complex diagenetic history can be established by optical mineralogy, using a petrographic microscope. Carbonate rocks predominantly consist of carbonate minerals such as calcite, aragonite or dolomite. Both the cement and the clasts (including fossils and ooids) of a carbonate sedimentary rock usually consist of carbonate minerals. The mineralogy of a clastic rock is determined by the material supplied by the source area, the manner of its transport to the place of deposition and the stability of that particular mineral. The resistance of rock-forming minerals to weathering is expressed by the Goldich dissolution series. In this series, quartz is the most stable, followed by feldspar, micas, and finally other less stable minerals that are only present when little weathering has occurred. The amount of weathering depends mainly on the distance to the source area, the local climate and the time it took for the sediment to be transported to the point where it is deposited. In most sedimentary rocks, mica, feldspar and less stable minerals have been weathered to clay minerals like kaolinite, illite or smectite. = Fossils = Among the three major types of rock, fossils are most commonly found in sedimentary rock. Unlike most igneous and metamorphic rocks, sedimentary rocks form at temperatures and pressures that do not destroy fossil remnants. Often these fossils may only be visible under magnification. Dead organisms in nature are usually quickly removed by scavengers, bacteria, rotting and erosion, but under exceptional circumstances, these natural processes are unable to take place, leading to fossilisation. The chance of fossilisation is higher when the sedimentation rate is high (so that a carcass is quickly buried), in anoxic environments (where little bacterial activity occurs) or when the organism had a particularly hard skeleton. Larger, well-preserved fossils are relatively rare. Fossils can be both the direct remains or imprints of organisms and their skeletons. Most commonly preserved are the harder parts of organisms such as bones, shells, and the woody tissue of plants. Soft tissue has a much smaller chance of being fossilized, and the preservation of soft tissue of animals older than 40 million years is very rare. Imprints of organisms made while they were still alive are called trace fossils, examples of which are burrows, footprints, etc. As a part of a sedimentary rock, fossils undergo the same diagenetic processes as does the host rock. For example, a shell consisting of calcite can dissolve while a cement of silica then fills the cavity. In the same way, precipitating minerals can fill cavities formerly occupied by blood vessels, vascular tissue or other soft tissues. This preserves the form of the organism but changes the chemical composition, a process called permineralization. The most common minerals involved in permineralization are various forms of amorphous silica (chalcedony, flint, chert), carbonates (especially calcite), and pyrite. At high pressure and temperature, the organic material of a dead organism undergoes chemical reactions in which volatiles such as water and carbon dioxide are expulsed. The fossil, in the end, consists of a thin layer of pure carbon or its mineralized form, graphite. This form of fossilisation is called carbonisation. It is particularly important for plant fossils. The same process is responsible for the formation of fossil fuels like lignite or coal. = Primary sedimentary structures = Structures in sedimentary rocks can be divided into primary structures (formed during deposition) and secondary structures (formed after deposition). Unlike textures, structures are always large-scale features that can easily be studied in the field. Sedimentary structures can indicate something about the sedimentary environment or can serve to tell which side originally faced up where tectonics have tilted or overturned sedimentary layers. Sedimentary rocks are laid down in layers called beds or strata. A bed is defined as a layer of rock that has a uniform lithology and texture. Beds form by the deposition of layers of sediment on top of each other. The sequence of beds that characterizes sedimentary rocks is called bedding. Single beds can be a couple of centimetres to several meters thick. Finer, less pronounced layers are called laminae, and the structure a lamina forms in a rock is called lamination. Laminae are usually less than a few centimetres thick. Though bedding and lamination are often originally horizontal in nature, this is not always the case. In some environments, beds are deposited at a (usually small) angle. Sometimes multiple sets of layers with different orientations exist in the same rock, a structure called cross-bedding. Cross-bedding is characteristic of deposition by a flowing medium (wind or water). The opposite of cross-bedding is parallel lamination, where all sedimentary layering is parallel. Differences in laminations are generally caused by cyclic changes in the sediment supply, caused, for example, by seasonal changes in rainfall, temperature or biochemical activity. Laminae that represent seasonal changes (similar to tree rings) are called varves. Any sedimentary rock composed of millimeter or finer scale layers can be named with the general term laminite. When sedimentary rocks have no lamination at all, their structural character is called massive bedding. Graded bedding is a structure where beds with a smaller grain size occur on top of beds with larger grains. This structure forms when fast flowing water stops flowing. Larger, heavier clasts in suspension settle first, then smaller clasts. Although graded bedding can form in many different environments, it is a characteristic of turbidity currents.The surface of a particular bed, called the bedform, can also be indicative of a particular sedimentary environment. Examples of bed forms include dunes and ripple marks. Sole markings, such as tool marks and flute casts, are grooves eroded on a surface that are preserved by renewed sedimentation. These are often elongated structures and can be used to establish the direction of the flow during deposition.Ripple marks also form in flowing water. There can be symmetric or asymmetric. Asymmetric ripples form in environments where the current is in one direction, such as rivers. The longer flank of such ripples is on the upstream side of the current. Symmetric wave ripples occur in environments where currents reverse directions, such as tidal flats. Mudcracks are a bed form caused by the dehydration of sediment that occasionally comes above the water surface. Such structures are commonly found at tidal flats or point bars along rivers. = Secondary sedimentary structures = Secondary sedimentary structures are those which formed after deposition. Such structures form by chemical, physical and biological processes within the sediment. They can be indicators of circumstances after deposition. Some can be used as way up criteria. Organic materials in a sediment can leave more traces than just fossils. Preserved tracks and burrows are examples of trace fossils (also called ichnofossils). Such traces are relatively rare. Most trace fossils are burrows of molluscs or arthropods. This burrowing is called bioturbation by sedimentologists. It can be a valuable indicator of the biological and ecological environment that existed after the sediment was deposited. On the other hand, the burrowing activity of organisms can destroy other (primary) structures in the sediment, making a reconstruction more difficult. Secondary structures can also form by diagenesis or the formation of a soil (pedogenesis) when a sediment is exposed above the water level. An example of a diagenetic structure common in carbonate rocks is a stylolite. Stylolites are irregular planes where material was dissolved into the pore fluids in the rock. This can result in the precipitation of a certain chemical species producing colouring and staining of the rock, or the formation of concretions. Concretions are roughly concentric bodies with a different composition from the host rock. Their formation can be the result of localized precipitation due to small differences in composition or porosity of the host rock, such as around fossils, inside burrows or around plant roots. In carbonate rocks such as limestone or chalk, chert or flint concretions are common, while terrestrial sandstones sometimes contain iron concretions. Calcite concretions in clay containing angular cavities or cracks are called septarian concretions. After deposition, physical processes can deform the sediment, producing a third class of secondary structures. Density contrasts between different sedimentary layers, such as between sand and clay, can result in flame structures or load casts, formed by inverted diapirism. While the clastic bed is still fluid, diapirism can cause a denser upper layer to sink into a lower layer. Sometimes, density contrasts occur or are enhanced when one of the lithologies dehydrates. Clay can be easily compressed as a result of dehydration, while sand retains the same volume and becomes relatively less dense. On the other hand, when the pore fluid pressure in a sand layer surpasses a critical point, the sand can break through overlying clay layers and flow through, forming discordant bodies of sedimentary rock called sedimentary dykes. The same process can form mud volcanoes on the surface where they broke through upper layers. Sedimentary dykes can also be formed in a cold climate where the soil is permanently frozen during a large part of the year. Frost weathering can form cracks in the soil that fill with rubble from above. Such structures can be used as climate indicators as well as way up structures.Density contrasts can also cause small-scale faulting, even while sedimentation progresses (synchronous-sedimentary faulting). Such faulting can also occur when large masses of non-lithified sediment are deposited on a slope, such as at the front side of a delta or the continental slope. Instabilities in such sediments can result in the deposited material to slump, producing fissures and folding. The resulting structures in the rock are syn-sedimentary folds and faults, which can be difficult to distinguish from folds and faults formed by tectonic forces acting on lithified rocks. Depositional environments The setting in which a sedimentary rock forms is called the depositional environment. Every environment has a characteristic combination of geologic processes, and circumstances. The type of sediment that is deposited is not only dependent on the sediment that is transported to a place (provenance), but also on the environment itself.A marine environment means that the rock was formed in a sea or ocean. Often, a distinction is made between deep and shallow marine environments. Deep marine usually refers to environments more than 200 m below the water surface (including the abyssal plain). Shallow marine environments exist adjacent to coastlines and can extend to the boundaries of the continental shelf. The water movements in such environments have a generally higher energy than that in deep environments, as wave activity diminishes with depth. This means that coarser sediment particles can be transported and the deposited sediment can be coarser than in deeper environments. When the sediment is transported from the continent, an alternation of sand, clay and silt is deposited. When the continent is far away, the amount of such sediment deposited may be small, and biochemical processes dominate the type of rock that forms. Especially in warm climates, shallow marine environments far offshore mainly see deposition of carbonate rocks. The shallow, warm water is an ideal habitat for many small organisms that build carbonate skeletons. When these organisms die, their skeletons sink to the bottom, forming a thick layer of calcareous mud that may lithify into limestone. Warm shallow marine environments also are ideal environments for coral reefs, where the sediment consists mainly of the calcareous skeletons of larger organisms.In deep marine environments, the water current working the sea bottom is small. Only fine particles can be transported to such places. Typically sediments depositing on the ocean floor are fine clay or small skeletons of micro-organisms. At 4 km depth, the solubility of carbonates increases dramatically (the depth zone where this happens is called the lysocline). Calcareous sediment that sinks below the lysocline dissolves; as a result, no limestone can be formed below this depth. Skeletons of micro-organisms formed of silica (such as radiolarians) are not as soluble and are still deposited. An example of a rock formed of silica skeletons is radiolarite. When the bottom of the sea has a small inclination, for example, at the continental slopes, the sedimentary cover can become unstable, causing turbidity currents. Turbidity currents are sudden disturbances of the normally quiet deep marine environment and can cause the near-instantaneous deposition of large amounts of sediment, such as sand and silt. The rock sequence formed by a turbidity current is called a turbidite.The coast is an environment dominated by wave action. At a beach, dominantly denser sediment such as sand or gravel, often mingled with shell fragments, is deposited, while the silt and clay sized material is kept in mechanical suspension. Tidal flats and shoals are places that sometimes dry because of the tide. They are often cross-cut by gullies, where the current is strong and the grain size of the deposited sediment is larger. Where rivers enter the body of water, either on a sea or lake coast, deltas can form. These are large accumulations of sediment transported from the continent to places in front of the mouth of the river. Deltas are dominantly composed of clastic (rather than chemical) sediment. A continental sedimentary environment is an environment in the interior of a continent. Examples of continental environments are lagoons, lakes, swamps, floodplains and alluvial fans. In the quiet water of swamps, lakes and lagoons, fine sediment is deposited, mingled with organic material from dead plants and animals. In rivers, the energy of the water is much greater and can transport heavier clastic material. Besides transport by water, sediment can be transported by wind or glaciers. Sediment transported by wind is called aeolian and is almost always very well sorted, while sediment transported by a glacier is called glacial till and is characterized by very poor sorting.Aeolian deposits can be quite striking. The depositional environment of the Touchet Formation, located in the Northwestern United States, had intervening periods of aridity which resulted in a series of rhythmite layers. Erosional cracks were later infilled with layers of soil material, especially from aeolian processes. The infilled sections formed vertical inclusions in the horizontally deposited layers, and thus provided evidence of the sequence of events during deposition of the forty-one layers of the formation. = Sedimentary facies = The kind of rock formed in a particular depositional environment is called its sedimentary facies. Sedimentary environments usually exist alongside each other in certain natural successions. A beach, where sand and gravel is deposited, is usually bounded by a deeper marine environment a little offshore, where finer sediments are deposited at the same time. Behind the beach, there can be dunes (where the dominant deposition is well sorted sand) or a lagoon (where fine clay and organic material is deposited). Every sedimentary environment has its own characteristic deposits. When sedimentary strata accumulate through time, the environment can shift, forming a change in facies in the subsurface at one location. On the other hand, when a rock layer with a certain age is followed laterally, the lithology (the type of rock) and facies eventually change. Facies can be distinguished in a number of ways: the most common are by the lithology (for example: limestone, siltstone or sandstone) or by fossil content. Coral, for example, only lives in warm and shallow marine environments and fossils of coral are thus typical for shallow marine facies. Facies determined by lithology are called lithofacies; facies determined by fossils are biofacies.Sedimentary environments can shift their geographical positions through time. Coastlines can shift in the direction of the sea when the sea level drops (regression), when the surface rises (transgression) due to tectonic forces in the Earth\\'s crust or when a river forms a large delta. In the subsurface, such geographic shifts of sedimentary environments of the past are recorded in shifts in sedimentary facies. This means that sedimentary facies can change either parallel or perpendicular to an imaginary layer of rock with a fixed age, a phenomenon described by Walther\\'s Law.The situation in which coastlines move in the direction of the continent is called transgression. In the case of transgression, deeper marine facies are deposited over shallower facies, a succession called onlap. Regression is the situation in which a coastline moves in the direction of the sea. With regression, shallower facies are deposited on top of deeper facies, a situation called offlap.The facies of all rocks of a certain age can be plotted on a map to give an overview of the palaeogeography. A sequence of maps for different ages can give an insight in the development of the regional geography. Gallery of sedimentary facies Sedimentary basins Places where large-scale sedimentation takes place are called sedimentary basins. The amount of sediment that can be deposited in a basin depends on the depth of the basin, the so-called accommodation space. The depth, shape and size of a basin depend on tectonics, movements within the Earth\\'s lithosphere. Where the lithosphere moves upward (tectonic uplift), land eventually rises above sea level and the area becomes a source for new sediment as erosion removes material. Where the lithosphere moves downward (tectonic subsidence), a basin forms and sediments are deposited. A type of basin formed by the moving apart of two pieces of a continent is called a rift basin. Rift basins are elongated, narrow and deep basins. Due to divergent movement, the lithosphere is stretched and thinned, so that the hot asthenosphere rises and heats the overlying rift basin. Apart from continental sediments, rift basins normally also have part of their infill consisting of volcanic deposits. When the basin grows due to continued stretching of the lithosphere, the rift grows and the sea can enter, forming marine deposits. When a piece of lithosphere that was heated and stretched cools again, its density rises, causing isostatic subsidence. If this subsidence continues long enough, the basin is called a sag basin. Examples of sag basins are the regions along passive continental margins, but sag basins can also be found in the interior of continents. In sag basins, the extra weight of the newly deposited sediments is enough to keep the subsidence going in a vicious circle. The total thickness of the sedimentary infill in a sag basin can thus exceed 10 km. A third type of basin exists along convergent plate boundaries – places where one tectonic plate moves under another into the asthenosphere. The subducting plate bends and forms a fore-arc basin in front of the overriding plate – an elongated, deep asymmetric basin. Fore-arc basins are filled with deep marine deposits and thick sequences of turbidites. Such infill is called flysch. When the convergent movement of the two plates results in continental collision, the basin becomes shallower and develops into a foreland basin. At the same time, tectonic uplift forms a mountain belt in the overriding plate, from which large amounts of material are eroded and transported to the basin. Such erosional material of a growing mountain chain is called molasse and has either a shallow marine or a continental facies. At the same time, the growing weight of the mountain belt can cause isostatic subsidence in the area of the overriding plate on the other side to the mountain belt. The basin type resulting from this subsidence is called a back-arc basin and is usually filled by shallow marine deposits and molasse. = Influence of astronomical cycles = In many cases facies changes and other lithological features in sequences of sedimentary rock have a cyclic nature. This cyclic nature was caused by cyclic changes in sediment supply and the sedimentary environment. Most of these cyclic changes are caused by astronomic cycles. Short astronomic cycles can be the difference between the tides or the spring tide every two weeks. On a larger time-scale, cyclic changes in climate and sea level are caused by Milankovitch cycles: cyclic changes in the orientation and/or position of the Earth\\'s rotational axis and orbit around the Sun. There are a number of Milankovitch cycles known, lasting between 10,000 and 200,000 years.Relatively small changes in the orientation of the Earth\\'s axis or length of the seasons can be a major influence on the Earth\\'s climate. An example are the ice ages of the past 2.6 million years (the Quaternary period), which are assumed to have been caused by astronomic cycles. Climate change can influence the global sea level (and thus the amount of accommodation space in sedimentary basins) and sediment supply from a certain region. Eventually, small changes in astronomic parameters can cause large changes in sedimentary environment and sedimentation. Sedimentation rates The rate at which sediment is deposited differs depending on the location. A channel in a tidal flat can see the deposition of a few metres of sediment in one day, while on the deep ocean floor each year only a few millimetres of sediment accumulate. A distinction can be made between normal sedimentation and sedimentation caused by catastrophic processes. The latter category includes all kinds of sudden exceptional processes like mass movements, rock slides or flooding. Catastrophic processes can see the sudden deposition of a large amount of sediment at once. In some sedimentary environments, most of the total column of sedimentary rock was formed by catastrophic processes, even though the environment is usually a quiet place. Other sedimentary environments are dominated by normal, ongoing sedimentation.In many cases, sedimentation occurs slowly. In a desert, for example, the wind deposits siliciclastic material (sand or silt) in some spots, or catastrophic flooding of a wadi may cause sudden deposits of large quantities of detrital material, but in most places eolian erosion dominates. The amount of sedimentary rock that forms is not only dependent on the amount of supplied material, but also on how well the material consolidates. Erosion removes most deposited sediment shortly after deposition. Stratigraphy That new rock layers are above older rock layers is stated in the principle of superposition. There are usually some gaps in the sequence called unconformities. These represent periods where no new sediments were laid down, or when earlier sedimentary layers were raised above sea level and eroded away. Sedimentary rocks contain important information about the history of the Earth. They contain fossils, the preserved remains of ancient plants and animals. Coal is considered a type of sedimentary rock. The composition of sediments provides us with clues as to the original rock. Differences between successive layers indicate changes to the environment over time. Sedimentary rocks can contain fossils because, unlike most igneous and metamorphic rocks, they form at temperatures and pressures that do not destroy fossil remains. Provenance Provenance is the reconstruction of the origin of sediments. All rock exposed at the Earth\\'s surface is subjected to physical or chemical weathering and broken down into finer grained sediment. All three types of rocks (igneous, sedimentary and metamorphic rocks) can be the source of sedimentary detritus. The purpose of sedimentary provenance studies is to reconstruct and interpret the history of sediment from the initial parent rocks at a source area to final detritus at a burial place. See also Back-stripping Deposition (geology) – Geological process in which sediments, soil and rocks are added to a landform or landmass Dunham classification Growth fault List of minerals – A list of minerals for which there are articles on Wikipedia List of rock types – A list of rock types recognized by geologists Sediment transport – The movement of solid particles, typically by gravity and fluid entrainment Shelly limestone Volcaniclastics References Notes Bibliography External links Basic Sedimentary Rock Classification, by Lynn S. Fichter, James Madison University, Harrisonburg.VI; Sedimentary Rocks Tour, introduction to sedimentary rocks, by Bruce Perry, Department of Geological Sciences, California State University at Long Beach .'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sed_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "453a157a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "320"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Sentence tokenize to extract long and short sentences\n",
    "punkt_st = nltk.tokenize.PunktSentenceTokenizer()\n",
    "sed_rk1 = punkt_st.tokenize(sed_3)\n",
    "len(sed_rk1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d05012a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Sedimentary rocks are types of rock that are formed by the accumulation or deposition of mineral or organic particles at the Earth's surface, followed by cementation.\",\n",
       " 'Sedimentation is the collective name for processes that cause these particles to settle in place.',\n",
       " 'The particles that form a sedimentary rock are called sediment, and may be composed of geological detritus (minerals) or biological detritus (organic matter).']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Sanity check: Let us see the first 3 sentences\n",
    "sed_rk1[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b0e139e",
   "metadata": {},
   "source": [
    "#### 1A\n",
    "Find the longest sentence you can, longer than 10 words, that the POS tagger tags correctly. Show the input and output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "71fe3777",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Separate the 'long' (>10 words) sentences from the short ones. \n",
    "\n",
    "sed_rk_long_sent = [] #list of long sentences\n",
    "sed_rk_short = [] #list of short sentences\n",
    "for sent in punkt_st.tokenize(sed_rk_txt):\n",
    "    if len(sent.split()) > 10: #condition for long sentences\n",
    "        sed_rk_long_sent.append(sent)\n",
    "    else:\n",
    "        sed_rk_short.append(sent) #condition for short sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1d8fd171",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sed_rk_long_sent) #how many long sentences do we have"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "af8d62b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Sedimentary rocks are types of rock that are formed by the accumulation or deposition of mineral or organic particles at the Earth's surface, followed by cementation.\""
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sed_rk_long_sent[0] #see the first sentence as example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b0b272b",
   "metadata": {},
   "source": [
    "### Part of speech tagger 1\n",
    "\n",
    "#### Using the nltk tagger\n",
    "- Part of Speech tagging algorithms take in words and tag them accordingly. \n",
    "    - Therefore we need to word-tokenize the text before feeding it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d2fa1f30",
   "metadata": {},
   "outputs": [],
   "source": [
    "#object function to tokenize the text\n",
    "\n",
    "default_wt = nltk.word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1f729107",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Word</th>\n",
       "      <th>POS tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sedimentary</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>rocks</td>\n",
       "      <td>NNS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>are</td>\n",
       "      <td>VBP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>types</td>\n",
       "      <td>NNS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>rock</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>that</td>\n",
       "      <td>WDT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>are</td>\n",
       "      <td>VBP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>formed</td>\n",
       "      <td>VBN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>by</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>accumulation</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>or</td>\n",
       "      <td>CC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>deposition</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>mineral</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>or</td>\n",
       "      <td>CC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>organic</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>particles</td>\n",
       "      <td>NNS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>at</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Earth</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>'s</td>\n",
       "      <td>POS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>surface</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>,</td>\n",
       "      <td>,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>followed</td>\n",
       "      <td>VBN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>by</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>cementation</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>.</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Word POS tag\n",
       "0    Sedimentary      JJ\n",
       "1          rocks     NNS\n",
       "2            are     VBP\n",
       "3          types     NNS\n",
       "4             of      IN\n",
       "5           rock      NN\n",
       "6           that     WDT\n",
       "7            are     VBP\n",
       "8         formed     VBN\n",
       "9             by      IN\n",
       "10           the      DT\n",
       "11  accumulation      NN\n",
       "12            or      CC\n",
       "13    deposition      NN\n",
       "14            of      IN\n",
       "15       mineral      NN\n",
       "16            or      CC\n",
       "17       organic      JJ\n",
       "18     particles     NNS\n",
       "19            at      IN\n",
       "20           the      DT\n",
       "21         Earth     NNP\n",
       "22            's     POS\n",
       "23       surface      NN\n",
       "24             ,       ,\n",
       "25      followed     VBN\n",
       "26            by      IN\n",
       "27   cementation      NN\n",
       "28             .       ."
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Select the first long sentence\n",
    "#Word tokenize the sentence and POS it\n",
    "nltk_pos_tagged_long = nltk.pos_tag(default_wt(sed_rk_long_sent[0])) #tag the long sentence\n",
    "sed_long_tag = pd.DataFrame(nltk_pos_tagged_long, columns=['Word', 'POS tag']) #Display it\n",
    "sed_long_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27fdbce7",
   "metadata": {},
   "source": [
    "#### 1B\n",
    "Find the shortest sentence you can, that the POS tagger tags correctly. Show the input and output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "18ab2ae5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Randomly select a short sentence\n",
    "short_sent = sed_rk_short[8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d3a1e96e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Word</th>\n",
       "      <td>Larger</td>\n",
       "      <td>,</td>\n",
       "      <td>well-preserved</td>\n",
       "      <td>fossils</td>\n",
       "      <td>are</td>\n",
       "      <td>relatively</td>\n",
       "      <td>rare</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POS tag</th>\n",
       "      <td>NNP</td>\n",
       "      <td>,</td>\n",
       "      <td>JJ</td>\n",
       "      <td>NNS</td>\n",
       "      <td>VBP</td>\n",
       "      <td>RB</td>\n",
       "      <td>JJ</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0  1               2        3    4           5     6  7\n",
       "Word     Larger  ,  well-preserved  fossils  are  relatively  rare  .\n",
       "POS tag     NNP  ,              JJ      NNS  VBP          RB    JJ  ."
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#nltk_pos_tagged_short = nltk.pos_tag(default_wt(sed_rk_short[8]))\n",
    "nltk_pos_tagged_short = nltk.pos_tag(default_wt(short_sent))\n",
    "sed_short_tag = pd.DataFrame(nltk_pos_tagged_short, columns=['Word', 'POS tag']).T\n",
    "sed_short_tag "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b19fba",
   "metadata": {},
   "source": [
    "For this sentence:\n",
    ">'Larger, well-preserved fossils are relatively rare.'\n",
    "\n",
    "- 'Larger' is adjective but the POS module tagged it as Proper Noun\n",
    "\n",
    "-I think it's because 'Larger' begins the sentence or/and is capitalized.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9df0722",
   "metadata": {},
   "source": [
    "## Question 2\n",
    "- Run a different POS tagger in Python. Process the same two sentences from question 1.\n",
    "    - Does it produce the same or different output?\n",
    "    - Explain any differences as best you can."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eab1225b",
   "metadata": {},
   "source": [
    "### Other part of speech taggers (BYOB, n-gram, Spacy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e9c9d7",
   "metadata": {},
   "source": [
    "#### B.Y.O.T: Build your own tagger\n",
    "\n",
    "*reference: Text Analytics with Python. Dipanjan 2019*\n",
    "\n",
    "- We leverage some classes provided by NLTK. \n",
    "- To evaluate the performance of our taggers, we use some test data from the treebank corpus in NLTK. \n",
    "- We will also be using some training data for training some of our taggers. \n",
    "- To start with, we will first get the necessary data for training and evaluating the taggers by reading in the tagged treebank corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "7d0ef67b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ('61', 'CD'), ('years', 'NNS'), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), ('the', 'DT'), ('board', 'NN'), ('as', 'IN'), ('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ('Nov.', 'NNP'), ('29', 'CD'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import treebank\n",
    "data = treebank.tagged_sents()\n",
    "train_data = data[:3500]\n",
    "test_data = data[3500:]\n",
    "print(train_data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f4b7f368",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.classify import NaiveBayesClassifier, MaxentClassifier\n",
    "from nltk.tag.sequential import ClassifierBasedPOSTagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "e0535b60",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Fit the Naive Bayes classifier to the treebank pos tagged corpus.\n",
    "nbt = ClassifierBasedPOSTagger(train=train_data, classifier_builder=NaiveBayesClassifier.train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "4b446e1a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9306806079969019\n",
      "[('Larger', 'NNP'), (',', ','), ('well-preserved', 'VBD'), ('fossils', 'NNS'), ('are', 'VBP'), ('relatively', 'RB'), ('rare', 'JJ'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "# evaluate tagger on test data and unknown (our short sentence)\n",
    "print(nbt.evaluate(test_data))\n",
    "print(nbt.tag(default_wt(sed_rk_short[8])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "90174703",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Word</th>\n",
       "      <td>Larger</td>\n",
       "      <td>,</td>\n",
       "      <td>well-preserved</td>\n",
       "      <td>fossils</td>\n",
       "      <td>are</td>\n",
       "      <td>relatively</td>\n",
       "      <td>rare</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POS tag_NaiveB</th>\n",
       "      <td>NNP</td>\n",
       "      <td>,</td>\n",
       "      <td>VBD</td>\n",
       "      <td>NNS</td>\n",
       "      <td>VBP</td>\n",
       "      <td>RB</td>\n",
       "      <td>JJ</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     0  1               2        3    4           5     6  7\n",
       "Word            Larger  ,  well-preserved  fossils  are  relatively  rare  .\n",
       "POS tag_NaiveB     NNP  ,             VBD      NNS  VBP          RB    JJ  ."
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Run the short sentence through the designed tagger\n",
    "nbt_tag_short = nbt.tag(default_wt(short_sent))\n",
    "sed_tag_short_naiveB = pd.DataFrame(nbt_tag_short, columns=['Word', 'POS tag_NaiveB']).T\n",
    "sed_tag_short_naiveB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "b71594cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Word</th>\n",
       "      <td>Sedimentary</td>\n",
       "      <td>rocks</td>\n",
       "      <td>are</td>\n",
       "      <td>types</td>\n",
       "      <td>of</td>\n",
       "      <td>rock</td>\n",
       "      <td>that</td>\n",
       "      <td>are</td>\n",
       "      <td>formed</td>\n",
       "      <td>by</td>\n",
       "      <td>...</td>\n",
       "      <td>at</td>\n",
       "      <td>the</td>\n",
       "      <td>Earth</td>\n",
       "      <td>'s</td>\n",
       "      <td>surface</td>\n",
       "      <td>,</td>\n",
       "      <td>followed</td>\n",
       "      <td>by</td>\n",
       "      <td>cementation</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POS tag_NaiveB</th>\n",
       "      <td>NNP</td>\n",
       "      <td>NNS</td>\n",
       "      <td>VBP</td>\n",
       "      <td>NNS</td>\n",
       "      <td>IN</td>\n",
       "      <td>NN</td>\n",
       "      <td>WDT</td>\n",
       "      <td>VBP</td>\n",
       "      <td>VBN</td>\n",
       "      <td>IN</td>\n",
       "      <td>...</td>\n",
       "      <td>IN</td>\n",
       "      <td>DT</td>\n",
       "      <td>JJ</td>\n",
       "      <td>POS</td>\n",
       "      <td>NN</td>\n",
       "      <td>,</td>\n",
       "      <td>VBN</td>\n",
       "      <td>IN</td>\n",
       "      <td>CD</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         0      1    2      3   4     5     6    7       8   \\\n",
       "Word            Sedimentary  rocks  are  types  of  rock  that  are  formed   \n",
       "POS tag_NaiveB          NNP    NNS  VBP    NNS  IN    NN   WDT  VBP     VBN   \n",
       "\n",
       "                9   ...  19   20     21   22       23 24        25  26  \\\n",
       "Word            by  ...  at  the  Earth   's  surface  ,  followed  by   \n",
       "POS tag_NaiveB  IN  ...  IN   DT     JJ  POS       NN  ,       VBN  IN   \n",
       "\n",
       "                         27 28  \n",
       "Word            cementation  .  \n",
       "POS tag_NaiveB           CD  .  \n",
       "\n",
       "[2 rows x 29 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##Run the long sentence through the designed tagger\n",
    "nbt_tag = nbt.tag(default_wt(sed_rk_long_sent[0]))\n",
    "sed_tag_long_naiveB = pd.DataFrame(nbt_tag, columns=['Word', 'POS tag_NaiveB']).T\n",
    "sed_tag_long_naiveB"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e8be52",
   "metadata": {},
   "source": [
    "##### N gram taggers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "6c1f8e0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# regex tagger\n",
    "from nltk.tag import RegexpTagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "7e644006",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define regex tag patterns\n",
    "patterns = [\n",
    "    (r'.*ing$', 'VBG'), # gerunds\n",
    "    (r'.*ed$', 'VBD'), # simple past\n",
    "    (r'.*es$', 'VBZ'), # 3rd singular present\n",
    "    (r'.*ould$', 'MD'), # modals\n",
    "    (r'.*\\'s$', 'NN$'), # possessive nouns\n",
    "    (r'.*s$', 'NNS'), # plural nouns\n",
    "    (r'^-?[0-9]+(.[0-9]+)?$', 'CD'), # cardinal numbers\n",
    "    (r'.*', 'NN') # nouns (default)\n",
    "]\n",
    "\n",
    "rt = RegexpTagger(patterns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "95036f50",
   "metadata": {},
   "outputs": [],
   "source": [
    "## N gram taggers\n",
    "from nltk.tag import UnigramTagger\n",
    "ut = UnigramTagger(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "195d11c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8607803272340013\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Word</th>\n",
       "      <td>Sedimentary</td>\n",
       "      <td>rocks</td>\n",
       "      <td>are</td>\n",
       "      <td>types</td>\n",
       "      <td>of</td>\n",
       "      <td>rock</td>\n",
       "      <td>that</td>\n",
       "      <td>are</td>\n",
       "      <td>formed</td>\n",
       "      <td>by</td>\n",
       "      <td>...</td>\n",
       "      <td>at</td>\n",
       "      <td>the</td>\n",
       "      <td>Earth</td>\n",
       "      <td>'s</td>\n",
       "      <td>surface</td>\n",
       "      <td>,</td>\n",
       "      <td>followed</td>\n",
       "      <td>by</td>\n",
       "      <td>cementation</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POS tag_Ngram</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>VBP</td>\n",
       "      <td>NNS</td>\n",
       "      <td>IN</td>\n",
       "      <td>NN</td>\n",
       "      <td>IN</td>\n",
       "      <td>VBP</td>\n",
       "      <td>VBN</td>\n",
       "      <td>IN</td>\n",
       "      <td>...</td>\n",
       "      <td>IN</td>\n",
       "      <td>DT</td>\n",
       "      <td>None</td>\n",
       "      <td>POS</td>\n",
       "      <td>None</td>\n",
       "      <td>,</td>\n",
       "      <td>VBD</td>\n",
       "      <td>IN</td>\n",
       "      <td>None</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        0      1    2      3   4     5     6    7       8   \\\n",
       "Word           Sedimentary  rocks  are  types  of  rock  that  are  formed   \n",
       "POS tag_Ngram         None   None  VBP    NNS  IN    NN    IN  VBP     VBN   \n",
       "\n",
       "               9   ...  19   20     21   22       23 24        25  26  \\\n",
       "Word           by  ...  at  the  Earth   's  surface  ,  followed  by   \n",
       "POS tag_Ngram  IN  ...  IN   DT   None  POS     None  ,       VBD  IN   \n",
       "\n",
       "                        27 28  \n",
       "Word           cementation  .  \n",
       "POS tag_Ngram         None  .  \n",
       "\n",
       "[2 rows x 29 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# testing performance of unigram tagger on both test and long sentence\n",
    "print(ut.evaluate(test_data))  #86% accuracy on the test data\n",
    "ngram_tag = ut.tag(default_wt(sed_rk_long_sent[0])) #Performance on 'unknown' data (my 'long' sentence)\n",
    "sed_tag_ngram = pd.DataFrame(ngram_tag, columns=['Word', 'POS tag_Ngram']).T\n",
    "sed_tag_ngram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "97fd73a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8607803272340013\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Word</th>\n",
       "      <td>Larger</td>\n",
       "      <td>,</td>\n",
       "      <td>well-preserved</td>\n",
       "      <td>fossils</td>\n",
       "      <td>are</td>\n",
       "      <td>relatively</td>\n",
       "      <td>rare</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POS tag_Ngram</th>\n",
       "      <td>None</td>\n",
       "      <td>,</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>VBP</td>\n",
       "      <td>RB</td>\n",
       "      <td>JJ</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    0  1               2        3    4           5     6  7\n",
       "Word           Larger  ,  well-preserved  fossils  are  relatively  rare  .\n",
       "POS tag_Ngram    None  ,            None     None  VBP          RB    JJ  ."
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# # testing performance of unigram tagger on both test data and short sentence\n",
    "print(ut.evaluate(test_data))\n",
    "ngram_tag = ut.tag(default_wt(sed_rk_short[8]))\n",
    "sed_tag_ngram = pd.DataFrame(ngram_tag, columns=['Word', 'POS tag_Ngram']).T\n",
    "sed_tag_ngram"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90e43f77",
   "metadata": {},
   "source": [
    "#### Spacy tagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "73b10f9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "63e827aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\olani\\anaconda3\\envs\\Machine Learning 1\\lib\\site-packages\\spacy\\util.py:275: UserWarning: [W031] Model 'en_core_web_sm' (2.2.0) requires spaCy v2.2 and is incompatible with the current spaCy version (2.3.5). This may lead to unexpected results or runtime errors. To resolve this, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate\n",
      "  warnings.warn(warn_msg)\n"
     ]
    }
   ],
   "source": [
    "import en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "39baa440",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_sm', parse=True, tag=True, entity=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "1439afbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"US unveils world's most powerful supercomputer, beats China.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "de0d4a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_nlp = nlp(sed_rk_long_sent[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "0fc0a38f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Word</th>\n",
       "      <td>Sedimentary</td>\n",
       "      <td>rocks</td>\n",
       "      <td>are</td>\n",
       "      <td>types</td>\n",
       "      <td>of</td>\n",
       "      <td>rock</td>\n",
       "      <td>that</td>\n",
       "      <td>are</td>\n",
       "      <td>formed</td>\n",
       "      <td>by</td>\n",
       "      <td>...</td>\n",
       "      <td>at</td>\n",
       "      <td>the</td>\n",
       "      <td>Earth</td>\n",
       "      <td>'s</td>\n",
       "      <td>surface</td>\n",
       "      <td>,</td>\n",
       "      <td>followed</td>\n",
       "      <td>by</td>\n",
       "      <td>cementation</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POS tag.spacy</th>\n",
       "      <td>JJ</td>\n",
       "      <td>NNS</td>\n",
       "      <td>VBP</td>\n",
       "      <td>NNS</td>\n",
       "      <td>IN</td>\n",
       "      <td>NN</td>\n",
       "      <td>WDT</td>\n",
       "      <td>VBP</td>\n",
       "      <td>VBN</td>\n",
       "      <td>IN</td>\n",
       "      <td>...</td>\n",
       "      <td>IN</td>\n",
       "      <td>DT</td>\n",
       "      <td>NNP</td>\n",
       "      <td>POS</td>\n",
       "      <td>NN</td>\n",
       "      <td>,</td>\n",
       "      <td>VBN</td>\n",
       "      <td>IN</td>\n",
       "      <td>NN</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tag type</th>\n",
       "      <td>84</td>\n",
       "      <td>92</td>\n",
       "      <td>87</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>92</td>\n",
       "      <td>95</td>\n",
       "      <td>87</td>\n",
       "      <td>100</td>\n",
       "      <td>85</td>\n",
       "      <td>...</td>\n",
       "      <td>85</td>\n",
       "      <td>90</td>\n",
       "      <td>96</td>\n",
       "      <td>94</td>\n",
       "      <td>92</td>\n",
       "      <td>97</td>\n",
       "      <td>100</td>\n",
       "      <td>85</td>\n",
       "      <td>92</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        0      1    2      3   4     5     6    7       8   \\\n",
       "Word           Sedimentary  rocks  are  types  of  rock  that  are  formed   \n",
       "POS tag.spacy           JJ    NNS  VBP    NNS  IN    NN   WDT  VBP     VBN   \n",
       "Tag type                84     92   87     92  85    92    95   87     100   \n",
       "\n",
       "               9   ...  19   20     21   22       23  24        25  26  \\\n",
       "Word           by  ...  at  the  Earth   's  surface   ,  followed  by   \n",
       "POS tag.spacy  IN  ...  IN   DT    NNP  POS       NN   ,       VBN  IN   \n",
       "Tag type       85  ...  85   90     96   94       92  97       100  85   \n",
       "\n",
       "                        27  28  \n",
       "Word           cementation   .  \n",
       "POS tag.spacy           NN   .  \n",
       "Tag type                92  97  \n",
       "\n",
       "[3 rows x 29 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Spacy tagger also uses the Penn Treebank notation for POS tagging.\n",
    "#I did not even need to word-tokenize the sentence.\n",
    "spacy_pos_tagged = [(word, word.tag_, word.pos) for word in sentence_nlp]\n",
    "pd.DataFrame(spacy_pos_tagged, columns=['Word', 'POS tag.spacy', 'Tag type']).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78439523",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "72f64e59",
   "metadata": {},
   "source": [
    "### 2A,B: Comparison of tags\n",
    "All the taggers algorithms use the Penn Tree Bank Notation\n",
    "- Both nltk and Spacy produced the same POS tags for the sentences.\n",
    "- 'Sedimentary' is labeled as JJS/JJ by nltk and Spacy taggers but as NNP by Naive Bayes.\n",
    "    - nltk/Spacy pos tag is more correct because sedimentary is a type of rock/qualifies the rock so it is a JJS(adjective)\n",
    "- 'Earth' is labelled as NNP by nltk but JJ by Naive Bayes\n",
    "    - nltk is more correct because Earth is a possessive noun in this case and less like an adjective that qualifies.\n",
    "\n",
    "- Compared to the nltk and Naive Bayes, the ngram produced the worst pos tags.\n",
    "    - the ngram failed to correctly tag many words even in the 6-word short sentence.\n",
    "\n",
    "- I think the error in the tags produced by BYOB and n-gram is because of the limited sized corpus I trained these taggers on. (i.e. only 3,500 length tagged words)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37a79d7b",
   "metadata": {},
   "source": [
    "## 3.\t\n",
    "- In a news article from this week’s news, find a random sentence of at least 10 words.\n",
    "    - a.\tLooking at the Penn tag set, manually POS tag the sentence yourself.\n",
    "    - b.\tNow run the same sentences through both taggers that you implemented for questions 1 and 2. Did either of the taggers produce the same results as you had created manually?\n",
    "    - c.\tExplain any differences between the two taggers and your manual tagging as much as you can.\n",
    "    \n",
    "Source of news sentence:  \n",
    "\n",
    "<a>https://www.reuters.com/business/biden-supply-chain-strike-force-target-china-trade-2021-06-08/<a>    \n",
    "\n",
    "'Officials also said the Department of Commerce is considering initiating a Section 232 investigation into the national security impact of neodymium magnet imports used in motors and other industrial applications, which the United States largely obtains from China.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "07ade7a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "news = 'Officials also said the Department of Commerce is considering initiating a Section 232 investigation into the national security impact of neodymium magnet imports used in motors and other industrial applications, which the United States largely obtains from China.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "53ee1566",
   "metadata": {},
   "outputs": [],
   "source": [
    "#news_nlp = nlp(news)\n",
    "#spacy_pos_tagged1 = [(word, word.tag_, word.pos) for word in news_nlp]\n",
    "#spacy_pos_tagged= [(word, word.tag_) for word in news_nlp]\n",
    "#news_pos_tagged_df = pd.DataFrame(spacy_pos_tagged, columns=['Word', 'POS tag.spacy'])\n",
    "#news_pos_tagged_df\n",
    "\n",
    "#news_pos_tagged_df.to_csv('news_pos_tagged_df.csv', index = False, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7cb9242",
   "metadata": {},
   "source": [
    "## 3A\n",
    "#### Manually tag the news text.\n",
    "\n",
    "Manual tagging:\n",
    "(Officials, NNS)\n",
    "(also, RB) \n",
    "(said, VBD)\n",
    "(the, DT)\n",
    "(Department, NNP)\n",
    "(of, IN)\n",
    "(Commerce, NNP)\n",
    "(is, VBZ)\n",
    "(considering, VBG)\n",
    "(initiating, VBG)\n",
    "(a, DT)\n",
    "(Section, NN)\n",
    " (232,JJ)\n",
    "(investigation, NN)\n",
    "(into, IN)\n",
    "(the, DT)\n",
    "(national, JJ)\n",
    "(security, JJ)\n",
    "(impact, NN)\n",
    "(of, IN)\n",
    "(neodymium, JJ)\n",
    "(magnet, JJ)\n",
    "(imports, NNS)\n",
    "(used, VBN)\n",
    "(in, IN)\n",
    "(motors, NNS)\n",
    "(and, CC)\n",
    "(other, JJ)\n",
    "(industrial, JJ)\n",
    "(applications, NNS)\n",
    "(,)\n",
    "(which, WDT)\n",
    "(the, DT)\n",
    "(United, NNP)\n",
    "(States, NNP)\n",
    "(largely, RB)\n",
    "(obtains, VBZ)\n",
    "(from, IN)\n",
    "(China, NNP)\n",
    "(.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "5c5d11df",
   "metadata": {},
   "outputs": [],
   "source": [
    "manual_tag = [('Officials', 'NNS'),\n",
    "('also', 'RB'),\n",
    "('said', 'VBD'),\n",
    "('the', 'DT'),\n",
    "('Department', 'NNP'),\n",
    "('of', 'IN'),\n",
    "('Commerce', 'NNP'),\n",
    "('is','VBZ'),\n",
    "('considering', 'VBG'),\n",
    "('initiating', 'VBG'),\n",
    "('a', 'DT'),\n",
    "('Section', 'NN'),\n",
    " ('232','JJ'),\n",
    "('investigation', 'NN'),\n",
    "('into', 'IN'),\n",
    "('the', 'DT'),\n",
    "('national', 'JJ'),\n",
    "('security', 'JJ'),\n",
    "('impact', 'NN'),\n",
    "('of', 'IN'),\n",
    "('neodymium', 'JJ'),\n",
    "('magnet', 'JJ'),\n",
    "('imports', 'NNS'),\n",
    "('used', 'VBN'),\n",
    "('in', 'IN'),\n",
    "('motors', 'NNS'),\n",
    "('and', 'CC'),\n",
    "('other', 'JJ'),\n",
    "('industrial', 'JJ'),\n",
    "('applications', 'NNS'),\n",
    "(',', 'Punctuation'),\n",
    "('which', 'WDT'),\n",
    "('the', 'DT'),\n",
    "('United', 'NNP'),\n",
    "('States', 'NNP'),\n",
    "('largely', 'RB'),\n",
    "('obtains', 'VBZ'),\n",
    "('from', 'IN'),\n",
    "('China', 'NNP'),\n",
    "('.', 'Punctuation')\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e62cc9f6",
   "metadata": {},
   "source": [
    "## 3B\n",
    "Now run the same sentences through both taggers that you implemented for questions 1 and 2. Did either of the taggers produce the same results as you had created manually?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "a11b2263",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Word.news</th>\n",
       "      <th>Spacy.tag</th>\n",
       "      <th>nltk.tag</th>\n",
       "      <th>Manual.tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Officials</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>also</td>\n",
       "      <td>RB</td>\n",
       "      <td>RB</td>\n",
       "      <td>RB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>said</td>\n",
       "      <td>VBD</td>\n",
       "      <td>VBD</td>\n",
       "      <td>VBD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Department</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Commerce</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>is</td>\n",
       "      <td>VBZ</td>\n",
       "      <td>VBZ</td>\n",
       "      <td>VBZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>considering</td>\n",
       "      <td>VBG</td>\n",
       "      <td>VBG</td>\n",
       "      <td>VBG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>initiating</td>\n",
       "      <td>VBG</td>\n",
       "      <td>VBG</td>\n",
       "      <td>VBG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>a</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Section</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>232</td>\n",
       "      <td>CD</td>\n",
       "      <td>CD</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>investigation</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>into</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>national</td>\n",
       "      <td>JJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>security</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>impact</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>neodymium</td>\n",
       "      <td>NN</td>\n",
       "      <td>JJ</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>magnet</td>\n",
       "      <td>NN</td>\n",
       "      <td>NN</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>imports</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>used</td>\n",
       "      <td>VBN</td>\n",
       "      <td>VBN</td>\n",
       "      <td>VBN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>in</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>motors</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>and</td>\n",
       "      <td>CC</td>\n",
       "      <td>CC</td>\n",
       "      <td>CC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>other</td>\n",
       "      <td>JJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>industrial</td>\n",
       "      <td>JJ</td>\n",
       "      <td>JJ</td>\n",
       "      <td>JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>applications</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "      <td>NNS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>,</td>\n",
       "      <td>,</td>\n",
       "      <td>,</td>\n",
       "      <td>Punctuation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>which</td>\n",
       "      <td>WDT</td>\n",
       "      <td>WDT</td>\n",
       "      <td>WDT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "      <td>DT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>United</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>States</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNPS</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>largely</td>\n",
       "      <td>RB</td>\n",
       "      <td>RB</td>\n",
       "      <td>RB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>obtains</td>\n",
       "      <td>VBZ</td>\n",
       "      <td>VBZ</td>\n",
       "      <td>VBZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>from</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "      <td>IN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>China</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "      <td>NNP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>.</td>\n",
       "      <td>.</td>\n",
       "      <td>.</td>\n",
       "      <td>Punctuation</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Word.news Spacy.tag nltk.tag   Manual.tag\n",
       "0       Officials       NNS      NNS          NNS\n",
       "1            also        RB       RB           RB\n",
       "2            said       VBD      VBD          VBD\n",
       "3             the        DT       DT           DT\n",
       "4      Department       NNP      NNP          NNP\n",
       "5              of        IN       IN           IN\n",
       "6        Commerce       NNP      NNP          NNP\n",
       "7              is       VBZ      VBZ          VBZ\n",
       "8     considering       VBG      VBG          VBG\n",
       "9      initiating       VBG      VBG          VBG\n",
       "10              a        DT       DT           DT\n",
       "11        Section        NN       NN           NN\n",
       "12            232        CD       CD           JJ\n",
       "13  investigation        NN       NN           NN\n",
       "14           into        IN       IN           IN\n",
       "15            the        DT       DT           DT\n",
       "16       national        JJ       JJ           JJ\n",
       "17       security        NN       NN           JJ\n",
       "18         impact        NN       NN           NN\n",
       "19             of        IN       IN           IN\n",
       "20      neodymium        NN       JJ           JJ\n",
       "21         magnet        NN       NN           JJ\n",
       "22        imports       NNS      NNS          NNS\n",
       "23           used       VBN      VBN          VBN\n",
       "24             in        IN       IN           IN\n",
       "25         motors       NNS      NNS          NNS\n",
       "26            and        CC       CC           CC\n",
       "27          other        JJ       JJ           JJ\n",
       "28     industrial        JJ       JJ           JJ\n",
       "29   applications       NNS      NNS          NNS\n",
       "30              ,         ,        ,  Punctuation\n",
       "31          which       WDT      WDT          WDT\n",
       "32            the        DT       DT           DT\n",
       "33         United       NNP      NNP          NNP\n",
       "34         States       NNP     NNPS          NNP\n",
       "35        largely        RB       RB           RB\n",
       "36        obtains       VBZ      VBZ          VBZ\n",
       "37           from        IN       IN           IN\n",
       "38          China       NNP      NNP          NNP\n",
       "39              .         .        .  Punctuation"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#A display to compare the automatic pos tags with the manual tagging side by side.\n",
    "manaul_tag_list = [token for word, token in manual_tag] #Extract pos tag from manually tagged text\n",
    "news_tag_list = [word for word, token in manual_tag] #Extract words of the news text that were tagged\n",
    "nltk_tag_list = [token for word, token in nltk.pos_tag(default_wt(news))] #Extract nltk pos tags\n",
    "spacy_tag_list = [token.tag_ for token in nlp(news)] #Extract Spacy pos tags\n",
    "\n",
    "#We now make a dataframe to compare those tags\n",
    "pos_tagged_df = pd.DataFrame({'Word.news': news_tag_list, 'Spacy.tag':spacy_tag_list, 'nltk.tag': nltk_tag_list, 'Manual.tag': manaul_tag_list})\n",
    "pos_tagged_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e870543",
   "metadata": {},
   "source": [
    "### 3B and C\n",
    "#### Comment on the tags\n",
    "1. Both taggers produced the same result\n",
    "2. More than 90% of the words have the same part of speech tags\n",
    "3. The part of speech tags that are different can also be ambiguous for human beings.\n",
    "4. I do note that if 'Section 232' was treated as a phrase, the taggers would have tagged '232' as an adjective of 'Section' like I did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7df056f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a43398c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "701987bb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fbdbe6e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06c63f30",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}