connector-sets-revised.lyx

#LyX 2.2 created this file. For more info see http://www.lyx.org/
\lyxformat 508
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass article
\begin_preamble
\usepackage{url} 
\usepackage{newunicodechar}
\newunicodechar{」}{~}
\newunicodechar{‐}{-}
\end_preamble
\use_default_options false
\begin_modules
theorems-ams
eqs-within-sections
figs-within-sections
\end_modules
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding utf8
\fontencoding global
\font_roman "times" "default"
\font_sans "helvet" "default"
\font_typewriter "courier" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks true
\pdf_pdfborder true
\pdf_colorlinks true
\pdf_backref false
\pdf_pdfusetitle true
\papersize default
\use_geometry true
\use_package amsmath 2
\use_package amssymb 2
\use_package cancel 1
\use_package esint 0
\use_package mathdots 1
\use_package mathtools 1
\use_package mhchem 0
\use_package stackrel 1
\use_package stmaryrd 1
\use_package undertilde 1
\cite_engine basic
\cite_engine_type default
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 0
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 2
\tocdepth 2
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header

\begin_body

\begin_layout Title
Connector Set Distributions 
\end_layout

\begin_layout Author
Linas Vepstas
\end_layout

\begin_layout Date
\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "50col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Plain Layout
Version 1: 
\begin_inset space \hfill{}
\end_inset

 11 May 2017
\begin_inset Newline newline
\end_inset

Version 2: 
\begin_inset space \hfill{}
\end_inset

 6 August 2017
\begin_inset Newline newline
\end_inset

Version 3: 
\begin_inset space \hfill{}
\end_inset

 12 September 2017
\begin_inset Newline newline
\end_inset

Version 4: 
\begin_inset space \hfill{}
\end_inset

 23 July 2018
\end_layout

\end_inset


\end_layout

\begin_layout Abstract
The goal of unsupervised learning of the grammar of natural language is
 to obtain, by automated means, a valid lexis of dependencies between words.
 In the Link Grammar formalism, these dependencies are represented as 
\begin_inset Quotes eld
\end_inset

connector sets
\begin_inset Quotes erd
\end_inset

 (also termed 
\begin_inset Quotes eld
\end_inset

disjuncts
\begin_inset Quotes erd
\end_inset

) that capture how a word connects to it's neighbors.
\end_layout

\begin_layout Abstract
One well-known model of statistical dependency parsing is the so-called
 MST-parse model of Deniz Yuret.
 It provides reasonable but imperfect results, and has several unsatisfying
 properties: it fails to labelled links and it fails to assign words to
 grammatical classes.
 Both of these limitations can be overcome by extracting connector sets
 from MST linkages, and then performing a statistical averaging over many
 observations.
 Arguments for why this might be a good approach to statistical language
 learning is given in a companion report; this report reports on experimental
 results.
\end_layout

\begin_layout Abstract
A collection of connector sets (disjuncts) and disjunct vectors are obtained
 from a large statistical sampling of MST-parsed sentences.
 This report surveys the general statistical landscape of these structures.
 The ultimate aim is to use such collections as input data to graph algorithms
 that can extract grammatical classes, perform word-sense disambiguation,
 obtain synonym-sets and provide accurate dependency parsing.
\end_layout

\begin_layout Abstract
This is a revised version of earlier reports.
 It provides a better overview, and provides additional results.
 This is not yet the final version; I'm still waiting for additional computation
s to complete.
 
\end_layout

\begin_layout Section
Introduction
\end_layout

\begin_layout Standard
This report characterizes the statistical distribution of word-disjunct
 pairs extracted from a large block of text, using unsupervised natural
 language learning techniques.
\end_layout

\begin_layout Standard
A disjunct is a sequence of words, resembling an N-gram, but containing
 grammatical information.
 The grammatical information in a disjunct is 
\begin_inset Quotes eld
\end_inset

complete
\begin_inset Quotes erd
\end_inset

, in the sense that it is sufficient to construct and accurate parser of
 language.
 This is unlike the situation encountered with Word2Vec
\begin_inset CommandInset citation
LatexCommand cite
key "Mikolov2013a"

\end_inset

 or AdaGram
\begin_inset CommandInset citation
LatexCommand cite
key "Bartunov2015"

\end_inset

, where one can discover 
\begin_inset Quotes eld
\end_inset

important
\begin_inset Quotes erd
\end_inset

 sequences of words, but where the route to the creation of a grammar is
 less clear.
 The algorithm used to extract disjuncts is also completely different from
 the neural-net techniques; it is a graph technique, as opposed to a gradient-de
scent technique.
 Despite this, there are exploitable conceptual similarities in building
 models of natural language.
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2018skippy"

\end_inset

 The goal of this report is to provide a fairly detailed statistical analysis
 of the distribution of disjuncts, thereby characterizing the structure
 of natural language at a grammatical, syntactic level.
\end_layout

\begin_layout Standard
The need for this report is to characterize the structure of the disjunct
 dataset to a sufficient degree that it can be used as input to other graph-base
d methods to extract grammatical classes and to perform word-sense disambiguatio
n.
 These techniques and algorithms are presented in 
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2018skippy"

\end_inset

.
\end_layout

\begin_layout Standard
The origin of the concept of a 
\begin_inset Quotes eld
\end_inset

disjunct
\begin_inset Quotes erd
\end_inset

 comes from the Link Grammar theory of semantic-syntactic parsing.
\begin_inset CommandInset citation
LatexCommand cite
key "Sleator1991,Sleator1993"

\end_inset

.
 Link Grammar is a form of dependency grammar; it is essentially equivalent
 to other forms of dependency grammar, in that Link Grammar dependencies
 can be algorithmically converted into other kinds of dependencies, as well
 as into phrase-structure grammars.
 In Link Grammar, a disjunct is defined as an ordered sequence of connectors;
 those connectors indicate how a word can attach or connect to other words,
 forming links.
 A reasonable conceptual model for a disjunct is to think of it as a jigsaw-puzz
le piece; the connectors correspond to the tabs and slots on the jigsaw-puzzle
 piece.
 Parsing a sentence then consists of assembling jigsaw pieces together,
 in such a way that there are no unconnected tabs or slots at the end of
 the process.
 The links between words then indicate the syntactic and semantic relationships
 between them.
\end_layout

\begin_layout Standard
A similar relationship between words can be obtained by performing a maximum-spa
nning-tree (MST) parse, where links between words are the ones that maximize
 the mutual information between the word-pairs.
\begin_inset CommandInset citation
LatexCommand cite
key "Yuret1998"

\end_inset

 The MST approach to natural language grammar has been well-explored, and
 provides a reasonable accurate model.
 The primary issue is that, when interpreted naively, it does not provide
 any labels that describe the relationships between words, such as 
\begin_inset Quotes eld
\end_inset

subject
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

object
\begin_inset Quotes erd
\end_inset

 relationships, which are considered to be key to the symbolic, linguistic
 grammatical structure of a language.
 
\end_layout

\begin_layout Standard
A bridge can be built between the the unlabeled MST representation of a
 parsed sentence, and the labeled Link Grammar parse of a sentence.
\begin_inset CommandInset citation
LatexCommand cite
key "Goertzel2014"

\end_inset

 This is done by starting with an unlabeled MST parse of a sentence, and
 then applying a label to each link, that label consisting of the word-pair,
 itself.
 In effect, one obtains a jigsaw-puzzle piece, where each tab and slot is
 labeled by the word that the tab/slot is allowed to connect to.
 After accumulating a large database of statistics on such jigsaw-puzzle
 pieces, one can compare them, looking for similarity.
 By clustering together similar pieces, one is effectively creating a dictionary
 or lexis of grammatical categories (parts of speech) together with the
 grammatical information as to how these can be assembled into grammatically
 correct, semantically meaningful sentences.
\end_layout

\begin_layout Standard
This approach appears to be sufficient to distinguish between different
 meanings associated with a word.
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2018stiching"

\end_inset

 The final outcome is not dissimilar to popular neural-net techniques, although
 the path taken is entirely different.
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2018skippy"

\end_inset

 The overall approach of discerning graph structure from large bodies of
 statistical data is hypothesized to be generalizable to domains far outside
 of linguistics, including genomics and proteomics.
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2017sheaves"

\end_inset


\end_layout

\begin_layout Standard
This report examines one specific dataset of word-disjunct pairs.
 It explores the overall distribution of words and disjuncts, which are
 unsurprisingly Zipfian in nature.
 It explores various different measures of similarity between words, such
 as cosine distance and entropic distance, either of which are employable
 for discerning the grammatical classes into which the word-disjunct pairs
 can be assigned.
 In effect, this provides the background needed for understanding how words
 are assigned to grammatical categories, in practice.
 It characterizes the nature and structure of natural language, when viewed
 from the path of disjuncts derived from MST parses derived from the mutual
 information (MI) distribution of word pairs.
 The analysis is similar in spirit to an earlier analysis of the statistics
 of word-pairs.
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2009"

\end_inset


\end_layout

\begin_layout Subsection
Recap
\end_layout

\begin_layout Standard
A detailed description of the process of obtaining (pseudo-)disjuncts from
 large unannotated text corpora is given elsewhere.
 This section provides a short, informal summary.
 
\end_layout

\begin_layout Standard
The story so far: Starting from a large text corpus, the mutual information
 (MI) of word-pairs are counted.
 This MI is used to perform a maximum spanning-tree (MST) parse (of a different
 subset of) the corpus.
 From each parse, a pseudo-disjunct is extracted for each word.
 The pseudo-disjunct is like a real LG disjunct, except that each connector
 in the disjunct is the word at the far end of the link.
\end_layout

\begin_layout Standard
So, for example, in in idealized world, the MST parse of the sentence "Ben
 ate pizza" would produce the parse Ben <–> ate <–> pizza and from this,
 we can extract the pseudo-disjunct (Ben- pizza+) on the word "ate".
 Similarly, the sentence "Ben puked pizza" should produce the disjunct (Ben-
 pizza+) on the word "puked".
 Since these two disjuncts are the same, we can conclude that the two words
 "ate" and "puked" are very similar to each other.
 Considering all of the other disjuncts that arise in this example, we can
 conclude that these are the only two words that are similar.
\end_layout

\begin_layout Standard
Any given word will have many pseudo-disjuncts attached to it.
 Each disjunct has a count of the number of times it has been observed.
 Thus, this set of disjuncts can be imagined to be a vector in a high-dimensiona
l vector space, which each disjunct being a single basis element.
 The similarity of two words can be taken to be the cosine-similarity between
 the disjunct-vectors.
 Other similarities are possible, and it seems that the entropic similarity,
 described below, is superior.
\end_layout

\begin_layout Standard
Equivalently, the set of disjuncts can be thought of as a weighted set:
 each disjunct has a weight, corresponding to the number of times it has
 been observed.
 A weighted set is more or less the same thing as a vector, and these two
 are treated as the same, in what follows.
 Note that the disjunct vectors are sparse: for any given word, almost all
 coefficients will have a count of zero.
 For example, the dataset that will be examined next has over a quarter
 of a million different pseudo-disjuncts in it; most words have fewer than
 a hundred disjuncts on them.
\end_layout

\begin_layout Subsection
Summary of results
\end_layout

\begin_layout Standard
The primary results reported below are these:
\end_layout

\begin_layout Standard
* Most scores and metrics that can be assigned to connector sets give a
 (scale-free) Zipfian ranking distribution, and are thus fairly boring.
 There are some oddities here and there.
\end_layout

\begin_layout Standard
* The greater the average number of observations per disjunct, the more
 grammatically acceptable (accurate) the disjunct seems to be.
 This is good news: it means that the general technique is not generating
 ungrammatical garbage.
\end_layout

\begin_layout Standard
* Connector sets can be given a mutual information score.
 The distribution for the MI scores appears to be Gaussian (
\emph on
i.e
\emph default
.
 a Bell curve).
 This comes as a bit of a surprise.
 I am not aware of what kind of network theory gives a natural rise to Gaussians.
\end_layout

\begin_layout Standard
* The MI score seems to be quite good at identifying words that participate
 in idioms, set phrases and institutional phrases.
\end_layout

\begin_layout Standard
* The average number of connectors per disjunct, which should have indicated
 the part-of-speech that the word belongs to, fails to do this.
 This seems to be due to the fact that the dataset is polluted with lists
 and tables (including tables-of-contents, and indexes), all of which are
 mis-interpreted as sentences by the processing software.
 This causes some very unusual disjuncts to be constructed.
\end_layout

\begin_layout Standard
* In the earlier sample, derived from Wikipedia, it became clear that there
 were very few verbs that aren't relationship verbs.
 Wikipedia articles describe concepts and events.
 The relationship between these require the copula and other relationship
 verbs: 
\begin_inset Quotes eld
\end_inset

is
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

has
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

was
\begin_inset Quotes erd
\end_inset

.
 Wikipedia is almost completely devoid of narrative verbs: 
\begin_inset Quotes eld
\end_inset

ran
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

jumped
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

hit
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

ate
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

thought
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

took
\begin_inset Quotes erd
\end_inset

.
 Thus, we discern two very different styles of human communication: the
 exchange of facts, and the exchange of stories.
 Narratives contain a far richer selection of verbs, and thus, for language
 learning, a text corpus of narratives is required.
 Ideally, this would be from young-adult literature, which is a bit more
 direct in its kinesthetic content than adult literature might be.
\end_layout

\begin_layout Standard
* Cosine similarity applied to connector sets seems to be an effective way
 of determining the grammatical similarity of words.
 Yet, it is not so unambiguously great, that other kinds of measures shouldn't
 be contemplated.
\end_layout

\begin_layout Standard
* Entropic similarity, essentially, a form of symmetric mutual information
 between words, appears to be an even better similarity measure.
 It is particularly appealing because it is naturally additive, and thus
 fits naturally into network theories derived from thermodynamic partition
 functions.
\end_layout

\begin_layout Subsection
Revision History
\end_layout

\begin_layout Standard
The original version of this report, dated 7 May 2017, was prepared on a
 painfully small dataset, which also (may have?) incorporated a fatal bug
 in the disjunct code: disjuncts were being assembled incorrectly, due to
 a reversed sign in the MI calculations.
 This bug was eventually uncovered, and so it seemed best to entirely discard
 the initial analysis, and instead repeat it with a newer and larger dataset
 that was correctly assembled.
 The revised analysis was done mostly in July 2017.
 Sadly, the discovery of this bug required that multiple large datasets
 be discarded and reconstructed.
 This caused a month of effort to be lost.
 Version 1 can be accessed by digging in git, and pulling up commit 27a66643a52c
0985adc5b38caf94fc25f5e2e684 (or maybe a bit earlier, circa late June 2017,
 as that is when the bug was spotted.).
\end_layout

\begin_layout Standard
Simultaneously, there was a lot of confusion about the efficacy of the cosine
 similarity measure.
 Initial work on cosine similarity used a filtered dataset, with the goal
 of filtering to reduce 
\begin_inset Quotes eld
\end_inset

noise
\begin_inset Quotes erd
\end_inset

 in the dataset, as well as to manage dataset size.
 It turns out that this filtering also had the undesired side-effect of
 destroying much of the 
\begin_inset Quotes eld
\end_inset

signal
\begin_inset Quotes erd
\end_inset

 as well – it rendered many grammatically unrelated words to be judged to
 be very similar.
 Between the accidental sign reversal, and the excessively strong data cuts,
 it was all very confusing, and has taken another month to recover from
 this – I'm back to where I was in May, just older and wiser, now.
 Versions 2 and 3 of this report provide the status quo.
 They can be found at commit bc48712d3c2ab71383961f4a50dd6b3b4c6fd75f in
 git.
\end_layout

\begin_layout Standard
Version 4 of this report expands it by considering an entropic similarity
 between words, intended to supplant the cosine similarity as an effective
 graph metric.
 The entropic similarity is a form of symmetric mutual information between
 words; it is given this oddball name so as to avoid confusion with a variety
 of other contexts in which mutual information appears.
 And so, Version 4 provides additional statistical analysis comparing the
 entropic similarity to the cosine similarity.
\end_layout

\begin_layout Section
Dataset Characterization
\end_layout

\begin_layout Standard
Some terminology and notation are introduced next, followed by a characterizatio
n of the dataset.
 This is followed by a statistical analysis of the word-disjunct pairs.
 The analysis of word-similarity is left for a later section.
\end_layout

\begin_layout Subsection
Terminology
\end_layout

\begin_layout Standard
It is useful to introduce some notation for counting words, disjuncts, and
 connectors.
 Let 
\begin_inset Formula $N(w)$
\end_inset

 be the number of times that the word 
\begin_inset Formula $w$
\end_inset

 has been observed, in the dataset.
 Let 
\begin_inset Formula $N(w,d)$
\end_inset

 be the number of times that the disjunct 
\begin_inset Formula $d$
\end_inset

 has been observed on word 
\begin_inset Formula $w$
\end_inset

.
 The pair 
\begin_inset Formula $(w,d)$
\end_inset

 is referred to as a 
\begin_inset Quotes eld
\end_inset

connector set
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

cset
\begin_inset Quotes erd
\end_inset

 in the text below.
 Thus, for a word 
\begin_inset Formula $w$
\end_inset

, there is a set 
\begin_inset Formula $(w,*)=\left\{ (w,d)|N(w,d)>0\right\} $
\end_inset

 of associated csets, called the 
\begin_inset Quotes eld
\end_inset

support
\begin_inset Quotes erd
\end_inset

 of the word.
 The size of this set can be written using the standard notation for set-sizes
 as 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

.
 Similarly, a disjunct 
\begin_inset Formula $d$
\end_inset

, is supported by the set 
\begin_inset Formula $(*,d)=\left\{ (w,d)|N(w,d)>0\right\} $
\end_inset

 of associated csets.
 
\end_layout

\begin_layout Standard
The primary contents of the database are the counts 
\begin_inset Formula $N(w,d)$
\end_inset

 and everything else of interest in this section can be obtained from this.
 Note that 
\begin_inset Formula $N(w,d)$
\end_inset

 can be understood as a matrix, where the disjuncts identify columns, and
 the words identify rows.
 In general, this is a very sparse matrix: the number of non-zero entries
 
\begin_inset Formula $\left|(*,*)\right|$
\end_inset

 is far less than the number of rows times the number of columns.
\end_layout

\begin_layout Standard
Every time a word is observed in an MST parse, a disjunct is extracted for
 it; thus, word observations and disjunct observations are on one-to-one
 correspondence.
 In notation:
\begin_inset Formula 
\[
\sum_{d}N(w,d)=N(w,*)=N(w)
\]

\end_inset

Similarly, the total number of times that a disjunct was observed is just
\begin_inset Formula 
\[
N(*,d)=\sum_{w}N(w,d)
\]

\end_inset


\end_layout

\begin_layout Standard
Frequencies can be obtained by dividing by the total number of observations,
 so that 
\begin_inset Formula $p(w,d)=N(w,d)/N(*)$
\end_inset

 and 
\begin_inset Formula $p(w)=N(w)/N(*)$
\end_inset

 with 
\begin_inset Formula $N(*)=\sum_{w}N(w)$
\end_inset

 the total number of observations of words.
\end_layout

\begin_layout Standard
A single disjunct is always composed of a fixed number of connectors, independen
tly of any observations; let 
\begin_inset Formula $C(d,c)$
\end_inset

 be the number of times that connector 
\begin_inset Formula $c$
\end_inset

 appears in disjunct 
\begin_inset Formula $d$
\end_inset

.
 Note that 
\begin_inset Formula $C(d,c)$
\end_inset

 is almost always either zero or one; however, a connector can appear more
 than once in a disjunct, so this count can rise to 2 or 3 or higher.
 The wild-card sum 
\begin_inset Formula $C(d,*)=\sum_{c}C(d,c)$
\end_inset

 is the total number of connectors in the disjunct; it is the vertex degree
 of all edges connecting to that disjunct.
 It is also useful to define 
\begin_inset Formula $C(d,+)$
\end_inset

 and 
\begin_inset Formula $C(d,-)$
\end_inset

 as the total number of right-linking and left-linking connectors.
\end_layout

\begin_layout Subsection
Dataset characterization
\end_layout

\begin_layout Standard
The rest of this report is based on a single dataset, called 'en_pairs_rfive_mtw
o'.
 It was built by performing MST parsing of of text from tranche-1 and 2,
 using word-pair statistics gathered from random parses of text from tranche-1,2
,3,4,5.
\end_layout

\begin_layout Standard
These 
\begin_inset Quotes eld
\end_inset

tranches
\begin_inset Quotes erd
\end_inset

 consist of unannotated text corpora downloaded from the net; the specific
 download scripts that were used are the 'download.sh' scripts located in
 github, at https://github.com/opencog/learn/download.
 Word-pair statistics are obtained by 
\begin_inset Quotes eld
\end_inset

random-tree parsing
\begin_inset Quotes erd
\end_inset

.
 A random-tree parse is generated by creating a random planar parse tree
 for a sentence; it is planar in the sense that no links cross.
 It is random in the sense that a uniform distribution is assumed on the
 space of all possible planar parses.
 For each link in such a parse, one simply records the two words at each
 end of the link, and increments the count of that particular word pair
 by one.
 The random-tree method for obtaining word-pair counts results in subtly
 different statistics than what would be obtained by a sliding window of
 some fixed width.
 In particular, the random-tree method will occasionally count long-distance
 links, longer than what would be seen in a sliding window.
 The difference in statistics between random-tree and sliding-window methods
 has not been characterized.
 It is presumed to be minor, and probably immaterial to subsequent results.
\end_layout

\begin_layout Standard
The word-pair dataset, and other similar datasets are summarized in the
 language learning diary
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2013"

\end_inset

, in the section titled 
\begin_inset Quotes erd
\end_inset

Dataset report 3 June 2017
\begin_inset Quotes erd
\end_inset

.
 It is repeated here, as it gives a hint of the foundation for the MST parses.
 The column labels in the table below are the same as those explained there.
 They are:
\end_layout

\begin_layout Description
Size The dimensions of the array.
 This is the number of unique, distinct words observed occurring on the
 left-side of a word pair, times the number of words occurring on the right.
 We expect the dimensions to be approximately equal, as most words will
 typically occur on both the left and right side of a pair.
 
\end_layout

\begin_layout Description
Pairs The total number of distinct word-pairs observed.
\end_layout

\begin_layout Description
Obs'ns The total number of observations of these pairs.
 Most pairs will be observed more than once.
 Distributions are typically Zipfian.
\end_layout

\begin_layout Description
Obs/pr The average number of times each word-pair was observed.
\end_layout

\begin_layout Description
Entropy The total entropy of these pairs in this dataset.
 Denote a word-pair as 
\begin_inset Formula $(w_{L},w_{R})$
\end_inset

 and 
\begin_inset Formula $p(w_{L},w_{R})$
\end_inset

 as the probability (normalized count) for observing the word-pair.
 The total entropy is then 
\begin_inset Formula $H=-\sum_{w_{L},w_{R}}p(w_{L},w_{R})\log_{2}p(w_{L},w_{R})$
\end_inset

.
\end_layout

\begin_layout Description
MI The total mutual information for the pairs in this dataset, defined as:
 
\begin_inset Formula $MI=\sum_{w_{L},w_{R}}p(w_{L},w_{R})\log_{2}\left[p(w_{L},w_{R})/p(w_{L},*)p(*,w_{R})\right]$
\end_inset


\end_layout

\begin_layout Standard
The 'en_pairs_rfive' word-pair dataset can then be summarized as:
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="2" columns="7">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Size
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Pairs
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Obs'ns
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Obs/pr
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Dataset
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
839K
\begin_inset Formula $\times$
\end_inset

851K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30.1M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.35G
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
44.9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18.54
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.84
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_rfive
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\noindent
The diary provides reports for other word-pair datasets, of varying sizes,
 and some for other languages.
 Perhaps the most interesting and surprising result is that the MI value
 is nearly independent of the size of the dataset, and the language, varying
 between about 1.8 and 3.0.
\end_layout

\begin_layout Standard
A collection of pairs can be thought of as a sparse matrix; the rows of
 the matrix can be taken to be vectors, as can the columns.
 As vectors, they can be thought of as inhabiting some Banach space, having
 an 
\begin_inset Formula $l_{p}$
\end_inset

-norm.
 Given a vector 
\begin_inset Formula $\vec{v}=\sum_{n}a_{n}\hat{e}_{n}$
\end_inset

 decomposed into basis elements 
\begin_inset Formula $\hat{e}_{n}$
\end_inset

 and real-number-valued coefficients 
\begin_inset Formula $a_{n}$
\end_inset

, the 
\begin_inset Formula $l_{p}$
\end_inset

-norm is defined as
\begin_inset Formula 
\[
l_{p}\left(\vec{v}\right)=\sqrt[p]{\sum_{n}\left|a_{n}\right|^{p}}
\]

\end_inset

For 
\begin_inset Formula $p=2$
\end_inset

, this is simply the (Euclidean) 
\begin_inset Quotes eld
\end_inset

length
\begin_inset Quotes erd
\end_inset

 of the vector.
 The 
\begin_inset Formula $p=1$
\end_inset

 norm is sometimes called the 
\begin_inset Quotes eld
\end_inset

Manhattan distance
\begin_inset Quotes erd
\end_inset

; here called simply the 
\begin_inset Quotes eld
\end_inset

count
\begin_inset Quotes erd
\end_inset

.
 For 
\begin_inset Formula $p=0$
\end_inset

, one is simply measuring the support of the vector; that is, the number
 of coefficients 
\begin_inset Formula $a_{n}$
\end_inset

 that are non-zero.
 In what follows, the 
\begin_inset Formula $\hat{e}_{n}$
\end_inset

 correspond to words, either to words on the left of a pair, or on the right.
 Thus, the table caption below is given by:
\end_layout

\begin_layout Description
Size The left and right dimensions, as before.
 
\emph on
Videlicet
\emph default
, the number of unique, distinctly different words observed on the left
 and the right side of a pair.
 Viewed as a matrix, this is the number of columns and rows in the matrix.
\end_layout

\begin_layout Description
Support The support is the average number of word-pairs that a word participates
 in (on the left, or on the right).
 Viewed as a matrix, this is the average number of non-zero entries in each
 row or column.
 Viewed as (row or column) vectors, this is the 
\begin_inset Quotes eld
\end_inset

support
\begin_inset Quotes erd
\end_inset

 of a (row or column) vector.
 Formally, this is the 
\begin_inset Formula $l_{0}$
\end_inset

 norm of each vector: 
\begin_inset Formula $\left|(w_{L},*)\right|=\sum_{w_{R}}\left(0<N(w_{L},w_{R})\right)$
\end_inset

 and likewise 
\begin_inset Formula $\left|(*,w_{R})\right|=\sum_{w_{L}}\left(0<N(w_{L},w_{R})\right)$
\end_inset

.
\end_layout

\begin_layout Description
Count The count is the average number of observations that a word-pair was
 observed, for a given word.
 Viewed as a matrix, this is the average value of each non-zero entry (averaged
 over rows, or columns).
 Viewed as vectors, this is the 
\begin_inset Formula $l_{1}$
\end_inset

 norm divided by the 
\begin_inset Formula $l_{0}$
\end_inset

 norm.
 The 
\begin_inset Formula $l_{1}$
\end_inset

 norm is just the wild-card counts 
\begin_inset Formula $N(w_{L},*)$
\end_inset

 and 
\begin_inset Formula $N(*,w_{R})$
\end_inset

, where as always, the wild-card counts are defined as 
\begin_inset Formula $N(w_{L},*)=\sum_{w_{R}}N(w_{L},w_{R})$
\end_inset

.
 The count shown in the table is then the average count: 
\begin_inset Formula $N(w_{L},*)/\left|(w_{L},*)\right|$
\end_inset

 for the rows, and likewise for the columns.
\end_layout

\begin_layout Description
Length The length is the average length of the row and column vectors.
 This is the 
\begin_inset Formula $l_{2}$
\end_inset

 norm divided by the 
\begin_inset Formula $l_{0}$
\end_inset

 norm.
 The 
\begin_inset Formula $l_{2}$
\end_inset

 norm is just the standard concept of the length of a vector in Euclidean
 space.
 Here, 
\begin_inset Formula $L(w_{L},*)=\sqrt{\sum_{w_{R}}N^{2}(w_{L},w_{R})}$
\end_inset

, and likewise 
\begin_inset Formula $L(*,w_{R})=\sqrt{\sum_{w_{L}}N^{2}(w_{L},w_{R})}$
\end_inset

.
 The length is interesting, because it 
\begin_inset Quotes eld
\end_inset

penalizes
\begin_inset Quotes erd
\end_inset

 word-pairs with only a small number of counts.
 The act of squaring the count has the effect of giving much higher 
\begin_inset Quotes eld
\end_inset

confidence
\begin_inset Quotes erd
\end_inset

 to large observation counts: a word-pair observed twice as often is given
 four times the credit.
 The length shown in this table is the 
\begin_inset Quotes eld
\end_inset

average
\begin_inset Quotes erd
\end_inset

 length: it is 
\begin_inset Formula $L(w_{L},*)/\left|(w_{L},*)\right|$
\end_inset

 for the rows, and likewise for the columns.
\end_layout

\begin_layout Standard
The support and count for the pairs are given below.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="3" columns="9">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Size
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Support
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Count
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Length
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Dataset
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
L
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
R
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
L
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
R
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
L
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
R
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
L
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
R
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Name
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
839K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
851K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
80.6K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
80.6K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
249
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
230
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28.2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_rfive
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The disjunct dataset is obtained by collecting statistics on MST parses
 obtained from the tables above.
 MST parsing is performed by considering the set of all-possible planar
 trees connecting the words in a sentence.
 One then selects the one single tree which maximizes the sum of the MI
 values between all word-pairs.
 This is the MST parse.
 Given an MST parts, one then 
\begin_inset Quotes eld
\end_inset

cuts
\begin_inset Quotes erd
\end_inset

 each link, and labels the cut end of the link with the word that it was
 previously connected to.
 The disjunct is then the collection of these cut, labeled half-edges.
 The collection is ordered, left-to-right, and a plus/minus indicator is
 used to half-edge as having been originally left going (minus) or right-going
 (plus).
 A word-disjunct pair is then the word, and the dangling links that go with
 it; the word can be thought of as sitting in the middle of the left-going
 connectors, and the right-going connectors.
 (In this sense, a word-disjunct pair resembles an N-gram, except that N
 is not fixed, and there might be skipped intermediate words).
 The dataset is created by accumulating counts for a large number of these
 word-disjunct pairs.
\end_layout

\begin_layout Standard
Just as before, the dataset of pair-counts can be viewed as a sparse matrix.
 The rows correspond to words, the columns to disjuncts.
 The table below summarizes this dataset.
 The table caption is the same as before, with the following additional
 values being reported:
\end_layout

\begin_layout Description
\begin_inset Formula $H_{left}$
\end_inset

,
\begin_inset Formula $H_{right}$
\end_inset

 The left and right entropies.
 These are defined as 
\begin_inset Formula $H_{right}=-\sum_{w}p(w,*)\log_{2}p(w,*)$
\end_inset

 and 
\begin_inset Formula $H_{left}=-\sum_{d}p(*,d)\log_{2}p(*,d)$
\end_inset

.
 Note that 
\begin_inset Formula $MI=H-H_{left}-H_{right}$
\end_inset

 holds, by definition.
 The left and right entropies were not reported for the word-pairs table,
 because these two are nearly equal, and are equal to half the difference
 between the entropy and the MI.
\end_layout

\begin_layout Standard
The table is then characterized as:
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="2" columns="9">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Size
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Csets
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Obs'ns
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Ob/cs
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $H_{left}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $H_{right}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Dataset
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
137K
\begin_inset Formula $\times$
\end_inset

6.24M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8.63M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18.5M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20.96
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9.71
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.90
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_rfive_mtwo
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Subsection
Dataset Analysis
\end_layout

\begin_layout Standard
The word-pair dataset contains 851964 words.
 Not all of these were observe during the subsequent MST parsing, as the
 size of the MST-parsed corpus was smaller.
 However, since the MST corpus was a subset of the word-pair corpus, it
 does not contain any words that were not previously observed.
\end_layout

\begin_layout Standard
The MST dataset has 137078 words that have disjuncts attached to them.
 These words have been observed a total of 18489594  times, for an average
 of 18489594/137078 = 21.70 observations per word.
 This dataset contains 6239997 different, unique disjuncts, for an average
 of 18489594 / 6239997 = 2.963 observations per disjunct.
 
\end_layout

\begin_layout Standard
The period (the punctuation mark denoting end-of-sentence) appears 849354.0
 times, suggesting that this many sentences were observed.
 Each sentence thus has an average of 18489594 / 849354 = 21.77 words per
 sentence.
 The precise statistics, however, are governed by the end-of-sentence detector,
 which ignores common abbreviations such as 
\begin_inset Quotes eld
\end_inset

Mr.
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

Mrs.
\begin_inset Quotes erd
\end_inset

, 
\emph on
etc
\emph default
.
 but may make mistakes for uncommon abbreviations, and for non-prose text,
 such as tables and charts.
\end_layout

\begin_layout Standard
The dataset contains 6239997 unique disjuncts, for an average of 18489594
 / 6239997 = 2.96 observations per disjunct.
 This last number that makes this dataset feel thin and sparse.
 Its not clear how accurate that perception is: an earlier dataset was about
 one-tenth to one-twentieth the size, in the number of words, disjuncts
 and observations, yet it had a ratio of 1.5 observations per cset.
 That is, making more than ten times the number of observations only doubled
 the observations per cset.
 This suggests that the distribution is Zipfian, so that large increases
 in observations result in only small changes to averages.
\end_layout

\begin_layout Standard
The dataset is sparse, in a completely different sense from the above.
 Viewing 
\begin_inset Formula $N(w,d)$
\end_inset

 as a matrix whose size is 
\begin_inset Formula $851964\times6239997$
\end_inset

, there are only a very small number of counts that are non-zero.
 Specifically, the fraction is 
\begin_inset Formula $8629163/(851964\times6239997)=1.623\times10^{-6}$
\end_inset

.
 The sparsity of this matrix can be defined as 
\begin_inset Formula $-\log_{2}$
\end_inset

 of this number; here, it is 
\begin_inset Formula $-\log_{2}1.623\times10^{-6}\approx16.60$
\end_inset

.
 The sparsity appears to increase with the number of observations: the previous,
 ten-times-smaller dataset had a sparsity of 15.
\end_layout

\begin_layout Standard
The total word-entropy for the dataset is defined as
\begin_inset Formula 
\[
H_{word}=-\sum_{w}p(w)\log_{2}p(w)
\]

\end_inset

and was measured to be 
\begin_inset Formula $H_{word}=9.71$
\end_inset

 bits.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
This and the following entropies were measured with the word-entropy-bits,
 disjunct-entropy-bits, etc.
 functions in 'disjunct-stats.scm' Alternately, the 'print-matrix-summary-report'
 function now reports this.
\end_layout

\end_inset

 The total entropy is much larger.
 It is defined as
\begin_inset Formula 
\[
H_{total}=-\sum_{w,d}p(w,d)\log_{2}p(w,d)
\]

\end_inset

and is measured to be 
\begin_inset Formula $H_{total}=20.96$
\end_inset

 bits.
 The disjunct entropy is dual to the word entropy:
\begin_inset Formula 
\[
H_{disjunct}=-\sum_{d}p(*,d)\log_{2}p(*,d)
\]

\end_inset

and is measured to be 
\begin_inset Formula $H_{disjunct}=19.14$
\end_inset

 bits.
 The total mutual information between the words and disjuncts is then 
\begin_inset Formula 
\[
MI_{cset}=\sum_{w,d}p(w,d)\log_{2}\frac{p(w,d)}{p(*,d)p(w,*)}=H_{word}+H_{disjunct}-H_{total}
\]

\end_inset

and is measured to be 
\begin_inset Formula $MI_{cset}=7.897$
\end_inset

 bits.
\end_layout

\begin_layout Standard
It is interesting to compare 
\begin_inset Formula $MI_{cset}$
\end_inset

 to 
\begin_inset Formula $MI_{word-pair}$
\end_inset

.
 The former is much larger, by just over 6 bits.
 There are several ways to think about this.
 Clearly, the connector-set database contains 
\begin_inset Quotes eld
\end_inset

more information
\begin_inset Quotes erd
\end_inset

; one might say that it is a distillation, an extraction.
 It is an amplification, but done so that the signal-to-noise ratio is much
 improved.
 MST parsing is clearly picking out some sort of structure from natural
 language.
 Consider it this way: 
\begin_inset Formula $2^{1.84}=3.58$
\end_inset

 and so the word-pair dataset contains on average 1.84 more bits of information
 than what one would expect to observe in a completely random, uniformly-distrib
uted word-salad, for which we expect 
\begin_inset Formula $MI=0$
\end_inset

.
 The word-disjunct dataset takes a large step in extracting signal from
 the noise: one gets 
\begin_inset Formula $7.897-1.84=6.057$
\end_inset

 more bits of information, for a factor of 
\begin_inset Formula $2^{6.057}=66.5$
\end_inset

.
 That is a respectable amount of noise reduction.
\end_layout

\begin_layout Section
Distributions
\end_layout

\begin_layout Standard
The previous section characterized the dataset in terms of its total size,
 entropy, and related single-number values that can be extracted from it.
 More interesting is the distribution of columns and rows, that is, the
 distribution of the words and the disjuncts.
 These result in various graphs, almost all of which are Zipfian in nature;
 and some that very much are not.
 The distributions obtained here can be contrasted with the distributions
 obtained from word-pair counts.
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2009"

\end_inset

 They are similar in many ways, and also strikingly different.
\end_layout

\begin_layout Standard
In the following, the term 
\begin_inset Quotes eld
\end_inset

connector set
\begin_inset Quotes erd
\end_inset

 is taken as a synonym for 
\begin_inset Quotes eld
\end_inset

disjunct
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Subsection
Connector-set distribution
\end_layout

\begin_layout Standard
Some connector-sets will be observed far more often than others.
 Likewise for the two sides of the connector-set: some words will have far
 more observations, and some disjuncts will be seen more often.
\end_layout

\begin_layout Standard
Two graphs, dual to one-another.
 The one on the left shows 
\begin_inset Formula $N(w,*)$
\end_inset

, ranked by count.
 The one on the right shows 
\begin_inset Formula $N(*,d)$
\end_inset

, also ranked.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Obtained by running (print-ts-rank sorted-word-obs outport) from the disjunct-st
ats.scm file, on the en_pairs_rfive_mtwo database.
 The second one prints sorted-dj-obs.
 The graphs generated with ranked.gplot
\end_layout

\end_inset

.
 The first follows the canonical Zipf distribution.
 The green line is an eyeballed, approximate fit, of exponent -1.1.
 The second has an exponent of about -0.85.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-word-obs.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename en-dj/ranked-dj-obs.eps
	width 50text%

\end_inset


\end_layout

\begin_layout Standard
The first ten words in the word ranking are: "LEFT-WALL" "," "." "the" "and"
 "to" "of" "a" """ "in".
 This is the ranking of how often these words appear, overall, in the MST-parsed
 corpus.
 The number of connections to LEFT-WALL should be equal to the number of
 sentences in the corpus, as the parser is set up to make one LEFT-WALL
 connection to a sentence.
 Most sentences will end in a period; some with question marks of other
 punctuation.
 Commas and the word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 can appear more than once in a sentence.
 The frequent occurrence of the straight double-quote mark is due to the
 fact that the corpus is heavily weighted with dialog: i.e.
 with fictional novels, where the characters are speaking a lot.
\end_layout

\begin_layout Standard
This list is repeated in the table below.
 The support is 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

, that is, the number of different kinds of disjuncts observed for that
 word.
 The count is 
\begin_inset Formula $N(w,*)$
\end_inset

, that is, the total number of times those disjuncts have been observed
 for that word.
 The frequency is just the count divided by 18489594.
 The length is 
\begin_inset Formula $\mbox{len}(w,*)=\sqrt{\sum_{d}N^{2}(w,d)}$
\end_inset

, that is, the root of the sum of the squares of the observations.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
A printing utility for these three is `show-counts` in the `disjunct-stats.scm`
 file.
\end_layout

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="6">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
support
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
frequency
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $-\log_{2}$
\end_inset

frequency
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
length
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
LEFT-WALL
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
64215
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
972963
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.05262
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.248
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
122353.9
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
,
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
243987
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
957593
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.05179
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.271
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25475.4
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
106195
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
849354
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.04594
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.444
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
55168.0
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
215324
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
727027
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.03932
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.669
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9264.2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
126861
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
420942
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.02277
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.457
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28694.7
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
117110
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
401967
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.02174
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.523
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11480.0
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
of
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
108951
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
371211
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.02008
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.638
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11047.5
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
102720
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
289631
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.01566
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.996
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6855.1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
"
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
51289
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
256785
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.01389
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6.170
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21388.8
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
64011
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
208745
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.01129
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6.469
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14758.1
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The main point of this table is to demonstrate the log-likelihood column.
 At this point, these numbers won't seem to have much meaning; however,
 they provide an overall scale that will be seen, repeatedly, in the analysis
 below.
 The range of magnitudes – 4 to 7 – is no accident, and similar ranges will
 be seen later.
 
\end_layout

\begin_layout Standard
The first ten pseudo-disjuncts in the disjunct-ranking are ""+" ",-" "the+"
 "He+" "The+" "the-" "LEFT-WALL-" "I+" "“+" "to-".
 The meaning of the plus and minus signs was explained above; but to recap:
 the disjunct 
\begin_inset Quotes eld
\end_inset

xxx+
\begin_inset Quotes erd
\end_inset

 means that there are many words that expect to be followed by the word
 
\begin_inset Quotes eld
\end_inset

xxx
\begin_inset Quotes erd
\end_inset

 (on the right).
 The disjunct 
\begin_inset Quotes eld
\end_inset

the-
\begin_inset Quotes erd
\end_inset

 means that there are many words that want to link to the word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 on the left.
 This is grammatically correct: 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 is a determiner, and it is always the dependent of some noun.
 The disjunct 
\begin_inset Quotes eld
\end_inset

The+
\begin_inset Quotes erd
\end_inset

 is at first appears to be grammatical garbage/nonsense: it states that
 there are many words that want to link to the word 
\begin_inset Quotes eld
\end_inset

The
\begin_inset Quotes erd
\end_inset

 on the right.
 Naively, this is never correct for English; determiners always precede
 the noun that they modify.
 The capitalization gives away what is really happening: the word 
\begin_inset Quotes eld
\end_inset

The
\begin_inset Quotes erd
\end_inset

 is a sentence opener, and it is being linked by the LEFT-WALL, indicating
 the start of the sentence; 
\emph on
ergo
\emph default
, it is lining backwards.
 Similar remarks apply to ""+" "the+" "He+" "I+" "“+": Clearly, the capitalized
 
\begin_inset Quotes eld
\end_inset

He
\begin_inset Quotes erd
\end_inset

 is a sentence opener, and 
\begin_inset Quotes eld
\end_inset

I
\begin_inset Quotes erd
\end_inset

 is plausibly so.
 The two different styles of quotation marks (symmetrically vertical and
 right-leaning) open up dialog in fictional novels, which make up a large
 portion of the corpus.
 
\end_layout

\begin_layout Standard
The above avoids the question of whether its is syntactically correct to
 link the LEFT-WALL to 
\begin_inset Quotes eld
\end_inset

The
\begin_inset Quotes erd
\end_inset

.
 This is determined not by raw frequency counts, but by mutual information.
 This is explored later.
 At this point we can only say that such a linking is frequent, and cannot
 judge whether it is correct.
\end_layout

\begin_layout Standard
The ranking of connector sets is shown below.
 It's a graph of the ranked counts 
\begin_inset Formula $N(w,d)$
\end_inset

.
 Recall that we define a 
\begin_inset Quotes eld
\end_inset

connector set
\begin_inset Quotes erd
\end_inset

 as the pairing of a word, and one particular disjunct that is associated
 with that word.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Graph of sorted-cset-obs, 
\emph on
op cit
\emph default
.
 
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-cset-obs.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
The top-ten connector-sets are "LEFT-WALL: He+;" "LEFT-WALL: The+;" "LEFT-WALL:
 “+;" "LEFT-WALL: "+;" ".: "+;" "LEFT-WALL: I+;" "and: ,-;" "LEFT-WALL: It+;"
 "LEFT-WALL: She+;" ".: ”+;"
\end_layout

\begin_layout Standard
These are hard to read, so, decoded: the four are connectors from the left
 side of the sentence to the words 
\begin_inset Quotes eld
\end_inset

He
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

The
\begin_inset Quotes erd
\end_inset

, and two different styles of double-quote marks: a right-leaning double-quote,
 and a vertical double-quote.
 This was commented on before: the corpus has many novels, and so many sentences
 will begin with quotes.
 Next comes a period which links to a double-quote on it's right.
 Last comes period followed by a leaning double-quote.
 Clearly, this is expected in the corpus.
 Also visible are the sentence openers 
\begin_inset Quotes eld
\end_inset

I
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

It
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

She
\begin_inset Quotes erd
\end_inset

.
 In that list is the word 
\begin_inset Quotes eld
\end_inset

and
\begin_inset Quotes erd
\end_inset

, which connects to the 
\emph on
left
\emph default
 to a comma.
 Not a surprise.
\end_layout

\begin_layout Standard
Not visible in the top-ten, but can be seen in the top-fifty are mirror-images,
 for example: ",: and+;" in 14th place, which states that the comma expects
 to be followed by the 
\begin_inset Quotes eld
\end_inset

and
\begin_inset Quotes erd
\end_inset

.
 The counts differ: the sixth-place "and: ,-;" had 32755 counts, while 14th
 place had 17745.
 Presumably this is due the comma having a different or more complex linkage
 about half the time.
 Other items in the top-50 that are not sentence openers include "”: ?-;"
 ",: but+;" ".: him-; "in: the+;" ".: ’+;" "”: .-;" "of: the+;".
 The first disjunct with more than one connector in is is "It: LEFT-WALL-
 was+;" which states that 
\begin_inset Quotes eld
\end_inset

It
\begin_inset Quotes erd
\end_inset

 wants to be a sentence opener, but wants to be followed by the word 
\begin_inset Quotes eld
\end_inset

was
\begin_inset Quotes erd
\end_inset

 on the right.
 Not surprising.
 Proceeding down the list, the disjuncts continue in this manner.
 This does not necessarily mean that they are 
\begin_inset Quotes eld
\end_inset

high quality
\begin_inset Quotes erd
\end_inset

, only that they were frequently observed in MST parses.
 It might be the case that the quality is given by the MI between the word
 and its disjunct; this is explored later.
\end_layout

\begin_layout Subsection
Word distribution
\end_layout

\begin_layout Standard
It is also interesting to turn the word distribution graph 
\begin_inset Quotes eld
\end_inset

on it's side
\begin_inset Quotes erd
\end_inset

.
 This is meant to be a simple exercise, so as to place some later graphs
 into context.
 Despite the simplicity, the analysis turns out to be somewhat surprising,
 and somewhat subtle.
 In particular, the last graph elicits some features in the dataset that
 are not otherwise easily visible.
 
\end_layout

\begin_layout Standard
In this dataset, there were 53076 words observed exactly once (out of a
 total of 18.5M observations of 137K words).
 This is quite something: of all the words observed, almost half were seen
 only once.
 More than half were seen twice, or less.
 These are presumably rare typos, foreign words, IPA pronunciation guides:
 any word that appears only once must be unusual; and yet, there are a lot
 of them! There are 17120 words that appear twice, 9081 that appear exactly
 3 times, 
\emph on
etc.

\emph default
 These counts are graphed below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Graph of binned-word-counts.dat, generated in disjunct-stats.scm 
\end_layout

\end_inset

 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-word-counts.eps
	width 70text%

\end_inset


\begin_inset CommandInset label
LatexCommand label
name "graph word frequency"

\end_inset


\end_layout

\begin_layout Standard
This graph indicates that most (almost all) words were observed less than
 100 times.
 In this dataset, there were only 9425 words that were observed 100 or more
 times, 5683 words that were observed 200 or more times, and 3263 words
 that were observed 400 or more times.
 Percentage-wise: about 6.8% of the words were observed 100 times or more.
 This should be enough to give confidence in the syntactic usage of the
 commonly-used English words; but most of the rest of this dataset includes
 oddities of various sorts, including place names and given names.
 The challenge will be to see if these can be grouped into grammatical categorie
s.
\end_layout

\begin_layout Standard
Writing 
\begin_inset Formula $N$
\end_inset

 for the number of times that some word was observed, it appears that there
 are approximately 
\begin_inset Formula $53076\times N^{-3/2}$
\end_inset

 words observed that many times.
 In formulas, the size of the set of words 
\begin_inset Formula $\left\{ w|N(w)=N\right\} $
\end_inset

is given by 
\begin_inset Formula 
\begin{equation}
\left|\left\{ w|N(w)=N\right\} \right|\sim N^{-3/2}\label{eq:word-distrib}
\end{equation}

\end_inset

where 
\begin_inset Formula $\left\{ w|\mbox{cond}\right\} $
\end_inset

 is a set of words (subject to the condition cond) and 
\begin_inset Formula $\left|\left\{ w|\mbox{cond}\right\} \right|$
\end_inset

 denotes the size of that set of words.
\end_layout

\begin_layout Standard
There is something interesting about this chart: it is more stable under
 varying dataset sizes than the Zipfian distribution.
 The slope of the Zipfian distribution changes, as datasets grow larger,
 typically trying to approach a slope of 1.0 very slowly.
 By contrast, the above 
\begin_inset Formula $N^{-3/2}$
\end_inset

 behavior seems to provide a much better description, even as the size of
 the dataset varies.
 I cannot demonstrate that assertion here, but have noticed it to be true
 when looking at other datasets.
\end_layout

\begin_layout Standard
The next graph belabors the point, and yet it's important.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Generated from binned-word-logli.dat
\end_layout

\end_inset

 It shows nothing new, but it does show it in a format that will be recur
 frequently, later.
 Thus, its worth understanding now.
 This graph shows exactly the same data as the previous graph: it 
\emph on
is
\emph default
 the same graph, except that the x-axis is now labeled differently, and
 some of the counts have been binned together.
 So first: note that 
\begin_inset Formula $-\log_{2}(1/18489594)=24.140$
\end_inset

 and so this is the location of the first spike on the far-right.
 Next, 
\begin_inset Formula $-\log_{2}(2/18489594)=23.140$
\end_inset

 and 
\begin_inset Formula $-\log_{2}(3/18489594)=22.555$
\end_inset

 are the locations of the second and third spikes: these correspond to words
 that have been observed 1,2 and 3 times.
 Words that have been observed exactly 
\begin_inset Formula $N$
\end_inset

 times will have a log-likelihood of 
\begin_inset Formula $-\log_{2}(N/18489594)=24.140-\log_{2}N$
\end_inset

.
 The formula 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:word-distrib"

\end_inset

, which was based on the graph above, can be rewritten as 
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
\left|\left\{ w|N(w)=N\right\} \right|\sim2^{-3/2\times\log_{2}N}
\]

\end_inset

which effectively predicts the height and location of the spikes.
 This is clearly demonstrated by the straight green line.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-word-logli.eps
	width 70text%

\end_inset


\begin_inset CommandInset label
LatexCommand label
name "logli distribution graph"

\end_inset


\end_layout

\begin_layout Standard
But then something else happens.
 As long as the bins are narrow, so that they are either full, or empty,
 then the nice power law holds.
 Once the bins become too wide to just hold single, discrete counts, but
 instead lump together different logli's, the apparent distribution changes.
 It is worthwhile to understand this phenomenon.
\end_layout

\begin_layout Standard
This graph was generated by bin-counting.
 The x-axis was divided into 1200 equal-sized bins, and whenever the log-likelih
ood of a word landed within a particular bin, the count was accumulated
 into that bin.
 On the right side of the graph, many (most) bins are empty, because they
 do not correspond to logarithms of integers; this results in the spikes.
 The width of each bin is (24-9)/1200, and so when 
\begin_inset Formula 
\[
\log_{2}N-\log_{2}(N-1)\approx\log_{2}\left(1+\frac{1}{N}\right)\approx\frac{1}{N\log2}<\frac{24-9}{1200}=\frac{15}{1200}
\]

\end_inset

then multiple counts will be shoved into one bin.
 For this chart, this happens when 
\begin_inset Formula $N\approx115$
\end_inset

 so there are about 115 distinct spikes, and then they merge when the logli
 is 
\begin_inset Formula $24.140-\log_{2}115\approx17.3$
\end_inset

 which is the spot where the above graph bends from the green line to the
 blue line.
 From this point on, when multiple counts are being jammed into one bin,
 we expect the distribution measure to be given by the Jacobian determinant
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant
\end_layout

\end_inset

 of the point measure.
 We can compute this explicitly.
 Changing notation slightly, write 
\begin_inset Formula 
\[
C(N)=KN^{-3/2}
\]

\end_inset

for the number of words that were observed 
\begin_inset Formula $N$
\end_inset

 times.
 This is the same formula as before.
 For this dataset, 
\begin_inset Formula $K=53076$
\end_inset

.
 The bincount at logli
\begin_inset Formula $=x$
\end_inset

 is 
\begin_inset Formula 
\[
\mbox{bincnt}(x)=\sum_{x\le\log_{2}\left(T/N\right)\le x+\epsilon}C(N)\Delta N
\]

\end_inset

For this dataset, 
\begin_inset Formula $T=18489594$
\end_inset

.
 The binsize used in the above graph was 
\begin_inset Formula $\epsilon=15/1200=1/80$
\end_inset

.
 In the above formula, 
\begin_inset Formula $\Delta N=1$
\end_inset

 is a notational trick to allow us to convert the sum into an integral when
 
\begin_inset Formula $N$
\end_inset

 gets large.
 We replace 
\begin_inset Formula $\sum$
\end_inset

 by 
\begin_inset Formula $\int$
\end_inset

 and replace 
\begin_inset Formula $\Delta N$
\end_inset

 by 
\begin_inset Formula $dN$
\end_inset

 and write
\begin_inset Formula 
\[
\mbox{bincnt}(x)\approx\int_{x}^{x+\epsilon}C(N)\frac{dN}{dy}dy
\]

\end_inset

Here, there is a change-of-variable to 
\begin_inset Formula $y=\log_{2}\left(T/N\right)$
\end_inset

 or equivalently 
\begin_inset Formula $N=T2^{-y}$
\end_inset

.
 The Jacobian determinant is then 
\begin_inset Formula $\left|dN/dy\right|=N\log2$
\end_inset

 and so
\begin_inset Formula 
\[
\mbox{bincnt}(x)\approx\frac{K\log2}{\epsilon\sqrt{T}}2^{x/2}
\]

\end_inset

Comparing this to the graph above, with logli
\begin_inset Formula $=x$
\end_inset

, we expect the bincounted region to have a slope of 0.5, and yet, the eyeballed
 fit above clearly shows 0.8.
 What's going on? WTF? Blame the data.
 Try again.
 The graph below shows exactly the same data, but this time there are only
 60 bins grand-total, and so only the first three spikes show.
 After that, the spikes merge together into bins.
 The green line is drawn exactly with the same slope and offset as before:
 that's because the first three spikes are in exactly the same locations
 as before, and have the same height.
 The blue line shows an eyeballed fit to the merged counts, and initially,
 it really does have a slope of 0.5, which is exactly what the Jacobian determina
nt was telling is it should be.
 Yayy! Declare victory and go home!
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-word-logli-60.eps
	width 60text%

\end_inset


\end_layout

\begin_layout Standard
The purple line is an eyeballed fit to what the data is doing, when the
 number of observations really does become large.
 The knee in the graph is at about logli
\begin_inset Formula $=15.5=24.140-\log_{2}N$
\end_inset

 or, equivalently at 
\begin_inset Formula $N=400$
\end_inset

.
 Thus, we have to revise the apparent distribution.
 It is, for this dataset:
\begin_inset Formula 
\[
\left|\left\{ w|N(w)=N\right\} \right|\sim\begin{cases}
N^{-3/2} & \mbox{for }N<400\\
N^{-1.8} & \mbox{for }N>400
\end{cases}
\]

\end_inset

It was noted before that there are 3263 words in the dataset that were observed
 400 or more times.
 
\end_layout

\begin_layout Standard
What does this mean? What is this saying? Its not entirely clear.
 It seems to suggest that there are about 3.3K words that are used preferentially
 more often than the rest.
 That is, they are used more often not only in absolute terms, but also
 in relative terms: the form a core of the vocabulary, enjoying a popularity
 exceeding the trend line for less-frequently used words.
\end_layout

\begin_layout Standard
Its not clear if this is a generic feature of the English language, or if
 this is peculiar to the particular corpus.
 Let's review the corpus again.
 The corpus comprises assorted late-19th and early 20th century texts from
 Project Gutenberg, a dozen sci-fi/fantasy novels, and a sampling of fan-fiction.
 These texts will contain stray markup, including tables of contents, chapter
 headings, indexes, figure captions and itemized lists.
 Quite often, ASCII artwork is used to delimit chapters or sections.
 Chapter headings are often written in all-upper-case.
 There are stray quotations in Latin, snippets of Latin prose and poetry.
 Travelogues will include miscellaneous foreign sayings and unusual place-names.
 All of this stuff adds up: it will be observed only once, twice, maybe
 a a few dozen times.
 It may as well be random text, from the point of view of the word-pair
 MI statistics, and from the point of view of the MST parser.
 There is no easy way to remove this 
\begin_inset Quotes eld
\end_inset

garbage
\begin_inset Quotes erd
\end_inset

 in any 
\emph on
a priori
\emph default
 fashion.
 It is there, and it is unavoidable.
 It is indistinguishable from random sentences.
 However, it seems that the fact that some significant portion is 
\begin_inset Quotes eld
\end_inset

ungrammatical
\begin_inset Quotes erd
\end_inset

 should not affect word-count statistics, unless it just so happens that
 there is a core of 3.3K vocabulary words, followed by 130K words that are
 given names, arcane terms, and other 
\begin_inset Quotes eld
\end_inset

junk
\begin_inset Quotes erd
\end_inset

.
 This does not seem plausible.
\end_layout

\begin_layout Standard
Thus, it is unclear on what the meaning of this knee in the graph really
 is, and how it should be explained.
 Note that this knee is NOT visible in the Zipfian distribution – nothing
 happens at 
\begin_inset Formula $N=400$
\end_inset

 - it is smooth as silk.
 It maybe could have been visible in the 
\begin_inset CommandInset ref
LatexCommand vref
reference "graph word frequency"

\end_inset

 graph, except that the far edge of that graph ends at exactly 
\begin_inset Formula $N=400$
\end_inset

, and does not continue past there! This seems to be a fairly subtle effect.
\end_layout

\begin_layout Subsection
Ranked average observations per disjunct
\end_layout

\begin_layout Standard
A more interesting distribution arises by looking at the average number
 of observations per disjunct (per word).
 That is, a single word may have hundreds of disjuncts, observed thousands
 of times; what is the average number of times that a disjunct is observed?
 By 
\begin_inset Quotes eld
\end_inset

average
\begin_inset Quotes erd
\end_inset

, it is explicitly meant 
\begin_inset Formula $N(w,*)/\left|(w,*)\right|$
\end_inset

, the number of observations divided by the support for those observations.
\end_layout

\begin_layout Standard
This number gives a hint of how 
\begin_inset Quotes eld
\end_inset

narrow
\begin_inset Quotes erd
\end_inset

 the grammatical usage of a word is.
 If the average is high, it suggests that the word just does not have very
 many disjuncts on it; the few that it does have are observed a lot.
 Recall that these disjuncts (pseudo-disjuncts) connect to individual words,
 and not to word-classes.
 Thus, if a disjunct is seen a lot, it probably connects to another word,
 forming a high-MI pair.
 This can be explicitly seen in the example further below.
\end_layout

\begin_layout Standard
A graph of the ranked average number of observations, per disjunct, per
 word, is shown on the left, below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with the sorted-avg list in disjunct-stats.scm
\end_layout

\end_inset

 The ranking is distinctly not Zipfian; this is confirmed by slicing the
 data three ways: excluding words with less than 400 observations (leaving
 3263 words), excluding words with less than 100 observations (leaving 9425
 words), and excluding words with less than 20 observations (leaving 25505
 words out of 137K words).
\end_layout

\begin_layout Standard
The graph on the right expresses an alternate view of the same idea: it
 shows a bin-count of all of the words.
 Reaffirming the graph on the left, it indicates that almost all words have
 an average disjunct observation count of less than four.
 It also conveys the sense that when the average disjunct count is greater
 than about four, that this is unusual, and perhaps meaningful in some way.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-avg.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename en-dj/binned-avg.eps
	width 50text%

\end_inset


\end_layout

\begin_layout Standard
The first ten on the ranked list are "*" "Literary" "Archive" "Gutenberg"
 "Notes" "...." "|" "Foundation" "Project" "Summary".
 This suggests that these all come from exactly the same parse of a small
 group of sentences having a very regular, formulaic structure, occurring
 repeatedly in multiple texts.
 One of those sentences is easily found; it begins as: 
\begin_inset Quotes eld
\end_inset

The Project Gutenberg Literary Archive Foundation has been created...
\begin_inset Quotes erd
\end_inset

 
\end_layout

\begin_layout Standard
Closer examination indicates that more or less all words having an average
 of more than three observations per disjunct are associated with the Project
 Gutenberg legal boilerplate.
 The first 80 entries in the 400+ list have an average observation count
 of above three, and they are all boilerplate words: "fee" "copies" "trademark"
 "agreement" "electronic" "copyright" "donations", and so on.
 This suggests that pretty much all of the 
\begin_inset Quotes eld
\end_inset

bump
\begin_inset Quotes erd
\end_inset

 on the above-left graph is entirely due to license boilerplate!
\end_layout

\begin_layout Standard
Its entertaining to look at some of these close-up.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
View disjuncts by saying (filter (lambda (cset) (< 10 (get-count cset)))
 (cog-incoming-by-type (Word "foo") 'Section)) where 10 is the minimum number
 of counts.
\end_layout

\end_inset

 The word 
\begin_inset Quotes eld
\end_inset

Prince
\begin_inset Quotes erd
\end_inset

 shows up 99th on the list, with an average of 2.885 observations per disjunct.
 It has a total of 773 different disjuncts on it.
 The top six disjuncts are just single links, shown in the table below.
 Clearly all are princes.
 This leaves 767 other disjuncts with far fewer counts.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
number of observations
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Andrew+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
626
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Vasili+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
149
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Andrew's+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
46
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Edouard+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
46
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Hans+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Bagration+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
The word 
\begin_inset Quotes eld
\end_inset

think
\begin_inset Quotes erd
\end_inset

 appears as number 140 on the list, with an average of 2.6104 observations
 per disjunct.
 It has 5462 disjuncts in total; only six are observed more than 200 times.
 The last in the list below is the first clear appearance of a transitive
 verb.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
number of observations
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1321
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
you-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
308
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
don't-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
265
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
244
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
do- you-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
211
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- it+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
202
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
The word 
\begin_inset Quotes eld
\end_inset

long
\begin_inset Quotes erd
\end_inset

 appears as 195 on the list, with an average of 2.3825 observations per disjunct.
 It has 5412 different disjuncts on it; only five are seen more than 200
 times.
 These are:
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
number of observations
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
385
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
as+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
335
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
how-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
236
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a- time+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
212
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
enough+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
201
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
The skewness appears to be very sharp.
 This suggests that we should not waste time looking at mean-square variations
 in the average, although we'll do this anyway.
 But first, its worth graphing the skewness directly.
 Again, this is done on a log-log graph, in a Zipfian way.
\end_layout

\begin_layout Subsection
Disjunct count distribution
\end_layout

\begin_layout Standard
The graph below shows the distribution of the disjunct observations on the
 five words 
\begin_inset Quotes eld
\end_inset

Prince
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

think
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

long
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

fact
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

from
\begin_inset Quotes erd
\end_inset

.
 Indeed, it looks Zipfian; since we know that all but the first three or
 four disjuncts are noise, this graph illustrates 
\begin_inset Quotes eld
\end_inset

pink noise
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

1/f noise
\begin_inset Quotes erd
\end_inset

.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with the dj-prince and dj-think etc.
 arrays in disjunct-stats.scm 
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-dj-prince.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Can one get a smoother distribution by summing together these two graphs?
 Sure...
 and one can sum together not just these two words, but all words (that
 have been observed at least 100 times).
 
\end_layout

\begin_layout Standard
That graph is shown below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with the accum-dj-all function in disjunct-stats.scm 
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-dj-counts.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Its kind of a strange graph.
 Yes, the x-axis of this graph does imply that there are thousands of words
 with more than a thousand disjuncts on them, and hundreds that have more
 than ten-thousand (unique, different) disjuncts on them! Exactly what does
 this mean? This is covered in the next section.
\end_layout

\begin_layout Subsection
Disjunct Support Distribution
\end_layout

\begin_layout Standard
Is it possible that some words have a large number of disjuncts on them?
 
\end_layout

\begin_layout Standard
Yes, it is.
 For example, the comma was observed to have 243987 unique, different disjuncts
 associated with it.
 The word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 has 215324 unique, different disjuncts, the word 
\begin_inset Quotes eld
\end_inset

and
\begin_inset Quotes erd
\end_inset

 has 126861.
 Rounding out this list are "to" "of" "." "a" "was" "LEFT-WALL".
 Its not clear what fraction of these disjuncts are grammatically valid,
 and what fraction are junk.
\end_layout

\begin_layout Standard
The graph below shows the distribution of the size of the support: the ranking
 of 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

.
 Again, the graph appears to be approximately Zipfian.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Generated by sorted-support in disjunct-stats.scm 
\end_layout

\end_inset

 The eye-balled fit has a slope of 0.9, but, from the eyeball-perspective,
 this is not all that different from a slope of 1.0.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-support.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Terminology: the 
\begin_inset Quotes eld
\end_inset

support
\begin_inset Quotes erd
\end_inset

 of a vector is the number of basis elements that have a non-zero coefficient.
 This is the set 
\begin_inset Formula $(w,*)$
\end_inset

 defined earlier.
 Equivalently, this is the size of the set of disjuncts associated with
 a word, when counted 
\emph on
without
\emph default
 multiplicity.
\end_layout

\begin_layout Subsection
Ranked Euclidean length (RMS Size)
\end_layout

\begin_layout Standard
A different distribution arises by looking at the ranked RMS sizes of the
 disjunct sets
\begin_inset Foot
status open

\begin_layout Plain Layout
Obtained by running (print-ts-rank sorted-lengths outport) from the disjunct-sta
ts.scm file 
\end_layout

\end_inset

.
 Here, the RMS size
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
The word 
\begin_inset Quotes eld
\end_inset

length
\begin_inset Quotes erd
\end_inset

 can be used to describe the root-mean-square size of the set of disjuncts
 associated with a word.
 That is, each element of the set is a disjunct, and that disjunct has a
 count, the number of times it has been observed.
 The root-mean-square of these counts can be taken as the set-size.
 But this set can also be interpreted as a vector, and so the RMS size is
 the same thing as the Euclidean length of the vector.
 Thus, the word 
\begin_inset Quotes eld
\end_inset

length
\begin_inset Quotes erd
\end_inset

 is sometimes used for the RMS size; they're the same thing.
\end_layout

\end_inset

 is computed by taking the root-mean-square of the counts on each disjunct
 in the set, that is, by computing 
\begin_inset Formula $\sqrt{\sum_{d}N(w,d)^{2}}$
\end_inset

 for each word 
\begin_inset Formula $w$
\end_inset

 and then ranking.
 Interpreting 
\begin_inset Formula $d$
\end_inset

 as a basis element of a vector space, this can be recognized as the Euclidean
 length of the count-vector.
\end_layout

\begin_layout Standard
The RMS size of the set is thus larger not only when more disjuncts have
 been observed, but also when most of the observations are made of only
 a small handful of disjuncts.
 That is, the RMS size should be relatively larger, if the word is less
 grammatically flexible.
 So for example, prepositions tend to be very flexible; adjectives, not
 so much.
 Thus, we expect adjectives to appear higher-up on this ranking list, than
 the observation-based list.
 And this might be true, relatively, but certainly not true absolutely.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-lengths.eps
	width 70text%

\end_inset


\begin_inset CommandInset label
LatexCommand label
name "graph: ranked length"

\end_inset


\end_layout

\begin_layout Standard
The first dozen words of the RMS-size-list are: "LEFT-WALL" "." "and" ","
 """ "”" "in" "to" "of" "“" "as" "the".
 Not that interesting: these are all words that were observed a lot in the
 text.
 The RMS size is dominated by the total number of observations of a word
 in text.
 In and of itself, its insufficient to indicate how 
\begin_inset Quotes eld
\end_inset

concentrated
\begin_inset Quotes erd
\end_inset

 the disjuncts are, how grammatically narrow a word is.
 For this, some other quantity is needed.
\end_layout

\begin_layout Standard
The slope appears to be exactly -1.0, continuing the scale-free trend.
\end_layout

\begin_layout Subsection
Mean-square to size ratio
\end_layout

\begin_layout Standard
More interesting is the ratio of mean-square size to the total size.
 In formulas, by ranking according to 
\begin_inset Formula 
\[
\frac{\sqrt{\sum_{d}N^{2}(w,d)}}{N(w,*)}=\frac{\sqrt{\sum_{d}p^{2}(w,d)}}{p(w,*)}
\]

\end_inset

This seems like the interesting ratio, because the Zipf exponent of -0.65
 would be doubled, when working with mean-square sizes, thus making the
 two rankings comparable.
\end_layout

\begin_layout Standard
Words high in this score will be words that have relatively few disjuncts
 on them, or at least, few that matter much, that rise above the level of
 noise.
 The first ten words on this list, excluding punctuation, and excluding
 all words with fewer than 100 observations: "It" "and" "but" "in" "Two"
 "as" "Notes" "not" "There" "Summary".
 A review of the input corpus shows that the word "Summary" appears in only
 one input text: Charles Darwin's 
\begin_inset Quotes eld
\end_inset

On the Origin of Species
\begin_inset Quotes erd
\end_inset

, and then only to indicate an actual summary! The word 
\begin_inset Quotes eld
\end_inset

Notes
\begin_inset Quotes erd
\end_inset

 appears in less than a dozen input texts, with almost every usage being
 formulaic and rigid.
 The grammatical usage of these two words in the input corpus is fairly
 constrained, and thus it is no surprise that these have relatively narrower
 disjunct distributions on them.
 
\end_layout

\begin_layout Standard
Excluding the capitalized words, and punctuation, in this list what remains
 is quite surprising.
 It is shown in the table below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Extracted from ranked-sqlen-norm.dat 
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
score
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1956
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1147
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
but
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1043
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
814
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
as
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
776
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
not
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
590
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
been
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
22
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
568
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
be
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
453
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
one
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
440
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
at
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
386
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
own
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset

 
\begin_inset space \qquad{}
\end_inset

  
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
score
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
34
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
329
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
of
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
35
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
328
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
36
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
324
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
with
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
37
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
319
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
other
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
38
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
315
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
it
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
42
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
284
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
when
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
43
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
275
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
for
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
45
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
271
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
he
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
46
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
265
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
have
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
49
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
248
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
that
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset

 
\end_layout

\begin_layout Standard
It is surprising because, grammatically, we expect most of these words to
 have a large number of varied disjuncts attached to them.
 We expect them to be diffuse, not sharp: we expect that these would have
 a large number of observations smeared over a large variety of disjuncts,
 instead of having their weight concentrated in only a handful of disjuncts,
 the way that 
\begin_inset Quotes eld
\end_inset

Prince
\begin_inset Quotes erd
\end_inset

 was, above.
 So what is going on, here?
\end_layout

\begin_layout Standard
The distribution, shown below, is a power-law.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-sqlen-norm.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Are there other interesting measures? One could contemplate the ratio of
 the mean-square size to the support 
\begin_inset Formula $\sum_{d}N(w,d)^{2}/\left|(w,*)\right|$
\end_inset

.
 Another possibility would be this, minus the average-squared, which would
 give the second moment, aka, the mean-square deviation from the average,
 specifically
\begin_inset Formula 
\[
\frac{\sum_{d}N(w,d)^{2}}{\left|(w,*)\right|}-\left[\frac{N(w,*)}{\left|(w,*)\right|}\right]^{2}=\frac{1}{\left|(w,*)\right|}\sum_{d}\left[N(w,d)-\frac{N(w,*)}{\left|(w,*)\right|}\right]^{2}
\]

\end_inset

Neither of these variations seem promising; they seem to offer up more of
 the same, at least on this dataset.
 A larger and more refined dataset might reveal otherwise.
\end_layout

\begin_layout Subsection
Mutual information
\end_layout

\begin_layout Standard
The concept of the 
\begin_inset Quotes eld
\end_inset

fractional mutual information
\begin_inset Quotes erd
\end_inset

 for a pair is interesting to explore.
 Define this as
\begin_inset Formula 
\begin{equation}
MI_{pair}(w,d)=-\log_{2}\frac{p(w,d)}{p(w,*)p(*,d)}\label{eq:mi-word-disjunct}
\end{equation}

\end_inset

This is 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 in the sense that the total MI for the set of all pairs can be written
 as
\begin_inset Formula 
\[
MI_{cset}=\sum_{w,d}p(w,d)MI_{pair}(w,d)
\]

\end_inset

Fractional MI is interesting because it usually has a reasonably nice distributi
on.
 For this particular dataset, it ranges from about -11 to +24.
 The distribution is shown in the graphs below.
 
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
These are graphed by binned-cset-mi and weighted-cset-mi in the disjunct-stats.sc
m file.
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-cset-mi.eps
	width 50col%

\end_inset


\begin_inset Graphics
	filename en-dj/weighted-cset-mi.eps
	width 50col%

\end_inset


\end_layout

\begin_layout Standard
These graphs are generated by computing the value for 
\begin_inset Formula $MI_{pair}(w,d)$
\end_inset

 for each of the 8629163 
\begin_inset Formula $(w,d)$
\end_inset

 pairs (aka 'connector sets'), and approximating it's distribution by bin
 counting.
 In each graph above, there are 200 bins, each of width of about 35/200,
 and each pair is assigned to one of the bins, according to it's MI value.
 The graph on the left then shows how many pairs there are in each bin.
 The graph on the left is similar, but not the same: it sums the frequencies
 for all the pairs in each bin.
 In formulas: the graph on the left shows the value of
\begin_inset Formula 
\[
\mbox{sizeof }\left\{ (w,d)|MI_{pair}(w,d)\mbox{ is in bin}\right\} 
\]

\end_inset

while the graph on the right shows
\begin_inset Formula 
\[
\sum_{MI_{pair}(w,d)\mbox{ is in bin}}p(w,d)
\]

\end_inset

where '
\begin_inset Formula $x\mbox{ is in bin}$
\end_inset

' simply means 
\begin_inset Formula $lo\le x<hi$
\end_inset

 with the bin being the interval 
\begin_inset Formula $[lo,hi)$
\end_inset

.
\end_layout

\begin_layout Standard
Both of these graphs show 
\begin_inset Quotes eld
\end_inset

combs
\begin_inset Quotes erd
\end_inset

 in the right side.
 These combs are exactly the same combs as noted in the last figure in section
 
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"

\end_inset

 
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"

\end_inset

.
 The combs are due to the large number of words that have been observed
 only a small handful of times.
 In essence, the combs attest that the bulk of the high-MI pairs have been
 observed more than just a few times; 
\emph on
i.e.

\emph default
 the high MI values are meaningful.
\end_layout

\begin_layout Standard
Both graphs show an eyeballed fit in green.
 The left graph shows that the distribution can be approximated by a Gaussian
 (visually a parabola, due to the logarithmic scale), given by 
\begin_inset Formula $\exp-(0.23(MI_{pair}-11))^{2}/2$
\end_inset

.
 The shape of the graph on the right is harder to pin down.
 It has hints of parabolic behavior, yet the left edge appears more straight
 than curved, that is, to be exponential, given by 
\begin_inset Formula $2^{1.3MI_{pair}}=\exp0.9MI_{pair}$
\end_inset

.
 The eyeballed blue parabola is given by 
\begin_inset Formula $\exp-(0.15(MI_{pair}-12))^{2}/2$
\end_inset

, it's clearly missing a large excess in the middle of the graph.
\end_layout

\begin_layout Subsection
Marginal Mutual Information of Words
\end_layout

\begin_layout Standard
The marginal mutual information of a single word can be defined by summing
 the (fractional) mutual information between a word, and all of it's disjuncts:
\begin_inset Formula 
\[
MI_{word}(w)=\frac{1}{p(w)}\sum_{d}p(w,d)MI_{pair}(w,d)
\]

\end_inset

This is also written in the 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 style, so that, again, the total MI of the entire dataset can be written
 as
\begin_inset Formula 
\[
MI_{cset}=\sum_{w}p(w)MI_{word}(w)
\]

\end_inset

That is, 
\begin_inset Formula $MI_{word}(w)$
\end_inset

 is the fractional contribution of the word to the total MI.
 The fractional marginal MI is very convenient for comparing different words,
 since it factors out the frequency of how often a word is observed: the
 MI of two words with two very different frequencies can be directly compared.
 
\end_layout

\begin_layout Standard
As can be seen from the graph below, the fractional marginal MI ranges between
 +3 and +18 for this dataset.
 The total MI for the dataset is measured to be 7.8969 bits.
 The distribution can be visualized in two different ways.
 The graph on the left, below
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
This is graphed by sorted-word-mi-hi-p in disjunct-stats.scm 
\end_layout

\end_inset

 shows the ranked MI of the 9425 words that have been observed more than
 100 times in this dataset.
 Note that it is a semi-log plot; it is NOT Zipfian.
 Note that the MI seems to decay logarithmically, for a good long ways,
 and then drops off a cliff.
 
\end_layout

\begin_layout Standard
The graph on the right shows the distribution, for all 137078 words, bin-counted
 into 100 bins.
 The combs on the far right are again the same combs as noted in the last
 figure in section 
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"

\end_inset

 
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"

\end_inset

.
 The distribution appears to be Gaussian, and of approximately the same
 width as before, although located at a different center: the green line
 in the graph is the Gaussian given by 
\begin_inset Formula $\exp-(0.24(MI_{word}-20))^{2}/2$
\end_inset

 as compared to 0.23 for the non-marginal MI distribution above.
 The difference between 0.24 and 0.23 seems to be significant: changing one
 to the other seems to give a noticeably poorer fit.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-word-mi-hi-p.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename en-dj/binned-word-mi.eps
	width 50col%

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
What are the disjuncts like at either end of this distribution? The first
 few words in the ranking are: "LICENSE" "FULL" "formats" "BREACH" "AGREE"
 "WARRANTIES" "WARRANTY" and are clearly parts of the set phrases that make
 up the Project Gutenberg license agreement.
 The word 
\begin_inset Quotes eld
\end_inset

Prince
\begin_inset Quotes erd
\end_inset

, examined previously, is 7378th in this rank.
 
\end_layout

\begin_layout Standard
Its entertaining to look at some of these.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
View disjuncts by saying (filter (lambda (cset) (< 10 (get-count cset)))
 (cog-incoming-by-type (Word "foo") 'Section)) where 10 is the minimum number
 of counts.
\end_layout

\end_inset

 The word "LICENSE" is surely under-sampled, as, in this all-capitals form,
 it only appears in the Gutenberg boilerplate, and nowhere else.
 This we cannot expect accurate MST parses, and cannot expect accurate disjuncts
 for this word.
 Yet other set phrases quickly top the list.
 The word 
\begin_inset Quotes eld
\end_inset

San
\begin_inset Quotes erd
\end_inset

, number 33 in the list, is seen only with the disjuncts 
\begin_inset Quotes eld
\end_inset

Francisco+
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

Antonio+
\begin_inset Quotes erd
\end_inset

.
 The word 
\begin_inset Quotes eld
\end_inset

Tomb
\begin_inset Quotes erd
\end_inset

 is 40th in the list and has three disjuncts observed more than twice:
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="4" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct for 
\begin_inset Quotes eld
\end_inset

Tomb
\begin_inset Quotes erd
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Great- of+ Nazarick+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Underground- of+ Nazarick+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
The- Great- of+ Nazarick+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
It is already clear from this one example that the high marginal-MI words
 will be those that take part in idioms, 
\begin_inset Quotes eld
\end_inset

set phrases
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

institutional phrases
\begin_inset Quotes erd
\end_inset

, and that the disjunct identifies the words taking part in the setting.
 The word 
\begin_inset Quotes eld
\end_inset

prominently
\begin_inset Quotes erd
\end_inset

 appears 50th in the list, and suggests that it is only used only in a rather
 rigid and formulaic way:
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="3" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct for 
\begin_inset Quotes eld
\end_inset

prominently
\begin_inset Quotes erd
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
51
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
appear- whenever+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
51
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
without- displaying+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The word 
\begin_inset Quotes eld
\end_inset

Corps
\begin_inset Quotes erd
\end_inset

 appears 95th in the list.
 Note that both the English and French word-ordering appears: 
\begin_inset Quotes eld
\end_inset

Diplomatic Corps
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

Corps Diplomatique
\begin_inset Quotes erd
\end_inset

, as witness on the different sign on the disjuncts:
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct for 
\begin_inset Quotes eld
\end_inset

Corps
\begin_inset Quotes erd
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Supply-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Marine-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Medical-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Diplomatic-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Diplomatique+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
At the other extreme, the ten words with the lowest MI are: "it" "in" "of"
 "that" "to" "and" "the" "LEFT-WALL" "," "." ranging from 5.07 for "it" down
 to 3.33 for the period.
 These are already familiar from previous rankings: they occur with very
 high frequency; the disjunct lists on them will be lengthly variable, diffuse.
 
\end_layout

\begin_layout Standard
These samplings of disjuncts are sharply reminiscent of the technique of
 collocation used in corpus linguistics, but with a big, important difference.
 There, the linguist examines a window that is some 6-8 words wide, and
 examines the frequency of all phrases appearing within that window, containing
 the word-of-interest in the center.
 Here, we again have a word-of-interest, but this time, instead of seeing
 phrases, we see the grammatical structure revealed directly, by means of
 the disjuncts extracted from MST parses.
 The philosophical basis used to justify corpus linguistics, i.e.
 that of frequentism, is accepted and applied here as well.
 In this case, it is used to obtain grammatical structure.
\end_layout

\begin_layout Subsection
Mutual Information of Disjuncts
\end_layout

\begin_layout Standard
Symmetrically, one also has the mutual information of a disjunct, in comparison
 to all of the words it connects to:
\begin_inset Formula 
\[
MI_{disjunct}(d)=\frac{1}{p(*,d)}\sum_{w}p(w,d)MI_{pair}(w,d)
\]

\end_inset

Again, this is presented in the 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 style, so that the total MI of the entire dataset can be written as before:
\begin_inset Formula $MI=\sum_{d}p(*,d)MI_{disjunct}(d)$
\end_inset

The dataset contains 6239997 (6.24 million) unique disjuncts, observed for
 a total 18489594 (18.5 million) times.
 The distribution is shown below, with the disjuncts sorted into 200 equal-sized
 bins.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Use binned-dj-mi to get this.
 
\end_layout

\end_inset

 The combs on the far right are again the same combs as noted in the last
 figure in section 
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"

\end_inset

 
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"

\end_inset

.
 The green line shows an eyeballed fit to a Gaussian.
 As before, the width appears to be almost 1/4th, but not quite.
 It is given by 
\begin_inset Formula $\exp-(0.27(MI_{disjunct}-13.7))^{2}/2$
\end_inset

.
 That is, the mean MI is 13.7, and with a standard deviation of about 
\begin_inset Formula $\sqrt{1/0.27}=1.92$
\end_inset

.
 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-dj-mi.eps
	width 60col%

\end_inset


\end_layout

\begin_layout Standard
The origin of the noise that seems to build up for MI<10 is unclear.
 
\end_layout

\begin_layout Subsection
Fractional Entropy
\end_layout

\begin_layout Standard
There is a simpler variant than the mutual information, that is also worth
 understanding: the fractional contribution to the total entropy.
 This is given by the sum 
\begin_inset Formula 
\[
H_{word}(w)=-\frac{1}{p(w)}\sum_{d}p(w,d)\log_{2}p(w,d)
\]

\end_inset

This is written in the 
\begin_inset Quotes eld
\end_inset

fractional
\begin_inset Quotes erd
\end_inset

 style, so that the total entropy of the entire dataset can be written as
\begin_inset Formula 
\[
H_{cset}=\sum_{w}p(w)H_{word}(w)
\]

\end_inset

 Analogously, one also has the fractional contribution of the disjuncts:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
H_{disjunct}(d)=-\frac{1}{p(*,d)}\sum_{w}p(w,d)\log_{2}p(w,d)
\]

\end_inset

where, again, one has that 
\begin_inset Formula 
\[
H_{cset}=\sum_{d}p(d)H_{disjunct}(d)
\]

\end_inset


\end_layout

\begin_layout Standard
The ranked fractional entropy is shown in the left graph below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Generated from sorted-word-ent in disjunct-stats.scm.
\end_layout

\end_inset

 It only shows those words that have been observed 100 times, or more.
 
\end_layout

\begin_layout Standard
It resembles the graph for the ranked fractional marginal MI, above.
 The graph on the right shows the distribution of the entropy, for all of
 the words.
 This affirms (or explains?) the sharp knee in the graph on the left: the
 knee occurs because almost all words have a large disjunct-entropy.
 Remarkably, the distribution is anti-Gaussian, in that it appears to diverge,
 the larger the entropy.
 In that respect, it cannot even be a proper distribution, as it cannot
 be normalized to a probability of 1.0 - the distribution increases without
 bound! Yet, as the graph illustrates, that is what it seems to be.
 The green curve is the anti-Gaussian, given by 
\begin_inset Formula $\exp(0.27(FME-13.7))^{2}/2$
\end_inset

.
 As before, it has a width of approximately 1/4th.
 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-word-ent.eps
	width 50text%

\end_inset


\begin_inset Graphics
	filename en-dj/binned-word-ent.eps
	width 50text%

\end_inset


\end_layout

\begin_layout Standard
The top-ranked words (which have been observed 100 times or more) are "clawing"
 "manages" "noting" "anyways" "circling" "choke" "neared" "urging" "pursuing"
 "exited".
 This is not a list that has been exposed with other statistics.
 Almost all of these are verbs, a grammatical class that never appeared
 in any of the previous lists.
 Why these? 
\end_layout

\begin_layout Standard
All of these words have an entropy of exactly 
\begin_inset Formula $24.14021=\log_{2}18489594$
\end_inset

.
 Since there are a total of 
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
18489594 observations of disjuncts in this dataset, the only way in which
 this entropy is possible is if every disjunct on these words was observed
 once and only once.
\end_layout

\begin_layout Standard
Looking deeper into the disjunct set, the sensation that the words with
 the highest entropies are almost all verbs continues quite strikingly.
 A sampling is given in the table below.
 All of these words have a large support (i.e.
 have at least 100 observations of disjuncts on each), but each disjunct
 is observed only once, rarely twice; almost never more than that.
 For example, 
\begin_inset Quotes eld
\end_inset

gripped
\begin_inset Quotes erd
\end_inset

 has only one disjunct that was observed three times, seven that were observed
 twice, and 320 that were observed only once.
 A quick examination shows that many, maybe most, are grammatically reasonable,
 for example, for 
\begin_inset Quotes eld
\end_inset

gripped
\begin_inset Quotes erd
\end_inset

: (He- staff+) is the disjunct observed three times.
 Four of the seven observed twice are (had- him+) (she- his+ hand+) (He-
 staff+ tightly+) (his+ hand+) and look to be a part of rather formulaic
 sentences (possibly from the fan-fiction part of the corpus).
 The other three disjuncts observed twice are strange nonsense: (hands-
 Ross- at+ edge+ vanity+) (hand- jaw- other- top+) (LEFT-WALL- jockeyings-
 by+ them+).
 These seem to be the result of failed MST parses, depending, presumably,
 on word-pairs that were witnessed only a few times.
\end_layout

\begin_layout Standard
This observation, and the table below, reinforces the need to the urgency
 of clustering words: by clustering words together into grammatical categories,
 this should increase the number of observations of category-pairs, improving
 MST parses, as well as increasing the number of observations of disjuncts,
 hopefully drowning out the weak and bizarre disjuncts.
 Given that the high-entropy words seem to be predominantly verbs, this
 suggests that clustering will be absolutely required to pick out the grammatica
l form of verbs.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="6">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{3}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{2}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{1}$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.1402
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
clawing
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
102
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.1293
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
41
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
licking
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
183
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.1241
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
79
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
toys
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
124
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.1202
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
164
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
grabbing
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
300
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.0846
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
567
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gripped
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
328
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The table employs a bit of notation worth reviewing.
 Recall the definition of the set that supports a word: 
\begin_inset Formula 
\[
(w,*)=\left\{ (w,d)|N(w,d)>0\right\} 
\]

\end_inset

The notation 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

 was used to indicate the size of this set.
 Extend this notation as
\begin_inset Formula 
\[
\left|(w,*)\right|_{k}=\mbox{sizeof }\left\{ (w,d)|N(w,d)\ge k\right\} 
\]

\end_inset

That is, 
\begin_inset Formula $\left|(w,*)\right|_{k}$
\end_inset

 is the size of the support for when the disjuncts on word 
\begin_inset Formula $w$
\end_inset

 have been seen at least 
\begin_inset Formula $k$
\end_inset

 times.
\end_layout

\begin_layout Standard
From this table, it is now clear why 
\begin_inset Quotes eld
\end_inset

large entropy
\begin_inset Quotes erd
\end_inset

 can be intuitively understood to mean 
\begin_inset Quotes eld
\end_inset

many possibilities
\begin_inset Quotes erd
\end_inset

.
 Each of these words was seen in a very broad setting of possibilities:
 in a sense, the broadest possible.
 Each of these words was observed with a vary large set of different disjuncts,
 and this set was as spread-out as possible: the vast majority of disjuncts
 were observed exactly once.
 
\end_layout

\begin_layout Subsubsection*
More terms
\end_layout

\begin_layout Standard
Some curious terms show up in relating the fractional mutual information
 to the fractional entropy.
 Expanding out the above summations, one obtains
\begin_inset Formula 
\[
MI_{word}(w)=H_{word}(w)+\log_{2}p(w,*)+\frac{1}{p(w)}\sum_{d}p(w,d)\log_{2}p(*,d)
\]

\end_inset

 The last term is bizarre...
\end_layout

\begin_layout Subsection
Vertex degrees and hubiness
\end_layout

\begin_layout Standard
Vertex degrees can be defined as the average number of connectors per disjunct.
 In principle, the vertex degree is an excellent indicator of the part of
 speech.
 For example, determiners, adjectives and adverbs typically have a degree
 of one: they have one connector, which is modifying the noun (or verb)
 that they act on.
 By contrast, nouns typically have a degree of two: one connector to attach
 to a verb, another to a modifier, and that's it.
 Verbs have a degree of three: one connector to a subject, one to a direct
 object, a third to an indirect object or a modifier.
 Of course, nouns might have two or more modifiers, or maybe zero modifiers;
 verbs are also quite variable, but the general concept of vertex degree
 is appealing.
 Closely related to this is the idea of 
\begin_inset Quotes eld
\end_inset

hubiness
\begin_inset Quotes erd
\end_inset

, which can be defined as the second moment of the degree.
 
\end_layout

\begin_layout Standard
Thus, its worth looking at this.
 Define the average degree as
\begin_inset Formula 
\[
K(w)=\frac{\sum_{d,c}N(w,d)C(d,c)}{N(w,*)}
\]

\end_inset

This is graphed, below, for all words that have at least 100 observations.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with sorted-avg-connectors in disjunct-stats.scm 
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-avg-connectors.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
This graph is unexpected.
 Having an average number of connectors that exceed 5 or 6 is intuitively
 surprising.
 In proper grammar, it would be hard to reach even this degree without having
 a transitive or ditranstive verb with several modifiers, and a particle
 or preposition.
 The first ten items are out of control.
 What's happening here? The first ten items in the ranking are: "|" "#"
 "+" "||" "_" "ASCII" "electronically" "Northup" "disclaimer" "u".
 The first five are presumably formatting markup or possibly decorative markup
 in the texts.
 Three of the words appear to be license boilerplate.
 The name "Northup" appears in only one text in the corpus: an autobiography,
 
\begin_inset Quotes eld
\end_inset

Twelve Years a Slave
\begin_inset Quotes erd
\end_inset

, by Solomon Northup.
 Besides the title-page, the word Northup appears repeatedly in the table
 of contents, which is bound to make for awkward parsing.
 This presumably 
\begin_inset Quotes eld
\end_inset

explains
\begin_inset Quotes erd
\end_inset

 why the average connector count would be 11.2 for this word.
\end_layout

\begin_layout Standard
Moving further down this list, many of the words and symbols suggest that
 they appear in tables or lists embedded in the corpus.
 The 
\begin_inset Quotes eld
\end_inset

grammar
\begin_inset Quotes erd
\end_inset

 of tables and lists is necessarily awkward, and seems unlikely to get much
 of a meaningful parse from the MST parser.
 This is further strengthened by an earlier analysis of a Wikipedia-based
 dataset, re-reported below.
\end_layout

\begin_layout Subsubsection*
Vertex degree in Wikipedia
\begin_inset CommandInset label
LatexCommand label
name "subsec:Vertex-degree-in"

\end_inset


\end_layout

\begin_layout Standard
A similar analysis, performed on a much smaller dataset consisting entirely
 of Wikipedia articles found a similar behavior.
 The first ten items in the ranking were: "-" "de" - "y" ":" "(" ")" "General"
 "Department" "x" "Act".
\end_layout

\begin_layout Standard
Consider 
\begin_inset Quotes eld
\end_inset

de
\begin_inset Quotes erd
\end_inset

.
 There are 12 observations of the disjunct 
\begin_inset Quotes eld
\end_inset

Janeiro+
\begin_inset Quotes erd
\end_inset

.
 There are 9 observations of the disjunct 
\begin_inset Quotes eld
\end_inset

la+
\begin_inset Quotes erd
\end_inset

.
 There are 51 observations of a disjunct that has 117 connectors on it!!
 This starts out as 
\begin_inset Quotes eld
\end_inset

Diego- Francisco- Francisco- Alonso- Carlos- Fernández- Carlos-
\begin_inset Quotes erd
\end_inset

 and ends with 
\begin_inset Quotes eld
\end_inset

Figueroa+ (+ (+ y+
\begin_inset Quotes erd
\end_inset

 suggesting that there were possibly 51 really bad parses of a very long
 table of Spanish kings, which was mistaken for being a single sentence.
 Clearly, its junk; its frequently-occurring junk, which suggests that the
 table was repeatedly transcluded in maybe 51 different Wikipedia pages.
 
\end_layout

\begin_layout Standard
Similarly, 
\begin_inset Quotes eld
\end_inset

Department
\begin_inset Quotes erd
\end_inset

 has 18 observations of a disjunct with 41 connectors on it.
 It starts with 
\begin_inset Quotes eld
\end_inset

Education- Education- Health- Services- Services- Immigration-
\begin_inset Quotes erd
\end_inset

 and ends with 
\begin_inset Quotes eld
\end_inset

Veterans+ of+ Treasury+ Treasury+
\begin_inset Quotes erd
\end_inset

, again suggesting a bad parse of a table mistaken for a sentence, and included
 in 18 different Wikipedia pages.
 
\end_layout

\begin_layout Standard
The list continues in a similar way, for quite a while.
 The green line suggests that if some 30 or so pathological cases are ignored,
 the system settles down to a more respectable behavior.
 Entries 30 through 50 in the rankings are "Bay" "Street" "Island" "of"
 "century" "right" "Game" "Georgian" "or" "a" ";" "near" "Party" "team"
 "law" "Australia" "her" "research" "Church" "east" "Government".
 Notable is a preponderance of capitalized words, suggesting more tables
 of various sorts, and a complete lack of verbs.
 A spot-check of words like 
\begin_inset Quotes eld
\end_inset

team
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

law
\begin_inset Quotes erd
\end_inset

 shows that the pathological behavior continues.
 Several conclusions are possible.
\end_layout

\begin_layout Standard
One conclusion is that there is a severe shortage of verbs in Wikipedia
 articles, and this makes sense: its primarily descriptive, rather than
 active: running, jumping, hitting, putting, mixing, giving, setting are
 not the kinds of verbs that are required to describe a typical encyclopedia
 topic.
 
\end_layout

\begin_layout Standard
Another conclusion is that perhaps the number of observations of pairs are
 insufficient to get deep, reliable MST parsing.
 Junk links get used because there were not enough appropriate word-pairs
 seen to give a good-quality MST parse.
 A related conclusion is that the connector-set dataset is also too thin:
 The grammatically reasonable connectors are observed not even a few dozen
 times, barely pushing them out of the noise-floor of onesie-twosie observations
 of junk.
 
\end_layout

\begin_layout Standard
So: bigger datasets, and an urgent need for non-Wikipedia content.
 Fiction, and presumably teen fiction should be filled with the kinds of
 active verbs describing human motions and actions, and should be absent
 of tables and lists masquerading as sentences.
\end_layout

\begin_layout Subsection
Hubiness
\end_layout

\begin_layout Standard
Similar to the above, hubiness can be defined as the second moment of the
 connector count:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
hub(w)=\frac{\sum_{d,c}N(w,d)C^{2}(d,c)}{N(w,*)}-K^{2}(w)
\]

\end_inset

Given the earlier Zipfian results on the average degree 
\begin_inset Formula $K(w)$
\end_inset

, it should be no surprise that a ranked listing of words by hubiness is
 very nearly identical to the listing for average degree.
 This is, after all, what the scale-free nature of the Zipfian distribution
 really means.
 Not only is the ranking nearly the same, but one also has the approximate
 equality 
\begin_inset Formula $hub(w)\approx2K(w)$
\end_inset

 to some ten or twenty percent.
 
\end_layout

\begin_layout Section
Grammatical Classes
\end_layout

\begin_layout Standard
The primary intended use of the connector set dataset is to provide input
 to the measurement of the syntactic (and semantic) similarity of words,
 so that they can be organized into grammatical classes (colloquially, into
 parts of speech).
 Two words are grammatically similar if they share many disjuncts in common,
 that is, if they are used in sentences in much the same walk.
 Thus, for example, one might imagine that the words 
\begin_inset Quotes eld
\end_inset

run
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

trot
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

jog
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

walk
\begin_inset Quotes erd
\end_inset

 might get used in similar ways, sometimes being almost interchangeable.
 Thus, it is interesting to develop metrics by which similarity might be
 gauged.
\end_layout

\begin_layout Standard
A number of different metrics are possible, and many are proposed in the
 section that follows.
 Two are of particular interest.
 The first is the cosine similarity; this is interesting, if for no other
 reason than that it is widely defined and used.
 The second is the entropic similarity.
 Thus lurks in many contexts under the name of 
\begin_inset Quotes eld
\end_inset

mutual information
\begin_inset Quotes erd
\end_inset

; however, the term 
\begin_inset Quotes eld
\end_inset

mutual information
\begin_inset Quotes erd
\end_inset

 has already been heavily used in the previous sections, and the entropic
 similarity is something different; it resembles the cosine similarity,
 but has much nicer properties, when one is concerned with maximizing the
 information content over large datasets.
 
\end_layout

\begin_layout Standard
A related, but very important task, when judging the similarity of words,
 and attempting to classify them into distinct grammatical classes is that
 sometimes the same word, 
\emph on
viz
\emph default
, the same text-string can belong to two different parts of speech.
 A casual perusal of any dictionary will show that many words can be used
 both as nouns, and as verbs.
 Other words can be nouns or adjectives; yet others can be subclasses: 
\emph on
e.g.

\emph default
 transitive and intransitive verbs.
 Disjuncts can be thought of as hyper-fine-grained parts-of-speech identifiers,
 and so one task of categorization is to place them into larger, less crude
 categories.
 Another important realization is to observe that different disjuncts correlate
 strongly with different semantic meanings; for a particularly brutal example,
 the noun 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

 has a semantic meaning that is unrelated to the past tense of the verb
 
\begin_inset Quotes eld
\end_inset

to see
\begin_inset Quotes erd
\end_inset

.
 Which is intended is readily discerned from context; disjuncts encode content.
\end_layout

\begin_layout Standard
A more extensive discussion of the relationship between meaning and disjuncts
 is given elsewhere;
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2018stiching"

\end_inset

 it is touched on only briefly below, as it informs what properties of a
 similarity metric are desirable, and which are not.
\end_layout

\begin_layout Subsection
Matrix Factorization
\end_layout

\begin_layout Standard
One can pose the problem of organizing words into grammatical categories
 as a matrix factorization problem.
 In the current framework, this means factorizing the matrix
\begin_inset Formula 
\[
p\left(w,d\right)=\sum_{g\in G}p^{\prime}\left(w,g\right)p^{\prime\prime}\left(g,d\right)
\]

\end_inset

so that 
\begin_inset Formula $p^{\prime}\left(w,g\right)$
\end_inset

 gives the probability that the word 
\begin_inset Formula $w$
\end_inset

 belongs to the grammatical category 
\begin_inset Formula $g$
\end_inset

, while 
\begin_inset Formula $p^{\prime\prime}\left(g,d\right)$
\end_inset

 gives the probability that the disjunct 
\begin_inset Formula $d$
\end_inset

 is appropriate for use with the grammatical category 
\begin_inset Formula $g$
\end_inset

.
 This factorization feeds back into itself, as the disjuncts are necessarily
 composed of words, which must now be reassigned into classes.
 
\end_layout

\begin_layout Standard
There are many widely discussed algorithms for performing the above factorizatio
n, falling under the general name of 
\begin_inset Quotes eld
\end_inset

low-rank matrix approximation
\begin_inset Quotes erd
\end_inset

 (LRMA) techniques.
 These are discussed in greater detail in 
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2018skippy"

\end_inset

.
\end_layout

\begin_layout Subsection
Clustering
\end_layout

\begin_layout Standard
An alternative to applying LRMA techniques, one can try to guess at the
 grammatical categories by looking for words that are similar to each other,
 and so invoke clustering algorithms to obtain grammatical categories.
 As clustering is inherently point-wise and not global, it is a generally
 weaker technique.
 That is, clustering generally looks at just pairs of words, and 
\begin_inset Quotes eld
\end_inset

ignores
\begin_inset Quotes erd
\end_inset

 other evidence in making the decision as to whether two words might go
 into the same grammatical category.
 Worse, if the clustering problem is posed naively, then it has a naive
 and incorrect answer: each word can belong to only one class.
 This ignores the fact that a word might belong to multiple classes 
\emph on
viz.

\emph default
 that class membership is a probability 
\begin_inset Formula $p^{\prime}\left(w,g\right)$
\end_inset

.
 It also ignores the possibility that an observed word-vector might be the
 sum of observations of two very different words that just happen to have
 the same spelling: a canonical example being 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

, which can be the past tense of 
\begin_inset Quotes eld
\end_inset

to see
\begin_inset Quotes erd
\end_inset

, the noun that is a cutting tool, and the verb that is the cutting action.
 Although in call cases, one has 
\begin_inset Formula $w=\mbox{saw}$
\end_inset

, the collection of disjuncts associated to each are likely to be very different.
 That is, the distribution 
\begin_inset Formula $p^{\prime\prime}\left(g,d\right)$
\end_inset

 for 
\begin_inset Formula $g=\mbox{`cutting tool`}$
\end_inset

 will be very different from 
\begin_inset Formula $p^{\prime\prime}\left(g,d\right)$
\end_inset

 for 
\begin_inset Formula $g=\mbox{`past-tense-observation`}$
\end_inset

.
 
\end_layout

\begin_layout Section
Disjunct Cosine Similarity
\begin_inset CommandInset label
LatexCommand label
name "subsec:Disjunct-Cosine-Similarity"

\end_inset


\end_layout

\begin_layout Standard
The cosine similarity between two vectors is simply their inner product.
 In this case, given two words 
\begin_inset Formula $w_{1}$
\end_inset

 and 
\begin_inset Formula $w_{2}$
\end_inset

, it is given by 
\begin_inset Formula 
\begin{equation}
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}N(w_{1},d)N(w_{2},d)}{\mbox{len}(w_{1})\mbox{len}(w_{2})}\label{eq:cosine-sim}
\end{equation}

\end_inset

where 
\begin_inset Formula $\mbox{len}(w)$
\end_inset

 is the root-mean-square length (Euclidean length) of the connector-set
 vector:
\begin_inset Formula 
\[
\mbox{len}(w)=\sqrt{\sum_{d}N^{2}(w,d)}
\]

\end_inset

This section explores the distribution of the cosine similarity for the
 
\noun on
en_pairs_rfive_mtwo
\noun default
 dataset (same as above).
\end_layout

\begin_layout Subsubsection
Rank distribution
\end_layout

\begin_layout Standard
The 
\noun on
en_pairs_rfive_mtwo
\noun default
 dataset (same as above) contains 797 words whose length is greater than
 or equal to 128.
 The ranking-by-length was already shown up above, in the graph 
\begin_inset CommandInset ref
LatexCommand nameref
reference "graph: ranked length"

\end_inset


\begin_inset CommandInset ref
LatexCommand vpageref
reference "graph: ranked length"

\end_inset

.
 The similarity between all pairs of these was computed; this resulted in
 
\begin_inset Formula $797\times796/2=317206$
\end_inset

 pairs.
 These can be sorted and ranked.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
See the good-sims and ranked-sims in disjunct-stats.scm file.
\end_layout

\end_inset

 They are shown below.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-sims.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
Well, that's new! The similarity ranking is very well fit by a parabola
 at the high end, and a cubic at the low end! In the above, the green line
 is given by 
\begin_inset Formula $1-\log^{2}(rank)/160$
\end_inset

 and while the blue line is given by 
\begin_inset Formula $-\log^{3}(rank/317206).$
\end_inset


\end_layout

\begin_layout Subsubsection
Hyperbolic space
\end_layout

\begin_layout Standard
Recall that
\begin_inset Formula 
\[
\cos\theta=1-\frac{\theta^{2}}{2}+\frac{\theta^{4}}{4!}-\cdots
\]

\end_inset

and so the above graph is stating that 
\begin_inset Formula $\theta\approx\log\left(rank\right)/8.9$
\end_inset

 is a reasonable approximation for the similarity angle, for the first thousand
 pairs, or so.
 Lest there be any hope that the graph as a whole is sine-wave-like, the
 high-rank tail goes as the cube of the log, not the square.
 
\end_layout

\begin_layout Standard
The similarity angle can be taken as a metric or as a distance.
 If the relationships between words is thought of as a network graph, one
 can ask how many nearest neighbors there are, as a function of the distance.
 Since the rank is effectively a count of nearest neighbors, and since 
\begin_inset Formula $\theta$
\end_inset

 is effectively a distance, one has that 
\begin_inset Formula 
\[
V\sim e^{r}
\]

\end_inset

where 
\begin_inset Formula $V$
\end_inset

 is the number of nearest neighbors (the 
\begin_inset Quotes eld
\end_inset

volume
\begin_inset Quotes erd
\end_inset

) and 
\begin_inset Formula $r$
\end_inset

 is the distance (so 
\begin_inset Formula $r=\theta$
\end_inset

 is just a change of notation).
 This volume-to-radius relationship should be compared to the volume-to-radius
 relationships for flat spaces for fixed dimension.
 For flat, Euclidean two-dimensional space, one has 
\begin_inset Formula $A=\pi r^{2}$
\end_inset

 the area of a circle.
 In three dimensions, 
\begin_inset Formula $V=4\pi r^{3}/3$
\end_inset

.
 For flat 
\begin_inset Formula $N$
\end_inset

-dimensional space, 
\begin_inset Formula $V\sim r^{N}$
\end_inset

.
 Clearly, the 
\begin_inset Quotes eld
\end_inset

space
\begin_inset Quotes erd
\end_inset

 of the network graph cannot be flat.
 This is in contrast to spaces with constant negative curvature, where the
 volume does grow exponentially as a function of the radius.
 Thus, one can hypothesize that the language graph can be mapped (almost)
 isometrically to a hyperbolic surface (for example, the Poincaré disk).
\end_layout

\begin_layout Standard
This conjecture promptly poses various questions: does the network graph
 have constant negative curvature, or is it bumpy? Numerically, what is
 the curvature? Does it embed better into hyperbolic 3-space or hyperbolic
 4-space? There is no such thing as a hyperbolic 
\begin_inset Formula $N$
\end_inset

-space for classical Riemann spaces, for 
\begin_inset Formula $N<2$
\end_inset

; yet, for the network graph, presumably one has 
\begin_inset Formula $N=0$
\end_inset

.
 How should this be described? 
\end_layout

\begin_layout Subsubsection
Size Cutoffs
\end_layout

\begin_layout Standard
The network graph has some nodes and links that are observed very frequently,
 while others are observed only very rarely, sometimes only once or twice.
 As mentioned in the section 
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Vertex-degree-in"

\end_inset

 on the vertex degree, the rarely-observed disjuncts appear to be associated
 with bad parses of lists, tables (such as tables of contents, product tables,
 pricing tables) and indexes.
 Therefore, it is appealing to imagine that by removing this 
\begin_inset Quotes eld
\end_inset

crud
\begin_inset Quotes erd
\end_inset

, one could improve the quality of the dataset.
 One obvious way of performing a cleanup is to banish infrequently-observed
 words.
\end_layout

\begin_layout Standard
In the 
\noun on
en_pairs_rfive_mtwo
\noun default
 dataset, there are 427 words observed 256 or more times, these form 
\begin_inset Formula $427\times426/2=90951$
\end_inset

 pairs.
 There are 245 words observed 512 or more times; these form 
\begin_inset Formula $245\times244/2=29890$
\end_inset

 pairs.
 There are 130 words observed 1024 or more times, these form 
\begin_inset Formula $130\times129/2=8385$
\end_inset

 pairs.
 The ranking for these are shown below.
 There are 69 words observed 2048 or more times; these form 
\begin_inset Formula $69\times68/2=2346$
\end_inset

 pairs.
 The ranking of these are shown below.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/ranked-graded-sims.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
The graph below shows the distribution of cosine similarity.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Produced by binned-good-sims.
 
\end_layout

\end_inset

 There are 797 words for which 
\begin_inset Formula $128<\mbox{len}(w)$
\end_inset

.
 This shows the distribution of the 317206 word-pairs formed from these
 words, for which 
\begin_inset Formula $0.1\le\mbox{sim}(w_{1},w_{2})$
\end_inset

.
 The eyeballed fit is for 
\begin_inset Formula $\exp(-3.5\times\mbox{sim})$
\end_inset

.
 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-sims.eps
	width 70text%

\end_inset


\end_layout

\begin_layout Standard
One can also be less restrictive.
 The graph below shows the distribution of 1058120 pairs, with the length
 restriction loosened to 
\begin_inset Formula $4<\mbox{len}(w)$
\end_inset

.
 It is, however, an incomplete dataset; not all possible pairs were considered.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-dj/binned-sims-all.eps
	width 60col%

\end_inset


\end_layout

\begin_layout Standard
It has a rather different structure to it.
 Its not clear what this means; this may be an artifact of it being an incomplet
e dataset...
 
\end_layout

\begin_layout Subsubsection
Top Ten Lists
\end_layout

\begin_layout Standard
Its worth getting an idea of what the most similar words are, and why they
 are considered similar.
\end_layout

\begin_layout Standard
The top-ten similarity pairs (for which 
\begin_inset Formula $128<\mbox{len}(w)$
\end_inset

) are: 'Stats ..
 Category' 'Notes ..
 Summary' 'Stats ..
 Rating' 'Category ..
 Rating' 'She ..
 He' 'Category ..
 Fandom' 'Stats ..
 Fandom' 'Category ..
 Character' 'Stats ..
 Character' 'she ..
 he'.
 The first one has a cosine of exactly 1.0, and the rest are above 0.975.
 
\end_layout

\begin_layout Standard
The two words 
\begin_inset Quotes eld
\end_inset

Stats
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

Category
\begin_inset Quotes erd
\end_inset

 have exactly one disjunct: it is 'LEFT-WALL- :+' That is, these appear
 as the first word in a sentence, and are immediately followed by a colon.
 The word 'Notes' has 27 distinct disjuncts, but only three are observed
 more than 6 times: 'Chapter- or LEFT-WALL- or End-' That is, 'Notes' appears
 either all by itself, or as the phrase 'Chapter Notes' or as 'End Notes'.
 The word 'Summary' has 47 distinct disjuncts, but only two appear more
 than 6 times: 'Chapter- or LEFT-WALL-' i.e.
 either as a solitary word, or as the phrase 'Chapter Summary'.
 This is why 'Notes' and 'Summary' are considered to be so similar.
\end_layout

\begin_layout Standard
Continuing in this fashion: 'Rating' has 7 disjuncts, four of which are
 observed more than 6 times: '(LEFT-WALL- :+) or (LEFT-WALL- :+ Audiences+)
 or (LEFT-WALL- :+ Explicit+) or (LEFT-WALL- :+ Mature+)' This suggests
 it appears in a table of some sort.
 The first disjunct '(LEFT-WALL- :+)' accounts for its high similarity to
 both 'Stats' and 'Category'.
 So, yes, all these words have been discovered to behave in a grammatically
 similar fashion; however, this behavior is somewhat boring: they arise
 from the regularity of some table listing.
\end_layout

\begin_layout Standard
The first non-capitalized word-pair appears at position ten.
 Of the top thirty, the table below shows the non-capitalized word-pairs.
 
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="13" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
cosine
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word-pair
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.977
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'she ..
 he'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.955
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'guess ..
 suppose'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.954
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'city ..
 house'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.950
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'would ..
 might'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.945
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
're ..
 non'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.944
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'village ..
 city'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.931
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'father ..
 mother'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.928
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'village ..
 house'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.928
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'don ..
 sama'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.923
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'world ..
 city'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.921
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'son ..
 daughter'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
29
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.918
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'suppose ..
 hope'
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The table below shows several more pairs worth closer examination.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="8" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rank
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
cosine
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word-pair
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.915
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'though ..
 but'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.914
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'should ..
 might'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
54
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.893
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'should ..
 must'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
61
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.885
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'when ..
 until'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
67
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.881
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'leave ..
 take'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
68
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.8794
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'believe ..
 think'
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
69
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.879
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
'in ..
 by'
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Subsubsection
Vector Structure
\end_layout

\begin_layout Standard
It is worth looking at a few of the vectors, to see what form they actually
 have.
 
\end_layout

\begin_layout Standard
The word 'she' has 27578 distinct disjuncts on it; the word 'he' has 57330
 disjuncts.
 The table below shows the top-ranked disjuncts for each.
 It makes quite clear why the two have a high similarity.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Print the disjuncts with the `print-disjuncts` function.
\end_layout

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="9" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
she
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
he
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2501
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
said+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4673
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
said+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1060
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
was+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2643
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
was+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
983
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
had+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1959
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
had+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
656
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
asked+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1293
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
could+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
593
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
”-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1281
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
asked+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
474
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
could+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
909
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
that- was+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
372
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
that- was+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
813
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
”-
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The list above contains the pairs 'guess ..
 suppose' and also 'suppose ..
 hope'.
 The word 'guess' has 859 disjuncts on it.
 The word 'suppose' has 805 disjuncts on it.
 The word 'hope' has 1735 disjuncts on it.
 The table below indicates why their cosine distance is close.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="10" columns="6">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
guess
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
suppose
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
hope
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
859 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
805 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1735 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
272
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
377
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
393
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
57
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
66
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- you+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
160
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- you+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- it's+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
45
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
77
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
that+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- you're+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
39
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- that+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
59
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
of+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- so+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- it+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
55
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- that+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
could-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- ?+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
51
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
we- that+ you+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- you+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- ,+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- so+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The pair 're ..
 non' seems strange, but closer examination shows that both are usually
 followed by a hyphen, and that this accounts for all of the observed similarity
 between these two.
 The pair 'don ..
 sama' is also strange.
 The similarity is due entirely to their linking to dashes and other punctuation.
 It seems to be due to a Finnish text that has snuck into the corpus.
\end_layout

\begin_layout Standard
Some more verbs are worth looking at.
 Here is 'leave ..
 take', below.
 Almost all of the similarity is due the infinitive form; the only other
 shared disjunct appearing in the table is 'to- it+', although there are
 more common disjuncts at lower counts.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
leave
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
take
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3017 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5693 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
298
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
619
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
40
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- the+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
105
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
would-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
95
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- care+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
took-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
88
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
will-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
you-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
87
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- a+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- him+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
74
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- it+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- room+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
70
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- her+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- it+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
67
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- it+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The pair 'believe ..
 think' is as follows; several kinds of constructions are clearly shared.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="10" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
believe
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
think
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2190 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5462 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
339
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1321
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
I-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
116
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- that+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
308
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
you-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
94
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
265
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
don't-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
79
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to- that+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
244
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
54
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
that+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
211
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
do- you-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
51
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- you+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
202
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- it+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
51
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
 I- it+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
168
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- you+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The first prepositions in the list are 'in ..
 by' and so are worth a look.
 Most of the similarity is accounted for by having them precede a determiner.
 Why there is a strong link to a determiner is unclear.
 Whether there is also a strong link between the determiner and a subsequent
 noun is also unclear.
 
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="10" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
by
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
64011 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20907 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12484
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
the+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2064
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4776
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
497
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4720
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
his+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
271
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
followed-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1529
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
front+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
266
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
his+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1429
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
her+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
252
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
surrounded-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1113
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
this+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
214
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
which+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1092
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
my+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
207
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
means+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The pair 'would ..
 might' is accompanied by 'should ..
 might' and 'should ..
 must'.
 The disjuncts are shown below.
 The reason for the similarity is readily apparent, and corresponds with
 what might be expected.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="9" columns="8">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
would
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
should
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
might
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
must
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20039 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6388 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6861 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5860 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1648
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
have+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1064
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
have+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
654
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
have+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
666
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
have+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1510
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
840
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
468
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
558
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
be+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
621
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
it- be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
415
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
154
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
he- have+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
215
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
it- be+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
445
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
he- have+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
246
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I- be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
150
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
it- be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
195
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
you-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
315
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
he- be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
155
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
we-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
91
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
he- be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
183
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
I-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
290
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
not+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
134
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
he- be+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
89
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
I-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
137
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
we-
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The words 'city', 'house', 'village' and 'world' are all similar.
 These are shown below.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="12" columns="8">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
city
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
house
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
village
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
world
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1332 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3330 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1019 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell multicolumn="1" alignment="center" valignment="top" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3024 total obs
\end_layout

\end_inset
</cell>
<cell multicolumn="2" alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
disjunct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
258
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
598
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
157
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
646
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
96
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- ,+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
275
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- ,+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
57
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
265
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- .+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
94
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
226
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
57
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- ,+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
202
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- ,+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- and+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
80
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
street+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
158
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in- the-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
this-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
79
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
into- the-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- Mbonga+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
117
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in- the- .+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- of+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
58
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- and+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
112
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in- the- ,+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
The- the- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
54
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
his-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
their-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
82
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
this-
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- the- .+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
48
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
my-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- and+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
43
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
.+
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
13
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a-
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
45
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- was+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- of+
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
39
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the- is+
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The above table clearly indicates why the cosine similarity between these
 four words is high.
 Yet, it is also disappointing: the primary reason is that they all take
 the determiner 'the', and occur at the end of phrases (the- ,+) or at the
 end of sentences (the- .+).
 This seems superficial, at best.
 After that, there's not so much similarity, and what there is still falls
 back onto the presence of determiners.
\end_layout

\begin_layout Standard
The similarity above is predicated on the frequent pairing of a word with
 a determiner.
 The pairing itself is correct, but perhaps not entirely significant: the
 word-pairs that needed to appear in the MST parse to extract these disjuncts
 had almost abysmally low MI values.
 They were clearly high enough to allow the MST parse to move forward, but
 are not otherwise terribly promising.
 They are shown below.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="9" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word-pair
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
fractional mi
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the city
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.7827
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the house
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.4228
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the village
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.6635
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the world
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.0578
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
city .
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.0167
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
house .
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.0095
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
village .
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.8765
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
world .
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.2255
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset CommandInset label
LatexCommand label
name "Table:prototype table"

\end_inset


\end_layout

\begin_layout Standard
Recall that, for word-pairs, that MI scores below 4 are considered to be
 quite poor.
 That the scores above are poor is not surprising: determiners link with
 almost any noun, but not to verbs: this is enough to get them into the
 2-3 range for MI.
 By contrast, the period can appear after almost any word, except maybe
 for prepositions, adjectives and adverbs.
 This is apparently enough to drive the MI positive, but that's all.
\end_layout

\begin_layout Standard
The above suggests that using the raw observation count to compute the cosine
 similarity is perhaps not the best way to compute similarity.
 
\end_layout

\begin_layout Subsection
Score Cosine
\end_layout

\begin_layout Standard
The above results suggest that perhaps the correct way to assign a score
 is to total up the word-pair MI scores for each disjunct, and use that
 to form a cosine.
 That is, define
\begin_inset Formula 
\[
sc(w,d)=\sum_{c\in d_{+}}mi(w,c)+\sum_{c\in d_{-}}mi(c,w)
\]

\end_inset

where 
\begin_inset Formula $d_{+}\subseteq d$
\end_inset

 is the subset of connectors that connect to the right, and 
\begin_inset Formula $d_{-}\subseteq d$
\end_inset

 is the subset of connectors that connect to the left.
 Here, 
\begin_inset Formula $c\in d$
\end_inset

 is simply one of the words that the connector is connecting to.
 Then, 
\begin_inset Formula $mi(w,c)$
\end_inset

 and 
\begin_inset Formula $mi(c,w)$
\end_inset

 are the word-pair MI scores.
 The cosine similarity might then be defined as 
\begin_inset Formula 
\[
\mbox{sos}(w_{1},w_{2})=\frac{\sum_{d}sc(w_{1},d)sc(w_{2},d)}{\sqrt{\sum_{d}sc{}^{2}(w_{1},d)}\sqrt{\sum_{d}sc^{2}(w_{2},d)}}
\]

\end_inset


\end_layout

\begin_layout Standard
The idea here is that connectors to punctuation, and connectors to determiners,
 which might be observed quite often, but have a naturally low MI score,
 will contribute relatively little to the overall cosine similarity.
 
\end_layout

\begin_layout Standard
Why is this a reasonable thing to do? Well, the previous two tables already
 suggested that the four words 'city' 'house' 'village' and 'world' should
 be placed into the same grammatical class, simply because of the high frequency
 with which they connect to a determiner.
 More subtle differences carrying a semantic signal might be getting washed
 out in this process.
 Perhaps the sos cosine score would be more sensitive to those processes.
\end_layout

\begin_layout Section
Word Senses
\end_layout

\begin_layout Standard
Words, considered as text strings, often correspond to multiple senses.
 Conversely, many words have similar senses.
 One would like to induce what these senses and word categories are, based
 on the observed counts.
\end_layout

\begin_layout Subsection
Prototype Theory
\end_layout

\begin_layout Standard
Another way to interpret the table 
\begin_inset CommandInset ref
LatexCommand vpageref
reference "Table:prototype table"

\end_inset

 is that the highest MI entry defines the 
\begin_inset Quotes eld
\end_inset

prototype
\begin_inset Quotes erd
\end_inset

 for the class of similar words.
 That is, the cosine score suggests that (world, city, village, house) should
 be taken as a grammatical category.
 The highest-MI entry suggests that 
\begin_inset Quotes eld
\end_inset

world
\begin_inset Quotes erd
\end_inset

 should be taken as the prototype for this class.
 
\end_layout

\begin_layout Standard
Here, the intended sense for 
\begin_inset Quotes eld
\end_inset

prototype
\begin_inset Quotes erd
\end_inset

 is that of 
\begin_inset Quotes eld
\end_inset


\begin_inset CommandInset href
LatexCommand href
name "prototype theory"
target "https://en.wikipedia.org/wiki/Prototype_theory"

\end_inset


\begin_inset Quotes erd
\end_inset

 of cognitive semantics, which suggests that a good representative of a
 class provides the underlying 
\begin_inset Quotes eld
\end_inset

meaning
\begin_inset Quotes erd
\end_inset

 of an otherwise flat description of the extensional and intensional qualities
 of a set.
 To create a bridge to the statistical tools available here, one might say
 that an intensional or an extensional description of a class is a flat,
 unweighted collection of features or members.
 In fact, some features or members are more important than others, so that
 a 
\begin_inset Quotes eld
\end_inset

robin
\begin_inset Quotes erd
\end_inset

 is a better representative of 
\begin_inset Quotes eld
\end_inset

birds
\begin_inset Quotes erd
\end_inset

 than a 
\begin_inset Quotes eld
\end_inset

penguin
\begin_inset Quotes erd
\end_inset

 is.
 There are at least two ways of judging a 
\begin_inset Quotes eld
\end_inset

better representative
\begin_inset Quotes erd
\end_inset

: frequency, and MI.
 A frequency approach means that 
\begin_inset Quotes eld
\end_inset

robins
\begin_inset Quotes erd
\end_inset

 are more prototypical of birds simply because cultural English speakers
 (Americans, Europeans) see robins more frequently than penguins, and so
 are acculturated to this prototype.
 The MI approach means that robins are more prototypical of birds because
 they have a high MI, when correlating intensional and extensional qualities.
 That is, a large value for 
\begin_inset Formula 
\[
mi(robin)=\sum_{feat\in bird-features}mi(robin,feat)
\]

\end_inset

states that robins have many bird-like qualities.
 Likewise, feathers are more representative of birds than weight or color,
 suggesting a large value for 
\begin_inset Formula 
\[
mi(feather)=\sum_{bird\in birds}mi(bird,feather)
\]

\end_inset

where the individual MI score was computed over all pairs (thing, feature)
 over the class of all things and all features.
 
\end_layout

\begin_layout Standard
What determines a category itself? Wikipedia summarizes this aptly:
\end_layout

\begin_layout Itemize
maximize the number of attributes shared by members of the category, and
 
\end_layout

\begin_layout Itemize
minimize the number of attributes shared with other categories.
\end_layout

\begin_layout Standard
Taking 
\begin_inset Quotes eld
\end_inset

attribute
\begin_inset Quotes erd
\end_inset

 to be a disjunct, and 
\begin_inset Quotes eld
\end_inset

member
\begin_inset Quotes erd
\end_inset

 to be a word, this suggests that a grammatical category is a collection
 of words that share many disjuncts (large overlap, large cosine similarity),
 while minimizing the disjuncts shared with other words (large MI).
\end_layout

\begin_layout Standard
The later point is perhaps the most important: MI scores provide a sense
 of 
\begin_inset Quotes eld
\end_inset

mutual exclusion
\begin_inset Quotes erd
\end_inset

: a word-disjunct pair has a high MI exactly when that disjunct is not shared
 with other words.
 This motivates a measure such as the 
\begin_inset Quotes eld
\end_inset

entropic similarity
\begin_inset Quotes erd
\end_inset

, given below.
\end_layout

\begin_layout Subsection
Entropic Similarity
\end_layout

\begin_layout Standard
The entropic similarity is similar to the cosine similarity, in that both
 start with the dot-product (inner product) between two vectors.
 As pointed in the 
\begin_inset Quotes eld
\end_inset

Gradient Descent vs.
 Graphical Models 
\begin_inset Quotes eld
\end_inset

 paper
\begin_inset CommandInset citation
LatexCommand cite
key "Vepstas2018skippy"

\end_inset

, the cosine similarity 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:cosine-sim"

\end_inset

 is inappropriate for probability spaces (because probability spaces are
 not orthogonal vector spaces).
 The entropic similarity seems to come closer to the desired form of similarity.
 Both definitions start with the dot-product of two vectors:
\begin_inset Formula 
\begin{equation}
\mbox{dot}\left(w_{1},w_{2}\right)=\sum_{d}p\left(w_{1},d\right)p\left(w_{2},d\right)\label{eq:dot-prod}
\end{equation}

\end_inset

with the disjunct 
\begin_inset Formula $d$
\end_inset

 playing the role of the basis elements.
 The cosine similarity of eqn 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:cosine-sim"

\end_inset

 can be written as 
\begin_inset Formula 
\[
\mbox{dot}\left(w_{1},w_{2}\right)=\frac{\mbox{dot}\left(w_{1},w_{2}\right)}{\sqrt{\mbox{dot}\left(w_{1},w_{1}\right)\mbox{dot}\left(w_{2},w_{2}\right)}}
\]

\end_inset

By comparison, the entropic similarity is defined as
\begin_inset Formula 
\[
\mbox{dmi}(w_{1},w_{2})=-\log_{2}\frac{\mbox{dot}\left(w_{1},w_{2}\right)\mbox{dot}\left(*,*\right)}{\mbox{dot}\left(w_{1},*\right)\mbox{dot}\left(*,w_{2}\right)}
\]

\end_inset


\end_layout

\begin_layout Standard
This looks perhaps a bit awkward.
 Some insight is gained by rewriting this in terms of a joint probability.
 Let 
\begin_inset Formula $M=\left[p\left(w,d\right)\right]$
\end_inset

 be the matrix whose rows correspond to words, and columns correspond to
 disjuncts.
 The value of any given matrix entry is then the frequency 
\begin_inset Formula $p\left(w,d\right)$
\end_inset

.
 The matrix 
\begin_inset Formula $MM^{T}$
\end_inset

 then has matrix entries
\begin_inset Formula 
\[
\left[MM^{T}\right]\left(w_{1},w_{2}\right)=\sum_{d}p\left(w_{1},d\right)p\left(w_{2},d\right)=\mbox{dot}\left(w_{1},w_{2}\right)
\]

\end_inset

Here, the notation 
\begin_inset Formula $M^{T}$
\end_inset

 denotes the matrix-transpose.
 Note that 
\begin_inset Formula $MM^{T}$
\end_inset

 is a symmetric matrix: 
\begin_inset Formula 
\[
\left[MM^{T}\right]\left(w_{1},w_{2}\right)=\left[MM^{T}\right]\left(w_{2},w_{1}\right)
\]

\end_inset

Normalizing this matrix, one obtains a joint probability matrix 
\begin_inset Formula $Q$
\end_inset

:
\begin_inset Formula 
\begin{equation}
Q\left(w_{1},w_{2}\right)=\frac{\left[MM^{T}\right]\left(w_{1},w_{2}\right)}{\sum_{u,v}\left[MM^{T}\right]\left(u,v\right)}=\frac{\left[MM^{T}\right]\left(w_{1},w_{2}\right)}{\left[MM^{T}\right]\left(*,*\right)}=\frac{\mbox{dot}\left(w_{1},w_{2}\right)}{\mbox{dot}\left(*,*\right)}\label{eq:symmetric transpose}
\end{equation}

\end_inset

This really is a joint probability, in that 
\begin_inset Formula $Q\left(*,*\right)=1$
\end_inset

 (by definition) and 
\begin_inset Formula $Q\left(w\right)=Q\left(w,*\right)=Q\left(*,w\right)$
\end_inset

 is a unique marginal probability (by symmetry).
 Here, it makes sense that the left and right marginal probabilities are
 the same: both are words, and there is nothing making the relationship
 asymmetric.
\end_layout

\begin_layout Standard
The point-wise symmetric mutual information between two words, aka the entropic
 similarity, can then be defined as
\begin_inset Formula 
\[
\mbox{dmi}(w_{1},w_{2})=\log_{2}\frac{Q\left(w_{1},w_{2}\right)}{Q\left(w_{1},*\right)Q\left(*,w_{2}\right)}=\log_{2}\frac{Q\left(w_{1},w_{2}\right)}{Q\left(w_{1}\right)Q\left(w_{2}\right)}
\]

\end_inset

Written in this fashion, this is clearly a form of mutual information between
 two words; we choose to call it the 
\begin_inset Quotes eld
\end_inset

entropic similarity
\begin_inset Quotes erd
\end_inset

 here, to avoid confusion with other definitions of 
\begin_inset Quotes eld
\end_inset

mutual information
\begin_inset Quotes erd
\end_inset

 throughout this text.
 The total entropic MI is then
\begin_inset Formula 
\[
\mbox{DMI}=\sum_{w_{1},w_{2}}Q\left(w_{1},w_{2}\right)\log_{2}\frac{Q\left(w_{1},w_{2}\right)}{Q\left(w_{1}\right)Q\left(w_{2}\right)}
\]

\end_inset


\end_layout

\begin_layout Subsection
Experimental Results
\end_layout

\begin_layout Standard
Both the cosine similarity 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:cosine-sim"

\end_inset

 and the entropic similarity are built upon the same dot-product 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:dot-prod"

\end_inset

.
 It is reasonable to think that the two should be correlated.
 This section explores that correlation.
\end_layout

\begin_layout Standard
Begin by a redo of the graphs from section 
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Disjunct-Cosine-Similarity"

\end_inset

.
 This uses the same dataset, the 
\noun on
en_pairs_rfive_mtwo,
\noun default
 but more conveniently trimmed down to a more manageable size 
\noun on
en_sh_two_sim.

\noun default
 
\end_layout

\begin_layout Standard
The first graphs shows the distribution of the 
\begin_inset Formula $N=640$
\end_inset

 top most frequently-observed words, all pairs, for a total of 
\begin_inset Formula $N(N-1)/2=204480$
\end_inset

 pairs; all pairs shown.
 This is effectively the same as the figures in section 4.1 but with two
 differences: there is no minimum-length cutoff, and the similarity scores
 have been extended down to zero (rather than a 0.1 cutoff, as before).
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-sims/binned-sims-dj-cosine.eps
	width 70col%

\end_inset


\end_layout

\begin_layout Standard
\align block
The fit of the linear section is eye-balled.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
See the `(mksims top-items)` and `sim-dj-cosine` in the `disjunct-stats.scm`
 file.
\end_layout

\end_inset

 It is not clear why the slope differs from the figures in section 4.1.
\end_layout

\begin_layout Standard
The corresponding figure for the entropic similarity is quite different:
 
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-sims/binned-sims-dj-mi.eps
	width 70col%

\end_inset


\end_layout

\begin_layout Standard
This shows the 
\begin_inset Formula $N=920$
\end_inset

 top words, for a total of 
\begin_inset Formula $N\left(N-1\right)/2=422740$
\end_inset

 pairs.
 The lone spike at the far left corresponds to an 
\begin_inset Formula $MI=-\infty$
\end_inset

, as appropriate for disjuncts with zero overlap (the cosine would be zero,
 too) For this dataset, this 
\begin_inset Formula $MI=-\infty$
\end_inset

 bin holds 14334 items.
 The smooth parabolic line is an eyeballed fit - a Gaussian, centered at
 an MI value of -3.5, and having a standard deviation of 3.
\end_layout

\begin_layout Standard
As both the entropic similarity and the cosine similarity are based on the
 same inner product, one expects them to be (highly?) correlated.
 The figures below correlation is show the correlation, for the top 
\begin_inset Formula $N=640$
\end_inset

 words:
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-sims/scatter-dj-mi-cos-dots.eps
	width 49col%

\end_inset


\begin_inset Graphics
	filename en-sims/scatter-dj-mi-cos.eps
	width 49col%

\end_inset


\end_layout

\begin_layout Standard
Both graphs show the same data.
 The left graph displays it as a point cloud; the right graph sorts the
 data by increasing entropic similarity, and then draws a line connecting
 the points, in sequential order.
 There is a very clear and very linear relationship between the two, but
 it is quite broad and noisy.
 There does seem to be a bounding-box effect: for any given log cosine,
 the entropic similarity is guaranteeed to lie in a fixed range, and vice-versa.
\end_layout

\begin_layout Standard
The distribution for the entropic similarity follows a square-root Zipfian
 distribution: 
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-sims/ranked-dj-mi.eps
	width 49col%

\end_inset


\end_layout

\begin_layout Standard
As always, the fit is eyeballed, but it appears to be going as 
\begin_inset Formula $ES=const.-\log_{2}\sqrt{rank}$
\end_inset

 (with the const presumably dependent on the size of the dataset).
 The sharp falloff is presumably an artifact of the limited dataset size
 (it has a total of 
\begin_inset Formula $422740$
\end_inset

 pairs for 
\begin_inset Formula $N=920$
\end_inset

 words, same as before).
\end_layout

\begin_layout Standard
The distribution of the cosine similarity is not as well-behaved.
 The figure on the left shows the log of the cosine, so as to make it more
 directly comparable to the entopic similarity, which is effectively logarithmic.
 The figure on the right shows the same data, but with a different vertical
 scale.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename en-sims/ranked-dj-log-cos.eps
	width 49col%

\end_inset


\begin_inset Graphics
	filename en-sims/ranked-dj-cos.eps
	width 49col%

\end_inset


\end_layout

\begin_layout Standard
A linear region, if any, doesn't even extend to the first 1000 ranked similariti
es.
 The figure on the right suggests a possibly parabolic fit, but that does
 not work out very well, either.
 (The cusp in the left figue is where the parabola crosses zero.) The right
 hand-side figure is the same sort of graph as given in section 
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Disjunct-Cosine-Similarity"

\end_inset


\end_layout

\begin_layout Standard
No data cutoffs were used to generate these figures.
 It is already clear that the vectors contain disjuncts with very low observatio
n counts, and that these low counts are often the result of low-quality
 MST parses.
 A leading cause for low-quality MST parses are infrequently-observed words.
 A given word might only occur in the corpus only a handful of times, which
 is not enough to establish it's syntactic behavior.
 Thus, disjuncts containing this word are likely to be incorrect, behaving
 no better than chance.
 it is not clear how much this affects judgements of word-similarity.
\end_layout

\begin_layout Standard
All very interesting, I think.
\end_layout

\begin_layout Section
\start_of_appendix
Other Similarity Scores
\end_layout

\begin_layout Standard
There are a number of different ways of judging similarity between a pair
 of words.
 It seems that these are limited only by the imagination.
 A variety of different similarity measures are given below.
 Each of these can be justified, to some degree, by hand-waving arguments
 and appeals to taste and conceptions of mathematical beauty.
 How to properly anchor these measures more firmly in a consistent framework
 is unclear.
\end_layout

\begin_layout Subsection
Quality Cosine
\end_layout

\begin_layout Standard
Some disjuncts are more important than others.
 This is indicated by the word-disjunct MI, and gives a different perspective
 than the word-disjunct frequency.
 This suggests that either a cosine score, or an MI-weighted overlap similarity
 may capture the grammatical similarity of words better.
\end_layout

\begin_layout Standard
The motivation for using cosine similarity is to find pairs of words that
 act in a grammatically similar fashion: they are used in the same way,
 with the same kinds of disjuncts.
 However, observational counts are subject to the vicissitudes of the input
 text: perhaps, the 
\begin_inset Quotes eld
\end_inset

quality
\begin_inset Quotes erd
\end_inset

 of the disjunct itself is better judged by it's MI.
 A disjunct could be judged as being 
\begin_inset Quotes eld
\end_inset

high quality
\begin_inset Quotes erd
\end_inset

 if 
\begin_inset Formula $mi(w,d)=MI_{pair}(w,d)$
\end_inset

 as defined in equation 
\begin_inset CommandInset ref
LatexCommand vref
reference "eq:mi-word-disjunct"

\end_inset

 is high.
 This motivates a 
\begin_inset Quotes eld
\end_inset

quality cosine
\begin_inset Quotes erd
\end_inset

: 
\begin_inset Formula 
\[
\mbox{qim}(w_{1},w_{2})=\frac{\sum_{d}mi(w_{1},d)mi(w_{2},d)}{\sqrt{\sum_{d}mi^{2}(w_{1},d)}\sqrt{\sum_{d}mi^{2}(w_{2},d)}}
\]

\end_inset

But what if the high quality is based on a pathetically small number of
 observations? Whenever one has a small number of observations, one has,
 almost automatically, a high MI value, simply because these two things
 are seen together, and nothing else is.
 Perhaps there should be some observational weighting.
 One can contemplate defining 
\begin_inset Formula $pi(w,d)=p(w,d)mi(w,d)$
\end_inset

 and so the cosine 
\begin_inset Formula 
\[
\mbox{pim}(w_{1},w_{2})=\frac{\sum_{d}pi(w_{1},d)pi(w_{2},d)}{\sqrt{\sum_{d}pi^{2}(w_{1},d)}\sqrt{\sum_{d}pi^{2}(w_{2},d)}}
\]

\end_inset

 The suitability of these different means of judging similarity is not clear.
\end_layout

\begin_layout Subsection
Overlap Similarity
\end_layout

\begin_layout Standard
Cosine similarity is perhaps too strict: it judges that two words are similar
 when they share not only the same collection of disjuncts, but have these
 occur with the same frequencies.
 Perhaps its enough to say that two words are grammatically similar, if
 they simply share the same set of disjuncts.
 This suggests ignoring the relative counts, or at least, sharply filtering
 them.
 Thus, the overlap similarity can be defined as 
\begin_inset Formula 
\[
\mbox{ovl}(w_{1,}w_{2})=\frac{\sum_{d}\sigma(N(w_{1},d))\sigma(N(w_{2},d))}{\sum_{d}\sigma(N(w_{1},d)+N(w_{2},d))}
\]

\end_inset

where 
\begin_inset Formula $\sigma(x)$
\end_inset

 is some sigmoid function.
 In the most basic case, its a step function: 
\begin_inset Formula $\sigma(x)=1$
\end_inset

 if 
\begin_inset Formula $x>C$
\end_inset

 for some constant 
\begin_inset Formula $C$
\end_inset

 and zero otherwise.
 Each term in the numerator sum is one only when both disjuncts are present.
 Each term in the denominator is one when either disjunct is present.
 
\end_layout

\begin_layout Standard
This similarity measure essentially boils down to a normalized cylinder-set
 measure.
 That is, instead of interpreting the disjuncts as a basis for a vector
 space, they can be taken as independent observations in a Cartesian product
 space.
 This makes more sense than pretending that these form a vector space.
 Why is that? What makes the cosine angle, and vector dot products so appealing,
 is that they are preserved by rotations; yet there really is no reason
 to expect or desire rotational symmetry.
 In short, the idea that we are dealing with vector spaces give the wrong
 idea of what's going on: its more appropriate to view the set of disjuncts
 as elements of a product space.
 Product spaces have a natural measure: the cylinder-set measure.
 The overlap similarity is essentially 
\begin_inset Formula $\mu(A\cap B)/\mu(A\cup B)$
\end_inset

 with 
\begin_inset Formula $\mu$
\end_inset

 the Borel measure, and the intersection/union being taken over the associated
 disjunct sets.
\end_layout

\begin_layout Standard
Continuing in this vein, what should be taken as the measure 
\begin_inset Formula $\mu(A)$
\end_inset

 of 
\begin_inset Formula $A$
\end_inset

? The counting measure is a natural choice; that is, 
\begin_inset Formula $\mu(A)=\left|A\right|$
\end_inset

 being the number of elements in set 
\begin_inset Formula $A$
\end_inset

.
 Because we also have a count associated with the members of the set, we
 can consider using the 
\begin_inset Formula $l_{p}$
\end_inset

 norms for the measure.
 That is, 
\begin_inset Formula $\left|A\right|$
\end_inset

 is just the 
\begin_inset Formula $l_{0}$
\end_inset

 norm; perhaps 
\begin_inset Formula $\mu(A)=\left\Vert A\right\Vert _{p}$
\end_inset

 could also work.
 For the purposes here, this may be enough; note, however, that only 
\begin_inset Formula $l_{0}$
\end_inset

 and 
\begin_inset Formula $l_{1}$
\end_inset

 satisfy one of the axioms of measure theory: namely, that when 
\begin_inset Formula $A\cap B=\varnothing$
\end_inset

 then 
\begin_inset Formula $\mu(A\cup B)=\mu(A)+\mu(B)$
\end_inset

.
 The other 
\begin_inset Formula $l_{p}$
\end_inset

 norms do not satisfy this.
\end_layout

\begin_layout Standard
The (opencog matrix) module currently implements an overlap similarity which
 computes the 
\begin_inset Formula $l_{0}$
\end_inset

-norm based similarity: 
\begin_inset Formula 
\[
\mbox{ovl}_{0}(w_{1},w_{2})=\frac{\left|\left\{ d\mbox{ s.t. }0<N\left(w_{1},d\right)\right\} \cap\left\{ d\mbox{ s.t. }0<N\left(w_{2},d\right)\right\} \right|}{\left|\left\{ d\mbox{ s.t. }0<N\left(w_{1},d\right)\right\} \cup\left\{ d\mbox{ s.t. }0<N\left(w_{2},d\right)\right\} \right|}
\]

\end_inset

where 
\begin_inset Formula $\left\{ d\mbox{ s.t. }0<N\left(w,d\right)\right\} $
\end_inset

 is the set of disjuncts 
\begin_inset Formula $d$
\end_inset

 such that the cset 
\begin_inset Formula $(w,d)$
\end_inset

 was observed at least once.
 As always, 
\begin_inset Formula $\left|\left\{ x\right\} \right|$
\end_inset

 is the number of elements in the set 
\begin_inset Formula $\left\{ x\right\} $
\end_inset

.
\end_layout

\begin_layout Standard
Some experimentation was done with this similarity measure, but the results
 were not particularly impressive.
 What does become quickly apparent is that the most frequently observed
 words will necessarily have a low similarity to the low-frequency words.
 This is because the high-frequency words will have a large number of disjuncts
 observed with them, thus causing the denominator to grow large.
 The numerator, by contrast, stays small, essentially limited by the number
 of disjuncts observed on the low-frequency word.
 This behavior is not what we want.
 Intuition suggests that we really do want to be able to compare words,
 independent of how frequently they occur.
\end_layout

\begin_layout Standard
This suggests a modified form of overlap similarity, by normalizing to a
 common count.
 That is, one should work with 
\begin_inset Formula $N(w,d)/N(w,*)$
\end_inset

, which can be understood as the normalized probability of observing a disjunct
 
\begin_inset Formula $d$
\end_inset

 on some word 
\begin_inset Formula $w$
\end_inset

.
 So, arrange the words such that 
\begin_inset Formula $N\left(w_{1},*\right)>N\left(w_{2},*\right)$
\end_inset

 and define 
\begin_inset Formula $K\left(w_{1},d\right)=N\left(w_{1},d\right)N\left(w_{2},*\right)/N\left(w_{1},*\right)$
\end_inset

 – this makes the counts on 
\begin_inset Formula $w_{1}$
\end_inset

 directly comparable to those on 
\begin_inset Formula $w_{2}$
\end_inset

, giving
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
\mbox{rovl}(w_{1},w_{2})=\frac{\left|\left\{ d\mbox{ s.t. }1\le K\left(w_{1},d\right)\right\} \cap\left\{ d\mbox{ s.t. }1\le N\left(w_{2},d\right)\right\} \right|}{\left|\left\{ d\mbox{ s.t. }1\le K\left(w_{1},d\right)\right\} \cup\left\{ d\mbox{ s.t. }1\le N\left(w_{2},d\right)\right\} \right|}
\]

\end_inset

That is, the condition 
\begin_inset Formula $1\le N\left(w_{2},d\right)$
\end_inset

 simply tests for the presence or absence of disjunct 
\begin_inset Formula $d$
\end_inset

 on 
\begin_inset Formula $w_{2}$
\end_inset

, while 
\begin_inset Formula $K\left(w_{1},d\right)$
\end_inset

 gives the probability of observing disjunct 
\begin_inset Formula $d$
\end_inset

 on 
\begin_inset Formula $w_{1}$
\end_inset

, if 
\begin_inset Formula $w_{1}$
\end_inset

 had been observed as often as 
\begin_inset Formula $w_{2}$
\end_inset

.
\end_layout

\begin_layout Standard
The above considerations suggest a more appropriate definition for overlap
 similarity that allows for frequency-independent observations: namely
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
\mbox{ovl}(w_{1,}w_{2})=\frac{\sum_{d}\sigma(f(w_{1},d))\sigma(f(w_{2},d))}{\sum_{d}\sigma(f(w_{1},d)+f(w_{2},d))}
\]

\end_inset

with 
\begin_inset Formula $f(w,d)=N(w,d)/N(w,*)$
\end_inset

.
 In this form, the numerator now starts to resemble the numerator of the
 cosine similarity measure: both the cosine numerator, and this numerator
 are computing a kind-of overlap or intersection of sets of disjuncts.
 The denominators differ.
 Based on gut intuition, the above measure may be be more suitable for judging
 similarity than the cosine measure, precisely because it emphasizes the
 set-like qualities of connector sets, as opposed to vector-like qualities.
 
\end_layout

\begin_layout Subsection
Disjunct Subsets
\end_layout

\begin_layout Standard
The above line of thinking suggests that another interesting comparison
 can be made by looking for a subset relationship between the disjuncts
 on different words.
 For example, transitive verbs should have all the connectors that intransitive
 verbs do, plus some more.
\end_layout

\begin_layout Standard
The overlay similarity can be adapted for this purpose: two sets obey a
 subset relation 
\begin_inset Formula $A\subset B$
\end_inset

 if 
\begin_inset Formula $A\cap(1-B)=\varnothing$
\end_inset

.
\end_layout

\begin_layout Standard
This re-affirms the previous observation: we expect connector-sets to behave
 in a set-like fashion, not a vector-like fashion.
\end_layout

\begin_layout Subsection
Quality Overlaps
\end_layout

\begin_layout Standard
Overlap alone may be too weak: again, some of the observed disjuncts may
 just be junk, or may correlate poorly with the word.
 This suggests several different approaches.
 One would be to trim the dataset to discard low-MI word-disjunct pairs,
 before computing overlap.
 Another would be to weight the overlap with the MI (discarding negative
 MI values).
\end_layout

\begin_layout Subsection
Connector Similarity
\end_layout

\begin_layout Standard
There is another, dual kind of similarity that is very different from the
 above proposals.
 A single disjunct is an ordered list of (pseudo-)connectors: 
\begin_inset Formula 
\[
d=\left(c_{1},c_{2},\cdots,c_{k}\right)
\]

\end_inset

where each pseudo-connector is a word, and a direction indicator.
 We can judge two connectors to be similar, and thus, two words to be similar,
 if they appear in a large number of disjuncts that would be identical,
 but for that one connector.
\end_layout

\begin_layout Standard
There are some practical difficulties of writing code to discern this.
\end_layout

\begin_layout Subsection
Word-Pair Cosine Similarity
\end_layout

\begin_layout Standard
To correctly gauge the advantage of disjunct-based techniques, one should
 compare them to the same kinds of measures, but applied to word-pairs,
 instead of word-disjunct pairs.
 For example, the disjunct-based cosine similarity, can be contrasted against
 the simpler word-pair-based cosine similarity.
 This is given by 
\begin_inset Formula 
\[
\mbox{sim}_{pair}(w_{1},w_{2})=\frac{\sum_{w}N_{pair}(w_{1},w)N_{pair}(w,w_{2})}{\mbox{len}(w_{1})\mbox{len}(w_{2})}
\]

\end_inset

where 
\begin_inset Formula $\mbox{len}(w)$
\end_inset

 is the root-mean-square length (Euclidean length) of the pair vector:
\begin_inset Formula 
\[
\mbox{len}(w)=\sqrt{\sum_{v}N_{pair}(w,v)N_{pair}(v,w)}
\]

\end_inset

and 
\begin_inset Formula $N_{pair}(w,v)$
\end_inset

 is the count of having observed the ordered word-pair 
\begin_inset Formula $(w,v)$
\end_inset

.
 Equivalently, writing the normalized frequency of observing a word pair
 as 
\begin_inset Formula $p(w,v)=N(w,v)/N(*,*)$
\end_inset

, this similarity can be written in the form 
\begin_inset Formula 
\[
\mbox{sim}_{pair}(w_{1},w_{2})=\frac{\sum_{w}p(w_{1},w)p(w,w_{2})}{\sqrt{\sum_{v}p(w_{1},v)p(v,w_{1})}\sqrt{\sum_{v}p(w_{2},v)p(v,w_{2})}}
\]

\end_inset

Note that this similarity measure is NOT symmetric: 
\begin_inset Formula $\mbox{sim}(w_{1},w_{2})\ne\mbox{sim}(w_{2},w_{1})$
\end_inset

.
 This is because it's built out of a manifestly non-symmetric count: 
\begin_inset Formula $p(w,v)\ne p(v,w)$
\end_inset

 and should really be written as 
\begin_inset Formula $p(w,v)=p(R;w,v)$
\end_inset

 with the relation 
\begin_inset Formula $R$
\end_inset

 encompassing all of the constraints of pair-wise word relationships (including,
 for example, that the pair might have been extracted from a random planar
 tree parse).
 Of course, one could construct a symmetrized similarity measure.
\end_layout

\begin_layout Standard
The point here is that this measure treats words as similar with they link,
 pair-wise, to the same kinds of words, with the same kinds of frequencies.
 This is not unlike the similarity that the disjunct-cosine is measuring,
 except that the disjunct carries additional grammatical information with
 it: it captures more complex relationships between the words in a sentence.
\end_layout

\begin_layout Subsection
Symmetric Cosine Information
\end_layout

\begin_layout Standard
The cosine similarity was defined as 
\begin_inset Formula 
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}N(w_{1},d)N(w_{2},d)}{\sqrt{\sum_{d}N^{2}(w_{1},d)}\sqrt{\sum_{d}N^{2}(w_{2},d)}}
\]

\end_inset

which, after dividing by 
\begin_inset Formula $N(*,*)$
\end_inset

 so that 
\begin_inset Formula $p(w,d)=N(w,d)/N(*,*)$
\end_inset

, gives the equivalent expression 
\begin_inset Formula 
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}p(w_{1},d)p(w_{2},d)}{\sqrt{\sum_{d}p^{2}(w_{1},d)}\sqrt{\sum_{d}p^{2}(w_{2},d)}}
\]

\end_inset

Comparing this to the expression for mutual information suggests that using
 the vector support, instead of the vector length, could be interesting.
 In particular, these might be interesting:
\begin_inset Formula 
\[
\mbox{com}(w_{1},w_{2})=-\log_{2}\frac{\sum_{d}p(w_{1},d)p(w_{2},d)}{p(w_{1},*)p(w_{2},*)}
\]

\end_inset

Written this way, it resembles a kind of mutual information.
\end_layout

\begin_layout Subsection
Symmetric Mutual Information (Entropic Similarity)
\end_layout

\begin_layout Standard
The symmetric mutual information, and related quantities, can be considered
 by working with the symmetric matrix 
\begin_inset Formula $Q$
\end_inset

 defined in eqn 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:symmetric transpose"

\end_inset

.
 This can be more closely related to the cosine by noting that it shares
 the vector dot product as a basic building block.
 That it, let 
\begin_inset Formula 
\[
\mbox{dot}(w_{1},w_{2})=\sum_{d}p(w_{1},d)p(w_{2},d)
\]

\end_inset

define a process between two 
\begin_inset Quotes eld
\end_inset

independent
\begin_inset Quotes erd
\end_inset

 processes 
\begin_inset Formula $w_{1}$
\end_inset

 and 
\begin_inset Formula $w_{2}$
\end_inset

.
 Looked at this way, the canonical mutual information would then be 
\begin_inset Formula 
\[
\mbox{dmi}(w_{1},w_{2})=-\log_{2}\frac{\mbox{dot}(w_{1},w_{2})\mbox{dot}(*,*)}{\mbox{dot}(w_{1},*)\mbox{dot}(*,w_{2})}
\]

\end_inset

One can normalize the process using the cosine similarity; the resulting
 process is different, and thus has a different value for the mutual information
:
\begin_inset Formula 
\[
\mbox{cmi}(w_{1},w_{2})=-\log_{2}\frac{\mbox{sim}(w_{1},w_{2})\mbox{sim}(*,*)}{\mbox{sim}(w_{1},*)\mbox{sim}(*,w_{2})}
\]

\end_inset

The 
\begin_inset Formula $\mbox{sim}(*,*)$
\end_inset

 serves to normalize the entire calculation, so that one is effectively
 computing with 
\begin_inset Formula $\mbox{norm-sim}(w_{1},w_{2})=\mbox{sim}(w_{1},w_{2})/\mbox{sim}(*,*)$
\end_inset

.
 Some experiments using cmi were performed, and it quickly became apparent
 that cmi does 
\begin_inset Quotes eld
\end_inset

the wrong thing
\begin_inset Quotes erd
\end_inset

: it singles out pairs that have a lot to do with each other, but little
 to do with anything else.
 Which discriminates 
\emph on
against
\emph default
 the clusters of similar words, which is not what we want.
 We don't want strange, unusual pairings; we want common, likely pairings.
 The level of discrimination can be severe: some really, really bad pairings
 show up, but only because they are unlike anything else.
 This score is strong when sim is weak, and is essentially picking up the
 tail-end of the sim distribution.
\end_layout

\begin_layout Subsection
Mutual Length
\end_layout

\begin_layout Standard
One can be inspired to write some crazy concoctions: 
\begin_inset Formula 
\[
\mbox{mim}(w,d)=-\log_{2}\frac{p(w,d)}{\sum_{w,d}p^{2}(w,d)}
\]

\end_inset

I don't know what to call these; the first seems to be some kind of 
\begin_inset Quotes eld
\end_inset

cosine information
\begin_inset Quotes erd
\end_inset

, the second, some sort of 
\begin_inset Quotes eld
\end_inset

mutual length
\begin_inset Quotes erd
\end_inset

 device.
\end_layout

\begin_layout Section
Cosine Similarity scatterplots
\end_layout

\begin_layout Standard
The scatterplot in 
\begin_inset CommandInset ref
LatexCommand ref
reference "fig:Cosine-similarity-scatterplot"

\end_inset

visualizes the cosine similarity between 797 word-pairs.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Generated with 'scatter.scm'
\end_layout

\end_inset

 The rows and columns are ranked by frequency of word occurrence, so that
 the single-most-frequently occurring word is at the upper-left, with less
 and less frequently occurring words proceeding to the right, and downwards.
 The color scale is such that red represents 1.0, yellow represents 0.75,
 green is exactly 0.5, and blue is 0.25 or less, fading to black.
 This is for the same 
\noun on
en_pairs_rfive_mtwo
\noun default
 dataset as above, and examined in detail in section 
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Disjunct-Cosine-Similarity"

\end_inset


\begin_inset CommandInset ref
LatexCommand vpageref
reference "subsec:Disjunct-Cosine-Similarity"

\end_inset

.
 That is, it computes the similarity between the 797 words whose length
 is greater than 128.
 Note that cosine similarity is symmetric; this figure is necessarily symmetric
 about the diagonal.
 The diagonal can be seen as a red line, representing a similarity of 1.0.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Float figure
placement h
wide false
sideways false
status open

\begin_layout Plain Layout
\align center
\begin_inset Caption Standard

\begin_layout Plain Layout
Cosine similarity scatterplot
\begin_inset CommandInset label
LatexCommand label
name "fig:Cosine-similarity-scatterplot"

\end_inset


\end_layout

\end_inset


\begin_inset Graphics
	filename scat/scat-cosine-big.png
	width 100col%

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
The tartan pattern indicates that some words are very unlike others, and
 that most words are very unlike one-another.
 
\end_layout

\begin_layout Standard
It would be nice to re-arrange (permute) the rows and columns to bring the
 matrix into quasi-diagonal form.
 This is easier said than done.
 The graph 
\begin_inset CommandInset ref
LatexCommand ref
reference "fig:Quasi-diagonal-cosine-similarity"

\end_inset

 shows the same data, with a different ranking.
 This time, the first word is the the LEFT-WALL, and the next word is the
 one word, out of 796, that has the highest cosine similarity to the period.
 Next comes the word, out of the remaining 795, that has the highest cosine
 similarity to the last.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed with the `dranked` function, and the `dranked-long` list graphed,
 from the `disjunct-stats.scm` file.
\end_layout

\end_inset

 This continues on down the list, so that word-pairs with the highest similarity
 are next to each-other, in the list.
 Effectively, we get the two-highest similarities for each word: the highest
 being to the word right before, and second-highest to the word right after.
 The list is organized so that the highest similarity pair occurs in the
 upper-left.
 The pairs least-similar to anything else end up on the lower-right.
 The color scheme is as before.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Float figure
wide false
sideways false
status open

\begin_layout Plain Layout
\align center
\begin_inset Caption Standard

\begin_layout Plain Layout
Quasi-diagonal cosine similarity
\begin_inset CommandInset label
LatexCommand label
name "fig:Quasi-diagonal-cosine-similarity"

\end_inset


\end_layout

\end_inset


\begin_inset Graphics
	filename scat/scat-fcos.png
	width 100col%

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
This ranking exposes block-diagonal structure in the dataset.
 The block in the upper-right consists entirely of punctuation and various
 capitalized words.
 The middle portions are quite interesting.
 The full list of 797 words is shown in table 
\begin_inset CommandInset ref
LatexCommand vpageref
reference "tab:Block-ranked-words"

\end_inset

.
 Its sort of interesting to read.
 So, the word that is most similar to the LEFT-WALL is the double-dash –
 perhaps not to surprising, as the double dash is often used to start new
 sentence phrases.
 That various forms of punctuation follow is not surprising.
 
\end_layout

\begin_layout Standard
After this are various capitalized words.
 As noted previously, capitalized words are observed far less frequently
 than their u-capitalized counterparts – usually only one capitalized word
 per sentence! Thus, capitalized words have far fewer disjuncts on them,
 thus making it easier for them to appear superficially similar, when they
 really should not be.
 Despite this, it is reassuring to see 
\begin_inset Quotes eld
\end_inset

Where Who What Why How
\begin_inset Quotes erd
\end_inset

 occur in succession.
 Other confidence-instilling sequences include 
\begin_inset Quotes eld
\end_inset

2 1 4 3 A
\begin_inset Quotes erd
\end_inset

 and some names: 
\begin_inset Quotes eld
\end_inset

Mai Demelza Richard John George
\begin_inset Quotes erd
\end_inset

.
 Then another reassuring sequence: 
\begin_inset Quotes eld
\end_inset

nothing something anything everything
\begin_inset Quotes erd
\end_inset

.
 Why 
\begin_inset Quotes eld
\end_inset

George
\begin_inset Quotes erd
\end_inset

 should be similar to 
\begin_inset Quotes eld
\end_inset

nothing
\begin_inset Quotes erd
\end_inset

 is best left unasked.
 But we'll ask anyway.
 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Float table
wide false
sideways false
status open

\begin_layout Plain Layout
\begin_inset Caption Standard

\begin_layout Plain Layout
Block-ranked words
\begin_inset CommandInset label
LatexCommand label
name "tab:Block-ranked-words"

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
LEFT-WALL – ...
 ..
 ....
 *** Two Bit - The “ And But So Perhaps Though When As Then While If Have
 Can Did Do Will Thank Are Where Who What Why How Or Now Here This That
 It There He She I You We They Some One Most Instead However Besides Well
 Oh Ah No Yes Yeah Good Ham 2 1 4 3 A Mai Demelza Richard John George nothing
 something anything everything it there who which that what where when until
 as because since if before while little small large good great second woman
 man child boy girl king house city village world town country ship sun
 latter land others children people men things them us me him her his my
 your their our the this every another leave take get find see hear speak
 move stop fight be make give show keep bring meet follow call ask tell
 help go live stay talk turn run change use read say understand remember
 know think suppose guess hope promise believe mean am myself fear thought
 knew saw felt found called happy fine certain new single word moment minute
 year day night evening morning way book letter party distance light fire
 forest room air sea earth ground floor table subject story line case question
 place time thing person family soul heart body own wife brother sister
 father mother daughter son hands face feet mouth eyes hand mind friends
 friend thoughts arm arms chest staff words office past water door window
 truth same whole best present first last least once home back down up at
 over on in by from upon under through into for with such having like quite
 only also still born gone already been seen done taken known given brought
 made making taking after above among between of all near against to will
 can cannot may must might would should could did does was is seems seemed
 appeared used began tried wanted needed meant came went turned continued
 said says spoke had has always never ever usual possible well soon far
 much late bad strong big young white black dark blue closed opened open
 hold put stand sleep answer try return come enough ready trying going supposed
 beginning rest sound corner direction presence name side power nature one
 most part kind sort number lot bit couple matter group other two three
 four five ten several many some out length force death life voice hair
 husband lips head shoulder eye bed business point state view sense feeling
 thinking saying now indeed therefore however sir yes God dear lady poor
 old dead wrong right here doing glad afraid sure sorry coming close hard
 long short real different clear probably certainly simply just almost not
 hardly easily feel have shall wish need want seem next following Queen
 captain wall future general London work himself herself smiled sighed nodded
 asked replied cried got stood sat looked look smile chance reason means
 longer doubt idea end edge bottom top account instead perhaps and or but
 though although then thus yet even really heard left lost passed met were
 are do love told gave took held followed behind within half an raised free
 better less more rather often true alone again ) too so very pretty a its
 these those anyone you we they he she North Jack Pitch Ross Jim Dallas
 Telzey Goth Sir san together along off away around round about how why
 – ; , : … ! .
 ? — All From In On By To With After For It’s It's Not Just Even At See
 paragraph spirit cause middle sight front spite charge form attention throat
 breath money law human few hundred thousand years hours minutes days times
 later ago ' " ” 」 ’ _ etc Mr Mrs Dr sama don ul looking especially nor
 either each any no finally suddenly slowly quietly Mary King Lord Lady
 Miss Captain I'm I’m My Her His Your ( ‘ Illustration Stats Category Rating
 Fandom Character Warning Is Really Please Father Mother May course being
 without let set sitting drawing living high heavy cold public fact particular
 full relief order hour individual General outside both talking able happened
 exactly else hell non re self S t six natural than fell deep wide deal
 agree don’t don't didn’t didn't received paid care drop provide York agreement
 ape medium author laws terms works forward forth shook enjoyed originally
 access permission instance chapter comment St Princess Old New Summary
 Notes Our THE # * Of { including Prince notes states copies copy ‐ YOU
 THIS OF OR Section CHAPTER Chapter + | Project eBook eBooks Gutenberg }
 P tax States fee electronic associated Archive Foundation United & copyright
 Literary donations License trademark distribution refund Van distributing
 Ooal Additional Posted archive Tags 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The block-ranked words are a kind of 
\begin_inset Quotes eld
\end_inset

stream-of-consciousness
\begin_inset Quotes erd
\end_inset

 with regard to similarity.
 This is shown in the table below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Created with the `prt-sim` function.
\end_layout

\end_inset

 So, the numbers are all quite similar to each other, but the letter 'A'
 is not very similar to '3'.
 It just so happens that 'A' is more similar to '3', than any other subsequent
 word.
 'A' and 'Mai' are not similar, but then all of the given names are quite
 similar.
 Next, it turns out that 'George' and 'nothing' really don't have very much
 to do with one-another.
 And so the block-diagonal structure is exposed.
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="23" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Pair
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Similarity
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2 ..
 1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.881
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1 ..
 4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.791
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4 ..
 3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.835
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3 ..
 A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.375
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A ..
 Mai
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.351
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Mai ..
 Demelza
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.615
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Demelza ..
 Richard
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.515
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Richard ..
 John
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.602
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
John ..
 George
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.648
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
George ..
 nothing
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.342
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
nothing ..
 something
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.483
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
something ..
 anything
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.549
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
anything ..
 everything
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.369
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
everything ..
 it
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.496
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
it ..
 there
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.743
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
there ..
 who
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.416
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
who ..
 which
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.618
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
which ..
 that 
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.580
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
that ..
 what
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.730
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
what ..
 where
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.722
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
where ..
 when
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.830
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
when ..
 until
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.886
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
Reading through the list of 797 words in table 
\begin_inset CommandInset ref
LatexCommand vpageref
reference "tab:Block-ranked-words"

\end_inset

 reveals a lot of interesting runs.
 All this suggests that cosine similarity really is doing the right thing.
 Equally reassuring are the failures of similarity: the end of the list
 of 797 words is populated with Gutenberg license boilerplate words.
 This is good: they are at the end of the list because they don't fit into
 any other grammatical usage patterns.
 They don't fit because they have been observed hundreds of times, which
 propels them into the high-frequency category; yet, since they always appear
 in set phrases, they can't be similar to anything else.
\end_layout

\begin_layout Subsection*
Filtering
\end_layout

\begin_layout Standard
An earlier draft of this report showed a very different figure, which turned
 out to have been created based on what seemed to be a valid assumption,
 but turns out not to be so wise.
 It deserves a some discussion here, which is resumed in a later section.
\end_layout

\begin_layout Standard
The idea (recaptured below) is that it can be a good thing to filter out
 noisy data.
 For example, one may choose to ignore words that have not been seen many
 times.
 This is the cut done up above: there are only 797 words observed with a
 length of 128 or greater.
 This is a convenient number only because it allows a 797-pixel-wide color
 plot to be made.
 A different cut might exclude words that were observed less than N times.
 
\end_layout

\begin_layout Standard
Another possibility, and this seems to be the fatal one, is to exclude disjuncts
 that are seen only a handful of times.
 Superficially, this seems like a very plausible thing to do.
 In practice, this turns out to be an almost completely disastrous cut.
 Setting it too high renders almost all capitalized words identical to one-anoth
er, having a similarity of exactly one.
 Capitalized words are observed only infrequently; discarding low-frequency
 disjuncts leaves the capitalized words almost naked, and thus essentially
 identical.
 Just about all of them connect to the LEFT-WALL, since they are the first
 word in a sentence, and so all of them share a very high observation count
 of links to LEFT-WALL, and low observation counts of links to anything
 else.
 Ergo: they are all similar, since they all start sentences.
\end_layout

\begin_layout Standard
The damage doesn't stop there.
 It turns out that such trimming also raises similarity across the board:
 it ends up so that everything seems to be pretty darn similar to everything
 else, thus erasing too much of the 
\begin_inset Quotes eld
\end_inset

signal
\begin_inset Quotes erd
\end_inset

 that we are looking for.
 The original hope was to raise the signal-to-noise ratio by applying judicious
 cuts; but hope is not enough.
 Injudicious cuts can destroy the signal all too easily.
 Careful data analysis is needed; blind trust is not enough.
\end_layout

\begin_layout Standard
\begin_inset CommandInset bibtex
LatexCommand bibtex
bibfiles "lang"
options "tufte"

\end_inset


\end_layout

\end_body
\end_document