learn-lang-diary-part-one.lyx

#LyX 2.3 created this file. For more info see http://www.lyx.org/
\lyxformat 544
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass article
\begin_preamble
\usepackage{url} 
\end_preamble
\use_default_options false
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding utf8
\fontencoding global
\font_roman "times" "default"
\font_sans "helvet" "default"
\font_typewriter "cmtt" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\use_microtype false
\use_dash_ligatures false
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks true
\pdf_pdfborder true
\pdf_colorlinks true
\pdf_backref false
\pdf_pdfusetitle true
\papersize default
\use_geometry false
\use_package amsmath 2
\use_package amssymb 2
\use_package cancel 1
\use_package esint 0
\use_package mathdots 1
\use_package mathtools 1
\use_package mhchem 0
\use_package stackrel 1
\use_package stmaryrd 1
\use_package undertilde 1
\cite_engine basic
\cite_engine_type default
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 0
\use_minted 0
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\is_math_indent 0
\math_numbering_side default
\quotes_style english
\dynamic_quotes 0
\papercolumns 1
\papersides 1
\paperpagestyle default
\listings_params "basicstyle={\ttfamily},basewidth={0.5em}"
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header

\begin_body

\begin_layout Title
Language Learning Diary - Part One
\end_layout

\begin_layout Date
2014-2020
\end_layout

\begin_layout Author
Linas Vepstas
\end_layout

\begin_layout Abstract
The language-learning effort involves research and software development
 to implement the ideas described in ArXiv abs/1401.3372
\begin_inset CommandInset citation
LatexCommand cite
key "Goertzel2014"
literal "true"

\end_inset

.
 This document contains supplementary notes and a loosely-organized semi-chronol
ogical diary of results.
\end_layout

\begin_layout Abstract
Because this document has repeatedly become overly large, it has been split
 into multiple sub-documents.
 Notable ones include the report on connector sets, the preliminary report
 on grammatical classes, and several reports on word-pairs.
 What remains here are assorted ad hoc commentary.
\end_layout

\begin_layout Abstract
Because what remains here is still too long to manage, the diary resumes
 in Part Two, as of 2021.
\end_layout

\begin_layout Section*
Introduction
\end_layout

\begin_layout Standard
The language-learning effort involves research and software development
 to implement the ideas described in in ArXiv abs/1401.3372
\begin_inset CommandInset citation
LatexCommand cite
key "Goertzel2014"
literal "true"

\end_inset

.
 This document contains supplementary notes and a loosely-organized semi-chronol
ogical diary of results.
 Its not actually chronological: in general, it is organized so that theory
 precedes data analysis.
 Usually.
\end_layout

\begin_layout Standard
The initial stages of this work require the extraction of word-pair probabilitie
s from raw text, and the use of these to induce a Link Grammar
\begin_inset CommandInset citation
LatexCommand cite
key "Sleator1991,Sleator1993"
literal "true"

\end_inset

.
 This extends prior work on MST parsers
\begin_inset CommandInset citation
LatexCommand cite
key "Yuret1998"
literal "true"

\end_inset

, by inducing link types for word-pair relations.
\end_layout

\begin_layout Standard
Later stages further extend beyond what is possible with Link Grammar by
 inducing synonymous words and phrases.
 The goal here is to unify into a consistent framework various techniques
 for unsupervised semantic discovery that have already been proven in narrower
 contexts
\begin_inset CommandInset citation
LatexCommand cite
key "Poon2009,Lin1998,Lin2001"
literal "true"

\end_inset

.
\end_layout

\begin_layout Standard
The first section of this document is a review of various definitions of
 probabilities that can be obtained from natural language text.
 This is followed by a roughly chronological diary of further observaions
 and results.
 Many revisions are made out of chronological order.
\end_layout

\begin_layout Subsection*
Lexical Attraction, Mutual Information, Interaction Information
\end_layout

\begin_layout Standard
The goal of this section is to clarify some of the formulas used by Deniz
 Yuret in his PhD thesis 
\begin_inset Quotes eld
\end_inset


\emph on
Discovery of Linguistic Relations Using Lexical Attraction
\emph default

\begin_inset Quotes erd
\end_inset

, MIT 1998 (
\begin_inset CommandInset href
LatexCommand href
name "http://www2.denizyuret.com/pub/yuretphd.pdf"
target "http://www2.denizyuret.com/pub/yuretphd.pdf"
literal "false"

\end_inset

).
 These formulas are vitally important, because they provide a strong tool
 when working with text; this has been shown by Yuret in his thesis, as
 well as by many others, as well as by my own practical experience with using
 them.
\end_layout

\begin_layout Standard
Possibly the most useful formula is the one in the middle of page 40.
 By the time that we get to it, the terms 
\begin_inset Quotes eld
\end_inset

mutual information
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

lexical attraction
\begin_inset Quotes erd
\end_inset

 are being used interchangeably.
 This formula states the 
\begin_inset Formula $MI(x,y)$
\end_inset

 for two words 
\begin_inset Formula $x$
\end_inset

 and 
\begin_inset Formula $y$
\end_inset

; yet it is manifestly not symmetric in 
\begin_inset Formula $x$
\end_inset

 and 
\begin_inset Formula $y$
\end_inset

, since 
\begin_inset Formula $x$
\end_inset

 is the word on the left, and 
\begin_inset Formula $y$
\end_inset

 is the word on the right.
 By contrast, textbook (wikipedia) definitions of MI are symmetric in their
 variables.
 Below I try to dis-entangle the resulting confusion a bit, and give a more
 correct derivation of the formula.
 The key is to observe that the formula contains an implicit pair-wise relations
hip between two words, and that there are actually three variables: two
 words, and their relationship.
 If this implicit relationship is made explicit, then the confusion evaporates.
 It also opens the door to talking about the MI (or the interaction information
 InI) of more complex relationships, not just pair-wise ones.
 
\end_layout

\begin_layout Standard
Being able to correctly write down the MI and the InI for complex relationships
 is important for NLP: relationtionships can be labelled by types (subject,
 object) and by word classes (noun, verb), and have various dependency constrain
ts between them.
 Thus, we need to be able to talk both about a labelled directed graph,
 and the entropy or mutual information contained in it's various sub-graphs.
\end_layout

\begin_layout Standard
In defense of Yuret, he does say, on page 22, that 
\begin_inset Quotes eld
\end_inset

lexical attraction is the likelihood of a syntactic relation.
\begin_inset Quotes erd
\end_inset

 However, the relation starts becoming implicit by eqn 12 on page 29.
 An unexplained leap is then made from eqn 12 to the formula on page 40.
 The below gets fairly pedantic; this seems unavoidable to avoid confusion.
\end_layout

\begin_layout Subsubsection*
Definitions 
\end_layout

\begin_layout Standard
Let 
\begin_inset Formula $P(R(w_{l},w_{r}))$
\end_inset

 represent the probability (frequency) of observing two words, 
\begin_inset Formula $w_{l}$
\end_inset

 and 
\begin_inset Formula $w_{r}$
\end_inset

 in some relationship or pattern 
\begin_inset Formula $R$
\end_inset

.
 Typically, 
\begin_inset Formula $R$
\end_inset

 can be a (link-grammar) linkage of type 
\begin_inset Formula $t$
\end_inset

 connecting word 
\begin_inset Formula $w_{l}$
\end_inset

 on the left to word 
\begin_inset Formula $w_{r}$
\end_inset

 on the right; implicitly, both 
\begin_inset Formula $w_{l}$
\end_inset

 and 
\begin_inset Formula $w_{r}$
\end_inset

 occur in the same sentence.
 The goal of this discussion is to enable relations 
\begin_inset Formula $R$
\end_inset

 that are more general than this; for now, though, 
\begin_inset Formula $R$
\end_inset

 is a word-pair occurring in a single sentence.
\end_layout

\begin_layout Standard
The simplest dependency grammar language model has only one type 
\begin_inset Formula $t$
\end_inset

, the ANY type.
 This is the type that Yuret uses: it makes no distinction at all between
 subject, object relations (that is, all dependencies are unlabelled), and
 it does not make a head-dependent distinction (all dependencies are bi-directio
nal).
 Thus, in what follows, we do the same: initially, the relation 
\begin_inset Formula $R(w_{l},w_{r})$
\end_inset

 is simply the statement that the words 
\begin_inset Formula $w_{l}$
\end_inset

 and 
\begin_inset Formula $w_{r}$
\end_inset

 are connected by an unlabelled, un-directed edge.
 For this simplest case, what 
\begin_inset Formula $R(w_{l},w_{r})$
\end_inset

 does is to capture that 
\begin_inset Formula $w_{l}$
\end_inset

 is to the left of 
\begin_inset Formula $w_{r}$
\end_inset

.
 
\end_layout

\begin_layout Standard
In what follows, the relation 
\begin_inset Formula $R=R(w_{l},w_{r})$
\end_inset

 refers to a generic two-word relation, and not necessarily this simplest
 one.
 To regain Yuret's formula, use the simplest relation, the ordered word-pair
 relation, given just above.
\end_layout

\begin_layout Standard
The quantity of interest is the (unconditional) probability 
\begin_inset Formula $P(R(w_{l},w_{r}),w_{l},w_{r})$
\end_inset

 of observing the two words 
\begin_inset Formula $w_{l}$
\end_inset

 and 
\begin_inset Formula $w_{r}$
\end_inset

 in a relation 
\begin_inset Formula $R=R(w_{l},w_{r})$
\end_inset

.
 To correctly understand and work with this quantity, some care must be
 taken with the notation for several related probabilities.
 First, one has 
\begin_inset Formula $P(w)$
\end_inset

, the probability of observing the word 
\begin_inset Formula $w$
\end_inset

 in the data sample.
 Next, one has 
\begin_inset Formula $P(S(w_{1},w_{2}),w_{1},w_{2})$
\end_inset

, the probability that the two words occur in the same sentence.
 Again, 
\begin_inset Formula $S(w_{1},w_{2})$
\end_inset

 denotes a relation between the two words; it differs from 
\begin_inset Formula $R(w_{1},w_{2})$
\end_inset

 in that the word-order does not matter.
 A third kind of pair relation is the unconditional probability of observing
 two words, which can be 
\emph on
defined
\emph default
 as 
\begin_inset Formula $P(w_{1},w_{2})=P(w_{1})P(w_{2})$
\end_inset

.
 In this case, instead of assuming independence of two random variables,
 we define them to be so.
 This is possible, because we have a notation for specifying when there
 is a correlation.
 That is, if there was some correlation (relation) 
\begin_inset Formula $C(w_{1},w_{2})$
\end_inset

 between them, then one should write this explicitly, as 
\begin_inset Formula $P(C,w_{1},w_{2})=P(C(w_{1},w_{2}),w_{1},w_{2})$
\end_inset

.
 The notation here allows the various needed probabilities to be defined
 without ambiguity.
 
\end_layout

\begin_layout Standard
Thus, assumptions of independent variables are now replaced by a notational
 infrastrcture.
 Note, in particular, that if one uses a frequentist definition for the
 probabilities (as will be done in what follows), then the probabilities
 are not independent of the data sample from which they are drawn.
 Thus, all probabilities here have an implicit dependence on the data sample.
 This dependency is not explicitly shown.
 Some care must be taken to use the same data sample throughout.
 
\end_layout

\begin_layout Standard
The above notation allows the definition of conditional probabilities, in
 the conventional sense.
 For example, one has that 
\begin_inset Formula 
\[
P(R,w_{l},w_{r})=P(R|w_{l},w_{r})P(w_{l},w_{r})
\]

\end_inset

or that 
\begin_inset Formula 
\[
P(R|w_{l},w_{r})=\frac{P(R,w_{l},w_{r})}{P(w_{l},w_{r})}
\]

\end_inset

as the conditional probability of observing the relation 
\begin_inset Formula $R$
\end_inset

, given that it's component parts are observed.
 From the earlier definitions, the denominator factors, and so we conclude
 that the correct expression for the conditional probability is:
\begin_inset Formula 
\begin{equation}
P(R|w_{l},w_{r})=\frac{P(R,w_{l},w_{r})}{P(w_{l})P(w_{r})}\label{eq:cond-pair}
\end{equation}

\end_inset

This is the probability of observing the relationship 
\begin_inset Formula $R$
\end_inset

 given that the individual parts of the relationship have been observed.
 The relation 
\begin_inset Formula $R$
\end_inset

 includes all correlations between the two words: their ordering as well
 as their co-occurance in a sentence.
 
\end_layout

\begin_layout Standard
Take care, however: 
\begin_inset Formula $P(R|w_{l},w_{r})$
\end_inset

 is NOT the probability of seeing 
\begin_inset Formula $R$
\end_inset

, given that 
\begin_inset Formula $w_{l}$
\end_inset

 and 
\begin_inset Formula $w_{r}$
\end_inset

 occur in the same sentence.
 This would instead by given by 
\begin_inset Formula $P(R,w_{l},w_{r})/P(S,w_{l},w_{r})$
\end_inset

.
 This is an entirely different.
\end_layout

\begin_layout Subsubsection*
Frequentism - Counting words and pairs
\end_layout

\begin_layout Standard
In order to be usable, a computable definition for the probabilities must
 be given.
 For this, the definition can only be frequentist.
 That is, the probabilities are to be obtained from empircal data; from
 counting frequencies as they occur in data samples taken from nature.
 The frequency 
\begin_inset Formula $P(w)$
\end_inset

 of observing a word 
\begin_inset Formula $w$
\end_inset

 is obvious: 
\begin_inset Formula 
\[
P(w)=\frac{N(w)}{N(*)}
\]

\end_inset

where 
\begin_inset Formula $N(w)$
\end_inset

 is the count of observing word 
\begin_inset Formula $w$
\end_inset

 and 
\begin_inset Formula $N(*)$
\end_inset

 is the total number of words observed.
 That is, by definition, it is the wild-card summation 
\begin_inset Formula 
\[
N(*)=\sum_{w}N(w)
\]

\end_inset

How to count words is not entirely obvious, so even these definitions need
 care.
 There are several ways in which one can count words.
 One way is to simply count how many times a word occurs in the block of
 sample text.
 Another way is to count how many times a word occurs in parses of the sample
 text.
 These are not the same! For example, if a parse connects words by edges
 (by dependency-grammar relations), then one can count each word once, for
 each time that it occurs at the end of an edge.
 In this counting, the word-count is exactly double the word-pair count.
 A word is then counted multiple times, if it participates in multiple edges.
 If the sample text is parsed multiple times, then additional counts can
 result that way.
 To maintain consistency with the definitions given in the previous section,
 
\begin_inset Formula $N(w)$
\end_inset

 is defined to be the number of times that the word 
\begin_inset Formula $w$
\end_inset

 occurs in the data sample, and independent of any other relations that
 
\begin_inset Formula $w$
\end_inset

 might be engaged in.
 For now, it is assumed that the segmentation of the text sample into words
 is unambiguous.
 
\end_layout

\begin_layout Standard
Let 
\begin_inset Formula $F(S(w),w)$
\end_inset

 be the number of times (frequency) of observing word 
\begin_inset Formula $w$
\end_inset

 in any sentence 
\begin_inset Formula $S$
\end_inset

.
 This can be computed as
\begin_inset Formula 
\[
F(S,w)=\frac{N(w)}{NS}
\]

\end_inset

where 
\begin_inset Formula $N(w)$
\end_inset

 is the number of times a word 
\begin_inset Formula $w$
\end_inset

 was observed in a data sample, and 
\begin_inset Formula $NS$
\end_inset

 is the number of sentences in that same sample.
 This counts with 
\begin_inset Quotes eld
\end_inset

multiplicity
\begin_inset Quotes erd
\end_inset

, in that 
\begin_inset Formula $w$
\end_inset

 can appear in a sentence more than once.
 That is, 
\begin_inset Formula $F$
\end_inset

 is not a probability, rather, it is an expectation value of the number
 of times that a word is observed.
 This can be made explicit, by writing
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
F(S,w)=\frac{N(w)}{N(*)}\,\frac{N(*)}{NS}=P(w)L(S)
\]

\end_inset

with 
\begin_inset Formula $L(S)=F(S,*)$
\end_inset

 being the average sentence length (the expectation value of the number
 of words in a sentence).
 
\end_layout

\begin_layout Standard
Three different word-pair relationships are interesting.
 First, define the relation 
\begin_inset Formula $S(w_{1},w_{2})$
\end_inset

 as being the relation that both words 
\begin_inset Formula $w_{1}$
\end_inset

 and 
\begin_inset Formula $w_{2}$
\end_inset

 occur in the same sentence, but in arbitrary order.
 It is symmetric: 
\begin_inset Formula $S(w_{1},w_{2})=S(w_{2},w_{1})$
\end_inset

.
 Define 
\begin_inset Formula $A(w_{l},w_{r})$
\end_inset

 as being the relation that both words 
\begin_inset Formula $w_{l}$
\end_inset

 and 
\begin_inset Formula $w_{r}$
\end_inset

 occur in the same sentence, and that 
\begin_inset Formula $w_{l}$
\end_inset

 is to the left of 
\begin_inset Formula $w_{r}.$
\end_inset

 By this definition, the counts for the two are related: one has that 
\begin_inset Formula 
\[
N(S,w_{1},w_{2})=N(A,w_{1},w_{2})+N(A,w_{2},w_{1})
\]

\end_inset

This is the symmetrized count.
\end_layout

\begin_layout Standard
Neither of 
\begin_inset Formula $S$
\end_inset

 or 
\begin_inset Formula $A$
\end_inset

 is yet the relation 
\begin_inset Formula $R(w_{l},w_{r})$
\end_inset

 mentioned above, which is defined as being the relation that both words
 
\begin_inset Formula $w_{l}$
\end_inset

 and 
\begin_inset Formula $w_{r}$
\end_inset

 occur in the same sentence, that 
\begin_inset Formula $w_{l}$
\end_inset

 is to the left of 
\begin_inset Formula $w_{r}$
\end_inset

, and, most importantly, that there is a link-grammar link (of type 
\begin_inset Quotes eld
\end_inset

R
\begin_inset Quotes erd
\end_inset

) connecting the two.
 Observe that although 
\begin_inset Formula $A$
\end_inset

 can be deduced from 
\begin_inset Formula $S$
\end_inset

, there is no simple or obvious relation between 
\begin_inset Formula $S$
\end_inset

 and 
\begin_inset Formula $R$
\end_inset

; these are essentially independent relations.
\end_layout

\begin_layout Standard
The way that the statistics are collected for 
\begin_inset Formula $A$
\end_inset

 and for 
\begin_inset Formula $R$
\end_inset

 are different.
 To count the 
\begin_inset Formula $A$
\end_inset

-type relations, one tokenizes a sentence into words, and then, counts every
 possible word-pair in the sentence.
 Effectively, one draws a clique of edges between the words, and then counts
 each edge.
 The statistics for 
\begin_inset Formula $R$
\end_inset

 are collected by parsing the sentence into a random planar tree, and then
 counting the edges in the tree.
 The result for this counting is NOT the same as that for type-
\begin_inset Formula $A$
\end_inset

 edges.
 The reason for this is demonstrated in depth, in the section 
\begin_inset CommandInset ref
LatexCommand nameref
reference "sec:Edge-counting"

\end_inset

 
\begin_inset CommandInset ref
LatexCommand vpageref
reference "sec:Edge-counting"

\end_inset

, below.
\end_layout

\begin_layout Standard
Initially, there is only one link relation 
\begin_inset Quotes eld
\end_inset

R
\begin_inset Quotes erd
\end_inset

 between two words: this is the 
\begin_inset Quotes eld
\end_inset

ANY
\begin_inset Quotes erd
\end_inset

 link-type.
 However, in general, 
\begin_inset Quotes eld
\end_inset

R
\begin_inset Quotes erd
\end_inset

 can be other kinds of link-types.
 Note that 
\begin_inset Quotes eld
\end_inset

R
\begin_inset Quotes erd
\end_inset

 can also have a head-tail dependency order: either 
\begin_inset Formula $w_{l}$
\end_inset

 or 
\begin_inset Formula $w_{r}$
\end_inset

 can be the head-word of a directional link.
 Thus, there are three different symmetrizations that can be obtained from
 
\begin_inset Quotes eld
\end_inset

R
\begin_inset Quotes erd
\end_inset

: by failing to make a left-right distinction, by failing to make a head-tail
 distinction, and failing to do either.
\end_layout

\begin_layout Standard
The definition for the probability of observing a relation can be taken
 to be 
\begin_inset Formula 
\begin{equation}
P(R,w_{l},w_{r})=\frac{N(R,w_{l},w_{r})}{N(R,*,*)}\label{eq:prob-relation}
\end{equation}

\end_inset

where
\begin_inset Formula 
\[
N(R,*,*)=\sum_{w_{l},w_{r}}N(R,w_{l},w_{r})
\]

\end_inset

This can be roughly understood as being the conditional probability of observing
 the relation 
\begin_inset Formula $R(w_{l},w_{r})$
\end_inset

 between two specific words, given that the relation 
\begin_inset Formula $R$
\end_inset

 between any two words was seen.
\end_layout

\begin_layout Standard
Is it possible to define the unconditional probability 
\begin_inset Formula $P(R,*,*)$
\end_inset

 of seeing the relationship? The path to the answer is not entirely straight-for
ward.
 First consdier the probability 
\begin_inset Formula $P(S,w_{1},w_{2})$
\end_inset

 of seeing two words in the same sentence.
 This probability is defined just as in eqn 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:prob-relation"

\end_inset

; that is, 
\begin_inset Formula $P(S,w_{1},w_{2})=N(S,w_{1},w_{2})/N(S,*,*)$
\end_inset

.
 From this, one can define the frequency of seeing a relation in a sentence,
 as
\begin_inset Formula 
\[
F(R|S,w_{1},w_{2})=\frac{P(R,w_{1},w_{2})}{P(S,w_{1},w_{2})}
\]

\end_inset

This gives the expectation value of seeing the relation 
\begin_inset Formula $R$
\end_inset

 in a sentence, given that the two words are already known to be in the
 sentence.
 That this is an expectation value should be clear, as the relation might
 appear multiple times in one sentence (e.g.
 if one of the words is repeated).
  The sum
\begin_inset Formula 
\[
F(R|S,*,*)=\sum_{w_{l},w_{r}}F(R|S,w_{l},w_{r})
\]

\end_inset

then counts the average number of relations per sentence.
 For the any-type ordered-pair relation, clearly one must have that there
 are at least as many relations as there are words in the sentence, minus
 one, since each word must appear in at least one (distinct) relation.
 That is, 
\begin_inset Formula $F(S,*)-1\le F(R|S,*,*)$
\end_inset

 with 
\begin_inset Formula $F(S,*)$
\end_inset

 the expected length of a sentence.
 
\end_layout

\begin_layout Standard
Similarly, one can consider the ratio
\begin_inset Formula 
\[
F(S,w_{1},w_{2})=\frac{P(S,w_{1},w_{2})}{P(w_{1})P(w_{2})}
\]

\end_inset

which captures the frequency at which two words are seen in the same sentence.
 The summation 
\begin_inset Formula $F(S,*,*)$
\end_inset

 then counts how many pairs are seen per sentence.
 Assuming that the counting was performed with a uniform distribution, this
 should then equal the number of edges in a clique.
 That is, for a sentence of length 
\begin_inset Formula $m$
\end_inset

, there should be 
\begin_inset Formula $m(m-1)/2$
\end_inset

 word-pairs (edges) counted for that sentence.
 This should hold approximately, on average, so that 
\begin_inset Formula $F(S,*,*)\approx F(S,*)(F(S,*)-1)/2$
\end_inset

.
\end_layout

\begin_layout Standard
From the development above, it should be clear that it is not really possible
 to define a quantity 
\begin_inset Formula $P(R,*,*)$
\end_inset

 that is the 
\begin_inset Quotes eld
\end_inset

probability of seeing a relation
\begin_inset Quotes erd
\end_inset

.
 We can count the number 
\begin_inset Formula $N(R,*,*)$
\end_inset

 of times the relation occurs in a data sample.
 We can count the average number of times the relation is seen in a sentence.
 However, as long as the relation occurs at least once in the data sample,
 one would have to say that the 
\begin_inset Quotes eld
\end_inset

probability of seeing the relation in the data sample
\begin_inset Quotes erd
\end_inset

 is one.
 The problem is one of normalization: there is no universe, of which 
\begin_inset Formula $N(R,*,*)$
\end_inset

 is a fractional measure.
\end_layout

\begin_layout Standard
That said, once can still consider an interesting ratio:
\begin_inset Formula 
\[
F(R,*,*)=\sum_{w_{l},w_{r}}F(R,w_{l},w_{r})=\sum_{w_{l},w_{r}}\frac{P(R,w_{l},w_{r})}{P(w_{l})P(w_{r})}
\]

\end_inset

This can be interpreted as a kind-of centrality.
 So, for example, for the any-pair relation, every word in the data sample
 must participate in at least one such pair-relation, and thus, we expect
 that 
\begin_inset Formula $F(R,*,*)\approx1$
\end_inset

.
 The precise value is related to the tree-parse that is being used to generate
 the any-relation.
 If the (random) parse-tree is acyclic, then the number of edges is comparable
 to the number of words.
 If the parse-tree contains cycles, then there may be more relations than
 there are words.
\end_layout

\begin_layout Subsubsection*
Yuret's Mutual Information
\end_layout

\begin_layout Standard
Deniz Yuret introduces the concept of 
\begin_inset Quotes eld
\end_inset

lexical attraction
\begin_inset Quotes erd
\end_inset

.
 It is reviewed briefly, here.
 He defines a probability 
\begin_inset Formula $\mathcal{P}(w_{l},w_{r})$
\end_inset

 of seeing an ordered pair; as compared to the above, the relation is implicit.
 To make it explicit, one should write: 
\begin_inset Formula 
\begin{equation}
\mathcal{P}(w_{l},w_{r})=P(A(w_{l},w_{r}),w_{l},w_{r})\label{eq:Yuret-prob}
\end{equation}

\end_inset

which indicates the relation explicitly, as well as noting that the order
 of the positions in the relation matter.
 To avoid confusion, the cursive 
\begin_inset Formula $\mathcal{P}$
\end_inset

 is used for the Yuret notation, instead of the roman 
\begin_inset Formula $P$
\end_inset

 which is reserved for the definitions above.
\end_layout

\begin_layout Standard
The letter 
\begin_inset Formula $A$
\end_inset

 used here reminds us that in Yuret's work, the pair-counting method used
 is the clique-edge-counting mechanism, described above, rather than the
 random-planar-tree relation.
 One expects the two to be similar, but not the same.
\end_layout

\begin_layout Standard
Yuret also uses the notation 
\begin_inset Formula $\mathcal{P}(w_{l},*)$
\end_inset

 and 
\begin_inset Formula $\mathcal{P}(*,w_{r})$
\end_inset

 for wild-card summations, defined as 
\begin_inset Formula 
\[
\mathcal{P}(w_{l},*)=\sum_{w_{r}}\mathcal{P}(w_{l},w_{r})\qquad\mbox{and}\qquad\mathcal{P}(*,w_{r})=\sum_{w_{l}}\mathcal{P}(w_{l},w_{r})
\]

\end_inset

It is tempting to conflate 
\begin_inset Formula $\mathcal{P}(w_{l},*)$
\end_inset

 with 
\begin_inset Formula $P(w_{l})$
\end_inset

 but that would be wrong; not every possible word can occur on the 
\begin_inset Formula $w_{r}$
\end_inset

 position.
 This suggests a different, but tempting, error, that 
\begin_inset Formula $\mathcal{P}(w_{l},*)\le P(w_{l})$
\end_inset

.
 This is also not the case! A word might occur more frequently as the left
 side of a pair, than it does all by itself in the sample text.
 This follows from the frequentist definitions; the denominators for the
 two probabilities are not compatible; they do not range over the same universe.
\end_layout

\begin_layout Standard
Yuret defines the 
\begin_inset Quotes eld
\end_inset

lexical attraction
\begin_inset Quotes erd
\end_inset

 as 
\begin_inset Formula 
\begin{equation}
\mbox{MI}(w_{l},w_{r})=\log_{2}\,\frac{\mathcal{P}(w_{l},w_{r})}{\mathcal{P}(w_{l},*)\mathcal{P}(*,w_{r})}\label{eq:Yuret's MI}
\end{equation}

\end_inset

so that large positive MI is associated with words that rarely seen one
 without the other (e.g.
 '
\emph on
Northern Ireland
\emph default
' from his examples.) Note the absence of a minus sign in the above! See
 below for an explanation.
 Large-MI word pairs occur when 
\begin_inset Formula $\mathcal{P}(w_{l},w_{r})$
\end_inset

 is roughly comparable to 
\begin_inset Formula $\mathcal{P}(w,*)\approx\mathcal{P}(*,w)$
\end_inset

.
 
\end_layout

\begin_layout Standard
It is worth reviewing Yuret's example, at this point.
 He looks at the word pair 'Northern Ireland' and states (based on a particular
 corpus that was analyzed) that 
\begin_inset Formula $-\log_{2}P(\mbox{'Northern'})=12.60$
\end_inset

 and that 
\begin_inset Formula $-\log_{2}P(\mbox{'Ireland'})=14.65$
\end_inset

 and finally that 
\begin_inset Formula $-\log_{2}\mathcal{P}(\mbox{'Northern'},\mbox{'Ireland'})=16.13$
\end_inset

.
 What these numbers mean is that although either word alone occurs at a
 rate of roughly once in ten-thousand words, the word-pair together occurs
 at the rate of one in thirty-thousand or so: the word pair occurs almost
 as often as either word alone.
 Thus, the resulting MI is very large: 
\begin_inset Formula $\mbox{MI}=-16.13+12.60+14.65=11.12$
\end_inset

.
 The choice of sign in eqn 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:Yuret's MI"

\end_inset

 is such that words that co-occur have a large positive value.
 In practice, the distribution of the MI for word-pairs runs from about
 -15 to about +35, and, when ranked according to MI, the probabilities form
 a rounded mountain-peak, two-sided, each side being linear (Zipfian) with
 the peak at about MI=4 or 6.
 (See my other notes for a graph.)
\end_layout

\begin_layout Standard
References:
\end_layout

\begin_layout Itemize
Apparently, MI is introduced here: Kenneth Ward Church and Patrick Hanks.
 
\begin_inset Quotes eld
\end_inset

Word association norms, mutual information, and lexicography
\begin_inset Quotes erd
\end_inset

.
 Computational linguistics, 16(1):22–29, 1990.
 
\end_layout

\begin_layout Section*
1 January 2014
\end_layout

\begin_layout Standard
OK, after that side distraction, which helped clear up notation, back to
 the main show ...
\end_layout

\begin_layout Standard
The main show is this: We want to model language, and specifically, find
 a 'minimal' set of relations R that are accurately generative.
 The meaning of 'minimal' seems obvious, intuitively, but a lot harder to
 pin down mathematically.
 We need to pin it down to get an algorithm that works in a trust-worthy,
 understandable fashion.
\end_layout

\begin_layout Standard
So: what is the total space of relations, and how do we find it? The simplest
 model is then a Zipfian distribution of words, but placed in random order.
 This model has a total entropy of 
\begin_inset Formula 
\[
H=-\sum_{w}P(w)\log_{2}P(w)
\]

\end_inset

For a recent swipe at parsing a few hundred articles from the French wikipedia,
 I get H=7.2.
 This is on 17K words, observed a total of 35M times (actually, observerd
 each sentence 100 times, so really just 350K 'true' observations of words).
\end_layout

\begin_layout Standard
How does one count the entropy of the rule-set? Elucidating this is the
 goal-set.
\end_layout

\begin_layout Standard
But first, step back: describe the rules.
 
\end_layout

\begin_layout Standard
OK ...
 so, once again ...
 sentence structure is to be described via link-grammar, using disjoined
 conjunctions of connectors.
 This is theoretically sound, as it seems to be isomorphic to categorical
 grammars (via type-theory of the connectors; need a formal proof of this
 someday, but for now it seems 'obvious').
 Also link-grammar is fully compatible with dependency grammar.
 So lets move forward.
 But this is an old debate, off to the side, immaterial for now.
\end_layout

\begin_layout Subsection*
How to count relations
\end_layout

\begin_layout Standard
Consider a sentence with 
\begin_inset Formula $n$
\end_inset

 words in it, numbered 
\begin_inset Formula $w_{1},w_{2},\cdots,w_{n}$
\end_inset

 left to right.
 We want to constrain grammar by discovering a set of relations 
\begin_inset Formula $R(w_{1},w_{2},\cdots,w_{n})$
\end_inset

 such that 
\begin_inset Formula $P(R(w_{1},w_{2},\cdots,w_{n}))>0$
\end_inset

 when the sentence is grammatically valid (
\emph on
i.e.

\emph default
 such an 
\begin_inset Formula $R$
\end_inset

 exists), and 
\begin_inset Formula $P$
\end_inset

 is zero when no such 
\begin_inset Formula $R$
\end_inset

 exists (
\emph on
i.e.

\emph default
 the sentence is not grammatically valid.) The first and most obvious simplificati
on rule is to observe that 
\begin_inset Formula $R$
\end_inset

 can be replaced by 
\begin_inset Formula $R(W_{1},W_{2},\cdots,W_{n})$
\end_inset

 where each 
\begin_inset Formula $W_{k}$
\end_inset

 is a set of words.
 That is, instead of listing each sentences individually, we list certain
 classes of sentences.
 In other words, the relations 
\begin_inset Formula $R(w_{1},w_{2},\cdots,w_{n})$
\end_inset

 are in one-to-one correspondence with a list of grammatical sentences 
\begin_inset Formula $(w_{1},w_{2},\cdots,w_{n})$
\end_inset

, so simply listing all possible sentences is a very verbose way of specifying
 a grammar.
 It is linguistically 'obvious' that sentences fall into classes, and so
 the two relations 
\begin_inset Formula $R('this','is','a','dog')$
\end_inset

 and 
\begin_inset Formula $R('this','is','a','cat')$
\end_inset

 can be replaced by 
\begin_inset Formula $R('this','is','a',W_{n})$
\end_inset

 where 
\begin_inset Formula $W_{n}=\{'dog','cat'\}$
\end_inset

.
 In fact, 
\begin_inset Formula $W_{n}$
\end_inset

 can be a rather large set of nouns.
 
\end_layout

\begin_layout Standard
So ...
 the question is: what is the reduction of complexity, by performing this
 classification? What is the correct way of counting? I assume that 'complexity'
 is a synonym for 'entropy', so we are looking to do two things: enumerate
 the states of the system, and proivde a measure for complexity.
 So, lets consider a language with 
\begin_inset Formula $N$
\end_inset

 nouns, so that the cardinality of 
\begin_inset Formula $W_{n}$
\end_inset

 is 
\begin_inset Formula $|W_{n}|=N$
\end_inset

 and the only valid sentences are 
\begin_inset Formula $('this','is','a',w)$
\end_inset

 with 
\begin_inset Formula $w\in W_{n}$
\end_inset

 .
 Before simplification, we had 
\begin_inset Formula $N$
\end_inset

 relations 
\begin_inset Formula $R$
\end_inset

, one per sentence.
 We also had 
\begin_inset Formula $N+3$
\end_inset

 sets, each set containing a single word; the 
\begin_inset Formula $N$
\end_inset

 nouns, and the three words 
\begin_inset Formula $'this','is','a'$
\end_inset

.
 After simplification, we have one relation 
\begin_inset Formula $R$
\end_inset

, and four sets; three of the sets have cardinality 1, the fourth set has
 cardinality 
\begin_inset Formula $N$
\end_inset

.
 
\end_layout

\begin_layout Subsubsection*
Revision: July 2014
\end_layout

\begin_layout Standard
There seem to be several ways of counting.
 Some of these seem to give wrong answers.
 Some just seem wrong.
 This is all very confusing, so I've altered the entries to explicitly show
 the different ways of counting.
\end_layout

\begin_layout Paragraph*
Method 1 (naive counting):
\end_layout

\begin_layout Standard
One counting rule is to count set-membership relations on equal footing
 with structural relations.
 Thus, before simplification, we had 
\begin_inset Formula $N+3$
\end_inset

 sets, each a singleton, and thus 
\begin_inset Formula $N+3$
\end_inset

 set membership relations.
 After simplification, we have four sets, but still have 
\begin_inset Formula $N+3$
\end_inset

 set membership relations.
 Thus, this particular simplification step does not reduce the number of
 membership relations at all.
 This seems disconcerting...
 Let's provisionally go with this and see what happens.
 Thus, before simplification, we had 
\begin_inset Formula $2N+3$
\end_inset

 relations grand-total, and afterwords, we have 
\begin_inset Formula $N+4$
\end_inset

 relations grand-total.
\end_layout

\begin_layout Standard
What is the correct 'thermodynamic' picture of what's going on? In this
 toy problem, we have a grand-total state space of size 
\begin_inset Formula $(N+3)^{4}$
\end_inset

 since any of the 
\begin_inset Formula $N+3$
\end_inset

 words can appear in any of the four slots in a four-word sentence (micro-canonn
ical ensemble).
 The entropy, at 'infinite temperature' where all possible four-word sequences
 occur with equal probability is then 
\begin_inset Formula $4\log_{2}(N+3)$
\end_inset

.
 The entropy of the set of grammatical sentences is 
\begin_inset Formula $\log_{2}N$
\end_inset

 since there are only 
\begin_inset Formula $N$
\end_inset

 possible grammatical sentences.
 In this toy grammar, there are also invalid setences of length 1,2,3,5,6,7,...
 and so the total size of the space of word-sequences is clearly infinite.
 
\end_layout

\begin_layout Standard
OK, so the space of word-sequences is very concrete, and easy to describe
 and measure, at least for toy grammars.
 What about the space of relations? Well, the claim is that the entropies
 of the before-and-after models are 
\begin_inset Formula $\log_{2}(2N+3)$
\end_inset

 and 
\begin_inset Formula $\log_{2}(N+4)$
\end_inset

, respectively.
 Neither of these matches the entropy of the set of allowed sentences (which
 is 
\begin_inset Formula $\log_{2}N$
\end_inset

), so this seems paradoxical, and begs the questions 'did we count correctly?'
 and 'did we actually simplify anything by making the above change of description
n?' Hmm.
 The correct answer seems to be 'no' and 'no'.
 
\end_layout

\begin_layout Paragraph*
Method 2 (subtract one):
\end_layout

\begin_layout Standard
To 'fix' the oddball results above, an alternative counting methodology
 is to subtract 1 from the cardinality of every set.
 This would then give both 
\begin_inset Formula $\log_{2}N$
\end_inset

 as the entropy for both the before and after relation-sets.
 Thus, before, we had 
\begin_inset Formula $N$
\end_inset

 relations and 
\begin_inset Formula $N+3$
\end_inset

 sets, each of weight zero, for a total weighted-relation count of 
\begin_inset Formula $N$
\end_inset

.
 After, we have one relation and four sets; three of the sets have weight
 zero, one set has a weight of 
\begin_inset Formula $N-1$
\end_inset

 so the total weighted relations is again 
\begin_inset Formula $N$
\end_inset

.
 This seems to resolve the paradox.
 But why subtract one? That's a bizarre rule, almost unheard-of in information
 theory.
\end_layout

\begin_layout Paragraph*
Method 3 (naive log addition):
\end_layout

\begin_layout Standard
Total complexity is given by:
\begin_inset Formula 
\[
K=\log_{2}|Rel|+\sum_{W\in Wrds}\log_{2}|W|
\]

\end_inset

where 
\begin_inset Formula $Rel$
\end_inset

 is the set of relations, and 
\begin_inset Formula $Wrds$
\end_inset

 is the set of word-lists, and 
\begin_inset Formula $|W|$
\end_inset

 is the cardinality of each word-list.
 We then get, before simplification, 
\begin_inset Formula $|Rel|=N$
\end_inset

 and 
\begin_inset Formula $|W|=1$
\end_inset

 for each of the word-sets.
 The total complexity is thus 
\begin_inset Formula $K=\log_{2}N$
\end_inset

 as expected (i.e.
 equal to the log of the total number of possible sentences).
 After simplification, there is 
\begin_inset Formula $|Rel|=1$
\end_inset

 and 3 sets with 
\begin_inset Formula $|W|=1$
\end_inset

 and one set with 
\begin_inset Formula $|W|=N$
\end_inset

, thus yielding a total of 
\begin_inset Formula $K=\log_{2}N$
\end_inset

 again.
 This seems to give a plausible answer, and provides a plausible argument.
 
\end_layout

\begin_layout Paragraph*
Method 4 (relational complexity):
\end_layout

\begin_layout Standard
Treating each relation as being equally complex seems odd.
 It would seem to make more sense to have each relation contribute according
 to its complexity, so that the contribution of the relations to the total
 complexity is: 
\begin_inset Formula 
\[
\sum_{R\in Rel}C_{R}
\]

\end_inset

with 
\begin_inset Formula $C_{R}$
\end_inset

 the complexity of each relation, itself the log of some measure.
 But how do we measure complexity? Is it Kolmogorov complexity? There's
 no obvious 
\emph on
a priori
\emph default
 definition for this.
 The definition of this complexity would seem to depend on the particular
 algorithm machinery of the grammar; that is, on the 'programming language'
 used to represent the relation.
 This is the traditional ambiguity attached to the Kolmogorov complexity.
\end_layout

\begin_layout Paragraph*
Method 5 (corpus distribution):
\end_layout

\begin_layout Standard
Instead of measuring the complexity of a grammatical expression (in an as-yet
 unknown grammar), instead, use the corpus frequency as a proxy.
 For the above example, if the 
\begin_inset Formula $N$
\end_inset

 sentences are equi-distributed (i.e.
 occur equally likely in the corpus), then, before simplification, each
 of the relations has a complexity 
\begin_inset Formula 
\[
C_{R}=-\frac{1}{N}\log_{2}\frac{1}{N}
\]

\end_inset

so that, before simplification, 
\begin_inset Formula 
\[
K=\sum_{R\in Rel}C_{R}=\log_{2}N
\]

\end_inset

which again seems to be the desired answer.
 After simplification, there is one relation that applies to the entire
 corpus, so that 
\begin_inset Formula $C_{R}=0$
\end_inset

 after simplification.
 
\end_layout

\begin_layout Paragraph*
Method 6 (corpus word-counts):
\end_layout

\begin_layout Standard
If we are taking word-relation frequencies from the corpus, then we should
 be taking word-set frequencies from the corpus as well.
 That is, the word-set contribution 
\begin_inset Formula $\log_{2}|W|$
\end_inset

 is assuming an equi-distribution.
 This should be replaced by the corpus contribution 
\begin_inset Formula 
\[
-\sum_{w\in W}p(w)\log_{2}p(w)
\]

\end_inset


\end_layout

\begin_layout Paragraph*
Summary.
\end_layout

\begin_layout Standard
Provisionally, the last two methods seem to be the best way to move forward.
 To summarize, the complexity is given by 
\begin_inset Formula 
\begin{equation}
K=-\sum_{R\in Rel}P_{R}\log_{2}P_{R}-\sum_{W\in Wrds}\ \sum_{w\in W}P_{w}\log_{2}P_{w}\label{eq:counting complexity}
\end{equation}

\end_inset

where 
\begin_inset Formula $P_{R}=P(R)=P(R(W_{1},W_{2},\cdots,W_{n}))$
\end_inset

 is the probability of observing the relation 
\begin_inset Formula $R$
\end_inset

 in a sample corpus, and 
\begin_inset Formula $P_{w}=P(w|W)$
\end_inset

 is the probability of observing word 
\begin_inset Formula $w$
\end_inset

 in the corpus, conditioned on its appearance in the corpus having to do
 with it belonging to the word-class 
\begin_inset Formula $W$
\end_inset

.
\end_layout

\begin_layout Subsection*
Counting Link-Grammar Relations
\end_layout

\begin_layout Standard
Per link-grammar, each relation is decomposable into pair-wise relations;
 this is the so-called 'parse' of a sentence.
 If the relation is a single word-per-slot sentence relation, then the 'parse'
 is literal.
 We write 
\begin_inset Formula 
\begin{equation}
R(w_{1},w_{2},\cdots,w_{n})=\prod_{j,k,m}R_{\alpha}(w_{j},w_{k},t_{m})\ Q(R_{\alpha},R_{\beta},\cdots,R_{\omega})\label{eq:pair-decompose}
\end{equation}

\end_inset

where 
\begin_inset Formula $R_{\alpha}(w_{j},w_{k},t_{m})$
\end_inset

 is a single connected pair of words, connected by the connector 
\begin_inset Formula $t_{m}$
\end_inset

.
 The product symbol 
\begin_inset Formula $\prod$
\end_inset

 implies that all such binary relations must hold.
 The awkward 
\begin_inset Formula $Q(R_{\alpha},R_{\beta},\cdots,R_{\omega})$
\end_inset

 at the end is the additional no-links-cross constraint in the current link-gram
mar parser.
 Its a non-local constraint involving all of the binary relations.
 It also subsumes any 'post-processing' rules, although, for the language
 learnign exercise, there won't be any post-processing rules.
 At any rate, 
\begin_inset Formula $Q$
\end_inset

 is a place where higher ordrer constraints can be applied.
 In particular, the most genneral form for 
\begin_inset Formula $Q$
\end_inset

 should be 
\begin_inset Formula $Q(R_{\alpha},R_{\beta},\cdots,R_{\omega},w_{1},w_{2},\cdots,w_{n})$
\end_inset

 since, in principle, it could depend on the word-choice, although the no-links-
cross constraint does not.
 
\end_layout

\begin_layout Standard
Yuret proposes a way of discovering the pair-wise relations
\begin_inset CommandInset citation
LatexCommand cite
key "Yuret1998"
literal "true"

\end_inset

.
 He makes the implicit, unvoiced assumption that there is a single, unique
 connector type 
\begin_inset Formula $t_{m}$
\end_inset

 for every ordered pair of words 
\begin_inset Formula $w_{j},w_{k}$
\end_inset

; that is, that 
\begin_inset Formula $t_{m}=t_{m}(w_{j},w_{k})$
\end_inset

.
 Viz, specifically, that such connectors are in 1-1 correspondence with
 word-pairs.
 (I don't think he's aware of this assumption; I don't think anyone has
 ever before realized that he's making such an assumption; certainly, I
 haven't).
 Yuret then makes two claims: first, that the only possible grammatically
 correct parses are those of the above form (eqn 
\begin_inset CommandInset ref
LatexCommand eqref
reference "eq:pair-decompose"

\end_inset

) for which the relations 
\begin_inset Formula $R_{\alpha}(w_{j},w_{k},t_{m}(w_{j},w_{k}))$
\end_inset

 have been previously observed; secondly, that there is a natural ranking
 of such allowed parses by summing the total mutual information associated
 with each word-pair.
\end_layout

\begin_layout Standard
These two concepts give rise to the idea of minimum-spanning-tree parsers.
 Such parsers work in a two-step process: a training phase, and a parse
 phase.
 In the training phase, one gathers a lot of statistics about mutual information.
 The important point here is that this is unsupervised training.
 To parse, one first creates a graph clique, with every word connected to
 every other.
 One uses the gathered MI to define graph edge lengths.
 Finally, the correct parse is then the maximum spanning tree of the graph
 (maximizing the MI, summed over the tree edges in the graph).
\end_layout

\begin_layout Standard
Here, we use the same idea, but then take the next step.
 The spanning tree can be decomposed into a set of link-grammar disjuncts,
 one disjunct per word.
 The disjunct is merely a list of the connections that one word makes.
 It consists of the type, and the direction.
 The direction is left or right.
 The type is the 
\begin_inset Formula $t_{m}=t_{m}(w_{j},w_{k})$
\end_inset

 defined above.
 By parsing a large number of sentences, we can now automatically discover
 a large number of disjuncts, in an unsupervised manner.
\end_layout

\begin_layout Standard
The goal, the next step, is then to reduce the total number of disjuncts,
 and the total number of types, by clustering and discovering similarities.
\end_layout

\begin_layout Section*
3 January 2014
\end_layout

\begin_layout Subsection*
No-crossing Minimum Spanning Trees
\end_layout

\begin_layout Standard
It turns out that writing an algorithm for a no-crossing minimum spanning
 tree is surprisingly painful; enforcing the no-crossing constraint requires
 treatment of a number of special cases.
 But perhaps this is not actually required! R.
 Ferrer i Cancho in 
\begin_inset Quotes eld
\end_inset

Why do syntactic links not cross?
\begin_inset Quotes erd
\end_inset


\begin_inset CommandInset citation
LatexCommand cite
key "Ferrer2006"
literal "true"

\end_inset

 shows that, when attempting to arrange a random set of points on a line,
 in such a way as to minimize euclidean distances between connected points,
 one ends up with trees that almost never cross!
\end_layout

\begin_layout Standard
Other related references:
\end_layout

\begin_layout Itemize
Crossings are rare: Havelka, J.
 (2007).
 Beyond projectivity: multilingual evaluation of constraints and measures
 on non-projective structures.
 In: Proceedings of the 45th Annual Meeting of the Association of Computational
 Linguistics (ACL-07): 608-615.
 Prague, Czech Republic: Association for Computational Linguistics.
 
\end_layout

\begin_layout Itemize
Hubbiness is a better model of sentence complexity than mean dependency
 distance: Ramon Ferrer-i-Cancho (2013) 
\begin_inset Quotes eld
\end_inset

Hubiness, length, crossings and their relationships in dependency trees
\begin_inset Quotes erd
\end_inset

, ArXiv 1304.4086 — also states: maximum number of crossings is bounded above
 by mean dependency length.
 Also, mean dependency length is bounded below by variance of degrees of
 vertices (i.e.
 variance in number of connectors a word can have).
\end_layout

\begin_layout Itemize
Language tends to be close to the theoretical minimum possible dependency
 distance, if it was legal to re-arrange words arbitrarily.
 See Temperley, D.
 (2008).
 Dependency length minimization in natural and artificial languages.
 Journal of Quantitative Linguistics, 15(3):256-282.
 
\end_layout

\begin_layout Itemize
Park, Y.
 A.
 and Levy, R.
 (2009).
 Minimal-length linearizations for mildly context-sensitive dependency trees.
 In Proceedings of the North American Chapter of the Association for Computation
al Linguistics - Human Language Technologies (NAACL-HLT) conference.
 
\end_layout

\begin_layout Itemize
Sentences with long dependencies are hard to understand: The original claim
 is from Yngve, 1960, having to do with phrase-structure depth.
 See – Gibson, E.
 (2000).
 The dependency locality theory: A distance-based theory of linguistic complexit
y.
 In Marantz, A., Miyashita, Y., and O'Neil, W., editors, Image, Language, Brain.
 Papers from the first Mind Articulation Project Symposium.
 MIT Press, Cambridge, MA.
 
\end_layout

\begin_layout Itemize
(Cite this, its good) Mean dependency distance is a good measure of sentence
 complexity – for 20 languages – Haitao Liu gives overview starting from
 Yngve.
 
\begin_inset CommandInset citation
LatexCommand cite
key "Liu2008"
literal "true"

\end_inset

.
 Haitao Liu 
\begin_inset Quotes eld
\end_inset

Dependency distance as a metric of language comprehension difficulty
\begin_inset Quotes erd
\end_inset

, 2008, Journal of Cognitive Science, v9.2 pp 159-191 http://www.lingviko.net/JCS.pd
f 
\end_layout

\begin_layout Itemize
Sentences with long dependencies are rarely spoken: Hawkins, J.
 A.
 (1994).
 A Performance Theory of Order and Constituency.
 Cambridge University Press, Cambridge, UK.
 —-Hawkins, J.
 A.
 (2004).
 Efficiency and Complexity in Grammars.
 Oxford University Press, Oxford, UK.
 —-Wasow, T.
 (2002).
 Postverbal Behavior.
 CSLI Publications, Stanford, CA.
 Distributed by University of Chicago Press.
 
\end_layout

\begin_layout Itemize
Dependency-length minimization is universal: Richard Futrell, Kyle Mahowald,
 and Edward Gibson, 
\begin_inset Quotes eld
\end_inset

Large-scale evidence of dependency length minimization in 37 languages
\begin_inset Quotes erd
\end_inset

 (2015), doi: 10.1073/pnas.1502134112
\end_layout

\begin_layout Itemize
The longest links, observed statistically, are of length 6 or less.
 This is based on computing the mutual information of words at different
 distances for the Brown corpus.
 Xuedong Huang, Fileno Alleva, Hsiao-wuen Hon, Mei-Y uh Hwang, Kai-Fu Lee
 and Ronald Rosenfeld.
 The SPHINX-II Speech Recognition System: An Overview .
 Computer , Speech and Language , volume 2, pages 137–148, 1993.
\end_layout

\begin_layout Standard
So, rather than imposing no-crossing as a constraint on the parser, instead,
 let it find its own way into the grammar.
 Just implement a plain-old MST parser, punt on crossing.
\end_layout

\begin_layout Subsection*
Crossing and Copulas
\end_layout

\begin_layout Standard
Here's a great example where multiple lingusitic desires cause trouble.
 First, consider the planar diagram, no-links-cross, that indicates both
 the head-noun and the head-verb:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

        +-------->WV------->+
\end_layout

\begin_layout Plain Layout

        +---->Wd---->+      |
\end_layout

\begin_layout Plain Layout

        |      +<-Ds-+--Ss--+-->Pa->+
\end_layout

\begin_layout Plain Layout

        |      |     |      |       |
\end_layout

\begin_layout Plain Layout

    LEFT-WALL the  dog.n   was    black 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The above indicates the head-noun (
\family typewriter
Wd
\family default
), the head-verb (
\family typewriter
WV
\family default
) and a subject-verb relation (
\family typewriter
Ss
\family default
) and that these three for a cycle.
 Its still a DAG, because the arrows point down from the root.
 The 
\family typewriter
Pa
\family default
 link is a predicative-adjective link.
\end_layout

\begin_layout Standard
A plausible alternative diagram is
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

       +---->Wd---->+      
\end_layout

\begin_layout Plain Layout

       |            +-->adjcomp--->+
\end_layout

\begin_layout Plain Layout

       |            |              |
\end_layout

\begin_layout Plain Layout

       |      +<-Ds-+--Ss--+<-cop<-+
\end_layout

\begin_layout Plain Layout

       |      |     |      |       |
\end_layout

\begin_layout Plain Layout

   LEFT-WALL the  dog.n   was    black
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Here, the 
\family typewriter
adjcomp
\family default
 arrow indicates the adjectival complement, and 
\family typewriter
cop
\family default
 indicates the copula.
 Note that it is 
\emph on
impossible
\emph default
 to draw an arrow from the root to the head-verb, without forcing a link-crossin
g! Interesting, eh? 
\end_layout

\begin_layout Standard
The real question, the major question is: what happens to pair-wise MI?
 What would that show? It seems unlikely that MI(dog, black) is going to
 be all that high.
 But what about MI(physical object, color)? What is the correct way to compute
 MI(physical object, color)?
\end_layout

\begin_layout Standard
The above is perhaps the simplest-possible example of the confusion surrounding
 the transition from SSyntR to DSyntR (or to logical representations, a
 la opencog).
 That is, we want to convert the 
\family typewriter
adjcomp
\family default
 relation to 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  EvaluationLink
\end_layout

\begin_layout Plain Layout

     PredicateNode "has color"
\end_layout

\begin_layout Plain Layout

        ListLink
\end_layout

\begin_layout Plain Layout

           Concept "dog"
\end_layout

\begin_layout Plain Layout

           Concept "black"
\end_layout

\end_inset

 
\end_layout

\begin_layout Standard
Following Poon & Domingos, we really want to convert it to
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

  LambdaLink
\end_layout

\begin_layout Plain Layout

    VariableList
\end_layout

\begin_layout Plain Layout

       Variable $PHY
\end_layout

\begin_layout Plain Layout

       Variable $COL
\end_layout

\begin_layout Plain Layout

    AndLink
\end_layout

\begin_layout Plain Layout

       EvaluationLink
\end_layout

\begin_layout Plain Layout

          PredicateNode "has color"
\end_layout

\begin_layout Plain Layout

          ListLink
\end_layout

\begin_layout Plain Layout

             Variable $PHY
\end_layout

\begin_layout Plain Layout

             Variable $COL
\end_layout

\begin_layout Plain Layout

       InheritanceLink
\end_layout

\begin_layout Plain Layout

          Variable $PHY
\end_layout

\begin_layout Plain Layout

          Concept "physical object"
\end_layout

\begin_layout Plain Layout

       InheritanceLink
\end_layout

\begin_layout Plain Layout

          Variable $COL
\end_layout

\begin_layout Plain Layout

          Concept "color" 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Of course, this has to be done based on the evidence of many sentences,
 discovered to be qusi-synonymous (a la Dekang Lin DIRT).
\end_layout

\begin_layout Section*
11 January 2014
\end_layout

\begin_layout Subsection*
Clustering Redux
\end_layout

\begin_layout Standard
OK, so what is the very next algorithmic step? Up to here, we've generated
 a large number of unique disjuncts.
 Now what?
\end_layout

\begin_layout Standard
Back to counting.
 Lets do the French dictionary.
 The database fr_pairs contains table atoms_mi_snapshot.
 So:
\end_layout

\begin_layout Itemize

\family typewriter
select count(*) from atoms_mi_snapshot;
\family default
 returns 415532
\end_layout

\begin_layout Section*
15 January 2014
\end_layout

\begin_layout Subsection*
Embodied Learning
\end_layout

\begin_layout Standard
OK, so maybe learning syntax before emantics puts the cart before the horse.
 Can we learn a world-model first, and then gradually annotate and correct
 it as our linguistic comprehension improves? So, for example, can we start
 with a world-model obtained via document summarization? How do we annotate
 this model with newly discovered data?
\end_layout

\begin_layout Standard
Related question: how to automatically discover ontologies? Automated, unsupervi
sed concept, entity extraction? Semantic context change over time?
\end_layout

\begin_layout Standard
Steps: 
\end_layout

\begin_layout Enumerate
How do I extract entities out of a text? The extraction doesn't have to
 be perfect; having candidate entities is enough.
 How do I put a confidence rating on the entiy, and how do I discard the
 low-cnfidence ones?
\end_layout

\begin_layout Enumerate
Once entities are extracted, I want to start decorating them with attributes
 (adjectives, modifiers), to build a network.
\end_layout

\begin_layout Enumerate
Once a network is built, it needs to be factually reconciled, using logical
 reasoning and an ontology (is-a and has-a relations).
 Need to do this so that upon reading 
\begin_inset Quotes eld
\end_inset

colorless green ideas
\begin_inset Quotes erd
\end_inset

, we can deduce that ideas are either colorless or green, but not both.
\end_layout

\begin_layout Enumerate
How to automatically extract an ontology from free text?
\end_layout

\begin_layout Standard
The above seem to be the central steps/core issues for creating a world-model,
 unsupervised, from text.
\end_layout

\begin_layout Subsection*
Entropy
\end_layout

\begin_layout Standard
Some refresher notes: 
\end_layout

\begin_layout Itemize
\begin_inset Quotes eld
\end_inset

The Boltzmann distribution is the so-called canonical distribution, meaning
 it maximizes entropy subject to a constraint on the expected value of energy.
\begin_inset Quotes erd
\end_inset

 (viz, this is the MaxEnt principle.
 Except for MaxEnt, the constraint is not on energy, but rather a set of
 constraints obtained on some other theoretical grounds.)
\end_layout

\begin_layout Itemize
Define 
\begin_inset Quotes eld
\end_inset

Shanon Entropy
\begin_inset Quotes erd
\end_inset

 as 
\begin_inset Formula $S_{s}=-k_{B}\sum p\log p$
\end_inset


\end_layout

\begin_layout Itemize
The 
\begin_inset Quotes eld
\end_inset

Boltzmann Entropy
\begin_inset Quotes erd
\end_inset

 
\begin_inset Formula $S_{B}$
\end_inset

 is the Shanon entropy of the microcanonical ensemble: it maximizes the
 entropy (MaxEnt) for a fixed value of the energy.
 (MaxEnt: not the energy, but for a fixed set of constraints).
 (viz, 
\begin_inset Formula $S_{B}=k_{B}\log\left(\epsilon\frac{d\Omega}{dE}\right)$
\end_inset

 with 
\begin_inset Formula $\Omega$
\end_inset

 being number of states, E the energy, 
\begin_inset Formula $\epsilon$
\end_inset

 a constant of dimension energy to make arg of the log dimensionless.) (MaxEnt:
 replace E by the individual constraints.
 This suggests that there are many Boltzmann entropies: one for each constraint
 that is applied!)
\end_layout

\begin_layout Itemize
The 
\begin_inset Quotes eld
\end_inset

Gibbs Entropy
\begin_inset Quotes erd
\end_inset

 is the Shanon entropy, maximized for a system held to the constraint that
 energy is less-than-or-equal to E.
 (!) This gives 
\begin_inset Formula $S_{G}=k_{B}\log\Omega$
\end_inset

 (duhh, take 
\begin_inset Formula $p=1/\Omega$
\end_inset

 for 
\begin_inset Formula $\Omega$
\end_inset

 states.
 For a non-sharp cutoff, the Shannon entropy is primal.).
 (MaxEnt: one gets a different Gibbs entropy for each applied constraint.)
\end_layout

\begin_layout Itemize
Gibbs and Boltzmann entropies give different results for N-particle systems
 when N is very small.
 Viz, an off-by-one error for N.
 In some ways, 
\begin_inset Formula $S_{G}$
\end_inset

 is more correct (at low temp, quantum systems).
 See Jörn Dunkel and Stefan Hilbert (2014) 
\begin_inset Quotes eld
\end_inset

Consistent thermostatistics forbids negative absolute temperatures
\begin_inset Quotes erd
\end_inset

 Nature Physics DOI: 10.1038/NPHYS2815 
\end_layout

\begin_layout Subsection*
Why does Yuret's MST work?
\end_layout

\begin_layout Standard
There is an interesting simplification that happens with minimum-spanning
 tree parsers driven by entropy.
 If we use Yuret's definition of the MI of word-pairs, then, Yuret says
 (I should re-read his stuff) that we should maximize the entropy 
\begin_inset Formula 
\begin{equation}
\sum_{w_{l,}w_{r}}MI(w_{l},w_{r})\label{eq:MST entropy}
\end{equation}

\end_inset

Why? Why this, instead of the maybe 
\begin_inset Quotes eld
\end_inset

more obviously correct
\begin_inset Quotes erd
\end_inset

 sum: 
\begin_inset Formula 
\begin{equation}
\sum_{w_{l,}w_{r}}P(w_{l},w_{r})MI(w_{l},w_{r})\label{eq:wrong MST entropy}
\end{equation}

\end_inset

I think I can hand-wave the answer.
 The answer is that we don't really know the probability of 
\begin_inset Formula $P(w_{l},w_{r})$
\end_inset

 
\emph on
for the given sentence! 
\emph default
We know 
\begin_inset Formula $P(w_{l},w_{r})$
\end_inset

 for a large corpus, but its somewhat of a mistake to assume that this identical
 to what it would be for expressing a particular idea in a certain specific
 way.
 Its possible that, to express the idea, the only sentences that one could
 ever possibly use would have 
\begin_inset Formula $P(w_{l},w_{r})$
\end_inset

 that strongly deviate from a large-corpus average.
 Unfortunately, there is no easy way of knowing what this sentence-specific
 
\begin_inset Formula $P(w_{l},w_{r})$
\end_inset

 is.
 So, instead we make the uniform distribution assumption, that they're all
 the same, and thus get eqn 
\begin_inset CommandInset ref
LatexCommand eqref
reference "eq:MST entropy"

\end_inset

 instead of 
\begin_inset CommandInset ref
LatexCommand eqref
reference "eq:wrong MST entropy"

\end_inset

.
 Does Yuret ever make this argument himself? Dunno.
\end_layout

\begin_layout Standard
A supporting argument is that we also ignored 3,4,5-point interactions as
 well.
 Which brings us to the next point: why should we expect a link-parse to
 work better than an MST parse? Because Yuret-MST ignores the valence of
 words, whereas the disjuncts don't! The disjuncts provide a better, more
 accurate way of capturing valency!
\end_layout

\begin_layout Subsection*
Entity Extraction
\end_layout

\begin_layout Standard
See Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu Tal
 Shaked, Stephen Soderland, Daniel S.
 Weld, and Alexander Yates (2005) 
\begin_inset Quotes eld
\end_inset

Unsupervised Named-Entity Extraction from the Web: An Experimental Study
\begin_inset Quotes erd
\end_inset

.
 So: KNOWITALL utilizes a set of eight domain-independent extraction patterns
 to generate candidate facts.
 For example, the generic pattern “NP1 such as NPList2” ...
 
\begin_inset Quotes eld
\end_inset

cities such as Paris,...
\begin_inset Quotes erd
\end_inset


\begin_inset Quotes erd
\end_inset

 Of course, this is not really unsupervised, since it uses human-generated
 search patterns (
\begin_inset Quotes eld
\end_inset

such as
\begin_inset Quotes erd
\end_inset

) and also applies constraints (the targets must be proper nouns, which
 is not a-priori known).
\end_layout

\begin_layout Subsection*
Partition Function
\end_layout

\begin_layout Standard
Some notes about the Boltzmann distribution.
 Let 
\begin_inset Formula $F(w_{l},w_{r})$
\end_inset

 be a numeric score associated with the edge 
\begin_inset Formula $(w_{l},w_{r})$
\end_inset

 – for example, this might be (minus) the MI.
 By convention, one introduces a Lagrange multiplier 
\begin_inset Formula $\beta$
\end_inset

 and writes 
\begin_inset Formula $F=\beta f$
\end_inset

.
 The probability of a parse tree 
\begin_inset Formula $T$
\end_inset

 constructed solely out of edge-pairs is then defined as
\begin_inset Formula 
\[
P(T|S)=\frac{\exp\left(-\beta\sum_{(w_{l},w_{r})\in T}f(w_{l},w_{r})\right)}{Z(S)}
\]

\end_inset

Here, 
\begin_inset Formula $S$
\end_inset

 is the sentence: a sequence of words, and 
\begin_inset Formula $Z(S)$
\end_inset

 is the partition function: the sum of the probabilities of all different
 possible parses for that setence:
\begin_inset Formula 
\[
Z(S)=\sum_{T}\exp\left(-\beta\sum_{(w_{l},w_{r})\in T}f(w_{l},w_{r})\right)
\]

\end_inset

The MST parse is then the single parse 
\begin_inset Formula $T$
\end_inset

 that maximizes the probability 
\begin_inset Formula $P(T|S)$
\end_inset

 and this can be easily seen to be the parse that maximizes the sum 
\begin_inset Formula $\sum_{(w_{l},w_{r})\in T}f(w_{l},w_{r})$
\end_inset

 on the spanning tree 
\begin_inset Formula $T$
\end_inset

.
\end_layout

\begin_layout Standard
Written in this form, it suggests how parsing can be generalized to include
 other scores.
 If, for example, one has some other score 
\begin_inset Formula $g(w_{1},w_{2},w_{3})$
\end_inset

 over triples of words, then one sums as above, using 
\begin_inset Formula $g$
\end_inset

 in place of 
\begin_inset Formula $f$
\end_inset

.
 In general, one can consider scoring functions 
\begin_inset Formula $f=f(R;S)$
\end_inset

 for some relation 
\begin_inset Formula $R$
\end_inset

 over the sentence 
\begin_inset Formula $S$
\end_inset

.
 The prototypical example would be a scoring function that uses Link Grammar
 disjuncts for the relation 
\begin_inset Formula $R$
\end_inset

, together with some weighting for link lengths (so as to avoid long links),
 together with some weighting for word pairs (to give greater weight to
 idioms and set phrases, as opposed to grammatically valid but non-idiomatic
 parses).
\end_layout

\begin_layout Standard
In many papers on supervised training, it is common to call the relation
 
\begin_inset Formula $R$
\end_inset

 a 
\begin_inset Quotes eld
\end_inset

feature
\begin_inset Quotes erd
\end_inset

, and to consider many different possible features 
\begin_inset Formula $R$
\end_inset

.
 Thus, 
\begin_inset Formula $f=f_{R}(S)$
\end_inset

 becomes a vector 
\begin_inset Formula $\vec{f}$
\end_inset

 over the features 
\begin_inset Formula $R$
\end_inset

.
 The multiplier 
\begin_inset Formula $-\beta$
\end_inset

 is replaced by a vector of weights 
\begin_inset Formula $\vec{w}$
\end_inset

, and so one considers the probability 
\begin_inset Formula 
\[
P=\frac{\exp\left(\sum\vec{w}\cdot\vec{f}\right)}{Z(S)}
\]

\end_inset

Supervised training then uses a training corpus marked up with both a feature
 vector 
\begin_inset Formula $\vec{f}$
\end_inset

 and the correct parse; training consists of using various supervised learning
 algos to find the best-possible weight vector 
\begin_inset Formula $\vec{w}$
\end_inset

 that maximizes the fraction of correct parses (or optimizes the ROC curve,
 or other measure of accuracy).
 In unsupervised training, we don't have a training corpus, and thus, do
 not focus on optimizing the weight vector.
\end_layout

\begin_layout Standard
Now we do the funky chicken dance.
 Write
\begin_inset Formula 
\[
Z(S)=\det\,L(S)
\]

\end_inset

This is commonly done when working with fermions; this is the Berezin determinan
t or Berezin integral, so named because one may write 
\begin_inset Formula 
\[
\det A=\int\exp\left[-\theta A\eta\right]d\theta d\eta
\]

\end_inset

for Grassman variables 
\begin_inset Formula $\theta$
\end_inset

 and 
\begin_inset Formula $\eta$
\end_inset

 and 
\begin_inset Formula $A$
\end_inset

 a matrix.
 Here's the part that surprises me: Koo et al state that 
\begin_inset Formula $L(S)$
\end_inset

 can be taken to be a Laplacian matrix of a graph.
 Wow! The mind boggles.
\end_layout

\begin_layout Standard
References:
\end_layout

\begin_layout Itemize
Koo, T., Globerson, A., Carreras, X., and Collins, M.
 (2007).
 Structured prediction models via the matrix-tree theorem.
 In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural
 Language Processing and Computational Natural Language Learning (EMNLP-
 CoNLL) , pages 141–150, Prague, Czech Republic, June.
 Association for Computational Linguistics.
\end_layout

\begin_layout Section*
3 March 2014
\end_layout

\begin_layout Standard
Start again, after long distraction.
\end_layout

\begin_layout Subsection*
Finding patterns
\end_layout

\begin_layout Standard
To problem.
 Consider an alphabet of 
\begin_inset Formula $N=5$
\end_inset

 letters, 
\begin_inset Formula $\alpha=\{A,B,C,D,E\}$
\end_inset

 and a corpus built from those letters.
 The five letters occur with probability 
\begin_inset Formula $p(w)$
\end_inset

 with 
\begin_inset Formula $w\in\alpha$
\end_inset

.
 Assume the corpus consists entirely of pairs AB, CB and DE, each occurring
 equally often: so: 
\begin_inset Formula $p(A,B)=p(C,B)=p(D,E)=1/3$
\end_inset

 .
 From this, we can reconstruct that 
\begin_inset Formula $p(A)=p(C)=p(D)=p(E)=1/6$
\end_inset

 and 
\begin_inset Formula $p(B)=1/3$
\end_inset

.
 This follows because the corpus can be reduced to 
\begin_inset Formula $\{AB,CB,DE\}$
\end_inset

, so A occurs 1 out of 6 times, B two out of 6 times, etc.
 The total single-letter entropy is thus
\begin_inset Formula 
\begin{align*}
h_{SING} & =-\sum_{w\in\alpha}p(w)\log_{2}p(w)\\
 & =-\frac{4}{6}\log_{2}\frac{1}{6}-\frac{1}{3}\log_{2}\frac{1}{3}\\
 & =\frac{2}{3}-\log_{2}\frac{1}{3}=2.25163
\end{align*}

\end_inset

By contrast, in a random 2-letter corpus, we expect all possible letter
 pairs to occur equally often, i.e.
 
\begin_inset Formula $p(w)=1/5$
\end_inset

, which would result in 
\begin_inset Formula $h_{RAND}=-\log_{2}1/5=2.321928$
\end_inset

 and so we see that the total entropy for this corpus is less than the random
 corpus.
\end_layout

\begin_layout Standard
The total double-word entropy is
\begin_inset Formula 
\begin{align*}
h_{PAIR} & =-\sum_{w_{1},w_{2}\in\alpha}p(w_{1},w_{2})\log_{2}p(w_{1},w_{2})\\
 & =-\log_{2}\frac{1}{3}=1.5849625
\end{align*}

\end_inset

Compare this to 
\begin_inset Formula $h_{PR-RAND}=-\log_{2}1/25=4.643856$
\end_inset

 for the random 2-letter corpus.
 The pair-entropy is sharply lower.
\end_layout

\begin_layout Standard
What do we know about mutual information? We can also deduce that 
\begin_inset Formula $p(*,B)=2/3$
\end_inset

, 
\begin_inset Formula $p(A,*)=1/3$
\end_inset

 and so 
\begin_inset Formula 
\begin{align*}
MI(A,B) & =\log_{2}p(A,B)-\log_{2}p(A,*)-\log_{2}p(*,B)\\
 & =\log_{2}3/2=0.585
\end{align*}

\end_inset

and likewise 
\begin_inset Formula $MI(C,B)=MI(A,B)$
\end_inset

 while 
\begin_inset Formula $MI(D,E)=\log_{2}3=1.585$
\end_inset

.
 
\end_layout

\begin_layout Standard
By contrast, in a random 2-letter corpus, we expect all possible letter
 pairs to occur equally often, i.e.
 
\begin_inset Formula $p(w_{1},w_{2})=1/25$
\end_inset

, which would result in an 
\begin_inset Formula $MI(w_{1},w_{2})=\log_{2}1=0$
\end_inset

 for all word pairs.
 
\end_layout

\begin_layout Standard
Given this corpus, we wish to deduce the following answer: there is a cluster
 
\begin_inset Formula $\gamma=\{A,C\}$
\end_inset

 and two link relations 
\begin_inset Formula $R(\gamma,B)$
\end_inset

 and 
\begin_inset Formula $R(D,E)$
\end_inset

 occurring with probabilities 
\begin_inset Formula $p(\gamma,B)=p(A,B)+p(C,B)=2/3$
\end_inset

 and 
\begin_inset Formula $p(D,E)=1/3$
\end_inset

.
 Note that 
\begin_inset Formula $p(\gamma,*)=p(A,*)+p(C,*)=2/3$
\end_inset

 so that
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{align*}
MI(\gamma,B) & =\log_{2}p(\gamma,B)-\log_{2}p(\gamma,*)-\log_{2}p(*,B)\\
 & =\log_{2}3/2=0.585
\end{align*}

\end_inset

So how do we deduce this?
\end_layout

\begin_layout Standard
Well, consider the reduced space, with 
\begin_inset Formula $N=4$
\end_inset

 letters: 
\begin_inset Formula $\beta=\{\gamma,B,D,E\}$
\end_inset

.
 In this space, only two pairs are observed in the corpus, 
\begin_inset Formula $\gamma B$
\end_inset

 and 
\begin_inset Formula $DE$
\end_inset

 with probabilities as above.
 The single-letter probabilities are 
\begin_inset Formula $p(D)=p(E)=1/6$
\end_inset

 and 
\begin_inset Formula $p(\gamma)=p(B)=1/3$
\end_inset

.
 The single-letter entropy is
\begin_inset Formula 
\begin{align*}
h_{SING}^{red} & =-\sum_{w\in\beta}p(w)\log_{2}p(w)\\
 & =-\frac{2}{6}\log_{2}\frac{1}{6}-\frac{2}{3}\log_{2}\frac{1}{3}\\
 & =\frac{1}{3}-\log_{2}\frac{1}{3}=1.9182958
\end{align*}

\end_inset

This can be compared to the entropy of the random 4-word corpus: 
\begin_inset Formula $h_{RAND}^{red}=-\log_{2}1/4=2$
\end_inset

.
 Note that 
\begin_inset Formula 
\[
h_{RAND}^{red}-h_{SING}^{red}=0.081704>0.070298=h_{RAND}-h_{SING}
\]

\end_inset

In other words, the reduced corpus shows more order than the comparable
 unreduced corpus! Interesting! The above can be written as: 
\begin_inset Formula 
\[
h_{SING}-h_{SING}^{red}=0.333334>0.321928=h_{RAND}-h_{RAND}^{red}
\]

\end_inset


\end_layout

\begin_layout Standard
What about the reduced pair entropy? For this case, we have
\begin_inset Formula 
\begin{align*}
h_{PAIR}^{red} & =-\sum_{w_{1},w_{2}\in\beta}p(w_{1},w_{2})\log_{2}p(w_{1},w_{2})\\
 & =-\frac{2}{3}\log_{2}\frac{2}{3}-\frac{1}{3}\log_{2}\frac{1}{3}\\
 & =-\frac{2}{3}-\log_{2}\frac{1}{3}=0.9182958
\end{align*}

\end_inset

which can be compared to the random-pair entropy of 
\begin_inset Formula $h_{PR-RAND}^{red}=-\log_{2}1/16=4$
\end_inset

.
 The comparable reduction is 
\begin_inset Formula 
\[
h_{PR-RAND}^{red}-h_{PAIR}^{red}=3.081704>3.0588935=h_{PR-RAND}-h_{PAIR}
\]

\end_inset

So again, this wins, but not by a lot.
 Re-ordering, this can be written as:
\begin_inset Formula 
\[
h_{PAIR}-h_{PAIR}^{red}=0.6666667>0.643856=h_{PR-RAND}-h_{PR-RAND}^{red}
\]

\end_inset


\end_layout

\begin_layout Standard
So we seem to have two ways of winning: reducing the overall entropy, for
 for single letters, and for pairs, and also finding reductions that are
 strong, even compared to the reduced vocab.
\end_layout

\begin_layout Subsection*
Reductio ad absurdum? No.
\end_layout

\begin_layout Standard
What if we continue on this path, and (incorrectly) reduce to 
\begin_inset Formula $N=3$
\end_inset

 letters, with 
\begin_inset Formula $\delta=\{\gamma,\eta,D\}$
\end_inset

 where 
\begin_inset Formula $\eta=\{B,E\}$
\end_inset

? Then 
\begin_inset Formula $p(\eta)=p(B)+p(E)=1/2$
\end_inset

 
\begin_inset Formula 
\begin{align*}
h_{SING}^{rr} & =-\sum_{w\in\delta}p(w)\log_{2}p(w)\\
 & =-\frac{1}{6}\log_{2}\frac{1}{6}-\frac{1}{3}\log_{2}\frac{1}{3}-\frac{1}{2}\log_{2}\frac{1}{2}\\
 & =\frac{2}{3}-\frac{1}{2}\log_{2}\frac{1}{3}=1.4591479
\end{align*}

\end_inset

and the reduction inequality is
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
h_{SING}^{red}-h_{SING}^{rr}=0.4591479>0.4150375=h_{RAND}^{red}-h_{RAND}^{rr}
\]

\end_inset

So this inequality allows an inappropriate reduction to take place.
 That implies that we must not use the SING inequality to obtain reductions!
\end_layout

\begin_layout Standard
For the pairs, 
\begin_inset Formula $p(\gamma,\eta)=p(\gamma,B)=2/3$
\end_inset

 and 
\begin_inset Formula $p(D,\eta)=p(D,E)=1/3$
\end_inset

 and everything else being zero.
 Thus one gets:
\begin_inset Formula 
\begin{align*}
h_{PAIR}^{rr} & =-\sum_{w_{1},w_{2}\in\delta}p(w_{1},w_{2})\log_{2}p(w_{1},w_{2})\\
 & =-\frac{2}{3}\log_{2}\frac{2}{3}-\frac{1}{3}\log_{2}\frac{1}{3}\\
 & =-\frac{2}{3}-\log_{2}\frac{1}{3}=0.9182958
\end{align*}

\end_inset

so that 
\begin_inset Formula 
\[
h_{PAIR}^{red}-h_{PAIR}^{rr}=0\ngtr0.830075=h_{PR-RAND}^{red}-h_{PR-RAND}^{rr}
\]

\end_inset

Here, nothing is gained, so the pair inequality blocks the inappropriate
 reduction.
 Consider a different inappropraite reduction to 
\begin_inset Formula $N=3$
\end_inset

: let 
\begin_inset Formula $\epsilon=\{\zeta,B,E\}$
\end_inset

 with 
\begin_inset Formula $\zeta=\{D,\gamma\}$
\end_inset

 Then the pair probabilities are 
\begin_inset Formula $p(\zeta,B)=p(\gamma,B)=2/3$
\end_inset

 and 
\begin_inset Formula $p(\zeta,E)=p(D,E)=1/3$
\end_inset

 and again, there is no entropy reduction.
 The other groupings look to be equally ineffective.
\end_layout

\begin_layout Subsection*
Finding Patterns, General Formula
\end_layout

\begin_layout Standard
OK, recast the above section for the (semi-)general case of word-pairs (not
 structures in general).
 So, given a vocabulary of 
\begin_inset Formula $N$
\end_inset

 words, we have 
\begin_inset Formula $h_{RAND}=-\log_{2}\frac{1}{N}=\log_{2}N$
\end_inset

 and 
\begin_inset Formula $h_{RAND}^{red}=\log_{2}(N-1)$
\end_inset

 so that for large 
\begin_inset Formula $N,$
\end_inset

 
\begin_inset Formula $h_{RAND}-h_{RAND}^{red}=\log_{2}N/(N-1)=\log_{2}(1+1/(N-1))\approx1/N$
\end_inset

 and so we have a word-combine winner if we can combine words A and C into
 a cluster 
\begin_inset Formula $\gamma=\{A,C\}$
\end_inset

 such that 
\begin_inset Formula 
\begin{align*}
\frac{1}{N} & \lesssim h_{SING}-h_{SING}^{red}\\
 & =-\sum_{w\in\alpha}p(w)\log_{2}p(w)+\sum_{w\in\beta}p(w)\log_{2}p(w)\\
 & =-p(A)\log_{2}p(A)-p(C)\log_{2}p(C)+p(\gamma)\log_{2}p(\gamma)\\
 & =p(A)\log_{2}\left(1+\frac{p(C)}{p(A)}\right)+p(C)\log_{2}\left(1+\frac{p(A)}{p(C)}\right)
\end{align*}

\end_inset

where 
\begin_inset Formula $p(\gamma)=p(A)+p(C)$
\end_inset

.
 What's not clear: is this inequality 
\emph on
ever
\emph default
 broken? Or does it always hold? At any rate, from the previous example,
 it seems clear that we should not use the SING inequalities to obtain clusters.
\end_layout

\begin_layout Standard
For pairs, its clear that 
\begin_inset Formula $h_{PR-RAND}-h_{PR-RAND}^{red}\approx2/N$
\end_inset

 which follows as above, given that 
\begin_inset Formula $h_{PR-RAND}=2\log_{2}N$
\end_inset

, etc.
 The corresponding inequality is now
\begin_inset Formula 
\begin{align*}
\frac{2}{N}\lesssim & h_{PAIR}-h_{PAIR}^{red}\\
=-\sum_{w_{1},w_{2}\in\alpha} & p(w_{1},w_{2})\log_{2}p(w_{1},w_{2})+\sum_{w_{1},w_{2}\in\beta}p(w_{1},w_{2})\log_{2}p(w_{1},w_{2})\\
=-\sum_{w\in\alpha\backslash A,C} & \left[p(A,w)\log_{2}p(A,w)+p(C,w)\log_{2}p(C,w)-p(\gamma,w)\log_{2}p(\gamma,w)\right]\\
-\sum_{w\in\alpha\backslash A,C} & \left[p(w,A)\log_{2}p(w,A)+p(w,C)\log_{2}p(w,C)-p(w,\gamma)\log_{2}p(w,\gamma)\right]\\
 & -p(A,A)\log_{2}p(A,A)-p(C,A)\log_{2}p(C,A)+p(\gamma)\log_{2}p(\gamma)\\
 & -p(A,C)\log_{2}p(A,C)-p(C,C)\log_{2}p(C,C)
\end{align*}

\end_inset

So...
\end_layout

\begin_layout Section*
8 March 2014
\end_layout

\begin_layout Subsection*
Morphology
\end_layout

\begin_layout Standard
Notes: https://en.wikipedia.org/wiki/Nonconcatenative morphology
\end_layout

\begin_layout Section*
25 March 2014
\end_layout

\begin_layout Subsection*
Information-Theoretic Clustering
\end_layout

\begin_layout Standard
New references:
\end_layout

\begin_layout Itemize
http://www.cs.utexas.edu/users/inderjit/public_papers/kdd_cocluster.pdf Information-
Theoretic Co-clustering Inderjit S.
 Dhillon, Subramanyam Mallela, Dharmendra S.
 Modha 
\end_layout

\begin_layout Itemize
http://pdf.aminer.org/000/472/364/a_generalized_maximum_entropy_approach_to_bregma
n_co_clustering_and.pdf A Generalized Maximum Entropy Approach to Bregman
 Co-clustering and Matrix Approximation Arindam Banerjee, Inderjit Dhillon,
 Joydeep Ghosh, Srujana Merugu, Dharmendra S.
 Modha Journal of Machine Learning Research 8 (2007) 1919-1986 
\end_layout

\begin_layout Section*
30 March 2014
\end_layout

\begin_layout Standard
The below was going to be a brief note, but I'm turning it into a rough
 draft blog post.
 But after sleeping on it, it seems silly.
 
\end_layout

\begin_layout Subsection*
Freedom and Constraint
\end_layout

\begin_layout Standard
The concepts of freedom and constraint are central to the definition of
 algebra in mathematics.
 So for example, in group theory, the algebraic symbols denoting the elements
 of the group may be arranged freely, in any order desired.
 A given group is then defined as a 'presentation', a set of equivalences
 between different orderings.
 Thus, there is the notion of a 'free group', which is merely a set of symbols
 that can be written in arbitrary order, and no further constraints other
 than those of it being a group.
 Groups that aren't free are presented by a collection of equations, which
 state that one certain order of symbols is equivalent to another.
 One says that groups are 'equationally presented'.
 
\end_layout

\begin_layout Standard
A more complex example is the term algebra, where the terms may be arranged
 in free order; but the combination of the written symbols on the page are
 constrained to those of the 'signature' of the algebra.
 One then has the notion of an 'equational theory', which is a term algebra
 with additional equations between expressions, indicating which expressions
 should be taken as equivalent.
\end_layout

\begin_layout Standard
These have strong, and even precise analogues in linguistics.
 But first, continuing with the mathematical observations: the signature
 of a term algebra can be viewed as defining the 'syntax' of the symbolic
 notation: a Turing machine, tasked with the need to recognize the 'language'
 of the term algebra, would process input symbols one by one.
 It would appear that term algebras have a context-free syntax, and are
 thus recognizable by a push-down automata.
 That is, one must recognize the function symbol, the open and close parens,
 the commas separating arguments, and the constant symbols.
 The arbitrary-depth recursiveness is the only reason why the push-down
 is needed; otherwise the language seems 'almost regular'.
 (Hmm ...
 is there any formal definition/distinction of this case? i.e.
 for very simple constext-free languages, vs.
 'more complex' ones? Not that I know of ...)
\end_layout

\begin_layout Standard
In linguistics, similar notions of freedom and constraint arise, but seem
 to be more of a surprise and mystery to linguists.
 Thus, for example, in 
\begin_inset CommandInset citation
LatexCommand cite
key "Anderson2012"
literal "true"

\end_inset

, Anderson describes the syntax and morphotactics of Kwakw’ala, a Wakashan
 language of coastal British Columbia.
 The syntax of the language (that is, the order in which the words can appear
 in a sentence) is very strict: the verb must be followed by a subject,
 optionally followed by the object, and then a prepositional phrase.
 Similarly adjectives must always precede the noun.
 The language also has a rich morphology: words are assembled from stems
 and suffixes.
 The rules for assembling a word out of stems and affixes is referred to
 as the 'morphotactics'.
 In Kwakw’ala, it would appear that the morphotactics is utterly distinct
 from the syntax: here, object-denoting prefixes can precede verbs, adjective-de
noting suffixes follow a noun.
 Anderson finds this quite remarkable: the language has two distinct kinds
 of structure-imposing systems: the syntax and the morphtactics, and they
 are quite different.
 He notes that this dual structure in turn allows the same thing to be said
 in multiple ways.
 One may take meaning-parts, as morphemes, and glue them together morphtacticall
y into words, and aranging these in a sentence.
 Allternately, one may take the meaning-parts separately, as individual
 words, and glue them together into a sentence, having a different sequence
 of the meaning-parts.
\end_layout

\begin_layout Standard
The part that struck me with Anderson's analysis is the similarity of the
 phenomena to the analogous behaviour formalized in mathematics.
 Lets first look at a second example: Lithuanian has a rich morphotactical
 structure: verbs and adverbs are conjugated, nouns and adjectives are declined;
 the rules for doing so are rather fixed and uniform, making adjustments
 mostly for phonological reasons (i.e.
 with exceptions based on constraints that come from the natural flow of
 the sound sequences constrained by the use of vocal cords, mouth, tongue
 and lips).
 Curiously, Lithuanian is almost devoid of syntactic constraint: word-order
 can be chosen freely (in the mathemaitcal sense!), and the meaning of the
 resulting sentences are essentially the same (if I am allowed to gloss over
 the notion that different word orders can serve to highlight or emphasize
 different themes and rhemes).
 So again: a language with very distinct syntax and morphtactics; in this
 case, the syntax being almost absent.
 
\end_layout

\begin_layout Standard
I used the theory of Link Grammar for performing structural linguistic analysis.
 The theory was originally developed to model syntactic structure, but it
 also appears to be entirely adequate for morphotactic analysis as well
 (certainly, for 'agglutinative' or 'concatenative' languages, with ongoing
 research into more complex morphologies).
 From the point of view of a linguist, Link Grammar appears to be 'just
 another theory of syntax', being a kind of dependency grammar.
 From the point of view of a mathematician, the situation is entirely more
 remarkable.
 It appears that the mathematical definition of what constitutes a 'link
 grammar' is isomorphic to that of a 'categorical grammar', and that the
 correspondence is immediate and direct.
 Categorical grammars are interesting because they have a direct, formal
 mathematical definition that is studied and classified by mathematicians:
 roughly speaking, categorical grammars are 'non-symmetric compact closed
 monoidal categories'.
 The precise definition here has been championed by Bob Coecke ref [xxx]
 
\end_layout

\begin_layout Standard
It takes some study of category theory to understand what this means, but,
 roughly speaking, it means that sequences of sounds, morphemes, words are
 analyzed in sequential order: by means of short-distance groupings of left-righ
t arrangements.
 This may sound silly, as, of course, sequential things occur in a sequence,
 but it helps highlight the difference between dependency grammars and phrase-st
ructure grammars, or computer-science grammars in general.
 An example of a 'computer-science grammar' is the so-called 'context-free
 grammar'.
 A hallmark of such grammars is that they allow recursion to arbitrary depths.
 An English-language example would be the sequence of sentences: 
\begin_inset Quotes eld
\end_inset

This is a house
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

This thing is a house.
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

This thing is a thing that is a house.
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

This thing is a thing that is a thing that is ...
 a house.
\begin_inset Quotes erd
\end_inset

 The example is silly because no one ever talks that way.
 The phrase-structure analysis of this would be 
\begin_inset Quotes eld
\end_inset

(S (VP (NP (VP (NP ...
 )))))
\begin_inset Quotes erd
\end_inset

, with the hierarchical arrangement emphsized.
 Dependency grammars can also parse such sentences, but here, the arrangement
 of dependencies are in the form of arrows that point from head word to
 dependent word; the arrows are only rarely long-range, and usually point
 to the immediately-surrounding words.
 There is strong psycho-linguistic evidence for such local structure in
 language, see for example [xxxx].
 That is, the workings of the human mind is not recursive in nature, pushing
 and popping an arbitrarily deep stack as each new noun-phrase or verb-phrase
 is encountered.
 Indeed, psychological studies with constructed sentences similar to the
 above, but varying the 'thing' and 'house' at each depth, show that humans
 quickly loose track after just two or three nestings [need ref].
 In essence, the human mind is adapted for linear sequential analysis, and
 long-range order between words is challenging: this is the psycho-linguistic
 argument for dependency grammars.
 From the mathematical point of view, the statement is that human languages
 are not so much context-free, as they are non-symmetric compact closed
 monoidal categories.
 That Link Grammar is an example of the latter is why it seems so appropriate
 to use for syntactic and morpho-tactic structural analysis.
\end_layout

\begin_layout Standard
Which theories of language are mathematically isomorphic? That is, Link
 Grammar and categorical grammars seem to be isomorphic because there is
 a simple way of translating the one into the other, and vice-versa (although
 no formal mathematical proof of this has been written down).
 A mathematical proof of equivalence is a mechanical device: given one represent
ation, one turns a crank to obtain the other.
 More generally, its been argued that phrase-structure grammars and dependency
 grammars are equivalent in the same sense: there is an algorithm that converts
 the one into the other, and v.v.[where's the ref for this?].
 Does this mean that non-symmetric compact closed monoidal categories have
 context-free grammars as their internal language, and that every context-free
 language has a corresponding monoidal category? I think not, but the answer
 to that, the 'why not', and the 'what, then, is the difference?' is entirely
 unclear.
 Clarifying these relationships seems important for putting language study
 on a firmer basis.
\end_layout

\begin_layout Standard
Anyway, the point here was to clarify the boundaries between freedom and
 constraint.
 Traditional phrase-structure grammars were inspired by notions from 1960's-era
 computer science, but now seem slavishly wedded to the same ideas, to the
 detriment of closer linguistic understanding.
 Dependency grammars seem to be more psycho-linguistically valid, but have
 suffered from a lack of mathematical formalism that ellucidates freedom
 and constraint.
 This lack of formalism makes it hard to explain why some constructions
 are grammatically correct, and others are not.
 It also seems to draw an artificial and confusing line between syntactic
 and morphotactic structure, when, in fact, these really should be taken
 as a part of a continuum of structure.
 I see no reason why a single grammar could not also describe the allomorphic
 variations in pronunciation.
 After all, these are just a set of rules that govern how a morpheme is
 pronounced, and this is essentially a linear, sequential phenomenon, with
 only (mostly?) nearest-neighbor morphemes affecting one-another.
 The nearest-neighbor aspect of this fairly well screams out 'dependency'.
\end_layout

\begin_layout Standard
Another curious and interesting language-constraint structure emerges with
 the study of idioms and institutionalized, set phrases.
 Because these are 'phrases', built of 'words', it would naively seem that
 these lie in the domain of syntax.
 But this is misleading.
 Institutionalized utterances are those where neither the word-choice nor
 the word-order are directly governed by syntax alone, but instead seem
 to be frozen into a fixed form.
 So, one talks of the 'time of day', but never of 'pressure of air' or 'height
 of mountain' – 
\begin_inset Quotes eld
\end_inset

What pressure of air should I put in this tire?
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

What height of mountain do you plan to climb?
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

What time of day do you expect to come over?
\begin_inset Quotes erd
\end_inset

.
 There is nothing syntactic that prevents such a choice of wording, and
 the semantic meaning is more or less clear: its just that such word arrangement
s simply don't happen.
 Its as if the lexis for English has a phrase in it: 
\begin_inset Quotes eld
\end_inset

time-of-day
\begin_inset Quotes erd
\end_inset

, which should be treated as a single word, rather than the three words
 it is written as.
 This provides the first hint of the role of probability in this discussion:
 the probability of seeing the phrase 'height of mountain' in English approaches
 zero: in fact, this text that you are reading right now just might be the
 only place ever in the history of the world in which this phrase has appeared
 ...
 despite it being 'grammatically valid'.
 Freedom and constraint aren't just governed by true-false distinctions,
 but by probabilities.
 The question then is, 'what is the most natural way in which to express
 such probabilities?' 
\end_layout

\begin_layout Standard
The last is not just some idle intellectual question, but in fact, an engineerin
g question: the proper structure should have an immediate and direct effect
 on how well, and how quickly a language could be learned, via unsupervised
 machine-learning algorithms.
 A universal but naive attitude in the artificial-intelligence community
 is that 'oh.
 everything is a neural net, and we should use neural nets to build AI.'
 Less frequently, one may seem a similar attitude regarding Hidden Markov
 Models (HMMs).
 The fact that such naive approaches lead to algorithms that fail to converge
 quickly leads to ideas such as 'deep learning': a modification that explicitly
 splits a problem into layers, with explicit feedback between layers.
 Another variation used to escape the trap is to explicitly model what is
 un-known: this is the notion of maximum entropy (MaxEnt).
 Traditional AI was also founded on logic and reasoning, and, for many decades,
 AI was dominated by the exploration of boolean-valued logic.
 By this I mean anything with crisp, sharp truth values: whether first-order
 logic, boolean satisfiability, satisfiability-modulo-theories, stable-model
 semantics, and so on.
 Another corner was fuzzy logic, but that didn't seem to have legs.
 Notions of maximum entropy and probability can be unified: thus, one has
 Markov Logic Networks (MLN).
 What I'm wodering about here is that maybe none of these approaches are
 correct, because they are ignoring the actual structure that is in front
 of us.
\end_layout

\begin_layout Standard
So, perhaps, the correct approach is not to marry maximum entropy with first-ord
er logic, but to marry maximum entropy to dependency grammars (or, equivalently,
 to appropriate monoidal categories).
 The question then becomes: what is the appropriate monoidal category? Picking
 the wrong one will lead to disastrous machine learning performance (this,
 I think, is the lesson from neural networks).
 Picking something too easy doesn't get you far enough (the lesson of HMM's
 – excellent for certain classes of problems, but lacking in scale).
 There are more choices than that: but the chices, and their inter-connectedness
, and trade-offs, seem to be unarticulated.
 For any given monoidal category, there would seem to be some probabilistic
 model corresponding to that category's internal language.
 That is, there is a way of describing the transition probabilities from
 state to state.
 Indeeded, (finite) monoidal categories, in the form of acts, can be partly
 understood to be finite state machines acting on a set.
 The probabilistic generalization of this leads both to probabilistic and
 quantum finite automata, with the former having a strong resemblance, if
 not identity, to Markov chains, with the corresponding acts being HMM's.
 My hypothesis is that probabilistic dependency grammars will lead to machine
 learning algos that converge more rapidly than the similar-but-different
 HMM that can also be mapped onto the same problem.
 Unfortunately, my hypothesis is impedded by my lack of understanding of
 precisely, exactly how the different approaches named above may be equivalent,
 isomorphic, or merely similar.
\end_layout

\begin_layout Section*
2 April 2014
\end_layout

\begin_layout Subsection*
Link Grammar and Finite State Transducers
\end_layout

\begin_layout Standard
Claim: Finite state transducers, such as those used for morphological analysis,
 can be mapped to a Link Grammar.
 This implies that Link Grammar parsing can be used for morphological analysis,
 thus unifying syntactic parsing and morphological analysis into a unified
 framework.A finite state transducer (FST) is defined as:
\end_layout

\begin_layout Itemize
A set of states 
\begin_inset Formula $Q$
\end_inset


\end_layout

\begin_layout Itemize
A set 
\begin_inset Formula $\Sigma$
\end_inset

 of input symbols (surface form)
\end_layout

\begin_layout Itemize
A set 
\begin_inset Formula $\Gamma$
\end_inset

 of output symbols (lexicalized form)
\end_layout

\begin_layout Itemize
A transition function 
\begin_inset Formula $\delta\subset Q\times(\Sigma\cup\{\epsilon\})\times(\Gamma\cup\{\epsilon\})\times Q$
\end_inset


\end_layout

\begin_layout Standard
A member 
\begin_inset Formula $(r,a,b,s)\in\delta$
\end_inset

 should be thought of as the arrow from state 
\begin_inset Formula $r$
\end_inset

 to state 
\begin_inset Formula $s$
\end_inset

, the arrow being taken when the input symbol is 
\begin_inset Formula $a$
\end_inset

 and as a result producing the output symbol 
\begin_inset Formula $b$
\end_inset

.
 The corresponding link-grammar dictionary entry for this would be 
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     a.b: r- & s+;
\end_layout

\end_inset


\end_layout

\begin_layout Standard
This states that no linkage is possible, unless the previous link resulted
 in the emission of the r+ connector.
 No transition to the next state is possible, unless that state has an s-
 connector on it.
\end_layout

\begin_layout Standard
The current link-grammar notation a.b is awkward for printing, and perhaps
 some new style is needed to distinguish the output to be printed from the
 input that is recognized.
 Thus, perhaps, it would be better to invent a new notation, perhaps a$b
 to denote that a is recognized, and that b is printed.
 
\end_layout

\begin_layout Standard
Note that the above definition of link-grammar rules results in a very simple,
 linear linkage: state transitions follow one-another in linear order.
 Link grammar allows richer, more complex linkage diagrams, and so the question
 arises: can a given FST be compactified into a smaller system by making
 use of the richer possibilities that link-grammar offers? How can this
 compactification be achieved?
\end_layout

\begin_layout Standard
Suppose that the FST 
\begin_inset Formula $\delta$
\end_inset

 includes as a subset the state transitions 
\begin_inset Formula $\{(r,a,?,s),(s,\epsilon,\epsilon,t),(s,b,?,t),(t,c,?,u)\}$
\end_inset

.
 The symbol ? is used here as a don't-care state, as it is irrelevant to
 the discussion that follows.
 The above state transitions indicates that when the system is in state
 
\begin_inset Formula $s$
\end_inset

, it may spontaneously transition to state 
\begin_inset Formula $t$
\end_inset

, or may do so upon reading 
\begin_inset Formula $b$
\end_inset

.
 That is, the presence of 
\begin_inset Formula $b$
\end_inset

 is optional in the state transition.
 The 
\begin_inset Quotes eld
\end_inset

natural
\begin_inset Quotes erd
\end_inset

 way of indicating this with link-grammar notation is using the link-grammar
 dictionary entries: 
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

   a: r- & s+;
\end_layout

\begin_layout Plain Layout

   b: t+;
\end_layout

\begin_layout Plain Layout

   c: {t-} & s- & u+;
\end_layout

\end_inset

Because the transition 
\begin_inset Formula $(s,\epsilon,\epsilon,t)$
\end_inset

 reads no input, and produces no output, the state transitions would more
 likely be written as 
\begin_inset Formula $\{(r,a,?,t),(r,a,?,s),(s,b,?,t),(t,c,?,u)\}$
\end_inset

, that is, by collapsing the transition 
\begin_inset Formula $(s,\epsilon,\epsilon,t)$
\end_inset

 into the prior state.
 This would have the entries
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

   a: r- & (s+ or t+);
\end_layout

\begin_layout Plain Layout

   b: s- & t+;
\end_layout

\begin_layout Plain Layout

   c: t- & u+;
\end_layout

\end_inset


\end_layout

\begin_layout Standard
How should it be understood? These are, in fact, two distinct, inequivalent
 LG grammars, as can be seen by considering the parse of the strings 
\begin_inset Quotes eld
\end_inset

ac
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

abc
\begin_inset Quotes erd
\end_inset

 for the two cases.
\end_layout

\begin_layout Standard
When would weighting schemes interfere? when would output interfere?
\end_layout

\begin_layout Section*
15 April 2014
\end_layout

\begin_layout Subsection*
Elegant Normal Form
\end_layout

\begin_layout Standard
Or, more precisely, 
\begin_inset Quotes eld
\end_inset

Minimal Normal Form
\begin_inset Quotes erd
\end_inset

.
 Instead of writing out LG disjuncts in long strings of DNF or CNF, where
 they blow up into the thousands or tens of thousands, we really need to
 write then in Craig Holman's "Elegant Normal Form", ( http://www.patterncraft.com
/Blog/Blog-080609.html#ElegantNormalForm ) format.
 This is to be done by entropy minimization, in two different ways: first,
 ENF reduces the total count of terms, for just one single expression.
 Second, and maybe more important: different words will share significant
 subsets of the ENF expression.
 So, for example, the LG English dicts define:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     <verb-rq>: Rw- or ({Ic-} & Q- & <verb-wall>) or [()];
\end_layout

\end_inset

which is (1) in ENF, not DNF or CNF, and (2) shared by several dozen words.
 There should be a strong push to discover such common sub-expressions across
 many words.
\end_layout

\begin_layout Section*
28 April 2014
\end_layout

\begin_layout Subsection*
Isotopy
\end_layout

\begin_layout Standard
The concept of 
\begin_inset Quotes eld
\end_inset

isotopy
\begin_inset Quotes erd
\end_inset

 (https://en.wikipedia.org/wiki/Isotopy_(semiotics)) was introduced by Algirdas
 Greimas in 1966.
 Example: 
\begin_inset Quotes eld
\end_inset

I drink some water
\begin_inset Quotes erd
\end_inset

, with the meanings of 
\begin_inset Quotes eld
\end_inset

drink
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

water
\begin_inset Quotes erd
\end_inset

 re-inforcing one-another.
 But this is exactly what the Mihalcea WSD algo does, eh? 
\end_layout

\begin_layout Section*
14 May 2014
\end_layout

\begin_layout Subsection*
Tree similarity
\end_layout

\begin_layout Standard
\begin_inset Quotes eld
\end_inset

Similarity Evaluation on Tree-structured Data
\begin_inset Quotes erd
\end_inset

 Rui Yang Panos Kalnis Anthony K.
 H.
 Tung SIGMOD 2005 June 14-16, 2005.
 Quote from abstract: 
\begin_inset Quotes eld
\end_inset

propose to trans- form tree-structured data into an approximate numerical
 multidimensional vector
\begin_inset Quotes erd
\end_inset

.
 Funny – that's what Bob Coecke proposes for any kind of monoidal category:
 vector spaces being a special monoidal cat.
 Hmmm.
 
\end_layout

\begin_layout Standard
Approaches:
\end_layout

\begin_layout Itemize
Tree-edit distance: many variants proposed, all high cpu/memory intensive.
\end_layout

\begin_layout Itemize
convert tree to pre or post-order, and use string edit distance.
\end_layout

\begin_layout Itemize
Convert to binary tree.
 For combo trees, this makes sense, due to the associative property of most
 of the operators.
 In particular, in combo any oper that can have multi-sibilings is also
 associative and thus convertible to binary tree.
 What's more, trees with binary branch distance of zero really are equivalent
 for us: See Figure 4 in above reference.
 Yay! this fits very very well with combo!.
\end_layout

\begin_layout Section*
29 June 2014
\end_layout

\begin_layout Subsection*
Morphology Basic Claims
\end_layout

\begin_layout Standard
We have two tasks to address: the automated discovery of morpheme boundaries,
 and the automated discovery of 
\begin_inset Quotes eld
\end_inset

morphtactics
\begin_inset Quotes erd
\end_inset

, the syntax of connected morphemes.
 We make two claims: first.
 the automated discovery of morpheme boundaries can be accomplished by searching
 for breaks between word-parts that have the lowest mutual information.
 Second, the discovery of morphotactics is identical to the discovery of
 syntax, as outlined above.
\end_layout

\begin_layout Standard
The simplest approach to finding the breaks between morphemes is to randomly
 break up words into two parts.
 A worked example of this is given below.
 Several questions present themselves:
\end_layout

\begin_layout Itemize
To discover morphemes of words that split into three or more parts, is it
 better to always split pairwise, and then perform recursion, or is it easier
 to split into multiple parts immediately? Perhaps the answer is language-depend
ent?
\end_layout

\begin_layout Itemize
Does one obtain better mophological splits by immediately including morphtactic
 analysis, or can this be deferred?
\end_layout

\begin_layout Subsection*
Morphology Worked Example
\end_layout

\begin_layout Standard
OK, this will be tedious, but I see no alternative.
 Suppose we have the corpus 
\begin_inset Quotes eld
\end_inset

test gift tester testy gifty tester gifter
\begin_inset Quotes erd
\end_inset

 so that 
\begin_inset Quotes eld
\end_inset

tester
\begin_inset Quotes erd
\end_inset

 appears twice in the corpus.
 Explore all possible splits into two parts.
 The 4-letter splits split 3 ways, the 5-letter splits split 4 ways, etc.
 so there is a total of N(*,*)=3+3+5+4+4+5+5=29 pairs.
 All pairs appear once, except for tester, which appears twice.
 Viz.
\begin_inset Newline newline
\end_inset

 
\end_layout

\begin_layout Standard
P(x,y)=1/29 for (x,y) in {(t,est), (te,st), (tes,t), (g,ift), (gi,ft), (gif,t),
 (t,esty), (te,sty), (tes,ty), (test,y), (g,ifty), (gi,fty), (gif,ty), (gift,y),
 (g,ifter), (gi,fter), (gif,ter), (gift,er), (gifte,r)}
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

and
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(x,y)=2/29 for (x,y) in {(t,ester), (te,ster), (tes,ter), (test,er), (teste,r)}
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

There is a bit of a procedural error in the above; we would like to discover
 the 
\begin_inset Quotes eld
\end_inset

null suffix
\begin_inset Quotes erd
\end_inset

, that is, that 
\begin_inset Quotes eld
\end_inset

test
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

gift
\begin_inset Quotes erd
\end_inset

, with nothing following it, are morphemes, so that the possible suffixes
 are 
\begin_inset Quotes eld
\end_inset

-y
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

-er
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

-nothing
\begin_inset Quotes erd
\end_inset

.
 However, the above failed to count this possibility separately.
 Thus, given the above data, what we expect to find are two roots: 
\begin_inset Quotes eld
\end_inset

gif-
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

tes-
\begin_inset Quotes erd
\end_inset

 and three suffixes: 
\begin_inset Quotes eld
\end_inset

-t
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

-ty
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

-ter
\begin_inset Quotes erd
\end_inset

.
 This is not so bad.
 If we did split and count in such a way as to allow a null suffix, it would
 be ambiguus as to whether the stems end with a 
\begin_inset Quotes eld
\end_inset

t
\begin_inset Quotes erd
\end_inset

 or not.
 That is, the with-t and without-t stems would have been equally likely...
 Anyway, moving on...
 the possible splits are shown in the table below 
\begin_inset CommandInset ref
LatexCommand ref
reference "tab:Word-Split-Table"

\end_inset

:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
\begin_inset Float table
wide false
sideways false
status open

\begin_layout Plain Layout
\begin_inset Caption Standard

\begin_layout Plain Layout
Word Split Table
\begin_inset CommandInset label
LatexCommand label
name "tab:Word-Split-Table"

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset Tabular
<lyxtabular version="3" rows="20" columns="12">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
g
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gi
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gift
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gifte
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
t
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
te
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
test
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
teste
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
row total
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ifter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
fter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
er
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
r
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ifty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
fty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ift
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ft
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
t
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ester
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ster
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
esty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
sty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
est
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
st
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
column total
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
29
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Plain Layout
The above is a sparse matrix showing the possible word splits.
 empty cells contain a count of zero.
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Next, lets do the partial sums.
 Recall the notation for the partial summation of pairs.
 writing P(x,y) for the probability of observing the 
\emph on
ordered
\emph default
 pair of items (x,y), the partial sums are: 
\begin_inset Formula 
\[
P(x,*)=\sum_{y\in Y}P(x,y)
\]

\end_inset

 and 
\begin_inset Formula 
\[
P(*,y)=\sum_{x\in X}P(x,y)
\]

\end_inset

 The left-hand sums are the column totals in the table above, table 
\begin_inset CommandInset ref
LatexCommand ref
reference "tab:Word-Split-Table"

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(t,*) = (1+1+2)/29 = 4/29 = P(te,*) = P(tes,*)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(g,*) = (1+1+1)/29 = 3/29 = P(gi,*) = P(gif,*)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(test,*) = (1+2)/29 = 3/29
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(teste,*) = 2/29
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(gift,*) = 2/29
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(gifte,*) = 1/29
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

Next, the right-hand partial sums.
 These are the row totals for the table above, table 
\begin_inset CommandInset ref
LatexCommand ref
reference "tab:Word-Split-Table"

\end_inset

:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(*,est) = 1/29 = P(*,st) = P(*,esty) = P(*,sty) =P(,ift) = P(*,ft) = P(*,ifty)
 = P(*,fty) = P(*,ifter) = P(*,fter)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(*,t) = (1+1)/29 = 2/29 = P(*,ty) = P(*,y)
\end_layout

\begin_layout Standard
P(*,ester) = 2/29 = P(*,ster)
\end_layout

\begin_layout Standard
P(*,ter) = (1+2)/29 = 3/29 = P(*,er) = P(*,r)
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

Now, the compute the MI (we use log=log_2 in all cases below, for measuring
 the entropy in units of bits).
 Recall the definition of mutual information for 
\emph on
ordered
\emph default
 pairs, previously discussed and given above: 
\begin_inset Formula 
\[
MI(x,y)=\log_{2}\frac{P(x,y)}{P(x,*)P(*,y)}
\]

\end_inset

So, working these by hand:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(t,est) = log P(t,est)/P(t,*)P(*,est) = log (1/29)(29/4)(29/1) = log(29/4)
 = 2.857981
\end_layout

\begin_layout Standard
= MI(te,st) = MI(t,esty) = MI(te,sty) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(g,ift) = log P(g,ift)/P(g,*)P(*,ift) = log (1/29)(29/3)(29/1) = log(29/3)
 = 3.273018
\end_layout

\begin_layout Standard
=MI(gi,ft) = MI(g,ifty) = MI(gi,fty)=MI(g,ifter)=MI(gi,fter)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(tes,t) = log P(tes,t)/P(tes,*)P(*,t) = log (1/29)(29/4)(29/2) = log(29/8)
 = 1.857981
\end_layout

\begin_layout Standard
=MI(tes,ty)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gif,t) = log P(gif,t)/P(gif,*)P(*,t) = log (1/29)(29/3)(29/2) = log(29/6)
 = 2.273018
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(test,y) = log P(test,y)/P(test,*)P(*,y) = log (1/29)(29/3)(29/2) = log(29/6)
 = 2.273018
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gif,ty) = log P(gif,ty)/P(gif,*)P(*,ty) = log (1/29)(29/3)(29/2) = log(29/6)
 = 2.273018
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gift,y) = log P(gift,y)/P(gift,*)P(*,y) = log (1/29)(29/2)(29/2) = log(29/4)
 = 2.857981
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gif,ter) = log P(gif,ter)/P(gif,*)P(*,ter) = log (1/29)(29/3)(29/3) =
 log(29/9) = 1.688056
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gift,er) = log P(gift,er)/P(gift,*)P(*,er) = log (1/29)(29/2)(29/3) =
 log(29/6) = 2.273018
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gifte,r) = log P(gifte,r)/P(gifte,*)P(*,r) = log (1/29)(29/1)(29/3) =
 log(29/3) = 3.273018
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(t,ester) = log P(t,ester)/P(t,*)P(*,ester) = log (2/29)(29/4)(29/2) =
 log(29/4) = 2.857981
\end_layout

\begin_layout Standard
= MI(te,ster) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(tes,ter) = log P(tes,ter)/P(tes,*)P(*,ter) = log (2/29)(29/4)(29/3) =
 log(29/6) = 2.273018
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(test,er) = log P(test,er)/P(test,*)P(*,er) = log (2/29)(29/3)(29/3) =
 log(58/9) = 2.688056
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(teste,r) = log P(teste,r)/P(teste,*)P(*,r) = log (2/29)(29/2)(29/3) =
 log(29/3) = 3.273018
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

Phew.
 I think that's all of them.
 So, what can we conclude? The basic claim is that the morpheme boundaries
 occur at the places where the letters are the least sticky, the most likely
 to be de-correlated, i.e.
 those with the lowest MI.
 In the above, these are: MI(gif,ter)=1.69 followed by MI(tes,t)=MI(tes,ty)=1.86.
 These are the most likely splits for these three words.
 Lets look up each possible split, for each word.
 We get:
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="12">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Split
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Split
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Split
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Split
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Split
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Best
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gift
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(g,ift)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gi,ft)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,t)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,t)
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gifty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(g,ifty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gi,fty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,ty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gift,y)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,ty)
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gifter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(g,ifter)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gi,fter)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,ter)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.69
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gift,er)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gifte,r)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,ter)
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
test
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(t,est)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(te,st)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,t)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,t)
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
testy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(t,esty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(te,sty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,ty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(test,y)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,ty)
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tester
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(t,ester)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(te,ster)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,ter)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(test,er)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.69
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(teste,r)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,ter)
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The best results from the above table are summarized below
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Lowest MI split(s)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gift
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,t)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gifty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,ty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gifter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(gif,ter)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.69
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
test
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,t)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.86
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
testy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,ty)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.86
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tester
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(tes,ter)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

What looks like the best split has been found; it certainly matches what
 was expected.
 Yay! After this, link-type clustering proceeds just as before, as if these
 were distinct words.
 That is, the above has 6 distinct link types; clustering will then proceed
 discover one link type, between the cluster {gif, tes} and {t,ty,ter}.
\end_layout

\begin_layout Subsubsection*
Morfesssor
\end_layout

\begin_layout Standard
An alternative algorithm is presented in:
\end_layout

\begin_layout Itemize
Mathias Creutz Krista Lagus, 
\begin_inset Quotes eld
\end_inset

Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora
 Using Morfessor 1.0
\begin_inset Quotes erd
\end_inset

 http://users.ics.aalto.fi/mcreutz/papers/Creutz05tr.pdf 
\end_layout

\begin_layout Standard
That algorithm works only for concatenative languages, and does not provide
 a morphotactic structure; that is, it cannot learn the grammar governing
 the morphemes.
 It also requires several (plausible) assumptions about Bayesian priors.
 One assumption is that morpheme frequency follows a modified Zipfian distributi
on, this is used to make estimates for morphemes that are observed only
 once in the corpus.
 Another assumption is is that the morpheme length distribution can be approxima
ted by either a Poisson or a (two-parameter) gamma distribution.
\end_layout

\begin_layout Section*
12 July 2014
\end_layout

\begin_layout Subsection*
Link-type discovery, worked example
\end_layout

\begin_layout Standard
In keeping with the previous, lets look at a super-simplified version of
 link-type discovery, continuing immediately from the previous morpheme-discovery
 example.
 We begin with the initial observations, given in the table below:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Pair
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Initial Link Type
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
# observations
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif–t
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
GA
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif–ty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
GB
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif–ter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
GC
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes–t
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
TA
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes–ty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
TB
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes–ter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
TC
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The 
\begin_inset Quotes eld
\end_inset

initial link type
\begin_inset Quotes erd
\end_inset

 is handed out randomly; the actual letter string has no bearing on the
 outcome.
 Notice the above has 6 different, unique link types.
 These correspond to the following link-grammar dictionary, written in the
 classic link-grammar notation:
\begin_inset Newline newline
\end_inset


\begin_inset Float algorithm
placement H
wide false
sideways false
status open

\begin_layout Plain Layout
\begin_inset Caption Standard

\begin_layout Plain Layout
Morpheme grammar
\begin_inset CommandInset label
LatexCommand label
name "alg:Morpheme-grammar"

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: GA+ or GB+ or GC+;
\end_layout

\begin_layout Plain Layout

     tes.=: TA+ or TB+ or TC+;
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty: GB- or TB-;
\end_layout

\begin_layout Plain Layout

     =ter: GC- or TC-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
From the above initial dictionary, we want to deduce that a single link
 type is sufficient to full describe what is happening.
 That is, we wish to discover the following dictionary:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: LL+;
\end_layout

\begin_layout Plain Layout

     =t =ty =ter: LL-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

This is inuitively obvious, because the morphemes obviously form a clique:
 each stem has been observed with each suffix.
 Technically, this is a bipartite clique or complete bipartite graph of
 order (2,3).
 Here, we see it immediately; however, in general, it is very hard to search
 for bipartite cliques in a grammar; general algorithms are provably NP-complete
 and run in exponential time.
\end_layout

\begin_layout Standard
So how should we find grammar reductions? How is this to be done?
\end_layout

\begin_layout Standard
Out vocabulary consists of N=5 morphemes 
\begin_inset Formula $\alpha=$
\end_inset

{gif.=, tes.=.
 =t, =ty, =ter}.
 We begin by recomuting the MI for observed pairs, once-again starting with
 the initial corpus 
\begin_inset Quotes eld
\end_inset

test gift tester testy gifty tester gifter
\begin_inset Quotes erd
\end_inset

, same as before, with 
\begin_inset Quotes eld
\end_inset

tester
\begin_inset Quotes erd
\end_inset

 appearing twice in the corpus.
 This time, se split strictly according to the learned morphology.
 The word split table is:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="5" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
row total
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
ty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
t
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
column total
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

Note that this table is a stric subset of the previous table; the column
 and row totals are completely unchanged.
 However, the total number of observations has diminished from 29 to 7,
 and so all P amd MI values need to be recomputed.
 Proceeding long-hand, as before:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(x,y)=1/7 for (x,y) in {(tes,t), (gif,t), (tes,ty), (gif,ty), (gif,ter)}
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

and
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(x,y)=2/7 for (x,y) in {(tes,ter)}
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The partial sums are:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(gif,*) = (1+1+1)/7 = 3/7
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(tes,*) = (1+1+2)/7 = 4/7
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(*,t) = 2/7 = P(*,ty)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
P(*,ter) = 3/7
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The MI values are all different, as well:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gif,t) = log P(gif,t)/P(gif,*)P(*,t) = log (1/7)(7/3)(7/2) = log (7/6)
 = 0.222392
\end_layout

\begin_layout Standard
= MI(gif,ty) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(gif,ter) = log P(gif,ter)/P(gif,*)P(*,ter) = log (1/7)(7/3)(7/3) = log
 (7/9) = -0.362570 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(tes,t) = log P(tes,t)/P(tes,*)P(*,t) = log (1/7)(7/4)(7/2) = log (7/8)
 = -0.192645
\end_layout

\begin_layout Standard
= MI(tes,ty)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
MI(tes,ter) = log P(tes,ter)/P(tes,*)P(*,ter) = log (2/7)(7/4)(7/3) = log
 (7/6) = 0.222392
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

Note that three of the MI values are negative, and three are positive.
\end_layout

\begin_layout Standard
Following the previous formulas, we compute the total pair entropy:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{align*}
h_{PAIR}^{observed} & =-\sum_{w_{1},w_{2}\in\alpha}p(w_{1},w_{2})\log_{2}p(w_{1},w_{2})\\
 & =-\frac{5}{7}\log_{2}\frac{1}{7}-\frac{2}{7}\log_{2}\frac{2}{7}=2.521641
\end{align*}

\end_inset

This is a bit of a misnomer, or misleading; we are actually computing the
 link-entropy: so the set is actually 
\begin_inset Formula $\beta=\{GA,GB,GC,TA,TB,TC\}$
\end_inset

 the first five of which were observed once, and the last was observed twice.
 So really we should write:
\begin_inset Formula 
\[
h_{PAIR}^{observed}=-\sum_{t\in\beta}p(t)\log_{2}p(t)
\]

\end_inset

with 
\begin_inset Formula $p(t)$
\end_inset

 being the probability of observing link-type 
\begin_inset Formula $t$
\end_inset

.
\end_layout

\begin_layout Standard
The above is the observed entropy, given the corpus, and the grammar shown
 in listing 
\begin_inset CommandInset ref
LatexCommand ref
reference "alg:Morpheme-grammar"

\end_inset

.
 However, this grammar does not have any probability indicators attached
 to it, so that if it was used to generate a corpus, the entropy would be
 different.
 Basically, the probability of observing any of the link-types would be
 identical, and so the entropy would be:
\begin_inset Formula 
\begin{align*}
h_{PAIR}^{generated} & =-\sum_{t\in\beta}p(t)\log_{2}p(t)\\
 & =-\frac{6}{6}\log_{2}\frac{1}{6}=2.584963
\end{align*}

\end_inset

This is obtained by observing that there are 6 link types in the set 
\begin_inset Formula $\beta$
\end_inset

 and so, if chosen equi-probably, the resulting entropy is just 
\begin_inset Formula $\log_{2}6$
\end_inset

.
 For a given number 
\begin_inset Formula $N$
\end_inset

 of link types, the entropy of the generated grammar will be 
\begin_inset Formula $\log_{2}N$
\end_inset

, for this extremely simply type of grammar, where all disjuncts have only
 one connector in them.
 The generated entropy will always be maximal for the grammar, as the observed
 distribution will surely never be equi-distributed.
 Thus, we have as a general principle:
\begin_inset Formula 
\[
h^{observed}\le h^{generated}
\]

\end_inset

Note that the equi-distributed link-types is the same as having each of
 the words in the corpus appear with equal frequency.
 The morphemes, however, do NOT appear with equally frequency (although
 individually, all stems do, and all suffixes do).
 
\end_layout

\begin_layout Standard
Link type reductions can be many ways.
 In each case, we look to see if adding a new word to a category improves
 the score.
 The possibilities are:
\end_layout

\begin_layout Enumerate
Group =ter and =ty together.
\end_layout

\begin_layout Enumerate
Group =ter and =t together.
\end_layout

\begin_layout Enumerate
Group =ty and =t together.
\end_layout

\begin_layout Enumerate
Group gif.= and tes.= together.
\end_layout

\begin_layout Standard
After this, we have more reductions:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
1.a.
 Add =t to {=ter, =ty}, and finally group together gif.= and tes.=
\end_layout

\begin_layout Standard
1.b.
 Group together gif.= and tes.=, and finally, add =t to {=ter, =ty}
\end_layout

\begin_layout Standard
2.a, 2.b.
 3.a 3.b variations of above
\end_layout

\begin_layout Standard
4.a.
 Group =ter and =ty together, then add =t.
\end_layout

\begin_layout Standard
4.b.
 Group =ter and =t together, then add =ty.
\end_layout

\begin_layout Standard
4.c.
 Group =ty and =t together, then add =ter.
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

This gives 9 different orders in which the reductions can take place.
 Actually, only 6: case 1b and 4a are the same, as are 2b=4b and 3b=4c.
 Lets do at least some of them.
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Paragraph*
Case 1.
 
\end_layout

\begin_layout Standard
Let 
\begin_inset Formula $\gamma=\{\mbox{=ter, =ty}\}$
\end_inset

.
 Then the link types GB and GC need to be consolidated: GG={GB, GC} and
 likewise TT={TB, TC}.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: GA+ or GG+;
\end_layout

\begin_layout Plain Layout

     tes.=: TA+ or TT+;
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty =ter: GG- or TT-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(GA) = p(gif,t) = 1/7 = p(TA) = p(tes,t)
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(GG) = p(gif,ter) + p(gif,ty) = 2/7
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(TT) = p(tes,ter) + p(tes,ty) = 3/7
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So the observed entropy is now
\begin_inset Formula 
\[
h_{PAIR}^{red1.}=-\frac{2}{7}\log_{2}\frac{1}{7}-\frac{2}{7}\log_{2}\frac{2}{7}-\frac{3}{7}\log_{2}\frac{3}{7}=1.842371
\]

\end_inset

The generated entropy is 
\begin_inset Formula $h_{gen}^{red1.}=\log_{2}4=2$
\end_inset

 since there are four total link types in the reduced grammar.
 Pursuant to equation 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:counting complexity"

\end_inset

, we should add the log of the cardinality of the word-sets.
 Here, only one word-set has a cardinality greater than one: {=ty, =ter}.
 So, one gets:
\begin_inset Formula 
\[
h_{gen}^{wrds1.}=\log_{2}2=1
\]

\end_inset

The conditional entropy, based on the textual observations, is 
\begin_inset Formula 
\[
h_{obs}^{wrds1.}=-\frac{3}{5}\log_{2}\frac{3}{5}-\frac{2}{5}\log_{2}\frac{2}{5}=0.970951
\]

\end_inset

 
\end_layout

\begin_layout Paragraph*
Case 1.a.
 
\end_layout

\begin_layout Standard
Let 
\begin_inset Formula $\delta=\{\mbox{=ter, =ty, =t}\}$
\end_inset

.
 Then the link types GA and GG need to be consolidated: G={GA, GG} and likewise
 T={TA, TT}.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: G+;
\end_layout

\begin_layout Plain Layout

     tes.=: T+;
\end_layout

\begin_layout Plain Layout

     =t =ty =ter: G- or T-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(G) = p(gif,t) + p(gif,ty) + p(gif,ter) = 3/7
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(T) = p(tes,t) + p(tes,ty) + p(tes,ter) = 4/7
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So the observed entropy is now
\begin_inset Formula 
\[
h_{PAIR}^{red1.a.}=-\frac{3}{7}\log_{2}\frac{3}{7}-\frac{4}{7}\log_{2}\frac{4}{7}=0.985228
\]

\end_inset

The generated entropy is 
\begin_inset Formula $\log_{2}2=1$
\end_inset

 since there are only two link types in the grammar.
 The word-counting entropy for the set 
\begin_inset Formula $\delta$
\end_inset

 contributes an additional 
\begin_inset Formula 
\[
h_{gen}^{wrds1.a.}=\log_{2}3=1.584963
\]

\end_inset

while the observed entropy is 
\begin_inset Formula 
\[
h_{obs}^{wrds1.a.}=-\frac{4}{7}\log_{2}\frac{2}{7}-\frac{3}{7}\log_{2}\frac{3}{7}=1.556657
\]

\end_inset


\end_layout

\begin_layout Paragraph*
Case 1.b.
 
\end_layout

\begin_layout Standard
Let 
\begin_inset Formula $\gamma=\{\mbox{=ter, =ty}\}$
\end_inset

 as before, and 
\begin_inset Formula $\epsilon=\{\mbox{gif.=, tes.=}\}$
\end_inset

.
 The link types consolidate: EA={GA, TA} and EM={GG, TT}.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: EA+ or EM+;
\end_layout

\begin_layout Plain Layout

     =t: EA-;
\end_layout

\begin_layout Plain Layout

     =ty =ter: EM-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(EA) = p(gif,t) + p(tes,t) = 2/7
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(EM) = 5/7
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So that the entropy is
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
h_{PAIR}^{red1.b.}=-\frac{2}{7}\log_{2}\frac{2}{7}-\frac{5}{7}\log_{2}\frac{5}{7}=0.863121
\]

\end_inset

The generated entropy is 
\begin_inset Formula $\log_{2}2=1$
\end_inset

 since there are only two link types in the grammar.
 The word-set counting probability adds 
\begin_inset Formula 
\[
h_{gen}^{wrds.1.b.}=2\log_{2}2=2
\]

\end_inset

 while the observed probabilities are
\begin_inset Formula 
\[
h_{obs}^{wrds.1.b.}=-\frac{3}{7}\log_{2}\frac{3}{7}-\frac{4}{7}\log_{2}\frac{4}{7}-\frac{3}{5}\log_{2}\frac{3}{5}-\frac{2}{5}\log_{2}\frac{2}{5}=1.956179
\]

\end_inset

 
\end_layout

\begin_layout Paragraph*
Other cases.
\end_layout

\begin_layout Standard
Case 2.a.
 and case 2.b.
 are identical to cases 1.a.
 and 1.b.
 because =t and =ty are interchangeable, from the probability point of view.
\end_layout

\begin_layout Standard
Case 3.a.
 and case 3.b.
 are similar, but with different probabilities.
\end_layout

\begin_layout Standard
Case 4.
 and the subcases are different, but not illuminating.
\end_layout

\begin_layout Paragraph*
Final Case.
\end_layout

\begin_layout Standard
The final consolidation gives 
\begin_inset Formula $\gamma=\{\mbox{=ter, =ty,=t}\}$
\end_inset

, and 
\begin_inset Formula $\epsilon=\{\mbox{gif.=, tes.=}\}$
\end_inset

.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: LL+;
\end_layout

\begin_layout Plain Layout

     =t =ty =ter: LL-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(LL) = 7/7
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So that the entropy is
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
h_{PAIR}^{final}=-\frac{7}{7}\log_{2}\frac{7}{7}=0
\]

\end_inset

The generated entropy is 
\begin_inset Formula $\log_{2}1=0$
\end_inset

 since there is only one link type in the grammar.
 The word-set counting probability adds 
\begin_inset Formula 
\[
h_{gen}^{wrds.fin}=\log_{2}2+\log_{2}3=2.584963
\]

\end_inset

while the observed word count is 
\begin_inset Formula 
\[
h_{obs}^{wrds.fin}=-\frac{4}{7}\log_{2}\frac{2}{7}-\frac{3}{7}\log_{2}\frac{3}{7}-\frac{3}{7}\log_{2}\frac{3}{7}-\frac{4}{7}\log_{2}\frac{4}{7}=2.541885
\]

\end_inset


\end_layout

\begin_layout Paragraph*
Summary.
\end_layout

\begin_layout Standard
The table below summaries these results.
 The sum columns show the entropy according to the equation 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:counting complexity"

\end_inset

 for the observed frequencies, and the generated frequencies.
\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="10" columns="7">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{obs}^{red}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{gen}^{red}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{obs}^{wrds}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{gen}^{wrds}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{obs}^{red}+h_{obs}^{wrds}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{gen}^{red}+h_{gen}^{wrds}$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Initial
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.521641
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.584963
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.521641
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.584963
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case 1.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.842371
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.970951
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.813322
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case 1.a.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.985228
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.556657
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.584963
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.541885
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.584963
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case 1.b.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.863121
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.956179
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.819299
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Final
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.541885
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.584963
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.541885
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.584963
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case 3.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.950212
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.950212
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case 3.a.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.985228
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.556656
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.541884
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case 3.b.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.985228
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.985228
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.970456
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case 4.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.556657
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.985228
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.541885
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

Arghhh.
 Such a simple case, so much complexity...
 anyway, the case 3 and 4 are computed from the script 
\begin_inset Quotes eld
\end_inset

link-type/gifty.scm
\begin_inset Quotes erd
\end_inset

 in this same directory.
\end_layout

\begin_layout Standard
Conclusions: based purely on entropy maximization, all cases advance, but
 none go to the final case.
 But we are not imposing any 'complexity penalty' on this.
\end_layout

\begin_layout Standard
Results on some alternate distributions, for this ranking: 
\begin_inset Quotes eld
\end_inset

tester testy test gifter gifty gift
\begin_inset Quotes erd
\end_inset


\end_layout

\begin_layout Itemize
Pure Zipf: 
\begin_inset Formula $(rank)^{-1.0}$
\end_inset

: none advance (
\begin_inset Formula $h_{initial}=2.281979$
\end_inset

 and 
\begin_inset Formula $h_{final}=2.293598$
\end_inset

)
\end_layout

\begin_layout Itemize
Zipf 
\begin_inset Formula $(rank)^{-1.05}$
\end_inset

 :none advance (
\begin_inset Formula $h_{initial}=2.251204$
\end_inset

 and 
\begin_inset Formula $h_{final}=2.263603$
\end_inset

)
\end_layout

\begin_layout Itemize
Zipf 
\begin_inset Formula $(rank)^{-1.5}$
\end_inset

 :none advance (
\begin_inset Formula $h_{initial}=1.930661$
\end_inset

 and 
\begin_inset Formula $h_{final}=1.948128$
\end_inset

)
\end_layout

\begin_layout Standard
None of these advance because the initial and final entropies are so very
 close.
 But, as before, there are advnces, with the biggest ones to case 4.c and
 3.b.
 The alternative rankings 
\begin_inset Quotes eld
\end_inset

tester testy test gift gifty gifter
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

tester gifter testy gifty test gift
\begin_inset Quotes erd
\end_inset

 give only slightly different results.
\end_layout

\begin_layout Subsection*
Link-type discovery, better example
\end_layout

\begin_layout Standard
In the previous, the unified link-type discovery is inevitable, so a more
 complex version is needed, with a less-obvious outcome.
 So lets take the original example of link-type discovery, and add some
 confounding link types.
 We begin with the initial observations, plus some extras, given in the table
 below:
\begin_inset Newline newline
\end_inset


\begin_inset Float table
placement H
wide false
sideways false
status open

\begin_layout Plain Layout
\begin_inset Caption Standard

\begin_layout Plain Layout
Example Link Frequency Table
\begin_inset CommandInset label
LatexCommand label
name "tab:Example-Link-Frequency"

\end_inset


\end_layout

\end_inset


\begin_inset Tabular
<lyxtabular version="3" rows="9" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Pair
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Initial Link Type
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
# observations
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif–t
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
GA
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif–ty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
GB
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
gif–ter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
GC
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes–t
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
TA
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes–ty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
TB
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tes–ter
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
TC
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
blo–ty
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
BB
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
blo-fu
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
BF
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Plain Layout
Example distribution of link frequencies obtained from an example corpus.
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Standard
The addition of 
\begin_inset Quotes eld
\end_inset

bloty
\begin_inset Quotes erd
\end_inset

 to the link table, and with a strong weight, will tend to derail the consolidat
ion of the =ty suffix with the others.
 The addition of 
\begin_inset Quotes eld
\end_inset

blofu
\begin_inset Quotes erd
\end_inset

 helps make sure that there's some confusion about the 
\begin_inset Quotes eld
\end_inset

blo=
\begin_inset Quotes erd
\end_inset

 stem.
\end_layout

\begin_layout Standard
The corresponding link-grammar dictionary is: 
\begin_inset CommandInset label
LatexCommand label
name "gifty-grammar"

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Float algorithm
wide false
sideways false
status open

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout
\begin_inset Caption Standard

\begin_layout Plain Layout
Example morpheme grammar
\begin_inset CommandInset label
LatexCommand label
name "alg:Example-morpheme-grammar"

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: GA+ or GB+ or GC+;
\end_layout

\begin_layout Plain Layout

     tes.=: TA+ or TB+ or TC+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty: GB- or TB- or BB-;
\end_layout

\begin_layout Plain Layout

     =ter: GC- or TC-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
A more complex grammar showing morpheme linkages.
\end_layout

\end_inset


\end_layout

\begin_layout Standard
From the above initial dictionary, we hope to deduce one word class that
 contains gif.= and tes.= and another that contains =t and =ter; exactly how
 the rest plays out is unclear.
 Lets begin by starting with the un-clustered entropy, and then see what
 happens if we try various different clusters.
 So, as before, let 
\begin_inset Formula $\beta=\{GA,GB,GC,TA,TB,TC,BB,BF\}$
\end_inset

 and write:
\begin_inset Formula 
\begin{align*}
h_{PAIR}^{observed} & =-\sum_{t\in\beta}p(t)\log_{2}p(t)\\
 & =-\frac{6}{11}\log_{2}\frac{1}{11}-\frac{2}{11}\log_{2}\frac{2}{11}-\frac{3}{11}\log_{2}\frac{3}{11}\\
 & =2.845351
\end{align*}

\end_inset

with 
\begin_inset Formula $p(t)$
\end_inset

 being the probability of observing link-type 
\begin_inset Formula $t$
\end_inset

.
 Since there are 8 different link types, the generated entropy is 
\begin_inset Formula $h_{PAIR}^{generated}=\log_{2}8=3$
\end_inset

.
 The different between these two is 
\begin_inset Formula $h^{gen}-h^{obs}=0.154649$
\end_inset

.
 The observed corpus also has 8 words in it (not counting multiplicity):
 this is by design; before reduction, there is always exactly one link type
 for each morpheme pair.
\end_layout

\begin_layout Standard
Lets look at several cases:
\end_layout

\begin_layout Enumerate
Group gif.= and tes.= together.
\end_layout

\begin_layout Enumerate
Group gif.= and blo.= together.
\end_layout

\begin_layout Enumerate
Group =t and =ty together.
\end_layout

\begin_layout Enumerate
Group =ty and =fu together.
\end_layout

\begin_layout Enumerate
Group =ter and =fu together.
\end_layout

\begin_layout Standard
Here, we expect case 1 to go easily, cases 2 and 3 to to be ambiguous or
 blocked, case 4 to be weakly blocked, and case 5 to be strongly blocked.
 So, proceeding:
\end_layout

\begin_layout Paragraph*
Case 1.
\end_layout

\begin_layout Standard
Group gif.= and tes.= together.
 Let 
\begin_inset Formula $\gamma=\{\mbox{gif.=, tes.=}\}$
\end_inset

.
 Then the link types G* and T* need to be consolidated: A={GA,TA} and likewise
 B={GB,TB} and C={GC,TC}.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: A+ or B+ or C+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t: A-;
\end_layout

\begin_layout Plain Layout

     =ty: B- or BB-;
\end_layout

\begin_layout Plain Layout

     =ter: C-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(A) = p(gif,t) + p(tes,t) = 2/11 = p(B) = p(gif,ty) + p(tes,ty) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(C) = p(gif,ter) + p(tes,ter) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BB) = p(blo,ty) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blo,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So the observed entropy is now
\begin_inset Formula 
\[
h_{PAIR}^{red1.}=-\frac{4}{11}\log_{2}\frac{2}{11}-\frac{6}{11}\log_{2}\frac{3}{11}-\frac{1}{11}\log_{2}\frac{1}{11}=2.231270
\]

\end_inset

The generated entropy is 
\begin_inset Formula $h^{gen}=\log_{2}5=2.321928$
\end_inset

.
 The difference is 
\begin_inset Formula $h^{gen}-h^{obs}=0.090658.$
\end_inset

 This clearly brings the entropy closer to the theoretical (equidistributional)
 maximum; the grouping goes.
 However, 
\begin_inset Formula $h^{lang}=\log_{2}8=3$
\end_inset

 as before, since the generated language still has 8 words in it.
\end_layout

\begin_layout Paragraph*
Case 2.
 
\end_layout

\begin_layout Standard
Group gif.= and blo.= together.
 Let 
\begin_inset Formula $\delta=\{\mbox{gif.=, blo.=}\}$
\end_inset

.
 Then the link types GB and BB can be consolidated, because they share the
 common suffix =ty: B={GB,BB}.
 No other link consolidation is possible, without permitting impermissible
 (previously unseen) linkages.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= blo.=: GA+ or B+ or GC+ or BF+;
\end_layout

\begin_layout Plain Layout

     tes.=: TA+ or TB+ or TC+;
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty: B- or TB-;
\end_layout

\begin_layout Plain Layout

     =ter: GC- or TC-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

Note that this dictionary does allow several previously unobserved words:
 giffu, blot, bloter.
 This is what happens when one hypothesizes unions between classes that
 merely overlap, instead of being subsets.
 What happens next depends on whether the overlap was large, or small.
\end_layout

\begin_layout Standard
The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(GA) = p(gif,t) = 1/11 = p(GC) = p(TA) = p(TB) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(TC) = p(tes,ter) = 2/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(B) = p(gif,ty) + p(blo,ty) = 4/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blo,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So the observed entropy is now
\begin_inset Formula 
\[
h_{PAIR}^{red2.}=-\frac{5}{11}\log_{2}\frac{1}{11}-\frac{2}{11}\log_{2}\frac{2}{11}-\frac{4}{11}\log_{2}\frac{4}{11}=2.550341
\]

\end_inset

The generated entropy is 
\begin_inset Formula $h^{gen}=\log_{2}7=2.807355$
\end_inset

.
 The difference is 
\begin_inset Formula $h^{gen}-h^{obs}=0.257014$
\end_inset

.
 The entropy is not getting closer to the equidistributional maximum; this
 grammar is rejected.
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Paragraph*
Case 3.
 
\end_layout

\begin_layout Standard
Group =t and =ty together.
 Let 
\begin_inset Formula $\epsilon=\{\mbox{=t, =ty}\}$
\end_inset

 Then we may group G={GA, GB} and T={TA, TB}.
 The corresponding link-grammar dictionary is:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: G+ or GC+;
\end_layout

\begin_layout Plain Layout

     tes.=: T+ or TC+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t =ty: G- or T- or BB-;
\end_layout

\begin_layout Plain Layout

     =ter: GC- or TC-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The above again allows a new, unobserved word: 
\begin_inset Quotes eld
\end_inset

blot
\begin_inset Quotes erd
\end_inset

.
 The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(G) = p(gif,t) + p(gif,ty) = 2/11 = p(T) = p(tes,t) + p(tes,ty) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(GC) = p(gif,ter) = 1/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(TC) = p(tes,ter) = 2/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BB) = p(blo,ty) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blo,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So the observed entropy is now
\begin_inset Formula 
\[
h_{PAIR}^{red3.}=-\frac{6}{11}\log_{2}\frac{2}{11}-\frac{2}{11}\log_{2}\frac{1}{11}-\frac{3}{11}\log_{2}\frac{3}{11}=2.481715
\]

\end_inset


\begin_inset Newline newline
\end_inset

The equidistributional entropy is 
\begin_inset Formula $h^{gen}=\log_{2}6=2.584963$
\end_inset

.
 The difference is 
\begin_inset Formula $h^{gen}-h^{obs}=0.103248$
\end_inset

.
 This difference means we are getting closer to the maximum; the grouping
 is acceptable! Its really not much worse than case 1, which was unambiguous.
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Paragraph*
Case 4.
 
\end_layout

\begin_layout Standard
Group =ty and =fu together.
 Let 
\begin_inset Formula $\zeta=\{\mbox{=ty, =fu}\}$
\end_inset

.
 Then we must group B={BB,BF} together.
 The dictionary is:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: GA+ or GB+ or GC+;
\end_layout

\begin_layout Plain Layout

     tes.=: TA+ or TB+ or TC+;
\end_layout

\begin_layout Plain Layout

     blo.=: B+;
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty =fu: GB- or TB- or B-;
\end_layout

\begin_layout Plain Layout

     =ter: GC- or TC-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

No new unobserved words are allowed by this grouping! The observed pair
 probabilities are:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(GA) = p(gif,t) = 1/11 = p(GB) = p(GC) = p(TA) = p(TB) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(TC) = p(tes,ter) = 2/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(B) = p(blo,ty) + p(blo,fu) = 4/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The observed entropy is then:
\begin_inset Formula 
\[
h_{PAIR}^{red4.}=-\frac{5}{11}\log_{2}\frac{1}{11}-\frac{2}{11}\log_{2}\frac{2}{11}-\frac{4}{11}\log_{2}\frac{4}{11}=2.550341
\]

\end_inset

Curiously, this entropy is identical to the completely different case 2.
 The equidistributional entropy is 
\begin_inset Formula $h^{gen}=\log_{2}7=2.807355$
\end_inset

 and the difference is thus 
\begin_inset Formula $h^{gen}-h^{obs}=0.257014$
\end_inset

 which is sharply further away from the equidistributional maximum.
 Thus, this grouping is rejected.
 This is perhaps surprising ...
 First, this grammar did not generate any new unobserved words; thus, it
 is a faithful grammar.
 Also, it succeeds in reducing the total number of link-types, and thus is
 naively acceptable for that reason.
 However, the frequency distribution of th generated grammar move away from
 the observed frequency distribution, leading to the rejection.
 This begs a question: when and how might we annotate the grammar with frequency
 information?
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Paragraph*
Case 5.
 
\end_layout

\begin_layout Standard
Group =ter and =fu together, so that 
\begin_inset Formula $\eta=\{\mbox{=ter, =fu}\}$
\end_inset

.
 It does not appear that any link types get consolidated! This is not much
 of a grouping, then ...
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: GA+ or GB+ or GC+;
\end_layout

\begin_layout Plain Layout

     tes.=: TA+ or TB+ or TC+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty: GB- or TB- or BB-;
\end_layout

\begin_layout Plain Layout

     =ter =fu: GC- or TC- or BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

Many new, unobserved words are allowed: bloter, giffu, tesfu.
 The observed pair probabilities are:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(GA) = p(gif,t) = 1/11 = p(GB) = p(GC) = p(TA) = p(TB) 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(TC) = p(tes,ter) = 2/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BB) = p(blo,ty) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blu,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The observed entropy is then:
\begin_inset Formula 
\[
h_{PAIR}^{red5.}=-\frac{6}{11}\log_{2}\frac{1}{11}-\frac{2}{11}\log_{2}\frac{2}{11}-\frac{3}{11}\log_{2}\frac{3}{11}=2.845351
\]

\end_inset

This is identical to the unreduced entropy: no surprise, because no link
 consolidation was performed.
 The equidistributional entropy is the same as well: 
\begin_inset Formula $h^{gen}=\log_{2}8=3$
\end_inset

 since there are still 8 link types.
 The language entropy increased: there are now 11 possible words in the
 language, so 
\begin_inset Formula $h^{lang}=\log_{2}11=3.459432$
\end_inset

.
 This is a very unsatisfying situation: the difference in entropies is no
 better or worse than the starting point, and so this seems like a reasonable
 sideways slide, and yet, this grammar allows a bunch of nonsense words
 to be generated.
 That seems wrong.
\end_layout

\begin_layout Paragraph*
Summary
\end_layout

\begin_layout Standard
Of the 5 cases, three are blocked (cases 2,4,5), and two are acceptable
 (1,3).
 Case 1 looks to be the best.
 Lets see what might happen next:
\end_layout

\begin_layout Itemize
Case 1a.
 Group =t and =ty
\end_layout

\begin_layout Itemize
Case 1b.
 Group =t and =ter
\end_layout

\begin_layout Itemize
Case 1c.
 Group =ty and =ter
\end_layout

\begin_layout Standard
Case 1a resembles Case 3 so we expect it to advance.
 Likewise for case 1c.
 Its reasonable to guess that case 1b will be the strongest.
 Lets try some of these.
\end_layout

\begin_layout Paragraph*
Case 1a.
\end_layout

\begin_layout Standard
Group =t and =ty.
 The link merges are T={A,B}; the resulting grammar is:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: T+ or C+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t =ty: T- or BB-;
\end_layout

\begin_layout Plain Layout

     =ter: C-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

This grammar allows a new unobserved word: 
\begin_inset Quotes eld
\end_inset

blot
\begin_inset Quotes erd
\end_inset

.
 The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(T) = p(gif,t) + p(tes,t) + p(gif,ty) + p(tes,ty) = 4/11 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(C) = p(gif,ter) + p(tes,ter) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BB) = p(blo,ty) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blo,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So the observed entropy is now
\begin_inset Formula 
\[
h_{PAIR}^{red1.}=-\frac{4}{11}\log_{2}\frac{4}{11}-\frac{6}{11}\log_{2}\frac{3}{11}-\frac{1}{11}\log_{2}\frac{1}{11}=1.867634
\]

\end_inset

The generated entropy is 
\begin_inset Formula $h^{gen}=\log_{2}4=2$
\end_inset

.
 The difference is 
\begin_inset Formula $h^{gen}-h^{obs}=0.132366$
\end_inset

 which is not closer than the previous delta of 
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.090658, so this is rejected.
\end_layout

\begin_layout Paragraph*
Casse 1b.
\end_layout

\begin_layout Standard
Group =t and =ter.
 This consolidates links T={A,C} and so the dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: T+ or B+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t =ter: T-;
\end_layout

\begin_layout Plain Layout

     =ty: B- or BB-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

This does not generate any new words.
 The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(A) = p(gif,t) + p(tes,t) + p(gif,ter) + p(tes,ter) = 5/11 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(B) = p(gif,ty) + p(tes,ty) = 2/11 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BB) = p(blo,ty) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blo,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

So the observed entropy is now
\begin_inset Formula 
\[
h_{PAIR}^{red1.}=-\frac{5}{11}\log_{2}\frac{5}{11}-\frac{2}{11}\log_{2}\frac{2}{11}-\frac{3}{11}\log_{2}\frac{3}{11}-\frac{1}{11}\log_{2}\frac{1}{11}=1.789929
\]

\end_inset

The generated entropy is 
\begin_inset Formula $h^{gen}=\log_{2}4=2$
\end_inset

.
 The difference is 
\begin_inset Formula $h^{gen}-h^{obs}=0.210071$
\end_inset


\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
 which does not get closer; the best still stands at 0.090658.
 This is surprising: it sems to be blocking the discovery of the clique.
\end_layout

\begin_layout Paragraph*
Case 1c.
\end_layout

\begin_layout Standard
Group =ty and =ter.
 This consolidates T={B,C}, so the dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: A+ or T+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t: A-;
\end_layout

\begin_layout Plain Layout

     =ty =ter: T- or BB-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

This does not generate any new words.
 The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(A) = p(gif,t) + p(tes,t) = 2/11 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(C) = p(gif,ty) + p(tes,ty)+ p(gif,ter) + p(tes,ter) = 5/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BB) = p(blo,ty) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blo,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The changes are the same as for case 1b.
 Again, this is blocked.
\end_layout

\begin_layout Paragraph*
Case 1f.
\end_layout

\begin_layout Standard
This is the 
\begin_inset Quotes eld
\end_inset

final
\begin_inset Quotes erd
\end_inset

 case: group together =t =ty =ter into one.
 This consolidates T={A,B,C} so that 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: T+;
\end_layout

\begin_layout Plain Layout

     blo.=: BB+ or BF+;
\end_layout

\begin_layout Plain Layout

     =t =ty =ter: T- or BB-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

This allows new words 
\begin_inset Quotes eld
\end_inset

blot
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

bloter
\begin_inset Quotes erd
\end_inset

.
 The observed pair probabilities become: 
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(T) = p(gif,t) + p(tes,t) + p(gif,ty) + p(tes,ty)+ p(gif,ter) + p(tes,ter)
 = 7/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BB) = p(blo,ty) = 3/11
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
p(BF) = p(blo,fu) = 1/11
\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The observed entropy is 
\begin_inset Formula 
\[
h_{PAIR}^{red1.}=-\frac{7}{11}\log_{2}\frac{7}{11}-\frac{3}{11}\log_{2}\frac{3}{11}-\frac{1}{11}\log_{2}\frac{1}{11}=1.240671
\]

\end_inset

Hmm.
 3 link types
\end_layout

\begin_layout Paragraph*
Summary
\end_layout

\begin_layout Standard
Movement to Cases 1a, 1b and 1c are all blocked.
 This seems surprising.
 The relatively high-frequency observation of =ter makes the distribution
 of the consolidated grammar to deviate strongly from the distribution of
 the observed corpus.
 This seems like an undesired effect, as the point of learning how to simplify
 the grammar is to obtain a smaller grammar, rather than to preserve the
 the distribution of the corpus.
 Mostly.
\end_layout

\begin_layout Standard
Intuition suggests that the grammar for 
\begin_inset Quotes eld
\end_inset

common
\begin_inset Quotes erd
\end_inset

 cases should be consolidated.
 The grammar for quite rare cases should indeed be handled distinctly.
 To avoid this seemingly perverse outcome, perhaps the grammar should contain
 frequency information, which is to be consolidated appropriately.
 This is truly tedious, but seems to be necessary.
 So we have to start from scratch.
\end_layout

\begin_layout Standard
And we do, below, and its a total failure, as now, the corpus frequencies
 are recorded faithfully, so the consolidation process doesn't tell us anything
 we didn't know.
 Its the same calculation done differently.
\end_layout

\begin_layout Subsection*
Worked Link Consolidation Example, with Frequencies (XXX Fail)
\end_layout

\begin_layout Standard
(XXX The below fails, don't bother reading it).
 So we start all over again, using the same corpus frequencies as before,
 namely, those of table 
\begin_inset CommandInset ref
LatexCommand ref
reference "tab:Example-Link-Frequency"

\end_inset

.
 The grammar is essentially identical to that of 
\begin_inset CommandInset ref
LatexCommand ref
reference "alg:Example-morpheme-grammar"

\end_inset

, except that it is now annotated with probabilities.
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.=: (GA+)(1/11) or (GB+)(1/11) or (GC+)(1/11);
\end_layout

\begin_layout Plain Layout

     tes.=: (TA+)(1/11) or (TB+)(1/11) or (TC+)(2/11);
\end_layout

\begin_layout Plain Layout

     blo.=: (BB+)(3/11) or (BF+)(1/11);
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty: GB- or TB- or BB-;
\end_layout

\begin_layout Plain Layout

     =ter: GC- or TC-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The above only annotates the +-going links; it seems like annotating the
 –going links would cause double-counting.
 This is somewhat confusing, since the probability has nothing to do with
 directionality.
 A better notation is not obvious.
 Lets go through the cases as before.
\end_layout

\begin_layout Paragraph*
Case 1.
\end_layout

\begin_layout Standard
Group gif.= and tes.= together.
 Let 
\begin_inset Formula $\gamma=\{\mbox{gif.=, tes.=}\}$
\end_inset

.
 Then the link types G* and T* need to be consolidated: A={GA,TA} and likewise
 B={GB,TB} and C={GC,TC}.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= tes.=: (A+)(2/11) or (B+)(2/11) or (C+)(3/11);
\end_layout

\begin_layout Plain Layout

     blo.=: (BB+)(3/11) or (BF+)(1/11);
\end_layout

\begin_layout Plain Layout

     =t: A-;
\end_layout

\begin_layout Plain Layout

     =ty: B- or BB-;
\end_layout

\begin_layout Plain Layout

     =ter: C-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The observational probabilities are unchanged, as the dictionary probabilities
 have no bearing on the parsing.
 However, the entropy of the generated language is different, as it is no
 longer 
\begin_inset Formula $\log_{2}5$
\end_inset

 but instead 
\begin_inset Formula 
\[
h^{gen}=-\frac{6}{11}\log_{2}\frac{1}{11}-\frac{2}{11}\log_{2}\frac{2}{11}-\frac{3}{11}\log_{2}\frac{3}{11}
\]

\end_inset

That is, it is now identical to 
\begin_inset Formula $h^{observed}$
\end_inset

.
 No surprise, as we made it like that, by encoding the frequency information
 in the dictionary.
\end_layout

\begin_layout Paragraph*
Case 2.
 
\end_layout

\begin_layout Standard
Group gif.= and blo.= together.
 Let 
\begin_inset Formula $\delta=\{\mbox{gif.=, blo.=}\}$
\end_inset

.
 Then the link types GB and BB can be consolidated, because they share the
 common suffix =ty: B={GB,BB}.
 No other link consolidation is possible, without permitting impermissible
 (previously unseen) linkages.
 The dictionary becomes
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

     gif.= blo.=: (GA+)(1/11) or (B+)(4/11) or (GC+)(1/11) or (BF+)(1/11);
\end_layout

\begin_layout Plain Layout

     tes.=: (TA+)(1/11) or (TB+)(1/11) or (TC+)(2/11);
\end_layout

\begin_layout Plain Layout

     =t: GA- or TA-;
\end_layout

\begin_layout Plain Layout

     =ty: B- or TB-;
\end_layout

\begin_layout Plain Layout

     =ter: GC- or TC-;
\end_layout

\begin_layout Plain Layout

     =fu: BF-;
\end_layout

\begin_layout Plain Layout

\end_layout

\end_inset

The generated entropy is
\begin_inset Formula 
\[
h^{gen}=-\frac{5}{11}\log_{2}\frac{1}{11}-\frac{4}{11}\log_{2}\frac{4}{11}-\frac{2}{11}\log_{2}\frac{2}{11}
\]

\end_inset

which is identical to the corpus entropy, again.
 Not surprising, I guess ...
 we seem to be doing the same calculation, but in a different way.
 Dohh.
 Never mind ...
 
\end_layout

\begin_layout Subsection*
Alternate Distributions
\end_layout

\begin_layout Standard
Instead of looking for an equi-distribution, how about a Zipf distribution,
 which seems far more plausible? The distribution is 
\begin_inset Formula 
\[
p(k,n)=\frac{1}{kH_{n}}
\]

\end_inset

 where the normalization is 
\begin_inset Formula $H_{n}=\sum_{k=1}^{n}1/n$
\end_inset

.
 The entropy is then 
\begin_inset Formula 
\begin{align*}
h_{n}^{Zipf} & =-\sum_{k=1}^{n}p(k,n)\log_{2}p(k,n)\\
 & =\frac{1}{H_{n}}\sum_{k=1}^{n}\frac{\log_{2}kH_{n}}{k}\\
 & =\log_{2}H_{n}+\frac{1}{H_{n}}\sum_{k=1}^{n}\frac{\log_{2}k}{k}
\end{align*}

\end_inset

and the first few values are shown below.
 For comparison, 
\begin_inset Formula $h_{n}^{equi}=\log_{2}n$
\end_inset

 is also shown.
\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="8" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
n
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $H_{n}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{n}^{Zipf}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h_{n}^{equi}$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.918296
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.83333
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.435371
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.584963
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.033333
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.792488
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.283333
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.063860
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.321928
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.45
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.281979
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.584963
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.592857
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.463914
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.807355
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.717857
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.619715
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

The question is then: how would the above cases go if this was used as the
 deciding factor? This is shown below:
\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="10">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
# lnk
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{observed}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{equi}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{eq}-h^{obs}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
OK
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{Zipf}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{Zipf}-h^{obs}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
C1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
C2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Base
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.845351
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.154649
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.619715
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.225636
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.231270
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.321928
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.090658
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.063860
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.16741
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.550341
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.807355
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.257014
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.463914
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.086427
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.481715
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.584963
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.103248
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.281979
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.199736
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.550341
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.807355
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.257014
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.463914
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.086427
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.845351
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.154649
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.619715
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.225636
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1a.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.867634
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.132366
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.792488
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.075146
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1b.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.789929
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.210071
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.792488
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.002559
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1c.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.789929
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.210071
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.792488
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.002559
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1f.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.240671
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.584963
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.344292
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.435371
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.1947
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

There seem to be two different decision criteria to apply: 
\end_layout

\begin_layout Enumerate
Does the reduced entropy come closer to the Zipfian entropy?
\end_layout

\begin_layout Enumerate
Does the reduced entropy increase, relative to the Zipfian entropy?
\end_layout

\begin_layout Standard
The first is shown in column C1, the second in C2.
 Naively, C2 seems like a better chooser.
 Does it also work for the simple case (with the original 7-word corpus)?
 Lets see:
\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="5" columns="10">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Case
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
#lnk
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{observed}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{equi}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{eq}-h^{obs}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
OK
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{Zipf}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $h^{Zipf}-h^{obs}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
C1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
C2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Base
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.521641
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
2.584963
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.063322
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.281979
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.239662
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
1.842371
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.157629
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.792488
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.049883
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1a
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.985228
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.014772
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.918296
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.066932
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Y
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1b.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
0.863121
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.136879
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.918296
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.055175
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
N
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset Newline newline
\end_inset

Basically, this is really irritating.
\end_layout

\begin_layout Subsection*
Thoughts
\end_layout

\begin_layout Standard
What have we learned from the above?
\end_layout

\begin_layout Itemize
The problem of condensing together morphemes into classes which share common
 link types is the bipartite clique problem.
 It is a known-hard problem.
\end_layout

\begin_layout Itemize
Bad grammars increase the size of the language.
 This could be acceptable, if the increase is small.
 What's the criteria? Unclear.
\end_layout

\begin_layout Section*
Consciousness - 27 July 2014
\end_layout

\begin_layout Standard
Two works:
\end_layout

\begin_layout Itemize
Masafumi Oizumi, Larissa Albantakis, Giulio Tononi, 
\begin_inset Quotes eld
\end_inset

From the Phenomenology to the Mechanisms of Consciousness: Integrated Informatio
n Theory 3.0
\begin_inset Quotes erd
\end_inset

 (2014) PLOS Computational Biology, http://www.ploscompbiol.org/article/info%3Adoi
%2F10.1371%2Fjournal.pcbi.1003588
\end_layout

\begin_layout Itemize
Max Tegmark, Consciousness as a State of Matter (27 Feb 2014) arXiv:1401.1219v2
 [quant-ph]
\end_layout

\begin_layout Standard
Curious points and thoughts:
\end_layout

\begin_layout Itemize
CEI – 
\begin_inset Quotes eld
\end_inset

Cause-effect information
\begin_inset Quotes erd
\end_inset

 – Tononi – sound like a time-ordered variant of mutual information.
 How should this be defined? Answer: my guess is that its just like the
 mutual information defined in eqn
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:Relation MI"

\end_inset

, right? Because the relational complexity can deal with arbitrary structures,
 so that seems appropriate.
 
\end_layout

\begin_layout Section*
13 Sept 2014
\end_layout

\begin_layout Standard
The Zipfian distribution is typical of a scale-free network.
 So why is language scale-free? Crudely, because we attempt to recycle existing
 concepts/words.
\end_layout

\begin_layout Standard
Next, from this:
\end_layout

\begin_layout Itemize
Christoph Adami 
\begin_inset Quotes eld
\end_inset

Information-Theoretic Considerations Concerning the Origin of Life
\begin_inset Quotes erd
\end_inset

 http://arxiv.org/pdf/1409.0590v1.pdf
\end_layout

\begin_layout Standard
Come the following thoughts:
\end_layout

\begin_layout Itemize
Never assume a uniform distribution of parts.
 This makes it very unlikely that an imprtant assemblage of parts can arise
 at random.
 For Adami, this is used to argue that biotic and abiotic strings should
 have very siilar distributions (or rather, the converse: a non-uniform
 abotic distribution makes it much more likely to find a replictor with
 a similar distribution.) 
\end_layout

\begin_layout Itemize
The information content of (grammatical sentences of length L is 
\begin_inset Formula 
\[
I_{grammatical}=-\log_{2}(N_{grammatical}/N_{total})
\]

\end_inset

where 
\begin_inset Formula $N_{total}$
\end_inset

 is the total number of sentences of length L, assuming a 
\emph on
uniform
\emph default
 distribution of words picked from a vocabulary of D words.
 That is, 
\begin_inset Formula $N_{total}=D^{L}$
\end_inset

.
 But this is weird ...
 because the vocabulary isn't really a constant, and the natural distribution
 is not uniform, so its not clear what kind of 
\begin_inset Quotes eld
\end_inset

information
\begin_inset Quotes erd
\end_inset

 the above actually is ...
 
\end_layout

\begin_layout Section*
Thermodynamics - 24 March 2015
\end_layout

\begin_layout Standard
Some quick short notes: blog post: 
\begin_inset Quotes eld
\end_inset

Thermodynamics with Continuous Information Flow
\begin_inset Quotes erd
\end_inset

 https://johncarlosbaez.wordpress.com/2015/03/21/19395 with arxiv paper: http://ar
xiv.org/pdf/1402.3276v3.pdf Jordan M.
 Horowitz and Massimiliano Esposito study the master equation for a probability
 p(x,y) over two distributions X,Y, which are connected via a bipartite
 graph.
 The total system is also connected to a thermal bath.
 The eqn is
\begin_inset Formula 
\[
\frac{dp(x,y)}{dt}=\sum_{x',y'}H_{x,x'}^{y,y'}p(x',y')-H_{x',x}^{y',y}p(x,y)
\]

\end_inset

i.e.
 its Markovian; we've written two indexes, which makes it clearer when H
 is bipartite i.e.
\begin_inset Formula 
\[
H_{x,x'}^{y,y'}=\begin{cases}
H_{x,x'}^{y} & \mbox{\ if\ }y=y'\\
H_{x}^{y,y'} & \mbox{\ if\ }x=x'\\
0 & \mbox{\ otherwise}
\end{cases}
\]

\end_inset

The interesting part is the entropy, and the thermal bath, which is not
 in the master eqn(!) The total entropy is 
\begin_inset Formula $S_{tot}=S_{XY}+S_{env}$
\end_inset

.
 Per usual, the information entropy is 
\begin_inset Formula $S_{XY}=-\sum_{x,y}p(x,y)\log p(x,y)$
\end_inset

.
 Two tricks now happen: (1) taking the timer derivative of 
\begin_inset Formula $S_{XY}$
\end_inset

 results in something that naturally splits into an X piece and a Y piece.
 Trick (2) is that 
\begin_inset Formula $S_{env}$
\end_inset

 cannot be written down directly, but its time derivative can be; it is
 proportional to the heat current: 
\begin_inset Formula $\dot{S}_{env}=-\dot{Q}/T$
\end_inset

 Observer the tiny dot over S,Q these are the usuaul rate-of-change dots,
 (i.e.
 just rates, not functions we are taking time derivative of).
 Q is heat, Q-dot is heat flow, T is temp.
 Local detailed balance requires that 
\begin_inset Formula 
\[
\log\frac{H_{x,x'}^{y,y'}}{H_{x',x}^{y',y}}=\frac{-(E_{x,y}-E_{x',y'})}{kT}
\]

\end_inset

is the change in energy due to a state transition: the change in energy
 is supplied by the heat reservior.
 Where does this mystery equation come from? Answer:
\end_layout

\begin_layout Standard
Detailed balance requires that, when the system reaches equilibirum, that
 the transition rate into and out of the equlibrium state 
\begin_inset Formula $p_{i}=\pi_{i}$
\end_inset

 are equal:
\begin_inset Formula 
\[
H_{ji}\pi_{i}=H_{ij}\pi{}_{j}
\]

\end_inset

(there is NO repeated-index summation).
 Then, just write 
\begin_inset Formula $\pi_{i}=\exp-E_{i}/kT$
\end_inset

, and turn the crank.
 The general principle: 
\emph on
the log of the ratio of the forward and backward transition rates between
 two states must be proportional to the energy difference between those
 states! 
\end_layout

\begin_layout Standard
BTW the detailed-balance equation resembles Bayes Theorem, in that, if we
 wrote 
\begin_inset Formula $H_{ji}\rightarrow P(j|i)$
\end_inset

 and 
\begin_inset Formula $\pi_{i}\rightarrow P(i)$
\end_inset

, then detailed balance is written as 
\begin_inset Formula $P(j|i)P(i)=P(i|j)P(j)$
\end_inset

.
 So the master equation describes 
\begin_inset Quotes eld
\end_inset

non-equilibrium Bayes statistics
\begin_inset Quotes erd
\end_inset

, in a strange sense.
 Hmm.
 But, of course, this is just a Markov chanin/process.
\end_layout

\begin_layout Section*
26 March 2015
\end_layout

\begin_layout Standard
The Inverse Relationship Principle of Channel theory: 
\begin_inset Quotes eld
\end_inset


\emph on
Whenever there is an increase in available information there is a corresponding
 decrease in possibilities, and vice versa.
\emph default

\begin_inset Quotes eld
\end_inset

 Barwise, 
\begin_inset Quotes eld
\end_inset

Information and Impossibilities.
\begin_inset Quotes erd
\end_inset

 Notre Dame J.
 Formal Logic Volume 38, Number 4, 488-515.
 Barwise, Jon and Jerry Seligman 1997.
 
\begin_inset Quotes eld
\end_inset

Information Flow: The Logic of Distributed Systems
\begin_inset Quotes erd
\end_inset

.
 Cambridge: Cambridge University Press
\end_layout

\begin_layout Section*
Linear networks - 3 May 2015
\end_layout

\begin_layout Standard
Another Baez post: 
\begin_inset Quotes eld
\end_inset


\emph on
A Compositional Framework for Passive Linear Networks
\begin_inset Quotes erd
\end_inset

 
\emph default
blog: https://johncarlosbaez.wordpress.com/2015/04/28/a-compositional-framework-fo
r-passive-linear-networks/
\end_layout

\begin_layout Standard
So first, we have the table:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="5">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top" width="15col%">
<column alignment="center" valignment="top" width="15col%">
<column alignment="center" valignment="top" width="25col%">
<column alignment="center" valignment="top" width="25col%">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
mechanics
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
electronics
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
information geometry
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
geometric mechanics
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
q
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
position
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
charge
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
point on manifold
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\dot{q}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
velocity
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
current
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
entropy change
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tangent vector
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
p
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
momentum
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
flux linkage
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
temperature momentum
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
covector (vector in cotangent bundle)
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\dot{p}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
force
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
voltage
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
temperature
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
map from tangent bundle to cotangent bundle
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
principle of least action
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
principle of least power dissipation
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
?
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
principle of least action
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
This table is slightly oversimplified; the first four columns show only
 the linear case.
 The fifth column makes clear that force isn't really p-dot; that only holds
 when the manifold is flat.
 Anyway..
\end_layout

\begin_layout Standard
Key concepts: (*) monoidal categories are needed, and (*) symplectic geometry
 is needed.
\end_layout

\begin_layout Standard
Baez does the linear passive-component electronics example, viz a network
 of passive resistors, capacitors, inductors.
 For the resistor network, voltages at each node are taken from the field
 
\begin_inset Formula $\mathbb{F}=\mathbb{R}$
\end_inset

 while for the inductive network, the field is the field of rational functions
 of one variable 
\begin_inset Formula $\mathbb{F}=\mathbb{R}(t)$
\end_inset

 with 
\begin_inset Formula $t$
\end_inset

 time: i.e.
 voltage varying over time.
 A Dirichlet form is a quadratic form 
\begin_inset Formula 
\[
P(\phi)=\frac{1}{2}\sum_{i,j}\frac{(\phi_{i}-\phi_{j})^{2}}{r_{ij}}
\]

\end_inset

where 
\begin_inset Formula $r_{ij}$
\end_inset

 is the reistance (impedance) between nodes i and j, and 
\begin_inset Formula $\phi_{i}=\phi(i)$
\end_inset

 is the voltage at node i.
 (Actually, we should be summing over edges, so as to handle parallel resistors).
 Note that the space of Dirichlet forms is smaller than the space of quadratic
 forms: Dirichlet forms do not have diagonal entries.
 Note that P is (half) the power dissipation.
\end_layout

\begin_layout Standard
The principle of least power dissipation is this: Given fixed voltages 
\begin_inset Formula $\psi$
\end_inset

 on the boundary of the network, i.e.
 on the input/output terminals, the actual power dissipated is
\begin_inset Formula 
\[
Q(\psi)=\min_{\phi\in\mathbb{R}^{N},\phi|_{\partial N}=\psi}\;P(\phi)
\]

\end_inset

Notation: there are N nodes, so voltages live in 
\begin_inset Formula $\mathbb{R}^{N}$
\end_inset

.
 The boundary of the network (input/output terminals) is written as 
\begin_inset Formula $\partial N$
\end_inset

 and the voltages are held fixed at the boundary.
 Note that Q is also a Dirichlet form.
 Viz its a map 
\begin_inset Formula $Q:\mathbb{R}^{\partial N}\to\mathbb{R}$
\end_inset

.
 The black-box principle of equivalent resistor networks is that any two
 resistor networks are black-box equivalent when they have the same Q.
 
\end_layout

\begin_layout Standard
For the correct generalization to impedance, it is not enough to just replace
 
\begin_inset Formula $\mathbb{F}=\mathbb{R}$
\end_inset

 by 
\begin_inset Formula $\mathbb{F}=\mathbb{R}(t)$
\end_inset

 because this fails to deal with the time variation correctly.
 Put it another way: for the pure resistor network, we are free to fix voltages
 at both the input and output terminals arbitrarily; the internal currents
 are determined entirely by these.
 For the general case with impedance, we are not free to fix both voltages
 and currents at both the input and output terminals.
 Out of the total set of 
\begin_inset Formula $2\dim(\partial N)$
\end_inset

 voltages and currents, we can fix only half the set, i.e.
 a mixture of voltages, currents of 
\begin_inset Formula $\dim(\partial N)$
\end_inset

.
\end_layout

\begin_layout Standard
To handle this, we need to construct a symplectic vector space, with a symplecti
c form on it, and work in the Lagrangian subspace of it.
 Thus, we have 
\begin_inset Formula $\psi\in\mathbb{F}^{\partial N}$
\end_inset

 as the potentials at the network terminals, and 
\begin_inset Formula $dQ_{\psi}\in\left(\mathbb{F}^{\partial N}\right)^{*}$
\end_inset

 as the conjugate currents.
 Out of the total space 
\begin_inset Formula $\mathbb{F}^{\partial N}\oplus\left(\mathbb{F}^{\partial N}\right)^{*}$
\end_inset

 of states, the subspace of actually attainable states is 
\begin_inset Formula 
\[
{\rm Graph}\left(dQ\right)=\left\{ \left(\psi,dQ_{\psi}\right)\vert\psi\in\mathbb{F}^{\partial N}\right\} \subseteq\mathbb{F}^{\partial N}\oplus\left(\mathbb{F}^{\partial N}\right)^{*}
\]

\end_inset

The set of Lagrangian subspaces is an algebraic variety, the Lagrangian
 Grassmanian.
\end_layout

\begin_layout Standard
Baez primary result on impedance networks is that the black box is describable
 by the symplecitification of ..
 OK I don't get it.
\end_layout

\begin_layout Section*
Mining Grammatical Categories – 20 June 2015
\end_layout

\begin_layout Standard
Now that we have a database filled with disjunct statistics, how do we datamine
 that for grammatical categories, which is, after all, the main point of
 this exercise? Let me explain in several steps; at first illustrative,
 and then, more precisely.
 So first, consider a corpus containing these sentences:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard

\family sans
the big tree 
\end_layout

\begin_layout Standard

\family sans
a green tree 
\end_layout

\begin_layout Standard

\family sans
the big bush 
\end_layout

\begin_layout Standard

\family sans
a green bush
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
\noindent
I want to conclude that "tree" is a lot like "bush", and the two should
 be considered as being "similar enough to be merged into a common grammatical
 category".
 That is, the words "tree" and "bush" always occur in similar contexts,
 or even the same contexts.
 The word 
\begin_inset Quotes eld
\end_inset

context
\begin_inset Quotes erd
\end_inset

 here means 
\begin_inset Quotes eld
\end_inset

the dependency parse context
\begin_inset Quotes erd
\end_inset

, and not 
\begin_inset Quotes eld
\end_inset

the n-gram context
\begin_inset Quotes erd
\end_inset

.
 More precisely, it means 
\begin_inset Quotes eld
\end_inset

the accumulated statistics for the disjuncts obtained from MST dependency
 parses
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Standard
Suppose the following parses were observed:
\end_layout

\begin_layout Standard

\family typewriter
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

	 +---MA---+
\end_layout

\begin_layout Plain Layout

	 |   +-MB-+
\end_layout

\begin_layout Plain Layout

	 |   |    |
\end_layout

\begin_layout Plain Layout

	the big  tree 
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

	 +---MC---+
\end_layout

\begin_layout Plain Layout

	 |   +-MD-+
\end_layout

\begin_layout Plain Layout

	 |   |    |
\end_layout

\begin_layout Plain Layout

	 a green tree 
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

	 +---ME---+
\end_layout

\begin_layout Plain Layout
\align left

	 |   +-MF-+
\end_layout

\begin_layout Plain Layout

	 |   |    |
\end_layout

\begin_layout Plain Layout

	the big bush
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

	 +---MG---+
\end_layout

\begin_layout Plain Layout

	 |   +-MH-+
\end_layout

\begin_layout Plain Layout

	 |   |    |
\end_layout

\begin_layout Plain Layout

	 a green bush
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Recall that the above parses were obtained by performing a Maximum-Spanning-Tree
 (MST) parse based on word-pair mutual information (MI).
 The MST is obtained by considering the graph clique joining all words in
 the sentence, and then keeping only those edges that have the greatest
 MI between pairs of words.
 This is the 
\begin_inset Quotes eld
\end_inset

Yuret parse
\begin_inset Quotes erd
\end_inset

.
 The Yuret parse does not have labelled edges, and so we assign arbitrary
 (but unique!) link labels to the edges that were kept.
 Every unique word pair gets a unique link type.
 Then, using the standard Link Grammar theory, each link is broken into
 a + and a - connector, and the ordered set of connectors on a word are
 called a disjunct.
\end_layout

\begin_layout Standard
The disjuncts extracted from the above parses would then be:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

	tree: (MA- & MB-) or (MC- & MD-)
\end_layout

\begin_layout Plain Layout

	bush: (ME- & MF-) or (MG- & MH-)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
No two disjuncts are alike, so naively, these seem completely uncomparable.
 Of course, this is wrong; we need to compare the 
\begin_inset Quotes eld
\end_inset

decoded disjunct
\begin_inset Quotes erd
\end_inset

.
 The 
\begin_inset Quotes eld
\end_inset

decoded disjunct
\begin_inset Quotes erd
\end_inset

 is NOT a part of the standard Link Grammar theory, so let me explain it
 here: it is simply the disjunct where the connector is replaced by the
 word or word-class that it connects to.
 For example, 
\family typewriter
MA-
\family default
 connects to the word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

, so the 
\begin_inset Quotes eld
\end_inset

decoded connector
\begin_inset Quotes erd
\end_inset

 for 
\family typewriter
MA-
\family default
 is 
\family typewriter
$the$-
\family default
.
 So, the decoded disjuncts are then:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

	tree: ($the$- & $big$-) or ($a$- & $green$-)
\end_layout

\begin_layout Plain Layout

	bush: ($the$- & $big$-) or ($a$- & $green$-)
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Now we can see that the decoded disjuncts are identical, for this example.
 Based on this, we conclude that perhaps 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

bush
\begin_inset Quotes erd
\end_inset

 indeed belong to the same grammatical category.
 The remainder of the clustering algorithm is now 
\begin_inset Quotes eld
\end_inset

obvious
\begin_inset Quotes erd
\end_inset

: rewrite the dictionary so that it has a single entry for both words:
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

	tree bush: (MA- & MB-) or (MC- & MD-)
\end_layout

\end_inset

This leaves the ME+, MF+, 
\emph on
etc
\emph default
.
 connector dangling: thus, we need to search for all occurrences of ME+ and
 replace it by MA+, and likewise all occurrences of MF+ need to be replaced
 by MB+, and so on.
\end_layout

\begin_layout Subsubsection*
Similarity metrics
\end_layout

\begin_layout Standard
The above conveys the general idea, but is over-simplifies a few aspects.
 First of all, it is very unlikely that two words will appear in sentence
 contexts that are exactly identical.
 Secondly, some constructions may be very common, and others, very rare;
 that is, some disjuncts may be very common, and some very rare.
 So, for example: suppose we read a text which used the phrase 
\begin_inset Quotes eld
\end_inset


\emph on
the big idea
\emph default

\begin_inset Quotes erd
\end_inset

 a lot, but we also read an obscure linguistics text that said that 
\begin_inset Quotes eld
\end_inset


\emph on
a green idea sleeps furiously
\emph default

\begin_inset Quotes erd
\end_inset

.
 It would probably be a mistake to lump 
\begin_inset Quotes eld
\end_inset

idea
\begin_inset Quotes erd
\end_inset

 in with 
\begin_inset Quotes eld
\end_inset

tree, bush
\begin_inset Quotes erd
\end_inset

, given that 
\begin_inset Quotes eld
\end_inset

green idea
\begin_inset Quotes erd
\end_inset

 is a very rare construction.
 Thus, we need a better way of comparing collections of disjuncts.
 
\end_layout

\begin_layout Standard
One obvious way is to treat a collection of decoded disjuncts as a vector
 in a high-dimensional vector space.
 The similarity between two vectors could be given by the cosine between
 two vectors.
 Alternately, perhaps the vectors could be treated as points, and similarity
 be given by the distance between points.
 There are other possibilities; the best choice is not obvious; several
 need to be explored.
\end_layout

\begin_layout Standard
Thus, for example, let 
\begin_inset Formula $\{e_{1},e_{2},e_{3},\cdots\}$
\end_inset

 be the basis of a high-dimensional vector space.
 For the previous example, we let 
\begin_inset Formula $e_{1}$
\end_inset

 correspond to the decoded disjunct 
\family typewriter
($the$- & $big$-)
\family default
 while 
\begin_inset Formula $e_{2}$
\end_inset

 corresponds to 
\family typewriter
($a$- & $green$-)
\family default
.
 The word 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 is then some vector ...
 what vector should it be? There are several choices.
 Suppose that 
\family typewriter
($the$- & $big$-)
\family default
 was observed with a frequency 
\begin_inset Formula $p_{1}$
\end_inset

 and that 
\family typewriter
($a$- & $green$-)
\family default
 was observed with frequency 
\begin_inset Formula $p_{2}$
\end_inset

.
 The corresponding vector is then obviously 
\begin_inset Formula $p_{1}e_{1}+p_{2}e_{2}$
\end_inset

 and we can construct another vector that corresponds to the the word 
\begin_inset Quotes eld
\end_inset

bush
\begin_inset Quotes erd
\end_inset

, say, for example: 
\begin_inset Formula $q_{1}e_{1}+q_{2}e_{2}$
\end_inset

.
\end_layout

\begin_layout Standard
The dot-product between 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

bush
\begin_inset Quotes erd
\end_inset

 is then given by 
\begin_inset Formula $p_{1}q_{1}+p_{2}q_{2}$
\end_inset

, so that the larger the product, the closer the two words are.
 The cosine angle is 
\begin_inset Formula $(p_{1}q_{1}+p_{2}q_{2})/\left|p\right|\left|q\right|$
\end_inset

 where 
\begin_inset Formula $\left|p\right|=\sqrt{p_{1}^{2}+p_{2}^{2}}$
\end_inset

 and so on.
 The closer that the cosine is to 1.0, the closer the two words are.
 There are other possibilities: we have the Cartesian distance 
\begin_inset Formula 
\[
dist(tree,bush)=\sqrt{(p_{1}-q_{1})^{2}+(p_{2}-q_{2})^{2}}
\]

\end_inset

 and we can contemplate 
\emph on
lp
\emph default
-metrics as well.
\end_layout

\begin_layout Standard
None of the above metrics take into account the mutual information (MI)
 of the disjunct.
 This is almost surely a mistake.
 Due to the vagaries of MST parsing, there will be many disjuncts with a
 low MI value.
 This is not uncommon in sentences with prepositions, where MST gives some
 poor choices for the links to the prepositions, and thus results in disjuncts
 with low MI values.
 Recall, the higher the MI, the stronger the structure is.
 Thus, perhaps a better vector for 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 might be
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
tree=e_{1}m_{1}p_{1}+e_{2}m_{2}p_{2}
\]

\end_inset

The above seems to be the most entropic-like in its expression.
 However, the probabilities might weight the terms too strongly, and so a
 weaker weighting would be the below.
 It is not yet clear to me which of these expressions are the most 
\begin_inset Quotes eld
\end_inset

elegant
\begin_inset Quotes erd
\end_inset

, or which work the best...
 
\begin_inset Formula 
\[
tree=e_{1}(m_{1}-\log_{2}p_{1})+e_{2}(m_{2}-\log_{2}p_{2})
\]

\end_inset

Here 
\begin_inset Formula $m_{1}$
\end_inset

 and 
\begin_inset Formula $m_{2}$
\end_inset

 are the mutual information of the disjuncts 
\family typewriter
(MA- & MB-)
\family default
 and 
\family typewriter
(MC- & MD-)
\family default
, respectively.
 The last two seem to be closer to the intended spirit of the maximum entropy
 principle.
 There are even more possibilities, though.
\end_layout

\begin_layout Subsubsection*
Frequency and Mutual Information
\end_layout

\begin_layout Standard
The above section makes explicit use of the frequency and the mutual information
 of a disjunct.
 It is useful to define these.
 Given a disjunct 
\family typewriter
(MA- & MB-)
\family default
 let N
\family typewriter
(MA- & MB-)
\family default
 be the number of times that this disjunct has been observed.
 It will usually be an integer (except when obtained in certain unusual
 situations not discussed here).
 Let N
\family typewriter
(*- & *-)
\family default
 be the number of times that any two-connector disjunct has been observed,
 as long as both connectors point in the - direction.
 That is,
\begin_inset Formula 
\[
N(*-\&*-)=\sum_{c_{1}\in-,c_{2}\in-}N(c_{1}\&c_{2})
\]

\end_inset

the summation taking place over all connectors in the - direction.
 The frequency of observing 
\family typewriter
(MA- & MB-)
\family default
 is then
\begin_inset Formula 
\[
p(MA-\&MB-)=\frac{N(MA-\&MB-)}{N(*-\&*-)}
\]

\end_inset

The mutual information associated with the disjunct is then 
\begin_inset Formula 
\[
m(MA-\&MB-)=\log_{2}\frac{p(MA-\&MB-)}{p(MA-\&*-)p(*-\&MB-)}
\]

\end_inset

The reason for this possibly unexpected form was developed earlier in this
 diary.
 
\end_layout

\begin_layout Subsubsection*
Semantics
\end_layout

\begin_layout Standard
There is another interesting issue that arises in the above discussion:
 the problem of syntax-semantics correspondence.
 Consider, for example, the sentence 
\emph on

\begin_inset Quotes eld
\end_inset

the dog treed the squirrel
\emph default

\begin_inset Quotes erd
\end_inset

.
 Here the word 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 is used as a verb, meaning 
\begin_inset Quotes eld
\end_inset

the dog chased the squirrel up into the tree
\begin_inset Quotes erd
\end_inset

.
 Such sentences will cause the the word 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 to accumulate disjuncts that the word 
\begin_inset Quotes eld
\end_inset

bush
\begin_inset Quotes erd
\end_inset

 will not have.
 Likewise, 
\begin_inset Quotes eld
\end_inset


\emph on
I'm bushed
\emph default

\begin_inset Quotes erd
\end_inset

 is a verb usage that has no analogous 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 version.
 Thus, not only do the words 
\begin_inset Quotes eld
\end_inset

bush
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 have different sets of disjuncts, but the differences are hiding semantic
 differences ...
 
\end_layout

\begin_layout Standard
There are several strategies that can be used to deal with this.
 More on this later.
 
\end_layout

\begin_layout Subsubsection*
Finding word pairs
\end_layout

\begin_layout Standard
We need a good way of finding word-pairs that are likely to be related.
 I think that perhaps the pattern matcher may be ideal for this.
 Details are TBD...
 but the basic idea is that the hypergraph for 
\begin_inset Quotes eld
\end_inset

tree: (MA- & MB-)
\begin_inset Quotes erd
\end_inset

 is connected to 
\begin_inset Quotes eld
\end_inset

big
\begin_inset Quotes erd
\end_inset

 because MB- is connected to 
\begin_inset Quotes eld
\end_inset

big
\begin_inset Quotes erd
\end_inset

, and 
\begin_inset Quotes eld
\end_inset

big
\begin_inset Quotes erd
\end_inset

 is connected to other lg-connectors, which in turn are connected to other
 disjuncts, which are then connected to other words.
 Thus, we search the local neighborhood of 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

, which causes us to dsicover the word 
\begin_inset Quotes eld
\end_inset

big
\begin_inset Quotes erd
\end_inset

, and then we search the neighborhood of 
\begin_inset Quotes eld
\end_inset

big
\begin_inset Quotes erd
\end_inset

 to find candidates such as 
\begin_inset Quotes eld
\end_inset

bush
\begin_inset Quotes erd
\end_inset

 which might be comparable to 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

.
 This search graph is not small, but it is not large: There may be thousands
 of words that are two hops away from 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

, but not millions.
\end_layout

\begin_layout Subsubsection*
Putting it all together
\end_layout

\begin_layout Standard
These are the things that need to be done:
\end_layout

\begin_layout Enumerate
compute the MI for the disjuncts
\end_layout

\begin_layout Enumerate
pick a common noun, compute the similarity scores for that word and every
 word that is linked to it.
 created ranked graph of similarity.
\end_layout

\begin_layout Enumerate
repeat step 2 for several different similarity formulas
\end_layout

\begin_layout Enumerate
repeat steps 2,3 for several verbs, several adjectives, several adverbs,
 several determiners, several prepositions.
\end_layout

\begin_layout Enumerate
Write code for creating grouping words into grammatical clusters.
\end_layout

\begin_layout Enumerate
Pick the most promising metric, and start clustering in bulk.
\end_layout

\begin_layout Standard
Step 5 requires writing a lot of code; it can all be written before the
 final metric has been determined.
\end_layout

\begin_layout Subsubsection*
The end.
\end_layout

\begin_layout Standard
That's all for now.
 More later.
\end_layout

\begin_layout Section*
Not LSA – 1 July 2015
\end_layout

\begin_layout Standard
NotLSA – a way to do LSA-like things without actually using LSA (Latent
 Semantic Analysis).
 Two very low-brow approaches, maybe well-known in the industry; I have
 no idea.
 Both of these approaches attempt to automatically extract keywords from
 documents.
 What cool about this is that its ...
 unsupervised; requires no training, and is based on very simple, proven
 ideas.
 Obvious, even: compute the mutual information between pairs of things ...
 between words and documents, between words and word-pairs, etc.
 Heh.
 
\end_layout

\begin_layout Standard
But how do we do this? How do we compute the MI between a page of text,
 and a word? No way to answer this without diving into the details.
\end_layout

\begin_layout Subsubsection*
Text-keyword correlation
\end_layout

\begin_layout Standard
Lets take a text, say – 1000 pages of ..
 something.
 Some corpus.
 We want to compute the mutual information between the page itself, and
 the words on the page.
 We do this by analogy to MI of word pairs.
\end_layout

\begin_layout Standard
Call the 
\begin_inset Formula $k$
\end_inset

'th page 
\begin_inset Formula $g_{k}$
\end_inset

.
 Count the number of times that word 
\begin_inset Formula $w_{m}$
\end_inset

 appears on this page; let this count be 
\begin_inset Formula $N_{mk}$
\end_inset

.
 Define 
\begin_inset Formula $N_{m}=\sum_{k}N_{mk}$
\end_inset

 be the total number of times that the work 
\begin_inset Formula $w_{m}$
\end_inset

 appear in the document, and let 
\begin_inset Formula $N=\sum_{m}N_{m}$
\end_inset

 be the total number of words in the document.
 Then, as usual, define probabilities, so that 
\begin_inset Formula 
\[
p_{m}=P(w_{m})=N_{m}/N
\]

\end_inset

is the frequency of observing word 
\begin_inset Formula $w_{m}$
\end_inset

 in the entire corpus, and 
\begin_inset Formula 
\[
p_{mk}=P(w_{m}|g_{k})=N_{mk}/\sum_{m}N_{mk}
\]

\end_inset

be the (relative) frequency of the same word on page 
\begin_inset Formula $g_{k}$
\end_inset

.
 Notice that the definition of 
\begin_inset Formula $p_{mk}$
\end_inset

 is independent of the page size.
 Pages do not all have to be of the same size.
 Define the mutual information as 
\begin_inset Formula 
\[
\mbox{MI}(g_{k},w_{m})=-\log_{2}\frac{p_{mk}}{p_{m}}=-\log_{2}\frac{N_{mk}N}{\sum_{m}N_{mk}\sum_{k}N_{mk}}=-\log_{2}\frac{p(m,k)}{p(m,*)p(*,k)}
\]

\end_inset

This is essentially a measure of how much more often the word 
\begin_inset Formula $w_{m}$
\end_inset

 appears on page 
\begin_inset Formula $g_{k}$
\end_inset

 as compared to its usual frequency.
 The highest-MI words are essentially the topic words for the page.
 The right-most form introduces a new notation, to make it clear that it
 resembles the traditional pair-MI expression.
 The notation is 
\begin_inset Formula 
\[
p(m,k)=\frac{N_{mk}}{N}
\]

\end_inset

so that 
\begin_inset Formula 
\[
p(m,*)=\sum_{k}p(m,k)\qquad\mbox{and}\qquad p(*,k)=\sum_{m}p(m,k)
\]

\end_inset

are the traditional-looking pair-MI values.
 
\end_layout

\begin_layout Standard
TODO: – this does not have the feature-reduction/word-combing aspects of
 LSA...
\end_layout

\begin_layout Subsubsection*
Variants
\end_layout

\begin_layout Standard
Instead of working with words, we could work with word-pairs, which is a
 stand-in for working with (named) entities.
 Thus, we can identify if a named entity occurs in a document more often
 than average.
\end_layout

\begin_layout Section*
Unsupervised Morphology Learning References
\end_layout

\begin_layout Standard
Here's some:
\end_layout

\begin_layout Itemize
John Goldsmith, 
\begin_inset Quotes eld
\end_inset

The unsupervised learning of natural language morphology
\begin_inset Quotes erd
\end_inset

, Journal Computational Linguistics archive Volume 27 Issue 2, June 2001
 Pages 153-198 MIT Press http://delivery.acm.org/10.1145/980000/972668/p153-goldsmi
th.pdf
\end_layout

\begin_layout Itemize
John Goldsmith, 
\begin_inset Quotes eld
\end_inset

An algorithm for unsupervised learning of morphology
\begin_inset Quotes erd
\end_inset

 Natural Language Engineering Volume 12 / Issue 04 / December 2006, pp 353-371
 Cambridge University Press DOI: http://dx.doi.org/10.1017/S1351324905004055
 http://people.cs.uchicago.edu/~jagoldsm/Papers/algorithm.pdf
\end_layout

\begin_layout Itemize
Survey Article Unsupervised Learning of Morphology Harald Hammarström Lars
 Borin http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00050
\end_layout

\begin_layout Section*
Predicate-Argument structure
\end_layout

\begin_layout Standard
Here's one:
\end_layout

\begin_layout Itemize
The Darwinian evolution of natural language comes from a combination of
 Expressive FSM's and Lexical predicate-argument FSM's within the human
 brain.
 Shigeru Miyagawa, Robert C.
 Berwick and Kazuo Okanoya 
\begin_inset Quotes eld
\end_inset

The emergence of hierarchical structure in human language
\begin_inset Quotes erd
\end_inset

 Front.
 Psychol., 20 February 2013 | http://dx.doi.org/10.3389/fpsyg.2013.00071 http://alpha-
leonis.lids.mit.edu/wordpress/wp-content/uploads/2014/01/shigeru-berwick-kaz-fronti
ers13.pdf
\end_layout

\begin_layout Section*
Edge-counting 27 March 2017
\begin_inset CommandInset label
LatexCommand label
name "sec:Edge-counting"

\end_inset


\end_layout

\begin_layout Standard
Counting edges in a clique is not the same as counting edges in planar trees.
 The diagram below shows the clique of a four-word sentence.
 The 
\begin_inset Quotes eld
\end_inset

words
\begin_inset Quotes erd
\end_inset

 are 'a', 'b', 'c' and 'd'.
 There are a total of six edges, with one edge between every possible word-pair.
 Each edge occurs only once.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/four-clique.eps
	width 30col%

\end_inset


\end_layout

\begin_layout Standard
Pair counting in planar diagrams gives different results.
 The diagram below shows the twelve planar trees, containing no cycles,
 that can be formed by parsing a sentence of four words.
 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/edge-counting.eps
	width 40col%

\end_inset


\end_layout

\begin_layout Standard
The general formula for the number of different planar dependency parses
 is
\begin_inset Formula 
\[
\frac{1}{2n-1}{3n-1 \choose n-1}
\]

\end_inset

This formula is given by Deniz Yuret in 
\begin_inset Quotes eld
\end_inset


\begin_inset CommandInset href
LatexCommand href
name "Lexical Attraction Models of Language"
target "http://www2.denizyuret.com/pub/lex-attr/lam-iscis06.pdf"
literal "false"

\end_inset


\begin_inset Quotes erd
\end_inset

 ISCIS 2006 (http://www2.denizyuret.com/pub/lex-attr/lam-iscis06.pdf) 
\end_layout

\begin_layout Standard
It is important not to confuse this with the 
\begin_inset Quotes eld
\end_inset

matrix-tree theorem
\begin_inset Quotes erd
\end_inset

 aka 
\begin_inset CommandInset href
LatexCommand href
name "Kirchoff's Theorem"
target "https://en.wikipedia.org/wiki/Kirchhoff%27s_theorem"
literal "false"

\end_inset

, which counts the number of spanning trees of a graph.
 In short, it states that the number of spanning trees is equal to any cofactor
 of its Laplacian matrix.
 In our case, we are dealing with a complete graph (a clique) and so on
 might hope that 
\begin_inset CommandInset href
LatexCommand href
name "Cayley's formula"
target "https://en.wikipedia.org/wiki/Cayley's_formula"
literal "false"

\end_inset

 applies.
 In fact, neither theorem works, because we are interested in non-self-intersect
ing planar trees, constrained by linear word-order.
\end_layout

\begin_layout Standard
There are 36 edges grand total, and these are unequally distributed.
 The counts are: 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word-pair
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
count
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(ab)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(bc)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(cd)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(ac)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(bd)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(ad)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Note that the most frequent edges occur almost twice as often as the least-frequ
ent edges.
 The distribution, by length, is:
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="4" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Length
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Count
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Note that the progressively-longer edges get less frequent.
\end_layout

\begin_layout Standard
If graphs with cycles are also allowed, (but no edge crossings) then, in
 addition to the above, there are eleven more diagrams.
 These are shown below.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/loop-counting.eps
	width 40col%

\end_inset


\end_layout

\begin_layout Standard
Again, we count the number of edges, as before.
 The 'tree' column shows the counts from before; the loop count count the
 edges from the additional eleven diagrams; the total is just that.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word-pair
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
tree
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
loop
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
total
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(ab)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(bc)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(cd)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(ac)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(bd)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(ad)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Likewise, the number of arcs of the given length is now given below:
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="4" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Length
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Count
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
48
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
What are the actual distributions, for these two cases? Begin by counting
 the number of planar trees.
 I currently do not know of any published work on this, so below, I make
 a half-baked attempt to count these myself.
 Its ...
 incomplete.
 Maybe there's some simpler approach.
\end_layout

\begin_layout Subsection*
Counting planar tree graphs
\end_layout

\begin_layout Standard
Let's count the number of planar tree graphs; i.e.
 those without any loops.
 First, we need a generic formula for sentences of length N.
 This is not so very easy.
 The diagram below shows one way to count.
 (I think what follows is correct, but I might be making a mistake.
 I am unaware of any literature that presents this information).
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/all-count.eps
	width 50col%

\end_inset


\end_layout

\begin_layout Standard
Here, the star represents some planar tree connecting all of the words of
 a smaller sentence.
 Assume that there are 
\begin_inset Formula $T(n)$
\end_inset

 such trees, connecting 
\begin_inset Formula $n$
\end_inset

 words.
 Tree diagrams of of Type A are assembled by placing two adjacent smaller
 trees next to each other.
 Naively, one can then count how many such pairs there are; the issue is
 that the Type B diagram will occur multiple times in this pairing; we would
 rather NOT count it with this mutiplicity.
 To avoid this problem, we should only allow pairs, such as Type A, to be
 assembled of sub-parts of the shapes C and D.
 Because of the over-arching arc, these can never result in double-counting.
 However, counting only pairs results in an under-counting: graphs of Type
 B never occur.
 Thus, one should count pairs, triples, and so on – graphs of Type E.
 Now we have a way of getting the formula.
 Define 
\begin_inset Formula $D(n)$
\end_inset

 as the count of the number of planar trees, connecting 
\begin_inset Formula $n$
\end_inset

 words, having an arc connecting the first and last word: i.e.
 trees of type C or D.
 (Think 
\begin_inset Quotes eld
\end_inset

D = dome
\begin_inset Quotes erd
\end_inset

) One then has that 
\begin_inset Formula 
\[
D(n)=\sum_{j=1}^{n-1}T(j)T(n-j)
\]

\end_inset

It is convenient, here, to define 
\begin_inset Formula $T(1)=1$
\end_inset

.
 The first and last terms of this sum then correspond to trees of Type C,
 while the middle terms are trees of type D.
\end_layout

\begin_layout Standard
To count trees of Type E, is is convenient to break this up into the problem
 of counting chains of length 
\begin_inset Formula $k$
\end_inset

, so that there are 
\begin_inset Formula $C_{k}(n)$
\end_inset

 trees, consisting of a sequence of 
\begin_inset Formula $k$
\end_inset

 domes, making up a total of 
\begin_inset Formula $n$
\end_inset

 words.
 One then has that 
\begin_inset Formula 
\[
T(n)=D(n)+C_{2}(n)+\cdots+C_{n-1}(n)
\]

\end_inset

It's convenient, here, to define 
\begin_inset Formula $C_{1}(n)=D(n)$
\end_inset

.
 Writing down the 
\begin_inset Formula $C_{k}(n)$
\end_inset

's requires some combinatorial magic.
 The first one is 
\begin_inset Formula 
\[
C_{2}(n)=\sum_{j=2}^{n-1}D(j)D(n-j+1)
\]

\end_inset

Next comes
\begin_inset Formula 
\[
C_{3}(n)=\sum_{j=2}^{n-1}\;\sum_{m=2}^{n-j+1}D(j)D(m)D(n-j-m+2)
\]

\end_inset

which is awkward to write down.
 It's easier to count partitions of sets.
 Thus, what really is happening here is that the sums range over all 
\begin_inset Formula $k$
\end_inset

-way partitions of sets containing 
\begin_inset Formula $n+k-1$
\end_inset

 elements.
 Not the partition is NOT over 
\begin_inset Formula $n$
\end_inset

 elements: to get connected graphs, we have to identify end-points of each
 link in the chain.
 Thus, 
\begin_inset Formula 
\[
C_{k}(n)=\Pi_{\sigma}\cdots
\]

\end_inset

The table below summarizes the first few sums:
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="7">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $n$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $T(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $D(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $C_{2}(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $C_{3}(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $C_{4}(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $C_{5}(n)$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
45
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
123
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Either I am computing this wrong, or the sequences are not in OEIS.
 Surprising!
\end_layout

\begin_layout Subsection*
Counting planar loop graphs
\end_layout

\begin_layout Standard
The above process can be repeated, except that this time, we consider the
 planar graphs containing loops.
 To get started, consider the diagram below.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/all-loop.eps
	width 60col%

\end_inset


\end_layout

\begin_layout Standard
Here, the stars represent either 
\begin_inset Quotes eld
\end_inset

domed
\begin_inset Quotes erd
\end_inset

 diagrams, or the empty set (a set containing no words and no edges.
 The type F concatenates two domes, and puts a dome over those, in turn.
 Since both of the stars are domed (or empty), it is impossible to add any
 additional edges to this graph.
 So, for graphs constructed out of a pair domes (one or both possibly empty),
 Type F is all that there is.
 For three domes in a row, there are only three ways of adding edges: these
 are shown in type G and H in this diagram.
 Again, this exhauses all possibilities.
 This process constructs both looped and tree diagrams.
 The general idea is to repeat this, for sequences of four or more stars.
\end_layout

\begin_layout Standard
The counting is similar to that before.
 Let 
\begin_inset Formula $F(n)$
\end_inset

 count the number of domed graphs, connecting 
\begin_inset Formula $n$
\end_inset

 words.
 Let 
\begin_inset Formula $G_{2}(n)$
\end_inset

 count the number of type F graphs, made of two parts, and containing 
\begin_inset Formula $n$
\end_inset

 words.
 Consulting the diagram, we have
\begin_inset Formula 
\[
G_{2}(n)=F(n-1)+\sum_{k=2}^{n-1}F(k)F(n-k+1)+F(n-1)
\]

\end_inset

Likewise, let 
\begin_inset Formula $G_{3}(n)$
\end_inset

 count the number of graphs of type G and H, combined.
 Consulting the diagram, this has a more complex expression:
\begin_inset Formula 
\[
G_{3}(n)=\sum_{k=2}^{n-2}F(k)F(n-k)+\sum\sum F()F()F()...+\sum_{k=2}^{n-2}F(k)F(n-k)+...
\]

\end_inset

The total number of domed graphs having 
\begin_inset Formula $n$
\end_inset

 words is then 
\begin_inset Formula 
\[
F(n)=\sum_{k=2}^{n-1}G_{k}(n)
\]

\end_inset

Let 
\begin_inset Formula $S(n)$
\end_inset

 be the count of a string of domed graphs, but NOT having connecting arcs:
 that is, graphs of type A or E.
\end_layout

\begin_layout Standard
A table of these is given below.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="8">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $n$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $L(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $S(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $F(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $G_{2}(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $G_{3}(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $G_{4}(n)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $G_{5}(n)$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
x
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
156
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
x
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1162
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
x
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Subsection*
Effecect on Grammar Induction
\end_layout

\begin_layout Standard
There is a surprising effect on the quality of the learned grammars, as
 shown in the graph below, obtained from Andres Suarez
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Email on the lang-learn@gmail.com mailing list, 22 July 2019, titled 
\begin_inset Quotes eld
\end_inset

MI-threshold for MST-parses
\begin_inset Quotes erd
\end_inset

.
\end_layout

\end_inset

 via the Kolonin learning project.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/andres-mi-cut.png
	width 100col%

\end_inset


\end_layout

\begin_layout Standard
The graph shows the F1-score (geometric mean of recall and precision) for
 parse trees obtained four different ways.
 The recall and precision are relative to a 
\begin_inset Quotes eld
\end_inset

golden
\begin_inset Quotes erd
\end_inset

 corpus.
 The horizontal axis shows score dependence on an MI threshold: word pairs
 with an MI of less than the threshold are discarded, before the parse tree
 is generated.
\end_layout

\begin_layout Standard
The lines are labeled as:
\end_layout

\begin_layout Itemize

\series bold
Sequential 
\series default
– The parse 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

 is just a sequence of links connecting neighboring words.
 This is independent of the MI score of individual word-pairs, and so is
 a flat line.
\end_layout

\begin_layout Itemize

\series bold
WIN6
\series default
 – The parse tree is an MST tree obtained using previously-obtained word-pair
 MI scores from a 
\begin_inset Quotes eld
\end_inset

clique-pair counting
\begin_inset Quotes erd
\end_inset

 word-pair generation technique.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
The code base calls this 
\begin_inset Quotes eld
\end_inset

clique-pair counting
\begin_inset Quotes erd
\end_inset

.
\end_layout

\end_inset

 The counting is done by considering all word-pairs formed between the left-most
 word in a 7-wide window, and all the other words in that window.
 Thus, six word-pairs are counted for each window position, then the window
 is slid over by one.
\end_layout

\begin_layout Itemize
\noindent
\align block

\series bold
WIN6-odist
\series default
 – The parse tree is an MST tree obtained using previously-obtained word-pair
 MI scores from a weighted-window counting technique.
 This counts pairs with a distance penalty of 
\begin_inset Formula $\mbox{floor}\left(6/dist\right)$
\end_inset

, as shown below:
\begin_inset Newline newline
\end_inset


\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Plain Layout
\noindent
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="8" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
distance
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
weight
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\end_inset


\begin_inset Newline newline
\end_inset

That is, a word pair formed from nearest neighbors is given a weight of
 six, and so on.
\end_layout

\begin_layout Itemize

\series bold
LG-all
\series default
 – The parse tree is an MST tree obtained using previously-obtained word-pair
 MI scores from a 100 random planar tree parses created using the LG 
\begin_inset Quotes eld
\end_inset

any
\begin_inset Quotes erd
\end_inset

 language parser.
\end_layout

\begin_layout Standard
Word-pair MI values were obtained from the ULL Project 
\begin_inset Quotes eld
\end_inset

Gutenberg Children's Corpus
\begin_inset Quotes erd
\end_inset

.
 This consists of 200K sentences of average length 13.5, for a total of 2.7M
 words.
 The F1-scores were obtained by evaluatng 228 sentences (average length
 of 8) from the 
\begin_inset Quotes eld
\end_inset

golden corpus
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Standard
There are two rather surprising results here:
\end_layout

\begin_layout Enumerate
The MST tree always has a lower score than the sequential 
\begin_inset Quotes eld
\end_inset

tree
\begin_inset Quotes erd
\end_inset

, when compared against the golden parse tree.
 The reason for this is not currently understood.
\end_layout

\begin_layout Enumerate
The way in which the MI values were collected has a strong effect on the
 quality of the MST parse.
 This is unexpected.
\end_layout

\begin_layout Standard
Both of these results are unexpected, and lack an adequate explanation.
 Understanding the first is more important.
\end_layout

\begin_layout Section*
English Dataset Sample 28 April 2017
\end_layout

\begin_layout Standard
This section was moved to `word-pairs-redux.lyx` in July 2019, so that all
 the word-pair stuff would be in one place.
\end_layout

\begin_layout Section*
Connector Sets 7 May 2017 (revised July 2017)
\end_layout

\begin_layout Standard
Tho better manage the size of this diary, this has been moved to its own
 file.
 See the 
\begin_inset Quotes eld
\end_inset

connector-sets-revised.lyx
\begin_inset Quotes erd
\end_inset

 file.
\end_layout

\begin_layout Subsubsection*
Abstract
\end_layout

\begin_layout Standard
This is a report on a dataset of disjuncts and connector sets, extracted
 from MST parses of a batch of sentences.
 First, a recap of what these are, then a characterization of the database
 contents, and finally, a report on the grammatical similarity of words
 in the dataset.
 
\end_layout

\begin_layout Section*
MST parsing algos
\end_layout

\begin_layout Standard
There are multiple MST algos, some better than others.
 A short list with some references.
\end_layout

\begin_layout Itemize
The current implementation in the (opencog sheaf) directory is an MST algo
 for generating projective MST trees from undirected edges.
 Its a simple-minded projective adaptation of Borůvka's algo (see wikipedia
 for a description).
 I just measured it to run at 
\begin_inset Formula $O\left(n^{3}\right)$
\end_inset

 for 
\begin_inset Formula $n$
\end_inset

 words.
 See atomspace/opencog/bench-ms/README.md.
\end_layout

\begin_layout Itemize
The 
\begin_inset Formula $O\left(n^{3}\right)$
\end_inset

 is for the case of arbitrary-length links.
 If the scoring function is altered to give bad scores to link lengths >6
 long, then the algo kicks over to 
\begin_inset Formula $O\left(n^{2.3}\right)$
\end_inset

 after about 
\begin_inset Formula $8<n$
\end_inset

 or so.
 Awesome! See graphs in atomspace/opencog/bench-ms/ for a better look.
\end_layout

\begin_layout Itemize
This isn't bad, per se, since Yuret published his best projective MST algo
 which ran at 
\begin_inset Formula $O\left(n^{3}\right)$
\end_inset

 for 
\begin_inset Formula $n$
\end_inset

 words.
 So we are in the right ballpark...
\end_layout

\begin_layout Itemize
The state-of-the-art projective algo (for directed edges) is supposed to
 be Eisner's algo, which runs at 
\begin_inset Formula $O\left(n^{3}\right)$
\end_inset

 for 
\begin_inset Formula $n$
\end_inset

 words.
 So we are still in the right ballpark.
 Eisner, Jason, 1996.
 “Three New Probabilistic Models for Dependency Parsing.” In Proceedings
 of the 16th Conference on Computational Linguistics (CoLING 96) .
 Saarbruecken: ACL, 340–345.
 
\end_layout

\begin_layout Itemize
The Chu-Liu-Edmonds algorithm finds (non-projective) spanning trees in directed
 graphs.
 It is described by Ryan McDonald, Fernando Pereira, Kiril Ribarov Jan Hajič,
 
\begin_inset Quotes eld
\end_inset

Non-projective Dependency Parsing using Spanning Tree Algorithms
\begin_inset Quotes erd
\end_inset

 http://www2.denizyuret.com/ref/mcdonald/nonprojectiveHLT-EMNLP2005.pdf Its
 supposed to run in 
\begin_inset Formula $O\left(n^{2}\right)$
\end_inset

 time.
\end_layout

\begin_layout Itemize
An non-projective algorithm that is 
\begin_inset Quotes eld
\end_inset

super-linear
\begin_inset Quotes erd
\end_inset

 in the number of edges is described by Effi Levi, Roi Reichart, Ari Rappoport,
 
\begin_inset Quotes eld
\end_inset

Edge-Linear First-Order Dependency Parsing with Undirected Minimum Spanning
 Tree Inference
\begin_inset Quotes erd
\end_inset

 (2016) https://www.aclweb.org/anthology/P/P16/P16-1198.pdf Since this is edge-line
ar, I think that, for us, it claims 
\begin_inset Formula $O\left(n^{2}\right)$
\end_inset

 for 
\begin_inset Formula $n$
\end_inset

 words.
 (Since, for us, we don't know, a priori, if we have an edge, or not).
 Its also not projective.
 https://arxiv.org/pdf/1510.07482.pdf
\end_layout

\begin_layout Section*
Dataset report 3 June 2017
\begin_inset CommandInset label
LatexCommand label
name "sec:Dataset-report"

\end_inset


\end_layout

\begin_layout Standard
Some summary reports from various different datasets.
 The summary for the word-pair datasets was moved to the `word-pairs-redux.lyx`
 directory, so that all word-pair stuff lives together.
\end_layout

\begin_layout Subsection*
Disjunct datasets
\end_layout

\begin_layout Standard
Next, datasets that hold disjuncts.
 This section used to report more data, but it was all flawed: the MI had
 a minus sign in it, causing all computed disjuncts to be maximally bad.
 Despite this, the results were similar to the below: observations and entropy
 fit in line, as expected.
 The 
\begin_inset Formula $H_{left}$
\end_inset

 entropy values were lower, hovered around 15, and the MI was in the 3-5
 range, while 
\begin_inset Formula $H_{right}$
\end_inset

 was unchanged and fit in line.
 You can find the original data in the git commit 9244905afdff191a39af8c5a6deab5
92d5a1558c.
 
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="9" columns="9">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Size
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Sects
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Obs'ns
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Ob/sec
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $H_{left}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $H_{right}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Notes
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
37K x 291K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
446K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
661K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.48
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18.30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16.00
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.98
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_sim
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
137K x 6.24M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8.63M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18.5M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20.96
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9.71
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.90
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_rfive_mtwo
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
522K x 25.2M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
34.2M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
77.8M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
22.82
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20.92
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.09
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8.18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_rfive_mst
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
445K x 23.4M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31.9M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
69.4M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23.09
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.11
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8.16
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_cfive_mst
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
60K x 602K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
801K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.19M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.48
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18.86
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17.99
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.13
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9.26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
zen_pairs_mst
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
85K x 4.88M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.02M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17.8M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.54
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20.48
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17.06
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9.52
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6.10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
zen_pairs_three_mst
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
563K x 29.9M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
39.7M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_mpg_18
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
564K x 21.9M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
34.5M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
78.2M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
22.41
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20.00
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.12
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.72
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_mpg_53
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
An updated legend for the columns:
\end_layout

\begin_layout Description
Size The dimensions of the array.
 The left dimension counts words, the right dimension counts the number
 of unique, distinct pseudo-disjuncts.
\end_layout

\begin_layout Description
Sects The total number of distinct Sections observed.
\end_layout

\begin_layout Description
Obsn's The total number of observations of these Sections.
 Most Sections will be observed more than once.
 Distributions are typically Zipfian, as pointed out earlier.
\end_layout

\begin_layout Description
Obs/sec The average number of times each Section was observed.
\end_layout

\begin_layout Description
Entropy The total entropy of the Sections in this dataset, as defined previously
: for Sections 
\begin_inset Formula $(w,d)$
\end_inset

 it is 
\begin_inset Formula $H=-\sum_{w,d}p(w,d)\log_{2}p(w,d)$
\end_inset

.
\end_layout

\begin_layout Description
MI The total mutual information for the Sections in this dataset, as defined
 previously: 
\begin_inset Formula $MI=\sum_{w,d}p(w,d)\log_{2}\left[p(w,d)/p(w,*)p(*,d)\right]$
\end_inset


\end_layout

\begin_layout Description
\begin_inset Formula $H_{left}$
\end_inset

,
\begin_inset Formula $H_{right}$
\end_inset

 The left and right entropies, as defined previously.
 Note that 
\begin_inset Formula $MI=H-H_{left}-H_{right}$
\end_inset

 holds, by definition.
 Not given for the word-pairs table, because these two are nearly equal,
 and are half the difference between the entropy and the MI.
\end_layout

\begin_layout Standard
Note how the MI is considerably larger than that for the word-pairs.
 Higher MI implies a stronger correlation, and this is good: this suggests
 that the disjuncts are capturing meaningful structures in the language.
\end_layout

\begin_layout Standard
The behavior of the 
\begin_inset Quotes eld
\end_inset

zen
\begin_inset Quotes erd
\end_inset

 dataset might be explained by two issues that this dataset has.
 The smaller 
\begin_inset Quotes eld
\end_inset

zen_pairs_mst
\begin_inset Quotes erd
\end_inset

 dataset is tiny, with a large fraction (most?) words observed only once,
 most disjuncts observed only once, and so the high MI being a false signal,
 an artifact of the tiny size of the set.
 By contrast, the unexpectedly low MI on the 
\begin_inset Quotes eld
\end_inset

zen_pairs_three_mst
\begin_inset Quotes erd
\end_inset

 dataset might be blamed on the 3rd-party word segmentation tool.
 It is known not to be very accurate, and the low MI might be a by-product
 of that.
\end_layout

\begin_layout Standard
The descriptinos of the datasets can be found in the 
\begin_inset Quotes eld
\end_inset

connector-sets-revised.lyx
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

word-pairs-redux.lyx
\begin_inset Quotes erd
\end_inset

 files.
 There are several new datasets since then.
\end_layout

\begin_layout Description

\noun on
en_mpg_53
\noun default
 The disjuncts were built using the MPG parser, after discarding all word
 pairs with MI<5.3 (and then recomputing word-pair MI marginals).
 The stats, as given in the table above, are not all that different than
 
\series bold
\noun on
en_pairs_cfive_mst
\series default
\noun default
 and its not clear if this should be surprising or not...
 Both
\series bold
\noun on
 en_pairs_cfive_mst
\series default
\noun default
 and this are built from the same original word-pair stats.
 If a different set of word-pairs were obtained, how much would things differ?
\end_layout

\begin_layout Description

\noun on
en_mpg_18
\noun default
 As above, but after discarding all word pairs with MI<1.8.
\end_layout

\begin_layout Section*
Thresholding PCA Classifier
\end_layout

\begin_layout Standard
The next step is what I've called 
\begin_inset Quotes eld
\end_inset

clustering
\begin_inset Quotes erd
\end_inset

 in the past, but it really needs to be something more like factor analysis,
 or better yet, sparse PCA.
 Except that's not right, either.
\end_layout

\begin_layout Standard
What is needed is a recognizer,as follows.
 Consider 
\begin_inset Formula $\overrightarrow{b}=\sum_{n}b_{n}w_{n}$
\end_inset

 be a vector, with the 
\begin_inset Formula $w_{n}$
\end_inset

T being individual words, and the 
\begin_inset Formula $b_{n}$
\end_inset

 being weights.
 Plain-old Principal Component Analysis (PCA) computes real-valued weights
 
\begin_inset Formula $b_{n}$
\end_inset

.
 It's problematic, because potentially all of the weights are non-zero for
 all of the words.
 Sparse PCA computes real-valued weights 
\begin_inset Formula $b_{n}$
\end_inset

 such that only some small number of them are non-zero.
 This is much better.
 But what is really needed is a classifier: a set of 
\begin_inset Formula $b_{n}$
\end_inset

 that are either zero or one, indicating the membership of a word 
\begin_inset Formula $w_{n}$
\end_inset

 in some class of words.
 (Note, by the way, that a word might belong to multiple classes, for example,
 according to its part-of-speech, or it's meaning.) This suggests a neural-netish
 variant on iterative PCA, described below.
 But, before giving this, some general remarks.
\end_layout

\begin_layout Subsection*
Preliminary comments
\end_layout

\begin_layout Standard
The definition of PCA requires a matrix 
\begin_inset Formula $X$
\end_inset

 that connects columns and rows in some way.
 In the conventional definition, it is a matrix connecting variables and
 measurements.
 The variables (the features being measured) are organized in the columns;
 the measurements in rows.
 The PCA algorithm effectively computes the eigenvectors of the matrix 
\begin_inset Formula $X^{T}X$
\end_inset

, with 
\begin_inset Formula $X^{T}$
\end_inset

 denoting the transpose of 
\begin_inset Formula $X$
\end_inset

.
\end_layout

\begin_layout Standard
The matrix 
\begin_inset Formula $X^{T}X$
\end_inset

 is proportional to the covariance matrix between the different features
 being observed.
 The principal component is the the direction of the greatest variation
 in this matrix.
 
\end_layout

\begin_layout Standard
What plays the role of 
\begin_inset Formula $X$
\end_inset

 in the current situation, and how should the principal component be understood
 and interpreted? 
\end_layout

\begin_layout Standard
What we have on hand, foundationally, is the frequency matrix 
\begin_inset Formula $P$
\end_inset

 with components 
\begin_inset Formula $p(w,d)$
\end_inset

 connecting words with disjuncts.
 It was defined previously as 
\begin_inset Formula $p(w,d)=N(w,d)/N(*,*)$
\end_inset

, and where 
\begin_inset Formula $N(w,d)$
\end_inset

 is the number of times word 
\begin_inset Formula $w$
\end_inset

 has been observed with disjunct 
\begin_inset Formula $d$
\end_inset

.
 As noted earlier, 
\begin_inset Formula $N(w,d)$
\end_inset

 is very large and very sparse: typically 
\begin_inset Formula $200K\times4M$
\end_inset

 in recent datasets, with only 1 entry out of 
\begin_inset Formula $2^{15}$
\end_inset

 being non-zero.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
I plan to send out the revised, expanded statistical analysis 
\begin_inset Quotes eld
\end_inset

real soon now
\begin_inset Quotes erd
\end_inset

.
\end_layout

\end_inset

 The extreme sparsity indicates that a power-iteration algorithm will be
 the most efficient for implementing a PCA algorithm.
\end_layout

\begin_layout Standard
What we will examine will be the results on several different kinds of matrices
 
\begin_inset Formula $M$
\end_inset

 derived from (constructed from) the base data matrix 
\begin_inset Formula $P$
\end_inset

.
 In all cases, the features are words, and so in all cases, it is appropriate
 to write 
\begin_inset Formula $X=M^{T}$
\end_inset

; that is, we work mostly with the transpose of the matrix 
\begin_inset Formula $X$
\end_inset

 as usually given in standard texts.
 This follows from the standard Link Grammar dictionary: the word is followed
 by the disjuncts it can employ.
\end_layout

\begin_layout Subsection*
PCA of the frequency matrix
\end_layout

\begin_layout Standard
Should we identify 
\begin_inset Formula $P$
\end_inset

 and 
\begin_inset Formula $X^{T}$
\end_inset

, so that 
\begin_inset Formula $X^{T}X=PP^{T}$
\end_inset

? We can, but then we don't get what we want.
 Computing the principal component of this matrix for a recent dataset (see
 later section below), we get the following vector shown below.
 The 
\begin_inset Quotes eld
\end_inset

weight
\begin_inset Quotes erd
\end_inset

 gives the magnitude of the vector component.
 The other two columns are the support for the word, and the number of observati
owns, and are shown for comparison.
 
\end_layout

\begin_layout Standard
XXXX the table below is still from the broken corpus!!!! discard it!!!
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="16" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
weight
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $N(w,*)$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.9358
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2031
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
341112
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.1212
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1403
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
106378
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.1098
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1225
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
96276
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.1035
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1276
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
96308
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
”
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0920
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
506
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
45809
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
,
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0904
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1703
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
111982
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0842
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
958
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
73760
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
in
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0808
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
750
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
56751
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
of
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0783
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
890
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
64753
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
his
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0666
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
691
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
48728
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
it
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0567
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
606
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
44211
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
with
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0531
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
480
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33681
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
him
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0482
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
425
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30345
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
that
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0464
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
729
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
49714
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
for
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0450
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
479
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33092
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
What does this mean? What can we do with this? Why is the weight of the
 period so high? In essence, this vector is stating that the greatest variety
 of disjuncts are associated with the period.
 Since periods are sentence enders, and every sentence has one, a link to
 the end of the sentence will attach to just about any word.
 That is, the period almost single-handedly accounts for almost all of the
 variance of the disjuncts in the dataset.
 The rest of the list is filled out with words that also attach freely and
 easily to just about anything: 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 should attach only to nouns, but common nouns wildly outnumber all of the
 other parts of speech put together.
 Similar remarks for 
\begin_inset Quotes eld
\end_inset

and
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

to
\begin_inset Quotes erd
\end_inset

.
 The comma can connect in a large variety of situations, and the closing
 quotation mark ” behaves much like a sentence-ender (This particular dataset
 contains a lot of dialog).
 The two columns labeled as 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

 and 
\begin_inset Formula $N(w,*)$
\end_inset

 confirms this interpretation: so, 
\begin_inset Formula $\left|(w,*)\right|$
\end_inset

 is the total number of unique, different disjuncts that were observed with
 
\begin_inset Formula $w$
\end_inset

, and 
\begin_inset Formula $N(w,*)$
\end_inset

 is the summation over all of the counts with which these disjuncts were
 seen.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
If these numbers seem small, it is because they were taken from a sharply
 filtered dataset, the en_pairs_ttwo_mst dataset with the (50,30,10) cut
 applied.
 This cut is discussed later, below.
\end_layout

\end_inset

 The top words, in terms of the variety and number of disjuncts, are more
 or less the makeup of the principal component of 
\begin_inset Formula $PP^{T}$
\end_inset

.
 This should not be a surprise.
 Anyway, this is not what we wanted: we want to classify sets of similar
 words; discovering which words account for the greatest variation in disjuncts
 is of secondary interest.
 
\end_layout

\begin_layout Subsection*
PCA of the cosine similarity
\end_layout

\begin_layout Standard
We had previously defined the cosine similarity of two words as
\begin_inset Formula 
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}p(w_{1},d)p(w_{2},d)}{\sqrt{\sum_{d}p^{2}(w_{1},d)}\sqrt{\sum_{d}p^{2}(w_{2},d)}}
\]

\end_inset

and so, perhaps we should use this as the basis for judging the similarity
 of words.
 This suggests defining a matrix 
\begin_inset Formula $S$
\end_inset

 with matrix components 
\begin_inset Formula 
\[
S(w,d)=\frac{p(w,d)}{\sqrt{\sum_{d}p^{2}(w,d)}}
\]

\end_inset

and then setting setting 
\begin_inset Formula $X=S^{T}$
\end_inset

 so that 
\begin_inset Formula $X^{T}X=SS^{T}$
\end_inset

.
 The idea here is that PCA allows a whole-set analysis of similarity, rather
 than point-wise similarity.
 That is, for normal clustering algorithms, one computes a large number
 of values for 
\begin_inset Formula $\mbox{sim}(w_{1},w_{2})$
\end_inset

 and then employs a clustering algorithm to categorize these, word by word.
 Here, instead, perhaps PCA can reveal entire clusters in one gulp, by simultane
ously evaluating the similarity between all words in a cluster.
\end_layout

\begin_layout Standard
Power iteration converges at about half of the rate as for the frequency
 matrix, which is not a surprise, as the off-diagonal entries are closer
 to one-another.
 The PCA vector, however, is not all that different: 0.978 
\begin_inset Quotes erd
\end_inset

.
\begin_inset Quotes erd
\end_inset

 + 0.137 
\begin_inset Quotes eld
\end_inset

,
\begin_inset Quotes erd
\end_inset

 + 0.080 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 + 0.068 
\begin_inset Quotes eld
\end_inset

to
\begin_inset Quotes erd
\end_inset

 + 0.067 
\begin_inset Quotes eld
\end_inset

and
\begin_inset Quotes erd
\end_inset

 + 0.039 
\begin_inset Quotes eld
\end_inset

a
\begin_inset Quotes erd
\end_inset

 + ...
 and so on, the remaining entries filled out in roughly the same order,
 by the same words, as in the frequency PCA.
\begin_inset Foot
status open

\begin_layout Plain Layout
Created as follows:
\end_layout

\begin_layout Plain Layout
(define fsi (add-subtotal-filter psa 50 30 10)) 
\end_layout

\begin_layout Plain Layout
(define pci (make-cosine-matrix fsi))
\end_layout

\begin_layout Plain Layout
(define pti (make-power-iter-pca pci 'left-unit))
\end_layout

\begin_layout Plain Layout
(define lit (pti 'left-iterate feig 8)) 
\end_layout

\begin_layout Plain Layout
(pti 'left-print lit 20) 
\end_layout

\end_inset

 Why is this? It's worth taking a look at the matrix:
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Created with 
\end_layout

\begin_layout Plain Layout
(define poi (add-pair-cosine-compute fsi)) 
\end_layout

\begin_layout Plain Layout
(poi 'right-cosine WORD-A WORD-B) 
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="7" columns="7">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
,
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.549
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.731
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.6435
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.668
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.627
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
,
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.711
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.824
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.888
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.765
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.790
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.744
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.896
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.906
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.857
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.755
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
a
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
So what is this saying? There are plenty of pairs that have greater similarity;
 here's an arbitrary sampling: 
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="9" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
pair
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
sim
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(he, she)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.982
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(this, that)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.894
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(run, walk)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.878
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(big, small)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.908
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(high, low)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.910
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(soft,hard)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.809
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(easy,hard)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.846
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(easy,soft)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.749
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
So why aren't some of these in the PCA vector?
\end_layout

\begin_layout Subsubsection*
IMPORTANT: 
\end_layout

\begin_layout Standard
Some readers have misunderstood this section.
 We are NOT doing PCA to obtain similarity! We are examining it as an algo
 for CLUSTERING! That is, instead of doing k-means clustering, or agglomerative
 clustering, or something like that, the idea is/was to use a thresholded
 PCA for CLUSTERING! and NOT for similarity, because we've already got reasonabl
e adequate similarity.
 Higher-quality similarity might be nice, but that is of secondary importance,
 right now.
\end_layout

\begin_layout Subsection*
A bit of sheaf theory
\end_layout

\begin_layout Standard
I recently realized that much of what is being discussed here can be anchored
 in the vocabulary of a generic mathematical theory, namely, sheaf theory.
 Sheaves allow topological structure to be discussed in a local way: sheaves
 describe how the local neighborhoods of a point glue together, to form
 a manifold as a whole.
 Link Grammar disjuncts and connector sets are really just the stalks and
 germs of sheaf theory, in mild disguise.
 This can be seen as follows.
\end_layout

\begin_layout Standard
A standard way of expressing a graph is to list all of the vertices in the
 graph, and to list all of the edges.
 Knowing these, one knows the graph.
 However, this is a global description: One does not know the local structure
 until one looks at specific vertices, and what they attach to.
\end_layout

\begin_layout Standard
A different way of describing a graph is to make a list of pairs: a vertex
 
\begin_inset Formula $v$
\end_inset

 and all the edges that attach to it.
 More generally, one can consider pairs where a vertex 
\begin_inset Formula $v$
\end_inset

 is attached to a vertex 
\begin_inset Formula $w$
\end_inset

 by means of a path of length 
\begin_inset Formula $N$
\end_inset

 or less.
\begin_inset Formula 
\[
\left(\mbox{vertex }v,\left\{ w\mbox{ s.t. vertex }w\mbox{ is attached to }v\right\} \right)
\]

\end_inset

This describes the graph, as a whole, just as well as the simpler vertex+edge
 list does.
 However, the language is different: these pairs are presheaves, obeying
 all the axioms of a presheaf, e.g.
 the composition of restriction morphisms.
 They become a sheaf because they also obey the gluing or collation axiom
 as well: they can be glued together to form the original graph from which
 they were taken.
\end_layout

\begin_layout Standard
Thus, we can see that the set
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
\left(\mbox{vertex }v,\left\{ \mbox{ edges attached to }v\right\} \right)
\]

\end_inset

is the same thing as a Link Grammar dictionary entry.
\end_layout

\begin_layout Standard
To be more precise, we need to distinguish the graph-sheaf that arises for
 a single sentence (from the dependency parse of the sentence) from the
 sheaf that arises from the entire language.
 If we take the language to consist of the set of all possible sentences,
 then the sheafification is to parse each of the sentences in the language,
 to get a dependency graph for each sentence, create the individual (word,
 connector-set) lists, and then take the quotient, identifying together
 all words that have the same spelling.
 This gives the sheaf of the entire language.
\end_layout

\begin_layout Standard
From what I can tell, this realization that language can be sheafified is
 not new; when the language is not a natural language, but is instead first-orde
r logic, then it's sheafification gives the Kripke–Joyal semantics.
 According to Wikipedia, this was noted in 1965 for existential quantification.
 I don't know if this was ever noted for natural language before, but, as
 I've blathered on the mailing list before, this provides the 
\begin_inset Quotes eld
\end_inset

answer
\begin_inset Quotes erd
\end_inset

 to why the logic of Link Grammar appears to be modal logic: Link Grammar
 dictionary entries are sheaves, and the disjuncts are the different 
\begin_inset Quotes eld
\end_inset

possible worlds
\begin_inset Quotes erd
\end_inset

 that a given word can inhabit.
 For a natural-language sentence, 
\begin_inset Quotes eld
\end_inset

there exists
\begin_inset Quotes erd
\end_inset

 (existential quantification) a collection of disjuncts that can parse the
 sentence.
 Bingo.
\end_layout

\begin_layout Standard
I used to say that LG disjuncts had something to do with linear logic, because
 linear logic also has the general whiff of 
\begin_inset Quotes eld
\end_inset

possible worlds
\begin_inset Quotes erd
\end_inset

 around it.
 I now see that in fact its actually modal logic, and it is the language
 of sheaves that provides the direct route from Link Grammar, to modal logic.
 It would be very interesting to see all the details worked out.
\end_layout

\begin_layout Standard
Most interesting is perhaps this: the sentences of a language are observed
 with some a priori frequency or probability.
 What's the correct way of converting this to a probability distribution
 on the sheaf? Next, given a probability distribution on the sheaf, what
 is the corresponding probability distribution on the corresponding modal
 logic? 
\end_layout

\begin_layout Standard
It seems to me that one could make this very generic: every language, and
 not just first-order logic, but any language, as considered in model theory,
 has a set of sentences.
 This sentences are composed of the terms in their term algebra, and these
 terms and how they connect, define a graph.
 That graph can be viewed in terms of sheaves, germs, stalks, etale spaces.
 This implies that every model, of model theory, has a corresponding cohomology.
 Writing this out could be interesting.
 Perhaps this has already been done; perhaps this is what topos theory is.
 But I suspect that it's not been sufficiently popularized: certainly, the
 standard computer-science textbooks that tell you what a language is do
 not tell you that it has a cohomology associated with it.
 And yet, this seems blatantly obvious, in retrospect, and naggingly it
 might actually be important for some reason or another.
\end_layout

\begin_layout Standard
Anyway: this is not just all-talk, no-action.
 I've written some code that implements some sheaf-based parsing on the
 atomspace.
 It is in the 
\begin_inset CommandInset href
LatexCommand href
name "github atomspace repo"
target "https://github.com/opencog/atomspace/tree/master/opencog/sheaf"
literal "false"

\end_inset

.
 The README file there explains more.
\end_layout

\begin_layout Subsection*
Disjuncts are compositional
\end_layout

\begin_layout Standard
One reason that a disjunct representation of a graph is important is that
 disjuncts can be composed, so that the product is again a disjunct.
 This is in contrast to the vertex-edge model.
 where neither vertices nor edges can be composed to obtain a new vertex
 or edge.
 That is, disjuncts form a monoidal category, and, specifically, a compact-close
d category.
 This section tries to spell out very clearly what this means, if it is
 not already apparent.
 
\end_layout

\begin_layout Standard
So, for example: the Link Grammar parse for 
\begin_inset Quotes eld
\end_inset

this is a test
\begin_inset Quotes erd
\end_inset

 involves four disjuncts:
\end_layout

\begin_layout Itemize
this: S+
\end_layout

\begin_layout Itemize
is: S- & O+
\end_layout

\begin_layout Itemize
a: D+
\end_layout

\begin_layout Itemize
test: D- & O-
\end_layout

\begin_layout Standard
The determiner connectors D+ and D- can be composed to form a determiner
 D link, leaving a phrase that is still describable by a disjunct, a single
 object O- connector that can attach to verbs: 
\end_layout

\begin_layout Itemize
\begin_inset Quotes eld
\end_inset

a test
\begin_inset Quotes erd
\end_inset

: O-
\end_layout

\begin_layout Standard
This has only one connector, but it is a perfectly ordinary connector, not
 differing from that which might be found on a single word.
 That is, Link Grammar makes no particular distinction between words and
 word-phrases.
 Using the same argument, it is why Link Grammar can work for morpho-syntax.
 One can continue composing:
\end_layout

\begin_layout Itemize
\begin_inset Quotes eld
\end_inset

is a test
\begin_inset Quotes erd
\end_inset

: S-
\end_layout

\begin_layout Standard
which has a subject connector S- that can connect to any subject.
 One can also, perhaps foolishly, perform some net-very-sensical disjuncts:
\end_layout

\begin_layout Itemize
\begin_inset Quotes eld
\end_inset

is a
\begin_inset Quotes erd
\end_inset

: S- & O+ & D+
\end_layout

\begin_layout Standard
or
\end_layout

\begin_layout Itemize
\begin_inset Quotes eld
\end_inset

this ...
 a
\begin_inset Quotes erd
\end_inset

: S+ & D+
\end_layout

\begin_layout Standard
This last has to use elipses as an awkward notation to indicate the projectivity
 constraint.
 Projectivity can be discard, provided some other means ensures a tight
 parse.
\end_layout

\begin_layout Standard
The point here is that the category of disjuncts can be taken to be a monoidal
 category, i.e.
 a category with a tensor product 
\begin_inset Formula $\otimes$
\end_inset

, with tensoring simply being the writing of two disjuncts next to each
 other.
 As the first three examples illustrate, the typical usage is not only to
 tensor together two disjuncts, but also to contract some of the connectors,
 as well.
 
\end_layout

\begin_layout Standard
The contractability of connectors into links means that the Link Grammar
 forms a compact-closed category.
 I've been through this one too many times, so won't try to sketch this
 here.
 It is a good homework exercise for novices.
 
\end_layout

\begin_layout Standard
Bob Coecke has written repeatedly on this topic, any one of his papers on
 pregroup grammars or closed monoidal categories applied to linguistics
 is adequate to grasp the concept.
 His notation is easily and readily translated into Link Grammar notation.
 The primary insigh is to understand that the Link Grammar connector letters
 should be understood as type labels: they provide a simple, easy notational
 device, overcoming the notational complexity that is otherwise required
 when presenting categorial grammars.
\end_layout

\begin_layout Standard
What's the point of all this? Well categorical grammars are all the rage,
 and the fact that LG is a categorical grammar seems to be frequently overlooked
 or misunderstood.
\end_layout

\begin_layout Subsection*
Conclusions
\end_layout

\begin_layout Standard
Conclusions from the above:
\end_layout

\begin_layout Itemize
Pair-wise similarity is very promising.
\end_layout

\begin_layout Itemize
The cosine similarity measure penalizes better, more accurate measurements,
 because better measurements are more likely to find dissimilarity.
 We need a better measure.
\end_layout

\begin_layout Itemize
PCA and sparse PCA, in the naive sense, applied to frequencies, or to cosine
 similarities, are inappropriate for classification.
 Its still possible that perhaps PCA applied to some sigmoid of the cosine
 similarity (e.g.
 cosine to the fourth power) might work better, but the selection of this
 sigmoid seems ad-hoc, and not anchored in any principles.
\end_layout

\begin_layout Itemize
First principles suggest something Bayesian, based on the Gibbs measure,
 maybe some sort of hidden multi-variate logistic regression.
 Hidden, because we don't know the grammatical categories in any a priori
 sense; we must deduce them.
 
\end_layout

\begin_layout Itemize
It would be great if someone worked out the precise details in going from
 sheaves to modal logic.
 This was already done, in 1965, for topos theory; no one has done this
 for natural language, though.
\end_layout

\begin_layout Subsection*
Other similarity measures
\end_layout

\begin_layout Standard
Its worth noting that one of the other similarity measures, such as qim
 and pim, discussed previously, can also be treated in this way.
 Note also that a matrix can be constructed so that 
\begin_inset Formula $X^{T}X$
\end_inset

 becomes explicitly Markovian.
 This is given by 
\begin_inset Formula 
\[
A(w,d)=\frac{p(w,d)}{p(*,d)}\quad\mbox{ and }\quad B(d,w)=\frac{p(w,d)}{p(w,*)}
\]

\end_inset

and then setting 
\begin_inset Formula $X^{T}X=AB$
\end_inset

.
 This has the property that 
\begin_inset Formula 
\[
\sum_{w_{1}}\left[AB\right](w_{1},w_{2})=1
\]

\end_inset

That is, the matrix 
\begin_inset Formula $AB$
\end_inset

 is a Markov matrix.
 It is straight-forward to compute using the standard power-iteration algorithm
 employed throughout this section.
 
\end_layout

\begin_layout Subsection*
A modified PCA algorithm
\end_layout

\begin_layout Standard
This suggests a feed-foreward-neural-netish variant on iterative PCA, described
 below.
 I is entirely of my own design, cribbed from nowhere at all, just popped
 into my head as I sit still immobilized.
\end_layout

\begin_layout Standard
\paragraph_spacing double
\noindent
\begin_inset space \hspace{}
\length 1.3em
\end_inset

0.
 
\begin_inset space \space{}
\end_inset

Pre-condition, filter the data.
 See step 10, below, for what to filter, and why.
\end_layout

\begin_layout Enumerate
Start with 
\begin_inset Formula $b_{n}=1/\sqrt{\left|w\right|}$
\end_inset

 where 
\begin_inset Formula $\left|w\right|$
\end_inset

 is the number of unique words.
 This starting point is a unit-length vector, i.e.
 
\begin_inset Formula $\left|\overrightarrow{b}\right|=1$
\end_inset

.
 Its convenient to change notation, here, and write 
\begin_inset Formula $b(w)$
\end_inset

 for the value of 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

 at word 
\begin_inset Formula $w$
\end_inset

.
 That is, 
\begin_inset Formula $b(w_{n})=b_{n}$
\end_inset

 is the same thing.
 
\end_layout

\begin_layout Enumerate
Let 
\begin_inset Formula $M$
\end_inset

 be the matrix for which the PCA is to be computed, with matrix components
 
\begin_inset Formula $M(w,d)$
\end_inset

 for word 
\begin_inset Formula $w$
\end_inset

 and disjunct 
\begin_inset Formula $d$
\end_inset

.
 This matrix is derived from (defined in terms of) the frequency matrix
 
\begin_inset Formula $p(w,d)$
\end_inset

 describing the base dataset.
 Compute the double-sum 
\begin_inset Formula 
\[
s(v)=\left[MM^{T}b\right](v)=\sum_{d}M(v,d)\sum_{w}M(w,d)b(w)
\]

\end_inset

which is basically a pair of dot products.
 Its still a large, time-consuming computation, even for sparse vectors.
\end_layout

\begin_layout Enumerate
Normalize: set 
\begin_inset Formula $\overrightarrow{b}\leftarrow\overrightarrow{s}/\left|\overrightarrow{s}\right|$
\end_inset

 so that 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

 is of unit length.
 In theory, this is not needed; in practice, each iteration can sharply
 shrink the value of 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

, making it very small, eventually leading to exponent underflow.
 
\end_layout

\begin_layout Enumerate
Repeat these steps 
\begin_inset Formula $k$
\end_inset

 times: go to step 2 and run the summation again.
 The repetition here is the 'power iteration' or the 'von Mises iteration'
 method for computing the largest eigenvalue of 
\begin_inset Formula $\left[MM^{T}\right]$
\end_inset

.
 It is not guaranteed to converge, and if it does, it might not do so quickly.
 But we deal with this in the next step, so its sufficient to keep 
\begin_inset Formula $k$
\end_inset

 small, just enough to get a trend going.
 Another way to think of this is as a Markov process (specifically, a Markov
 chain).
 That is, the matrix 
\begin_inset Formula $\left[MM^{T}\right]$
\end_inset

 will behave essentially as a Markov chain, and iteration on it just identifies
 the primary Peron-Frobenius stable state (step 3 makes it Markovian, by
 preserving to total probability measure).
 That is, 
\begin_inset Formula $\left[MM^{T}\right]$
\end_inset

 defines a weighted adjacency matrix for a graph, and iteration creates
 a measure-preserving process (walk) on this graph.
\end_layout

\begin_layout Enumerate
After the above repetitions, apply some standard neural-net sigmoid function
 to 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

.
 That is, set 
\begin_inset Formula $b(w)\leftarrow\sigma(b(w))$
\end_inset

 for some sigmoid.
 This has the effect of driving some of the elements to zero, and others
 to one.
\end_layout

\begin_layout Enumerate
Repeat this 
\begin_inset Formula $m$
\end_inset

 times: go to step 2, and repeat steps 2-5.
 Viewing this as a dynamical system, the effect of the sigmoid function
 is to force the system into a block-diagonal form, with the vector 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

 identifying a highly-connected block.
 Another way to look at this is as a graph factorization algorithm: the
 vector 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

 is identifying a well-connected subgraph, which is only weakly connected
 to the rest of the graph.
 The vector (viewed as a measure-preserving dynamical system) is spending
 most of its time in one particular block.
 Again, 
\begin_inset Formula $\left[MM^{T}\right]^{k}$
\end_inset

, the 
\begin_inset Formula $k$
\end_inset

-th power iterated matrix from step 4, can be thought of as a surrogate
 for a weighted graph adjacency matrix.
 A third way of thinking of this is as an 
\begin_inset Formula $m$
\end_inset

-layer neural net, with the link weights between one layer and the next
 being given by 
\begin_inset Formula $\left[MM^{T}\right]^{k}$
\end_inset

.
 All three ways of looking at this are essentially equivalent: a measure-preserv
ing dynamical system, a chaotic and mixing process on a graph, or as an
 
\begin_inset Formula $m$
\end_inset

-layer neural net.
 Pick your favorite.
\end_layout

\begin_layout Enumerate
Classify.
 Pass the vector 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

 through the step function, i.e.
 
\begin_inset Formula $b(w)\leftarrow\Theta(b(w))$
\end_inset

 where 
\begin_inset Formula $\Theta(x)=0$
\end_inset

 if 
\begin_inset Formula $x<1/2$
\end_inset

 and 
\begin_inset Formula $\Theta(x)=1$
\end_inset

 if 
\begin_inset Formula $x>1/2$
\end_inset

.
 The step function is a super-sharp sigmoid.
 This step identifies and isolates an active, well-connected subgraph of
 
\begin_inset Formula $\left[MM^{T}\right]$
\end_inset

.
 It identifies a square block, of dimension 
\begin_inset Formula $\left|b\right|\times\left|b\right|$
\end_inset

 where 
\begin_inset Formula $\left|b\right|$
\end_inset

 is the total number of non-zero entries in this final 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

.
 To belabor the point: the block-matrix is explicitly 
\begin_inset Formula 
\[
B(v,w)=b(v)b(w)\sum_{d}M(v,d)M(w,d)
\]

\end_inset

The non-zero elements of this final 
\begin_inset Formula $\overrightarrow{b}$
\end_inset

 identify a class of words that can be considered to be grammatically similar
 or identical.
 This is the 
\begin_inset Quotes eld
\end_inset

clustering
\begin_inset Quotes erd
\end_inset

 step.
\end_layout

\begin_layout Enumerate
Associated with this class of words is a disjunct set, the 
\begin_inset Quotes eld
\end_inset

average disjunct
\begin_inset Quotes erd
\end_inset

 for the class.
 It can be taken to be the set 
\begin_inset Formula $\left\{ d|0<\sum_{w}b(w)N(w,d)\right\} $
\end_inset

.
 The observed counts associated with this set can be taken to be 
\begin_inset Formula $N(b,d)=\sum_{w}b(w)N(w,d)$
\end_inset

 and the frequencies similarly: 
\begin_inset Formula $p(b,d)=\sum_{w}b(w)p(w,d)$
\end_inset

.
 From here-on, the set of words 
\begin_inset Formula $b\equiv\{w|0\ne b(w)\}$
\end_inset

 can be treated as if it was an ordinary word, behaving like any other,
 with the indicated disjuncts, counts and frequencies.
\end_layout

\begin_layout Enumerate
Since words can have have multiple meanings, or rather, multiple different
 kinds of grammatical behaviors based on their part of speech, the identified
 words need to be subtracted, en block, from the matrix 
\begin_inset Formula $p(w,d)$
\end_inset

, and then the process repeated, to identify another class of words.
 Put another way, if 
\begin_inset Formula $b$
\end_inset

 is to be added to the set of words, as 
\begin_inset Quotes eld
\end_inset

just another word
\begin_inset Quotes erd
\end_inset

, then the frequencies 
\begin_inset Formula $p(b,d)$
\end_inset

 have to be subtracted from the matrix 
\begin_inset Formula $P$
\end_inset

, and shunted to this new 
\begin_inset Quotes eld
\end_inset

word
\begin_inset Quotes erd
\end_inset

, so as not to loose the overall normalization.
 That is, one must preserve the identity 
\begin_inset Formula $\sum_{w,d}p(w,d)=1$
\end_inset

.
 So define, in the next iteration 
\begin_inset Formula 
\[
p(w,d)\leftarrow\begin{cases}
p(b,d) & \mbox{ if }w=b\\
p(w,d)-b(w)p(b,d) & \mbox{ otherwise}
\end{cases}
\]

\end_inset

(Hmmm.
 This may not be right, its late and I'm tired).
 This still sums to the identity except that now some of the values might
 go negative, and we don't want that.
 
\end_layout

\begin_layout Enumerate
And so we get to what should be called step zero: We want to truncate, and
 discard the negative entries.
 This should have been carried out as an actual step 0: a pre-conditioning
 of the matrix: some noise filtering, e.g.
 discarding all words that were observed less than a handful of times, discardin
g rare or preposterous disjuncts.
 Pre-conditioning in this way will have the effect of removing some (possibly
 many) of the words from the matrix: the size of the matrix shrinks.
 This is the step where the actual dimensional reduction takes place: the
 size of the set of words is shrinking, as they get classified into sets.
 
\end_layout

\begin_layout Enumerate
Go to step 0 and repeat, until the preconditioning and noise-removal has
 left behind an empty matrix (or alternately, a matrix where all words have
 been classified into some group).
 So, for example, words which have only one part-of-speech or meaning would
 (hopefully should) get classified after just one step; words that are more
 complex, and have two parts of speech, would require at least two iterations.
 This is perhaps optimistic; I expect dozens of iterations to get anything
 vaguely accurate.
\end_layout

\begin_layout Enumerate
There's one more step.
 After the formation of the class 
\begin_inset Formula $b$
\end_inset

, we arrive at a situation where no (pseudo-)connectors connect to 
\begin_inset Formula $b$
\end_inset

 directly.
 Instead, all disjuncts connect to words inside of 
\begin_inset Formula $b$
\end_inset

.
 But this is a problem: we don't know if any given connector actually connects
 to some 
\begin_inset Formula $w\in b$
\end_inset

 or if it connects to the same 
\begin_inset Formula $w$
\end_inset

, but outside of 
\begin_inset Formula $b$
\end_inset

.
 (e.g.
 if 
\begin_inset Formula $b$
\end_inset

 are nouns, then does 
\begin_inset Quotes eld
\end_inset

saw+
\begin_inset Quotes erd
\end_inset

 connect to 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

 the noun, or 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

 the verb?) Thus, after some small number of iterations of step 11, there
 needs to be a re-parse of the entire text, using these newly discovered
 classes of words.
 
\end_layout

\begin_layout Standard
That's it.
 I think this should work fairly well.
 Clearly, there are many nested loops, and so this is potentially a very
 time-consuming computation.
 The number of iterations 
\begin_inset Formula $k$
\end_inset

 and 
\begin_inset Formula $m$
\end_inset

 need to be kept small, and the classification in step 11 needs to be kept
 greedy, because step 12 is expensive.
 An alternate strategy is to brutally precondition 
\begin_inset Formula $p(w,d)$
\end_inset

 to make it as small as possible; but this risks throwing out the baby with
 the bathwater: early on, we want to cluster together the rare, obscure,
 unused words as best as possible into large bins, and then devote large
 CPU resources to correctly classifying the remaining much smaller set of
 verbs and prepositions, which we know, 
\emph on
a priori,
\emph default
 to be complex and difficult, due to their grammatical variability.
\end_layout

\begin_layout Subsection*
Dataset
\end_layout

\begin_layout Standard
The previous dataset 
\noun on
en_pairs_sim
\noun default
, analyzed above, proved to be inadequate in many respects.
 Thus, data analysis here resumes with a different, considerably larger
 dataset, collected on a higher-quality corpus.
 This will be the 
\noun on
en_pairs_ttwo_mst
\noun default
 dataset, listed above.
 To recap, it is this one:
\begin_inset Newline newline
\end_inset


\end_layout

\begin_layout Standard
\begin_inset Tabular
<lyxtabular version="3" rows="2" columns="9">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Size
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Csets
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Obs'ns
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Ob/cs
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Entropy
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $H_{left}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $H_{right}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
MI
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Notes
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
176K x 3.4M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6.43M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14.3M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.23
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21.01
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14.91
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10.01
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-3.91
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_pairs_ttwo_mst
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\begin_inset Newline newline
\end_inset

To avoid accidental corruption of this dataset, a copy was made, in which
 assorted sporadic results are maintained.
 The copy is the 
\noun on
en_pairs_ttwo_sim
\noun default
 dataset.
\end_layout

\begin_layout Subsection*
Filtering, Step 0
\end_layout

\begin_layout Standard
The filtering performed in step 0 (described in step 10, above) removes
 some of the noise in the dataset.
 Basic filtering is implemented in the 
\noun on
(opencog analysis)
\noun default
 scheme module, and specifically in the 
\noun on
filter.scm
\noun default
 file.
 One can remove rows and columns that have subtotal counts less than a cutoff,
 and also remove individual entries that have fewer than some number of
 counts.
 By removing very infrequently observed connector sets, some of drivers
 of accidental similarity or dis-similarity between words should be ameliorated.
\end_layout

\begin_layout Standard
How much data does filtering actually discard? This dataset has 175559 rows.
 Each row corresponds to one unique, distinct word (columns correspond to
 disjuncts).
 Of these words, only 84984 were observed twice, or more: slightly less
 than half! Only 64882 words were seen three times or more; only 10% of
 the words were seen 32 times or more.
 The distribution is shown in the graph below.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
From the cnt-obs-rows function.
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/count-rows.eps
	width 70col%

\end_inset


\end_layout

\begin_layout Standard
The fraction of rows with more than 
\begin_inset Formula $N$
\end_inset

 observations drops a little faster than 
\begin_inset Formula $\sqrt{N}$
\end_inset

.
 Note that this graph is not scale-free; for larger datasets, the graph
 should progressively flatten.
 Since cumulative distributions are integrals of distributions, this is
 essentially the integral of some of the graphs shown before.
 A table of plausible cutoffs to use with this dataset is given below.
 There are three cuts one can make: discard words that are observed 
\begin_inset Formula $N$
\end_inset

 or fewer times; discard disjuncts that were observed 
\begin_inset Formula $N$
\end_inset

 or fewer times, and discard connector-sets (word-disjunct pairs) that were
 observed 
\begin_inset Formula $N$
\end_inset

 or fewer times.
 These three cuts are given in the first three columns; the resulting dataset
 is given in the remaining columns.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Stats can be gotten by creating the add-support-compute object on the filter
 object, and then invoking 'left-basis-size, 'right-basis-size, 'total-support
 and 'total-count.
 
\end_layout

\end_inset


\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="7">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Word cut
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Dj cut
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Cset cut
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Size
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Csets
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Obs'ns
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Ob/cs
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
176K x 3.4M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6.43M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14.3M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.23
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
85K x 1.15M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.10M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12.0M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.94
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
49K x 145K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.81M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9.51M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.39
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32.2K x 40.3K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.42M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8.73M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.60
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17.9K x 14.4K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.98M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8.05M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.07
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17.9K x 40.3K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.26M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8.48M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.75
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17.9K x 40.3K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
269K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.52M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20.5
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
50
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
13.1K x 14.4K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.88M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.86M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.19
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
50
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
13.1K x 14.4K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
256K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.41M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21.1
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
50
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
13.1K x 14.4K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
101K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.36M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
43.4
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The last cut seems plausible for further work: it suggests that each disjunct
 is observed a fairly strong number of times; and that given the word/disjunct
 ratio, a lot of words are using disjuncts in similar fashion; thus, there
 should be a lot of similarity.
\end_layout

\begin_layout Standard
Note, by the way, that the previous sections carefully described entropy
 and mutual information distributions that no longer hold for the cut dataset.
 Filtering changes these!
\end_layout

\begin_layout Subsection*
Power iteration, Steps 1-4
\end_layout

\begin_layout Standard
Step 1-4 are implemented in the 
\noun on
(opencog analysis)
\noun default
 scheme module, and specifically in the 
\noun on
thresh-pca.scm
\noun default
 file.
 The implementation uses lazy evaluation to avoid unneeded computation,
 and caching of evaluation results to avoid repeated evaluations.
 This seems like the best way of working with the extremely sparse matrices
 involved.
\end_layout

\begin_layout Standard
Iteration appears to converge very rapidly.
 After three iterations, the ranking, by weight in the vector, appears to
 be established.
 This is shown in the figures below.
 Six iterations are performed, and the words 
\begin_inset Formula $w$
\end_inset

 are then ranked according to the strength 
\begin_inset Formula $b(w)$
\end_inset

 in the sixth iteration.
 Then the values of 
\begin_inset Formula $b(w)$
\end_inset

 are plotted for the first five iterations, using this rank.
 After the third iteration, there is no discernible change in the weights.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/power-iter-all.eps
	width 45col%

\end_inset


\begin_inset Graphics
	filename images/power-iter-close.eps
	width 45col%

\end_inset


\end_layout

\begin_layout Standard
In this case, the principle component is revealed to be
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Computed as follows (simplified, various checks were done to verify correctness)
: 
\end_layout

\begin_layout Plain Layout
(define psa (make-pseudo-cset-api))
\end_layout

\begin_layout Plain Layout
(psa 'fetch-pairs)
\end_layout

\begin_layout Plain Layout
(define fsi (add-subtotal-filter psa 50 30 10))
\end_layout

\begin_layout Plain Layout
(define pti (make-power-iter-pca fsi))
\end_layout

\begin_layout Plain Layout
(define feig (pti 'make-left-unit (fsi 'left-basis))) 
\end_layout

\begin_layout Plain Layout
(define lit (pti 'left-iterate feig 4)) 
\end_layout

\begin_layout Plain Layout
(pti 'left-print lit 20) 
\end_layout

\end_inset


\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
word
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
weight
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
.
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.9358
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
the
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.1212
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
and
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.1098
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.1035
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
”
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0920
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
The significance and interpretation of this vector was already discussed
 in a previous section.
\end_layout

\begin_layout Section*
Reboot: 23 June 2017
\end_layout

\begin_layout Standard
The cosine similarity gives OK pair-wise similarity results; the overlap
 similarity gives noticeably worse results.
 There's no obvious matrix-based algorithm that will group together multiple
 words into a cluster, efficiently, in one shot.
 Its time for a rethink.
 
\end_layout

\begin_layout Standard
What are we trying to do here, really?
\end_layout

\begin_layout Itemize
Grammatical categories: By grouping multiple words into categories, we hope
 to discover grammatical categories of words that behave similarly, with
 respect to grammar.
\end_layout

\begin_layout Itemize
Compression: By grouping multiple words into categories, we hope to compress
 the size of the overall dataset of connector-sets, without losing much
 fidelity.
\end_layout

\begin_layout Itemize
Idioms: By observing word-disjunct pairs with high MI scores, we hope to
 discover idioms and set phrases.
 
\end_layout

\begin_layout Itemize
Meaning: By developing a coherent framework for working with graph sections
 (of which the connector sets are a special case), we hope to discover synonymou
s phrases.
\end_layout

\begin_layout Itemize
Reference resolution: By developing a coherent framework for working with
 graph sections, we hope to discover reference resolution (of pronouns and
 of given names) across multiple sentences.
\end_layout

\begin_layout Standard
These seem like they should be achievable.
 Preliminary results looking promising, but not yet great.
 What's the grand scheme of things?
\end_layout

\begin_layout Itemize
Discover that certain nouns refer to objects in the physical world.
\end_layout

\begin_layout Itemize
Discover that certain verbs refer to actions in the physical world.
\end_layout

\begin_layout Itemize
Discover that certain nouns refer to abstract, non-physical concepts.
\end_layout

\begin_layout Itemize
Discover the meaning of the verb-phrases 
\begin_inset Quotes eld
\end_inset

is-a
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

has-a
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

is-a-part-of
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

belongs-to
\begin_inset Quotes erd
\end_inset

 ...
 or, generally, discover the meanings of prepositional phrases.
 
\end_layout

\begin_layout Itemize
Perform reasoning on relationships; specifically, on 
\begin_inset Quotes eld
\end_inset

is-a
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

has-a
\begin_inset Quotes erd
\end_inset

, ...
 relationships.
\end_layout

\begin_layout Itemize
Develop a database of common-sense knowledge.
\end_layout

\begin_layout Itemize
Translate between multiple languages, by employing common-sense knowledge
 and reasoning.
\end_layout

\begin_layout Standard
The first two seem impossible without embodiment.
 The third bullet holds out hope that progress might be possible for textual-onl
y analysis.
 The fourth bullet asks for an algebraic structure to be discerned: 
\begin_inset Quotes eld
\end_inset

is-a
\begin_inset Quotes erd
\end_inset

 relations are symmetric: 
\begin_inset Quotes eld
\end_inset

A is-a B
\begin_inset Quotes erd
\end_inset

 if 
\begin_inset Quotes eld
\end_inset

B is-a A
\begin_inset Quotes erd
\end_inset

, and it should be possible to data-mine such symmetric relations.
 Likewise for 
\begin_inset Quotes eld
\end_inset

next-to
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

near
\begin_inset Quotes erd
\end_inset

 relationships.
 It is at least plausible that such relations could be data-mined, with
 no 
\emph on
a priori
\emph default
 knowledge of the words.
 Anyway, this provides the setting for the initial grammatical tasks.
 So, back to the initial grammatical tasks.
\end_layout

\begin_layout Standard
Idioms: based on preliminary evidence, we could make lists of idiomatic
 phrases, now, based only on high-MI word-disjunct relations.
 But these are useless, until we can build a list of synonymous words.
\end_layout

\begin_layout Standard
Discerning synonymous words based on grammatical usage is tricky.
 First, it is often antonyms that get observed: e.g.
 the (black,white) pairs reported above.
 To discern antonymy, we would also need to discern is-a relationships,
 apply common-sense reasoning, and notice that antonyms never describe the
 same objects, never describe the properties of the same objects.
 So, antonym detection appears to be an advanced topic.
\end_layout

\begin_layout Subsection*
Clustering
\end_layout

\begin_layout Standard
We could go full-speed ahead on trying to discern grammatical categories,
 but for several issues: 
\end_layout

\begin_layout Itemize
Merging two items into one necessarily entails a loss of information.
 That is, one necessarily has that 
\begin_inset Formula $-(p_{a}+p_{b})\log_{2}(p_{a}+p_{b})\le-p_{a}\log_{2}p_{a}-p_{b}\log_{2}p_{b}$
\end_inset

.
 That is, information is necessarily lost.
 How can we minimize information loss? 
\end_layout

\begin_layout Itemize
If a classification error is made early on, can it be spotted, and later
 corrected? What is the mechanism, and how might it work?
\end_layout

\begin_layout Standard
We can use the 
\begin_inset Quotes eld
\end_inset

information loss
\begin_inset Quotes erd
\end_inset

 to our advantage if the 
\begin_inset Quotes eld
\end_inset

lost information
\begin_inset Quotes erd
\end_inset

 is in fact just noise in the data.
 That is, the data is necessarily noisy, and the naive calculation of the
 entropy and mutual information encodes both that noise and the signal we
 are searching for.
 Clustering together items, and the associated information loss, is desirable
 if the loss results in the filtering out of noise.
 How can we characterize the noise in the observations?
\end_layout

\begin_layout Subsection*
Word Meanings
\end_layout

\begin_layout Standard
Take as an assumption that word-meaning is strongly correlated with grammatical
 usage.
 That is, 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

, the noun, has a different meaning than 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

, the verb.
 Thus, as a hypothesis, write
\begin_inset Formula 
\[
[w,m_{1}]=\{(w,d_{a}),\,(w,d_{b}),\,(w,d_{c}),\,\cdots\}
\]

\end_inset

that is, the word 
\begin_inset Formula $w$
\end_inset

 might have meaning 
\begin_inset Formula $m_{1}$
\end_inset

 whenever is used with any of the disjunts 
\begin_inset Formula $d_{k}$
\end_inset

 from the indicated set.
 The meaning 
\begin_inset Formula $m_{2}$
\end_inset

 of a word will be associated with a different set of disjuncts.
 In general, the sets 
\begin_inset Formula $[w,m_{1}]$
\end_inset

 and 
\begin_inset Formula $[w,m_{2}]$
\end_inset

 will overlap.
 
\end_layout

\begin_layout Standard
Individually, 
\begin_inset Formula $[w,m]$
\end_inset

 is just a set, and has no weights or probabilities associated with it.
 However, if the disjunct 
\begin_inset Formula $(w,d)$
\end_inset

 is observed, one can say that there is a frequency or probability 
\begin_inset Formula $p([w,m]|(w,d))$
\end_inset

 that the meaning 
\begin_inset Formula $m$
\end_inset

 of word 
\begin_inset Formula $w$
\end_inset

 is intended when 
\begin_inset Formula $(w,d)$
\end_inset

 is obeserved.
 This is written as a conditional probability, so that one has
\begin_inset Formula 
\[
\sum_{m}p([w,m]|(w,d))=1
\]

\end_inset

That is, given that 
\begin_inset Formula $(w,d)$
\end_inset

 was observed, there must be some meaning 
\begin_inset Formula $m$
\end_inset

 that was intended; the list of possible meanings is complete and exhaustive.
 I'm assuming that one possible meaning is 
\begin_inset Quotes eld
\end_inset

nonsense
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

junk
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

unknown
\begin_inset Quotes erd
\end_inset

; just add it to the list of possible meanings.
\end_layout

\begin_layout Standard
One of the tasks is to discover the complete set of meanings 
\begin_inset Formula $\{m_{i}\}$
\end_inset

 for a word.
 Another task is to discern the probabilities 
\begin_inset Formula $p([w,m]|(w,d))$
\end_inset

.
\end_layout

\begin_layout Subsection*
Word Classes
\end_layout

\begin_layout Standard
Any given word might belong to one of many different word classes (noun,
 verb, ...) and the collected disjunct usage observations on that word will
 in general be a linear combination of such different usages.
 Distinguishing word-classes require untangling these relationships.
\end_layout

\begin_layout Standard
The setup for this problem is mostly identical to the problem above, except
 that the 
\begin_inset Quotes eld
\end_inset

meaning
\begin_inset Quotes erd
\end_inset

 
\begin_inset Formula $m$
\end_inset

 is re-interpreted as the word-class 
\begin_inset Formula $g$
\end_inset

, short for 
\begin_inset Quotes eld
\end_inset

grammatical class
\begin_inset Quotes erd
\end_inset

.
 That is, the above did not specify the definition of 
\begin_inset Formula $m$
\end_inset

, rather, it was presented in general terms.
 Here, likewise, but a stricter definition is proposed for a 
\begin_inset Quotes eld
\end_inset

word class
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Standard
A word class 
\begin_inset Formula $g$
\end_inset

 is a set of words 
\begin_inset Formula $\{w_{i}\}$
\end_inset

, together with a set of disjuncts 
\begin_inset Formula $\{d_{j}\}$
\end_inset

, such that all words in the word-class are commonly used with any of the
 disjuncts in the disjunct-set.
 That is, 
\begin_inset Formula 
\[
g=\left(\{w_{i}\},\{d_{j}\}\right)
\]

\end_inset

 subject to the constraint that the vector
\begin_inset Formula 
\[
[w_{1},g]=\left\{ (w_{1},d_{a}),\,(w_{1},d_{b}),\,(w_{1},d_{c}),\,\cdots\right\} 
\]

\end_inset

 is judged to be similar to the vector
\begin_inset Formula 
\[
[w_{2},g]=\left\{ (w_{2},d_{a}),\,(w_{2},d_{b}),\,(w_{2},d_{c}),\,\cdots\right\} 
\]

\end_inset

according to some similarity measure (e.g.
 cosine similarity).
 The idea is that any of the words in 
\begin_inset Formula $g$
\end_inset

 use any of the disjuncts in 
\begin_inset Formula $g$
\end_inset

 in similar ways.
\end_layout

\begin_layout Standard
Any given word might belone to multiple different classes 
\begin_inset Formula $g$
\end_inset

.
 For example, the verb 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

 will belong to a different class than the noun 
\begin_inset Quotes eld
\end_inset

saw
\begin_inset Quotes erd
\end_inset

.
 As a general rule, whenever 
\begin_inset Formula $g_{1}\ne g_{2}$
\end_inset

 then the set of disjuncts in 
\begin_inset Formula $g_{1}$
\end_inset

 and 
\begin_inset Formula $g_{2}$
\end_inset

 will not overlap very much, if at all.
\end_layout

\begin_layout Subsection*
Assigning Word to Word Classes
\end_layout

\begin_layout Standard
Assigning a word to a word-class has a knock-on network effect.
 That is, words appear not only in isolation, but also as connectors in
 disjuncts.
 If two words are considered to be similar, then perhaps two connectors
 should be judged to be similar.
 If two connectors are similar, then perhaps the disjuncts they appear in
 are similar.
 If two disjuncts are similar, then perhaps some other pair of words can
 now be considered to be similar.
 The question arises of how far to follow this network effect, and how to
 assign cutoffs.
 
\end_layout

\begin_layout Standard
Note that the network can be traced in either one of two directions.
 Given a pair of similar words, one can ask if any of the disjuncts attached
 to those words are similar to one-another, or not.
 In the other direction, one can ask how similar connectors imply similar
 disjuncts.
 This is made more explicit below.
\end_layout

\begin_layout Standard
Starting with a single word, one can examine all of the disjuncts on that
 word, to see if any of them are similar.
 For example, the word 
\begin_inset Quotes eld
\end_inset

the
\begin_inset Quotes erd
\end_inset

 should have disjuncts 
\begin_inset Quotes eld
\end_inset

book+
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

novel+
\begin_inset Quotes erd
\end_inset

 on it, and one can ask if 
\begin_inset Quotes eld
\end_inset

book
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

novel
\begin_inset Quotes erd
\end_inset

 are similar.
 If so, they can be merged into a grammatical class 
\begin_inset Formula $g=\left\{ \mbox{book},\mbox{novel}\right\} $
\end_inset

 and the two disjuncts 
\begin_inset Quotes eld
\end_inset

book+
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

novel+
\begin_inset Quotes erd
\end_inset

 replaced by 
\begin_inset Formula $g+$
\end_inset

.
 The observation count (and likewise the probability) on (the, g+) should
 be the sum of N(the, book+) and N(the, novel+).
 The process is then repeated recursively, examining each of the words appearing
 in the disjuncts.
\end_layout

\begin_layout Standard
Alternately, one may walk the network in the 
\begin_inset Quotes eld
\end_inset

other
\begin_inset Quotes erd
\end_inset

 direction, and merge disjuncts as they appear in similar words.
 Let disjunct 
\begin_inset Formula $d$
\end_inset

 be a sequence of connectors: 
\begin_inset Formula $d=(c_{1},c_{2},c_{3,}\cdots)$
\end_inset

 and each connector is a word and a direction indicator: 
\begin_inset Formula $c=(w,\pm)$
\end_inset

.
 Given two similar words 
\begin_inset Formula $w_{a}$
\end_inset

 and 
\begin_inset Formula $w_{b}$
\end_inset

, one can trace through the connectors 
\begin_inset Formula $c_{a+}=(w_{a},+)$
\end_inset

 and 
\begin_inset Formula $c_{b+}=(w_{b},+)$
\end_inset

 and likewise for the - direction.
 One then forms the set of all disjuncts in which 
\begin_inset Formula $c_{a+}$
\end_inset

appears: 
\begin_inset Formula 
\[
\left\{ d_{k}=(c_{1},c_{2},c_{3},\cdots)|c_{j}=c_{a+}\,\mbox{for some }j\right\} 
\]

\end_inset

Then, given one 
\begin_inset Formula $d_{k}$
\end_inset

, one constructs 
\begin_inset Formula $\widetilde{d_{k}}$
\end_inset

 so that 
\begin_inset Formula $c_{b+}$
\end_inset

 replaces 
\begin_inset Formula $c_{a+}$
\end_inset

.
 One then constructs the set of all words that appear with 
\begin_inset Formula $\widetilde{d_{k}}$
\end_inset

 :
\begin_inset Formula 
\[
\upsilon=\left\{ w|N(w,\widetilde{d_{k}})>0\right\} 
\]

\end_inset

and ask whether any of these words already belong to the same grammatical
 class.
 If not, then they should be compared to one-another, to see if they might.
\end_layout

\begin_layout Standard
If a pair of words in 
\begin_inset Formula $\upsilon$
\end_inset

 already belong to the same grammatical class, then the two disjuncts 
\begin_inset Formula $d_{k}$
\end_inset

 and 
\begin_inset Formula $\widetilde{d_{k}}$
\end_inset

 can be merged into one.
 Do this by forming the grammatical class 
\begin_inset Formula $g=\{w_{a},w_{b}\}$
\end_inset

 and construct the connector 
\begin_inset Formula $c_{g+}=(g,+)$
\end_inset

.
 Then construct 
\begin_inset Formula $\overbrace{d_{k}}$
\end_inset

 so that that 
\begin_inset Formula $c_{g+}$
\end_inset

 replaces 
\begin_inset Formula $c_{a+}$
\end_inset

, and replace both 
\begin_inset Formula $d_{k}$
\end_inset

 and 
\begin_inset Formula $\widetilde{d_{k}}$
\end_inset

 by 
\begin_inset Formula $\overbrace{d_{k}}$
\end_inset

 in the relevant sections.
 The observation counts are copied over.
 The process is recursive, repeating for each pair of words judged sufficiently
 similar.
 Alternately, one might defer the creation of 
\begin_inset Formula $g$
\end_inset

 until one has walked enough of the network to determine general similarity.
\end_layout

\begin_layout Subsection*
Merging Words to form Word Classes
\end_layout

\begin_layout Standard
After the grammatical behavior of two words is considered to be similar,
 how should a merged word-class be created? How should the merger be performed?
 There are several different ways in which words can be merged together
 to form word classes.
 These are reviewed below.
\end_layout

\begin_layout Standard
Merging is not straight-forward, because the process needs to result in
 an orthogonalization of the space for grammatical behavior.
 That is, the disjunct counts on any given word might be partly representative
 of its behavior as one of many different kinds of fine-grained parts of
 speech.
 That is, the goal is to take the disjuncts on any given word, separate
 them into two classes, and merge one class into an existing (or new) grammatica
l class, while leaving the rest as-is, which might be subsequently re-organized
 into some other class, ad infinitum.
\end_layout

\begin_layout Subsubsection*
Linear merging
\end_layout

\begin_layout Standard
Linear merging treats words as vectors, computing their sum to define the
 new merged class, and then computing the perpendicular components as the
 left-over, un-accounted-for remainders.
 More precisely, the disjuncts are considered to be the basis elements of
 the vector space, and the count (or frequency) of each disjunct defines
 the vector.
\end_layout

\begin_layout Standard
Consider merging two words or word-classes 
\begin_inset Formula $w_{a}$
\end_inset

 and 
\begin_inset Formula $w_{b}$
\end_inset

 (that is, each of 
\begin_inset Formula $w_{a}$
\end_inset

 and 
\begin_inset Formula $w_{b}$
\end_inset

 can be either a word, or a word-class).
 Let 
\begin_inset Formula $M(w,d)$
\end_inset

 be a number associated with the word-disjunct pair 
\begin_inset Formula $(w,d)$
\end_inset

.
 Typically it will be the count 
\begin_inset Formula $N(w,d)$
\end_inset

 that the pair was observed (or equivalently, the normalized frequency 
\begin_inset Formula $p(w,d)=N(w,d)/N(*,*)).$
\end_inset

 The corresponding vector is then
\begin_inset Formula 
\[
\vec{w}=\sum_{d}M(w,d)\;\widehat{d}
\]

\end_inset

where 
\begin_inset Formula $\widehat{d}$
\end_inset

 is the basis element.
 The merged word-class can then be defined as 
\begin_inset Formula $\vec{w}_{c}=\vec{w}_{a}+\vec{w}_{b}$
\end_inset

.
 
\end_layout

\begin_layout Subsubsection*
Erasure vs.
 orthogonal replacement
\end_layout

\begin_layout Standard
After the merger, there are two alternatives for what to do with 
\begin_inset Formula $\vec{w}_{a}$
\end_inset

 and 
\begin_inset Formula $\vec{w}_{b}$
\end_inset

 in the dataset.
 One alternative is to remove both 
\begin_inset Formula $\vec{w}_{a}$
\end_inset

 and 
\begin_inset Formula $\vec{w}_{b}$
\end_inset

 entirely.
 The other alternative is to compute the components of 
\begin_inset Formula $\vec{w}_{a}$
\end_inset

 and 
\begin_inset Formula $\vec{w}_{b}$
\end_inset

 that are orthogonal to 
\begin_inset Formula $\vec{w}_{c}$
\end_inset

, and replace 
\begin_inset Formula $\vec{w}_{a}$
\end_inset

 and 
\begin_inset Formula $\vec{w}_{b}$
\end_inset

 by these orthogonal components.
 That is, given a vector 
\begin_inset Formula $\vec{v}$
\end_inset

 (which might be 
\begin_inset Formula $\vec{w}_{a}$
\end_inset

 or 
\begin_inset Formula $\vec{w}_{b}$
\end_inset

), compute 
\begin_inset Formula 
\[
\vec{v}_{\bot}=\vec{v}-\widehat{w}_{c}\left(\widehat{w}_{c}\cdot\vec{v}\right)
\]

\end_inset

 where 
\begin_inset Formula $\widehat{w}_{c}=\vec{w}_{c}/\left|\vec{w}_{c}\right|$
\end_inset

 is the normalized unit vector pointing in the 
\begin_inset Formula $\vec{w}_{c}$
\end_inset

 direction.
 
\end_layout

\begin_layout Standard
The goal of maintaining the orthogonal components is that perhaps 
\begin_inset Formula $\vec{w}_{a}$
\end_inset

 and 
\begin_inset Formula $\vec{w}_{b}$
\end_inset

 have admixtures of other grammatical categories in them; what these are
 cannot be known a-priori.
 The discard option effectively discards these admixtures, hiding them from
 later iterations.
 This kind of hiding/data-destruction seems undesirable.
\end_layout

\begin_layout Standard
The orthogonal component potentially has negative coefficients appearing
 in it; it seems that these must be zeroed out to preserve 
\begin_inset Quotes eld
\end_inset

physicality
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Subsubsection*
Linear overlap
\end_layout

\begin_layout Standard
This would work like linear merging, described above, except that the intersecti
on of the sets of disjuncts on these two words is computed first, and the
 vector basis is taken only over this intersected set.
 The intersection is presumably substantial, if the two words are grammatically
 similar.
 For the replacement step, one has three alternatives: total discard, which
 seems inappropriate (as the non-intersected disjuncts get discarded); partial
 discard, which discards only the intersected components, and orthogoanl
 replacement.
\end_layout

\begin_layout Subsection*
Noise
\end_layout

\begin_layout Standard
If an event is normally distributed, then we can characterize the uncertainty
 as being 
\begin_inset Formula $1/\sqrt{N}$
\end_inset

 after 
\begin_inset Formula $N$
\end_inset

 observations.
 We don't actually know if our observations are normally distributed.
 Its not even clear quite how to even obtain the distribution.
 But lets assume they are.
 Then, given that 
\begin_inset Formula $p(x,y)=N(x,y)/N(*,*)$
\end_inset

 and estimating the noise to go as 
\begin_inset Formula $\sqrt{N(x,y)}$
\end_inset

 we get that the error in the frequentist probability estimate is given
 by
\begin_inset Formula 
\[
\frac{N(x,y)\pm\sqrt{N(x,y)}}{N(*,*)}=p(x,y)\pm\sqrt{\frac{p(x,y)}{N(*,*)}}
\]

\end_inset

where only the pair observations 
\begin_inset Formula $N(x,y)$
\end_inset

 are considered to be noisy and the value of 
\begin_inset Formula $N(*,*)$
\end_inset

 is held fixed (the natural variation in it is ignored).
\end_layout

\begin_layout Standard
The error in the frequentist estimate of the entropy to be
\begin_inset Formula 
\[
-\log_{2}\left[p(x,y)\pm\sqrt{\frac{p(x,y)}{N(*,*)}}\right]=-\left[\log_{2}p(x,y)\right]\pm\frac{1}{\log2}\sqrt{\frac{1}{p(x,y)N(*,*)}}
\]

\end_inset

where the estimate 
\begin_inset Formula $\log(1+\epsilon)\approx\epsilon$
\end_inset

 is used for small 
\begin_inset Formula $\epsilon$
\end_inset

.
 Summing to estimate the total entropy, one gets
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
H\pm\Delta H=-\sum_{x,y}\left[p(x,y)\pm\sqrt{\frac{p(x,y)}{N(*,*)}}\right]\log_{2}\left[p(x,y)\pm\sqrt{\frac{p(x,y)}{N(*,*)}}\right]
\]

\end_inset

which expands out to
\begin_inset Formula 
\[
\Delta H=\frac{-1}{\sqrt{N(*,*)}}\sum_{x,y}\sqrt{p(x,y)}\left(\frac{1}{\log2}+\log_{2}p(x,y)\right)
\]

\end_inset


\end_layout

\begin_layout Standard
What might these values be, in practice? As a worked example, consider the
 word-pair (big, deal) in the 
\noun on
en_pairs_rthree
\noun default
 dataset.
 It is observed 1039 times, out of 638845863 pair observations (639M) total.
 Plugging and chugging, one gets 
\begin_inset Formula $H=-\log_{2}p(\mbox{big},\mbox{deal})=19.23$
\end_inset

 and 
\begin_inset Formula $\Delta H=0.552$
\end_inset

 which seems to be eminently reasonable.
 The value of 
\begin_inset Formula $\Delta H$
\end_inset

 depends only on 
\begin_inset Formula $N(x,y)$
\end_inset

 and 
\begin_inset Formula $N(*,*)$
\end_inset

 and is graphed below, for fixed 
\begin_inset Formula $N(*,*)=639M$
\end_inset

.
 It does not take very many observations to drive the uncertainty to a fairly
 small value.
\begin_inset Separator latexpar
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/error.eps
	width 60col%

\end_inset


\end_layout

\begin_layout Standard
foo
\end_layout

\begin_layout Subsection*
Error correction
\end_layout

\begin_layout Standard
If a classification error is made early on, can it be spotted, and later
 corrected? What is the mechanism, and how might it work?
\end_layout

\begin_layout Subsection*
Worked example
\end_layout

\begin_layout Standard
Both the cosine similarity and the overlap similarity suggested that the
 words 
\begin_inset Quotes eld
\end_inset

black
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

white
\begin_inset Quotes erd
\end_inset

 are similar.
 What happens when these are grouped together? We not only consider throwing
 both words into the same bag, but we then may want to consider what happens
 to other disjuncts, that connected to other words.
\end_layout

\begin_layout Standard
Also, when clustering, should we create a single common category 
\begin_inset Quotes eld
\end_inset

bw
\begin_inset Quotes erd
\end_inset

 holding both words, or should we create three categories: bw, white-prime
 and black-prime, where bw just has the common disjuncts, and white-prime
 and black-prime is what's left after taking differences?
\end_layout

\begin_layout Section*
English wordpair small dataset July 2017
\end_layout

\begin_layout Standard
This section moved to `word-pairs-redux.lyx` in July 2019 so that all word-pair
 stuff is in the same place.
\end_layout

\begin_layout Section*
Chinese character pair dataset July 2017
\end_layout

\begin_layout Standard
This section moved to `word-pairs-redux.lyx` in July 2019 so that all word-pair
 stuff is in the same place.
\end_layout

\begin_layout Section*
Idioms and word boundary detection
\end_layout

\begin_layout Standard
Higher-level structures in language are important.
 By 
\begin_inset Quotes eld
\end_inset

higher level
\begin_inset Quotes erd
\end_inset

 I mean both the problem of detecting idioms in English, and segmenting
 words in Chinese.
 I claim that applying traditional algorithms to sheaves is sufficient to
 get good results.
 
\end_layout

\begin_layout Standard
In English, one is interested in discovering idioms, entity names, set phrases
 and institutional phrases set in English: one is looking for a sequence
 of neighboring 
\begin_inset Quotes eld
\end_inset

words
\begin_inset Quotes erd
\end_inset

 that commonly occur together.
 Examples include: 
\begin_inset Quotes eld
\end_inset

Sun Trust Bank
\begin_inset Quotes erd
\end_inset

 (an entity name), 
\begin_inset Quotes eld
\end_inset

gone fishin
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

out to lunch
\begin_inset Quotes erd
\end_inset

 (set phrases), 
\begin_inset Quotes eld
\end_inset

blessing in disguise
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

dime a dozen
\begin_inset Quotes erd
\end_inset

 (idioms).
 The words are not necessarily sequential: there are set circumpositions:
 
\begin_inset Quotes eld
\end_inset

if...
 then...
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

first...
 second...
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

not only ...
 but also ...
\begin_inset Quotes erd
\end_inset


\end_layout

\begin_layout Standard
The Chinese word-segmentation problem is discerning when two hanzi characters
 belong to the same word, or not.
 It is similar to the problem of discerning idioms in English.
\end_layout

\begin_layout Subsubsection*
What is a word?
\end_layout

\begin_layout Standard
As background knowledge: there are multiple definitions of a word: Jerome
 Packard, in 
\begin_inset Quotes eld
\end_inset

The Morphology of Chinese A Linguistic and Cognitive Approach
\begin_inset Quotes erd
\end_inset

 (2000) Cambridge University Press lists the following:
\end_layout

\begin_layout Itemize
Orthographic word
\end_layout

\begin_layout Itemize
Sociological word 
\end_layout

\begin_layout Itemize
Lexical word 
\end_layout

\begin_layout Itemize
Semantic word
\end_layout

\begin_layout Itemize
Phonological word 
\end_layout

\begin_layout Itemize
Morphological word
\end_layout

\begin_layout Itemize
Syntactic word
\end_layout

\begin_layout Itemize
Psycholinguistic word
\end_layout

\begin_layout Subsubsection*
Sheaf structures are important
\end_layout

\begin_layout Standard
The proposal being advanced here is that the general sheaf-theoretic techniques
 can be used to discover all of these structures.
\end_layout

\begin_layout Standard
The simplest case would seem to be word-boundary detection in Chinese.
 Here, a word boundary might be one, two or three hanzi characters in a
 row.
 It seems that basic MST techniques should be enough to discover these.
 So, for example, given a hanzi sequence A B C D E, if the MST parse provides
 a link B-C and C-D but no link A-B and no link D-E, then the sequence BCD
 is a candidate for being identified as a word.
 However, observing this once is not statistics: only if the sequence BCD
 is observed many times, can one consider it to be a word.
 
\end_layout

\begin_layout Standard
More complex structures can be found using sheaf-theoretic techniques.
 The example below is taken from the existing Link-Grammar lexis to illustrate
 a search for circumpositions.
 Consider the sentence 
\begin_inset Quotes eld
\end_inset

I will do it if you say so
\begin_inset Quotes erd
\end_inset

.
 It has the parse:
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

    +------->WV------->+--MVs-+---CV->+
\end_layout

\begin_layout Plain Layout

    +--Wd--+-Sp*i+--I--+Osm+  +Cs+-Sp-+--O-+
\end_layout

\begin_layout Plain Layout

    |      |     |     |   |  |  |    |    |
\end_layout

\begin_layout Plain Layout

LEFT-WALL I.p will.v do.v it if you say.v so 
\end_layout

\end_inset

Inside of this, there is a single 
\begin_inset Quotes eld
\end_inset

germ
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

gerbe
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

disjunct
\begin_inset Quotes erd
\end_inset

 located at the word 
\begin_inset Quotes eld
\end_inset

if
\begin_inset Quotes erd
\end_inset

: 
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

         +Cs+
\end_layout

\begin_layout Plain Layout

         |  |
\end_layout

\begin_layout Plain Layout

        if  ? 
\end_layout

\end_inset

Extending out from this are numerous 
\begin_inset Quotes eld
\end_inset

sections
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

partial linkages
\begin_inset Quotes erd
\end_inset

.
 One of these is
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

   +--MVs-+---CV->+
\end_layout

\begin_layout Plain Layout

   |      +Cs+-Sp-+--O--+
\end_layout

\begin_layout Plain Layout

   |      |  |    |     |
\end_layout

\begin_layout Plain Layout

   ?     if  ?  say     ? 
\end_layout

\end_inset

The above structure might occur in many other sentences, and not just in
 this sentence.
 One can keep an eye out for this structure.
 If it occurs more often than usual, one can deduce that it is some set
 phrase or idiom.
 This particular example is not a set phrase in English, but it does illustrate
 how one can describe structure, and search for it, in a way that is more
 sophisticated than using n-grams.
\end_layout

\begin_layout Section*
Word boundaries - Chinese
\end_layout

\begin_layout Standard
So ...
 how does one find word-boundaries in Chinese? The basic idea is to count
 the frequency of patterns such as the below, where BCD are sequentially
 linked, and there are no links AB or DE.
 There may be additional links from the triple BCD going elsewhere, but
 not to neighboring words.
 Ideally, those links attach to just one morpheme.
 
\end_layout

\begin_layout Standard
\begin_inset listings
inline false
status open

\begin_layout Plain Layout

 ..--+         +-...
  +--...
\end_layout

\begin_layout Plain Layout

     |   +--+--+      |
\end_layout

\begin_layout Plain Layout

     |   |  |  |      |
\end_layout

\begin_layout Plain Layout

     A   B  C  D      E 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If this was a European language, we would expect any extra links to attach
 to the last morpheme; this is due to the morphology of Indo-European, where
 the semantic (meaning-carrying) stem is always to the left of the grammatically
 active suffixes.
 Note Japanese, although it has a minimal morphology, as also similarly
 structured; i.e.
 the suffix carries the syntactic structure.
 With Chinese, this is less obviously the case, and ideally, the correct
 attachment will be discovered.
\end_layout

\begin_layout Section*
Meaning
\end_layout

\begin_layout Standard
So here's one approach to meaning.
 It is already clear that disjuncts are correlated with meaning, so one
 provisional approach might be to assign each disjunct a unique meaning.
 Alternately, this can be used as a doorway to the intentional meaning of
 a word.
 
\end_layout

\begin_layout Standard
Consider the phrases 
\begin_inset Quotes eld
\end_inset

the big balloon
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

the red balloon
\begin_inset Quotes erd
\end_inset

, 
\begin_inset Quotes eld
\end_inset

the small ballon
\begin_inset Quotes erd
\end_inset

...
 The pseudo-disjuncts on balloon in these three cases would be 
\begin_inset Quotes eld
\end_inset

the- big-
\begin_inset Quotes erd
\end_inset

 
\begin_inset Quotes eld
\end_inset

the - red-
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

the- small-
\begin_inset Quotes erd
\end_inset

 (plus an additional connector to the verb).
 Examining this connector-by-connector, we expect that the MI for the word
 pair (the, balloon) to be small, while the MI for the word-pairs (big,
 balloon), (red, balloon) and (small, balloon) to be large(r).
 Its thus tempting to identify the set {big, red, small} as the set of intention
al attributes associated with 
\begin_inset Quotes eld
\end_inset

balloon
\begin_inset Quotes erd
\end_inset

.
 The strength of the MI values to each of the connectors might be taken
 as a judgement of how much that attribute is prototypical of the object
 (see other section on 
\begin_inset Quotes eld
\end_inset

prototype theory
\begin_inset Quotes erd
\end_inset

).
\end_layout

\begin_layout Standard
The disjuncts associated with 
\begin_inset Quotes eld
\end_inset

balloon
\begin_inset Quotes erd
\end_inset

 will also connect to a verb.
 These verb connectors may be taken as another set of intentional attributes,
 for example {floats, drifts, rose, popped}.
 It should be possible to distinguish these as an orthogonal set of attributes,
 in that one might observe 
\begin_inset Quotes eld
\end_inset

the- red- floats+
\begin_inset Quotes erd
\end_inset

 and 
\begin_inset Quotes eld
\end_inset

the- red- drifts+
\begin_inset Quotes erd
\end_inset

 but never observe 
\begin_inset Quotes eld
\end_inset

floats- drifts+
\begin_inset Quotes erd
\end_inset

.
\end_layout

\begin_layout Standard
Meaning bibliography: 
\end_layout

\begin_layout Itemize
\begin_inset Quotes eld
\end_inset

The Molecular Level of Lexical Semantics
\begin_inset Quotes erd
\end_inset

, EA Nida, (1997) International Journal of Lexicography, 10(4): 265–274.
 https://www.academia.edu/36534355/The_Molecular_Level_of_Lexical_Semantics_by_EA_
Nida 
\begin_inset CommandInset citation
LatexCommand cite
key "Nida97"
literal "true"

\end_inset


\end_layout

\begin_layout Itemize
Zellig Harris.
 
\begin_inset Quotes eld
\end_inset

Distributional structure.
\begin_inset Quotes erd
\end_inset

 Word, 10(23):146–162, 1954.
 J.
 R.
 Firth.
 A synopsis of linguistic theory, 1930–1955.
 In J.
 R.
 Firth (Ed.), Studies in linguistic analysis, pages 1–32.
 (1957) https://www.tandfonline.com/doi/pdf/10.1080/00437956.1954.11659520 –
 Cited by 2660 
\end_layout

\begin_layout Itemize
John R.
 Firth (1935) 
\begin_inset Quotes eld
\end_inset

The technique of semantics.
\begin_inset Quotes erd
\end_inset

 Transactions of the Philological Society 34(1).
 36–73.
 Quote: 
\begin_inset Quotes eld
\end_inset

the complete meaning of a word is always contextual, and no study of meaning
 apart from context can be taken seriously.
\end_layout

\begin_layout Itemize
John R.
 Firth (1957) A synopsis of linguistic theory 1930–1955.
 In Studies in linguistic analysis, 1–32.
 Oxford: Blackwell.
 Quote: 
\begin_inset Quotes eld
\end_inset

You shall know a word by the company it keeps
\begin_inset Quotes erd
\end_inset


\end_layout

\begin_layout Section*
Meaning Redux
\end_layout

\begin_layout Standard
(9 June 2018) I keep explaing, over and over, why K-means and SVD cannot
 be used.
 Here's a snapshot from a recent email, explaining it again:
\end_layout

\begin_layout Subsection*
Why SVM and K-means don't work
\end_layout

\begin_layout Standard
Here's WHY both SVM and K-means are fundamentally wrong, and are total failures
 for this particular task.
 Lets start with K-means.
 One minor issue with K-means is that you have to pick K in advance, but
 you don't know what K is.
 But whatever, that's not important.
 Its OK to guess that K=100 or so.
 The big problem is that K-means then takes the 
\emph on
MEAN
\emph default
 (the 
\emph on
AVERAGE
\emph default
) of the vectors.
 That's what the word "mean" in "K-means" means.
 But we already know, a priori, that taking the average is wrong -- it wipes
 out, erases the different word senses.
\end_layout

\begin_layout Standard
For example: the word-token "saw" is going to have a vector that contains
 disjuncts for both "cutting tool", "the verb cut" "the past tense of to
 see".
 With K-means, this word token can only be assigned to just one cluster:
 it will be the cluster for nouns, or the cluster for past tenses, or the
 cluster for cutting-manipulation-actions.
 No matter which cluster its assigned to, when the average/sum of the vector
 is merged into the cluster, the wrong disjuncts will be averaged in as
 well.
 
\end_layout

\begin_layout Standard
So, for example: lets assume k-means places "saw" into the "nouns cluster".
 After averaging, the noun cluster will now contain disjuncts for both past-tens
e verbs, and also disjuncts for present-tense manipulation-verbs.
 Clearly, noun-clusters should not contain these.
 Two bad things happen: (a) the noun cluster is polluted with verb-vector
 components, and (b) the vector has not been factorized, and so "saw" cannot
 also be placed into other clusters as well.
 
\end_layout

\begin_layout Standard
Ergo -- K-means is fundamentally incorrect -- it cannot correctly cluster
 linguistic data! 
\end_layout

\begin_layout Standard
Lets write some formulas: let 
\begin_inset Formula $v$
\end_inset

 be a vector.
 The MST observation counts give us 
\begin_inset Formula $v_{saw}$
\end_inset

.
 We know that, a priori,
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
v_{saw}=v_{tool}+v_{past-tense}+v_{cutting}
\]

\end_inset

However, we do NOT know what these parts: 
\begin_inset Formula $v_{tool}$
\end_inset

,
\begin_inset Formula $v_{past-tense}$
\end_inset

, 
\begin_inset Formula $v_{cutting}$
\end_inset

 what they are.
 We need to factorize them out.
 K-means erases them, lumps them all into one.
 It does not factorize.
\end_layout

\begin_layout Standard
SVM is a little bit better, if you use it correctly.
 I am not convinced that you are using it correctly.
 So, for example, lets say we had only four words: 
\begin_inset Formula $v_{saw}$
\end_inset

, 
\begin_inset Formula $v_{look}$
\end_inset

, 
\begin_inset Formula $v_{heard}$
\end_inset

 and 
\begin_inset Formula $v_{tool}$
\end_inset

.
 Suppose that SVM was told to decompose into three dimensions, and that
 the three that were picked were mostly pointing along the direction of
 
\begin_inset Formula $v_{look}$
\end_inset

 and 
\begin_inset Formula $v_{heard}$
\end_inset

 and 
\begin_inset Formula $v_{tool}$
\end_inset

 -- these were the three principle components.
\end_layout

\begin_layout Standard
Where should 
\begin_inset Formula $v_{saw}$
\end_inset

 go? In traditional SVDM, the three principle components are used to define
 three hyperplanes or classifiers, and so the single vector 
\begin_inset Formula $v_{saw}$
\end_inset

 would then be classified as to being either "on the same side of the hyperplane
 as 
\begin_inset Formula $v_{look}$
\end_inset

, and thus a part of the 
\begin_inset Formula $v_{look}$
\end_inset

 singular value", or "on the same side of the hyperplane as 
\begin_inset Formula $v_{tool}$
\end_inset

, and thus a part of the 
\begin_inset Formula $v_{tool}$
\end_inset

 singular value", etc.
\end_layout

\begin_layout Standard
But this is again wrong.
 We want to factor or decompose 
\begin_inset Formula $v_{saw}$
\end_inset

 along these three different principle components, thereby automatically
 discovering that some of the disjuncts on 
\begin_inset Formula $v_{saw}$
\end_inset

 are tool-like, that others are verb-like, etc.
\end_layout

\begin_layout Standard

\emph on
THIS
\emph default
 is where word-sense disambiguation comes from.
 It is 
\emph on
NOT
\emph default
 done in some pre-cleaner, pre-disambiguator stage.
 It is done at the clustering stage.
\end_layout

\begin_layout Standard
But, as I hope is now clear, both SVM and K-means are fundamentally wrong
 approaches, because both ERASE word-sense information from the dataset!
\end_layout

\begin_layout Standard
Now, IF you are very careful, you might be able to modify SVM, and after
 finding principle components, go back for a second pass, and perform the
 factorization needed to extract the different word-senses.
 Maybe.
 I can see/guess at a way of doing this, but its hard.
\end_layout

\begin_layout Subsection*
Dimensional reduction
\end_layout

\begin_layout Standard
There's a completely different issue - "dimensional reduction" which must
 not be ignored; its important, a big part of the task.
\end_layout

\begin_layout Standard
So: My large dataset has 24 million disjuncts in it -- that's the dimension
 of the vector space -- all vectors are 24M-dimensional.
 How do you perform dimensional reduction? Well, its "easy" -- if two disjuncts
 have connectors that belong to the same word-class, replace them by one
 disjunct in that word-class.
 (The vector space is now (24M minus one)-dimensional) Lets suppose that
 one of the disjuncts is
\end_layout

\begin_layout Standard

\family typewriter
bird: the- & saw-
\end_layout

\begin_layout Standard
\noindent
When doing the dimensional reduction, the saw- needs to be replaced by:
 ??? either 
\family typewriter
TOOL-
\family default
 or 
\family typewriter
PASTTENSE-
\family default
 or by 
\family typewriter
CUTTING-.
\family default
Obviously, that disjunct was obtained from an MST parses of childrens-lit
 sentences like "John saw the bird.
 Susan saw the bird too.
 Mary saw it also".
 When you dimensionally reduce the saw- in this disjunct, which cluster
 do you assign it to?
\end_layout

\begin_layout Standard
Well, if the text has the sentences: "John knew the bird was there.
 John heard the bird", and if clustering determined that "knew", "heard"
 belongs to 
\family typewriter
PASTTENSE
\family default
 then the dim reduction is clear:
\end_layout

\begin_layout Standard

\family typewriter
bird: the- & PASTTENSE-
\end_layout

\begin_layout Standard
\noindent
\align left
and we know that the following is wrong:
\end_layout

\begin_layout Standard

\family typewriter
bird-: the- & TOOL-
\end_layout

\begin_layout Standard
\noindent
The problem here is that both K-means and SVD are completely ignorant of
 the structure of the basis elements of the vector space.
 Both assume that the basis elements of the vector space are irreducible,
 atomic, indivisible.
 Its a natural assumption for some machine-learning tasks, but completely
 wrong for language-learning where we know, a priori, that the basis elements
 have structure.
 This is kind of a key idea from sheaves!
\end_layout

\begin_layout Section*
Merge Results 5 June 2018
\end_layout

\begin_layout Standard
After a very long hiatus, restart.
 All earlier merge data lost!? 
\end_layout

\begin_layout Standard
Here's a sample of automatically-discovered grammatical classes, using the
 '
\family typewriter
ortho-merge
\family default
' strategy from '
\family typewriter
gram-class.scm
\family default
'.
 I seem to have lost/corrupted a previous, larger dataset, so this was remade
 from scratch the last few days.
 Source dataset is '
\family typewriter
en_pairs_cfive_class
\family default
'.
 The merge parameters are: cosine-similarity-accept cutoff = 0.65; union-merge-fra
ction = 0.3.
\end_layout

\begin_layout Standard
This run took 48 hours (its very far from done, this is a snapshot), it
 found 230 words that it could classify into 38 classes.
 (Seven more words got classified as I prepared this, so the counts may
 be off.) The table below lists them exhaustively.
 Note that some words appear in multiple classes: for example, 
\begin_inset Quotes eld
\end_inset

mother father
\begin_inset Quotes erd
\end_inset

.
 Some words are clearly mis-classified, but there are not many of those.
 Some classes are a bit confusing as to their content, but most seem very
 clear.
 The classes are clearly semantic in nature; for example, there are two
 distinct classes of prepositions.
 The semantics is entertainingly insightful: 
\begin_inset Quotes eld
\end_inset

voice mother hands heart head father mind face feet
\begin_inset Quotes erd
\end_inset

 are parts of oneself, with some unexpected members: 
\begin_inset Quotes eld
\end_inset

mother, father
\begin_inset Quotes erd
\end_inset

 are not normally considered to be body parts, but are, in some sense, deeply,
 
\begin_inset Quotes eld
\end_inset

parts of oneself
\begin_inset Quotes erd
\end_inset

.
 Similarly, 
\begin_inset Quotes eld
\end_inset

wife arm daughter friend mouth friends brother
\begin_inset Quotes erd
\end_inset

 are mostly relatives and relationships, yet 
\begin_inset Quotes eld
\end_inset

arm mouth
\begin_inset Quotes erd
\end_inset

 are not.
 Perhaps the arm and mouth have a mind of their own, functioning a bit independe
ntly from the true self?
\end_layout

\begin_layout Standard
I'll try to run this a few more days, and present a newer report.
 While reviving this old code, I realized that the classification algorithm
 being used here has multiple faults and is a bit crude.
 I'm writing a nicer algo right now.
 I don't really know how to compare the quality of the algos, at this point.
\end_layout

\begin_layout Standard
Meanwhile, you should be able to get similar results, by applying the code
 in '
\family typewriter
gram-class.scm
\family default
' to a dataset that contains disjuncts derived from MST parses.
\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="38" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0pt">
<column alignment="left" valignment="top" width="80col%">
<column alignment="left" valignment="top" width="18col%">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Size
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Members
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Comments
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
43
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
village city question subject sea town girl public land French English fire
 King war boy air morning words others poor best second world door book
 heart body case night room whole light country people house children last
 present ground water family first other
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Nouns, mostly
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
for in from at on by of with all towards within near against under through
 over upon into 
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Prepositions
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
help hear keep leave take find get make see give say go 
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Personal verbs
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
fine word large moment certain small woman new good man great little
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Adjectives, mostly
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
10
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
fall action history character state position sense force knowledge pleasure
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
full nature part death power most some out one 
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
voice mother hands heart head father mind face feet
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Body-parts
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
till whether since because until where if when
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Time
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
will would might should may can must could
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Imperatives
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
or but perhaps nor though And while
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Conjunctions
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
wife arm daughter friend mouth friends brother
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Relatives
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
feel believe myself am know think
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Beingness
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
rest end body name side power 
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
heard taken given already done seen
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Past perfect - action
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
really always still also now
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
year place same day way
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
kept held called made found
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Possessive verbs
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
son arms own eyes 
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
her me him us
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Anaphora
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
heard felt knew saw
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Simple past - action
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
making such like
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
our its their
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Possesives - plural
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
five three four
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
during between among
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Prepositions
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
once least
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
thus sometimes
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Deduction
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
therefore indeed
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Deduction
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
is was
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to be - Singular
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
are were
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
to be - Plural
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
! ? 
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Sentence end
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
, ; 
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Punct
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
And The
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Sentence start 
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
they we
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Anaphora
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
cannot shall
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Imperatives
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
mother father
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
sort number
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
France England
\end_layout

\end_inset
</cell>
<cell alignment="left" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset

Here's a graph of the above distribution.
 Its on a log-log scale.
 It looks to be approximately Zipfian.
 That's no surprise.
 Total of 38 classes shown.
 
\begin_inset Note Note
status collapsed

\begin_layout Plain Layout
This and the remaining graphs built with 'word-classes.scm' prt-disjunct-distribu
tion, etc.
 
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/class-words.eps
	width 80text%

\end_inset

 
\end_layout

\begin_layout Standard
Here's a graph of the number disjuncts in each grammatical class.
 (There were 259 words classified into 42 classes when this was prepared).
 The 
\begin_inset Quotes eld
\end_inset

number of disjuncts
\begin_inset Quotes erd
\end_inset

 is the same as the 
\begin_inset Quotes eld
\end_inset

support
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Formula $l_{0}$
\end_inset

 norm of the vector.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/class-dj-support.eps
	width 80text%

\end_inset


\end_layout

\begin_layout Standard
Continuing, below is the total count of the number of observations of the
 disjuncts (There were 269 words classified into 44 classes when this was
 prepared).
 The 
\begin_inset Quotes eld
\end_inset

count of disjuncts
\begin_inset Quotes erd
\end_inset

 is the same as the 
\begin_inset Quotes eld
\end_inset

count
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Quotes eld
\end_inset

Manhatten distance
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Formula $l_{1}$
\end_inset

 norm of the vector.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/class-dj-count.eps
	width 80text%

\end_inset


\end_layout

\begin_layout Standard
And again, below is the length of each vector, viz, the root-mean-square
 count of the number of observations of the disjuncts (There were 269 words
 classified into 44 classes when this was prepared).
 The 
\begin_inset Quotes eld
\end_inset

RMS count of disjuncts
\begin_inset Quotes erd
\end_inset

 is the same as the 
\begin_inset Quotes eld
\end_inset

length
\begin_inset Quotes erd
\end_inset

 or 
\begin_inset Formula $l_{2}$
\end_inset

 norm of the vector.
 The initial part of this graph is the most Zipfian so far, with a slope
 of exactly 1.0, as eyeballed.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/class-dj-length.eps
	width 80text%

\end_inset


\end_layout

\begin_layout Standard
Conclusion: Looks good, more processing needed; comparison of experiments
 needed.
\end_layout

\begin_layout Section*
Merge Experiments
\end_layout

\begin_layout Standard
Three different merge experiments are being run.
 These are reported below.
 The summary is here:
\end_layout

\begin_layout Description
Block-5x5 Same as above, pushed out farther.
 This is run on a copy of the 
\family typewriter
en_pairs_cfive_mst
\family default
 dataset (all five MST tranches) on the LXC container.
 Using a cutoff of 20 observations minimum per word, this contains 62607
 words to be classified.
 The classifier is the 
\family typewriter
merge-ortho
\family default
 classifier, using a minimum cosine of 0.65 to propose a merge, and a fixed
 fraction of 0.3 for the union-merge.
 The agglomeration algorithm is the block-diagonal algorithm.
 (This dataset has 40M sections; viz about 80M atoms, viz about 120GB to
 load in full)
\end_layout

\begin_layout Description
Fuzz-5x2 This is run on a copy of the 
\family typewriter
en_pairs_cfive_mtwo
\family default
 dataset (only two MST tranches) and thus has fewer words: a total of 25505
 words with more than 20 observations.
 As above, this uses the 
\family typewriter
merge-ortho
\family default
 classifier, using a minimum cosine of 0.65 to propose a merge, and a fixed
 fraction of 0.3 for the union-merge.
 The agglomeration algorithm is the greedy algorithm.
 (this dataset has about 13M sections) Run this as: `(gram-classify-greedy-fuzz
 0.65 0.3 20)`.
\end_layout

\begin_layout Description
Discrim-5x2 This is run on a copy of the 
\family typewriter
en_pairs_cfive_mtwo
\family default
 dataset, as above: a total of 25505 words with more than 20 observations.
 This uses the 
\family typewriter
merge-discrim
\family default
 classifier, which is like the 
\family typewriter
merge-ortho
\family default
 classifier, but uses a variable fraction for union-merge.
 Because the variable fraction should behave nicely, the minimum cosine
 is set to 0.50.
 The agglomeration algorithm is the greedy algorithm.
 Run this as: `(gram-classify-greedy-discrim 0.5 20)`.
\end_layout

\begin_layout Standard
Basically, the last two are directly comparable: they differ only in the
 merge strategy.
 The first two are harder to compare: they use different datasets and different
 agglomeration algos.
 All three are using the screwy 
\family typewriter
merge-ortho
\family default
 classifier, which is almost right, but altered counts in a somewhat screwy
 way.
 Thus, these experiments need to be repeated ...
 again.
\end_layout

\begin_layout Standard
The table below is a 
\begin_inset Quotes eld
\end_inset

progress report
\begin_inset Quotes erd
\end_inset

 on 
\series bold
Fuzz-5x2
\series default
, as its being computed.
 Each row represents a snapshot in a different point in time for the computation.
 
\series bold
Fuzz-5x2
\series default
 crashed; the last row are the stats at the time of the crash.
\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="10" columns="8">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
num-classes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
num-words
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
doubles
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
singletons
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
dupes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
dup-cls
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
uniq-dup
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
cpu
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
46
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
429
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
259
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
12
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3400
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
56
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
482
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
487
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6230
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
65
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
533
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
682
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
15
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8800
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
70
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
581
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
915
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
18
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
36
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
11760
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
75
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
635
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1138
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
48
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14760
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
81
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
670
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1301
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
50
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17200
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
85
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
707
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1544
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
50
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20720
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
90
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
822
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1846
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
29
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
58
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25256
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
93
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
835
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1927
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
66
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
34
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26624
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Foot
status open

\begin_layout Plain Layout
Table entries obtained from `(prt-all-classes)` and `(prt-multi-members)`
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
The columns are as follows:
\end_layout

\begin_layout Description
num-classes Total number of grammatical categories (word classes).
\end_layout

\begin_layout Description
num-words Total number of words assigned to grammatical classes, with two
 or more words per class.
\end_layout

\begin_layout Description
doubles Total number of classes having exactly two words in them.
\end_layout

\begin_layout Description
singletons Total number of words examined, but could not be assigned to
 any existing grammatical class.
 These can be thought of as classes that have only one member; they may
 eventually grow to more than one member.
\end_layout

\begin_layout Description
dupes Total number of words belonging to more than one class.
 These roughly correspond to words that have been found to have more than
 one syntactic form (i.e.
 more than one 
\begin_inset Quotes eld
\end_inset

meaning
\begin_inset Quotes erd
\end_inset

 - aka 
\begin_inset Quotes eld
\end_inset

word-sense disambiguation
\begin_inset Quotes erd
\end_inset

)
\end_layout

\begin_layout Description
dup-cls Total number of classes that the dupes belong to.
 Thus, dupe-cls / dupes = average number of 
\begin_inset Quotes eld
\end_inset

meanings
\begin_inset Quotes erd
\end_inset

 that a multi-meaning word has.
 Currently, this appears to be always 2 in the above dataset.
\end_layout

\begin_layout Description
uniq-dup Total number of unique classes that have multi-meaning words in
 them.
\end_layout

\begin_layout Description
cpu The CPU-minutes accumulated so far.
 This is ad-hoc, it doesn't count for time spent in postgres, or inefficient
 parallelism.
 It just provides a scale for forward progress.
\end_layout

\begin_layout Standard
Some examples of multi-category words: 
\end_layout

\begin_layout Description
what belongs to <that as when if what before where because until> and also
 <what how why whether> – The first class appears to correspond to propostional
 words, the second to question words.
\end_layout

\begin_layout Description
her belongs to <his her> and <her him me us> – possesives and determiners.
\end_layout

\begin_layout Description
with belongs to <of in with for on by from into upon over through under
 among> and also <with like such having> – prepositions and membership-property
 words.
\end_layout

\begin_layout Standard
Here's a progress report for 
\series bold
Discrim-5x2
\series default
.
 A quick look-see shows that this is lower-quality; the cosine=0.50 seems
 to accept too much, mixing nouns and verbs, although it is better at placing
 given names into one category (for example)...
\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="9" columns="8">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
num-classes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
num-words
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
doubles
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
singletons
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
dupes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
dup-cls
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
uniq-dup
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
cpu
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
427
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
8
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
67
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
34
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2370
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
47
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
715
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
89
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
62
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
127
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4130
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
63
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
946
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
169
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
87
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
182
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
6375
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
82
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1145
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
19
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
343
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
106
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
222
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
41
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
9700
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
96
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1268
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
573
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
125
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
260
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
44
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
13420
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
106
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1415
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
27
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
757
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
136
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
282
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
49
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
16400
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
121
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1518
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
994
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
142
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
294
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
54
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20600
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
125
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1545
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1020
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
145
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
300
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
56
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21149
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
\begin_inset VSpace defskip
\end_inset


\end_layout

\begin_layout Standard
Its clear that 
\series bold
Discrim-5x2
\series default
 is assigning more words into fewer classes than 
\series bold
Fuzz-5x2
\series default
 is.
 Particularly notable is the much much smaller size of the 'doubles' classes
 and the 'singletons' classes.
 This crashed mysteriosuly after running for 21K seconds...
 `(compute-right-cosine (WordNode "”" #)
\end_layout

\begin_layout Standard
Below are distribution graphs for the word-clases obtained using the two
 different datasets, and three different classification schemes.
 The general similarity of the graphs is immediately apparent.
 One can conclude: 
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Graphs created with `word-classes/distribution.gplot` from stats generated
 with `word-classes.scm` 
\end_layout

\end_inset


\end_layout

\begin_layout Itemize
The different classifcation schemes all generate the same distribution of
 class sizes, and that distribution is very nearly purely Zipfian.
 (Upper-left graph)
\end_layout

\begin_layout Itemize
The distribution of disjuncts in the classes is bimodal, and the modality
 and inflection is the same for the 
\begin_inset Formula $l_{0}$
\end_inset

, 
\begin_inset Formula $l_{1}$
\end_inset

 and 
\begin_inset Formula $l_{2}$
\end_inset

-norms.
\end_layout

\begin_layout Itemize
The distribution of disjuncts is determined primarily by the dataset, and
 not by the classification algo.
 That is, Fuzz-5x2 and Discrim-5x2 are two different classifiers running
 on the same dataset; they are in many ways similar, and differing a bit
 from the larger dataset Block-5x5.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/words-in-class.eps
	width 45text%

\end_inset


\begin_inset Graphics
	filename word-classes/support-of-class.eps
	width 45text%

\end_inset


\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/count-of-class.eps
	width 45text%

\end_inset


\begin_inset Graphics
	filename word-classes/length-of-class.eps
	width 45text%

\end_inset


\end_layout

\begin_layout Standard
The overall lack of dramatic differences in the distributions is remarkable.
 Visual inspection of the classes indicates that they are all arriving at
 the same general and mostly-correct classification of words.
 It could be interesting to see how much these classifications differ; measuring
 this, however, is difficult, as they are in distinct datasets, and there's
 no infrastructure for that.
\end_layout

\begin_layout Subsection*
Quality evaluation
\end_layout

\begin_layout Standard
Evaluating the quality is hard.
 Quick looks suggest its all going as planned...
 unclear how to be quantititive, except by tedious hand-scoring.
\end_layout

\begin_layout Standard
One possibility for an objective external score is to compare these word-classes
 to WordNet sysnsets.
 
\end_layout

\begin_layout Section*
Connector distribution from MST parses
\end_layout

\begin_layout Standard
(9 June 2018) This is kind-of a repeat of earlier work reported in `connector-se
ts-revised.lyx` but is (a) graphed differently and is (b) for a different
 dataset.
 Its actually a commentary on the quality of data comeing out of MST.
 First graph: number of sections having N connectors.
 
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/dj-size.eps
	width 60text%

\end_inset


\end_layout

\begin_layout Standard
It shows how many sections there are that have the indicated number of connector
s on them.
 That is, each section has one and only one disjunct in it.
 Each disjunct can have N connectors in it.
 So, fixing N, how many sections are there that have N connectors? For 
\begin_inset Quotes eld
\end_inset

real
\begin_inset Quotes erd
\end_inset

 linnguistic data, we expect a much much sharper falloff.
 Determiners (the, a this, that ...) should have one connector.
 There are few of these, and that is what the chart above suggests.
 (This chart shows 30K such connectors: since this is pre-clustering, those
 30K correspond to some determiner, connecting to some specific word.)
\end_layout

\begin_layout Standard
Nouns should have 2 or 3 or 4: zero or one to a determiner, zero, one or
 two (maybe rarely three) to adjectives, one to a verb.
 Transitive verbs should have 3 or 4 connectors: one to the subject, one
 to the object, one to LEFT-WALL, zero or one to adverbs, particles, preposition
s, etc.
 Thus, six or more connectors should be very very rare.
 its not.
 This suggests that the MST parser is not producing enitrely believable
 data (but we knew that, already).
 
\end_layout

\begin_layout Standard
Perhaps the high-connector count disjuncts are observed only infrequently?
 The next graph shows the counts, weighted by the number of observations:
 i.e.
 how often that particular disjunct was observed.
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename word-classes/dj-wsize.eps
	width 60text%

\end_inset


\end_layout

\begin_layout Standard
Hmm.
 The good news: the observation counts for 1- and 2-connector disjuncts
 are much higher, with 3-connector disjuncts seen a lot less often.
 The number of 4-connector disjuncts remains dishearteningly high, and the
 fall-off is slower than before.
\end_layout

\begin_layout Standard
Both of the above graphs were generated by considering only the sections
 on word-clusters.
 This comes from the 
\series bold
Block-5x5
\series default
, when it had 700 words assigned to clusters.
 Its not clear if this pattern is true, in general, for all words.
 Note that, to obtain these 700 words, the top-most-frequent 1300 words
 were examined: so the 700 words are among the most frequent.
 Note that, due to the orthogonalization algorithm, not all of the counts
 are transferred from the words to the clusters; only some are.
\end_layout

\begin_layout Section*
Cross-connector stats
\end_layout

\begin_layout Standard
Ran the full cross-connector stats
\end_layout

\begin_layout Section*
Entropic-similarity
\end_layout

\begin_layout Standard
Some stats for entropic-similarity.
 There is good reason to think that the entropic similarity will provide
 a superior judgement of similarity, as compared to cosine similarity.
 The entropic similarity is defined as the MI between word-disjunct vectors.
 Thus, it is similar to the cosine similarity, in being founded on a vector
 dot-product, but has a different normalization in the denominator.
 It's defined below.
\end_layout

\begin_layout Standard
Consider `
\series bold
en_rfive_mtwo
\series default
`.
 There are: 
\end_layout

\begin_layout Itemize
Rows: 137078 – 
\emph on
viz
\emph default
 that many words.
 Viz 
\begin_inset Formula $\sum_{w}1=137078$
\end_inset


\end_layout

\begin_layout Itemize
Columns: 6239997 - 
\emph on
viz
\emph default
 that many unique disjuncts – i.e.
 atoms of the form 
\family typewriter
(Section (*) (dj))
\family default
 with * the wild-card, and 
\family typewriter
dj
\family default
 held fixed.
 Viz 
\begin_inset Formula $\sum_{d}1=6239997$
\end_inset


\end_layout

\begin_layout Itemize
Size: 8629163 - 
\emph on
viz
\emph default
 this many unique sections, viz explicit 
\family typewriter
(Section w dj)
\family default
 for fixed 
\family typewriter
(w,dj)
\family default
.
 
\emph on
Viz
\emph default
 
\begin_inset Formula $\sum_{w,d}\left[0<N(w,d)\right]=8629163$
\end_inset


\end_layout

\begin_layout Standard
Let 
\begin_inset Formula $N(w,d)$
\end_inset

 be the observation count of disjunct 
\begin_inset Formula $d$
\end_inset

 on word 
\begin_inset Formula $w$
\end_inset

.
 Then, for 
\series bold
en_rfive_mtwo
\series default
 we have:
\begin_inset Formula 
\[
N\left(*,*\right)=\sum_{w,d}N(w,d)=18489594.0
\]

\end_inset

viz 18.5M observations total, while
\begin_inset Formula 
\[
\sum_{u,w,d}N(u,d)N(w,d)=\sum_{d}N(*,d)N(*,d)=63598403588.0
\]

\end_inset

viz 63.6G.
 Note that
\begin_inset Formula 
\[
\sum_{d}\frac{N(*,d)}{N\left(*,*\right)}\frac{N(*,d)}{N\left(*,*\right)}=1.8603\times10^{-4}
\]

\end_inset

and 
\begin_inset Formula 
\[
-\log_{2}\sum_{d}\frac{N(*,d)}{N\left(*,*\right)}\frac{N(*,d)}{N\left(*,*\right)}=-\log_{2}1.8603\times10^{-4}=12.4\mbox{ bits}
\]

\end_inset

Does this quantity have a name? What is the name?
\end_layout

\begin_layout Standard
Define the dot-product (inner product) between words as
\begin_inset Formula 
\[
i\left(u,w\right)=\sum_{d}N(u,d)N(w,d)
\]

\end_inset

Normalize this into a bona-fide joint probability as
\begin_inset Formula 
\[
p(u,w)=i\left(u,w\right)/i\left(*,*\right)
\]

\end_inset

and define the entropic similarity as the MI between these two vectors:
\begin_inset Formula 
\[
MI\left(u,w\right)=\log_{2}\frac{p\left(u,w\right)}{p\left(u\right)p\left(w\right)}
\]

\end_inset

where the marginal probability is
\begin_inset Formula 
\[
p\left(u\right)=p\left(u,*\right)=\frac{i\left(u,*\right)}{i\left(*,*\right)}=\frac{1}{i\left(*,*\right)}\;\sum_{w,d}N(u,d)N(w,d)=\frac{1}{i\left(*,*\right)}\;\sum_{d}N(u,d)N(*,d)
\]

\end_inset

The total entropy from the marginal probability is 
\begin_inset Formula 
\[
H=-\sum_{w}p\left(w\right)\log p\left(w\right)
\]

\end_inset

and the total MI would be
\begin_inset Formula 
\[
MI=\sum_{u,w}p\left(u,w\right)\log_{2}\frac{p\left(u,w\right)}{p\left(u\right)p\left(w\right)}
\]

\end_inset


\end_layout

\begin_layout Standard
What's this like? Some pairs below.
 Note that 
\begin_inset Formula $-\log_{2}1.8603\times10^{-4}=12.3922\mbox{ bits}$
\end_inset

.
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="15" columns="7">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $u$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $w$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $-\log_{2}p\left(u\right)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $-\log_{2}p\left(w\right)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $-\log_{2}p(u,w)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $MI(u,w)$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\cos\left(u,w\right)$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
other
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
same
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21.52
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
22.13
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
27.13
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.1232
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.5866
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
nice
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
fine
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
26.94
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25.21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
35.24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.5210
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.5256
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
him
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
me
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
20.81
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21.34
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25.50
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.2583
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.7842
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
men
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
women
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23.69
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
25.79
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
33.24
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.8489
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.6077
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
up
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
down
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
22.46
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23.31
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
27.87
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.5085
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.5630
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
found
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
called
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23.62
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.57
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
32.09
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.7066
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.5576
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
came
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
went
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.19
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
24.72
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31.32
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.1930
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.5900
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
eyes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
hand
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23.47
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23.19
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
28.91
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
5.3589
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.7284
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
men
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
nice
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
38.71
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.477
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0233
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
men
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
went
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
36.08
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-0.061
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0249
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
nice
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
went
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
38.22
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
+1.0490
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0353 
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
called
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
eyes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
38.14
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-2.499
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0049
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
nice
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
eyes
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
40.21
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
-2.193
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.0038
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
nice
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
called
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
35.63
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
+3.4810
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
0.3794
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
Casual observation suggests that the MI and the cosine similarity are correlated
; when one is high, so is the other.
 A detailed examination of this is in section 6 approx page 40 of the March
 2019 version of `connector-sets-revised.pdf`.
\end_layout

\begin_layout Subsection*
Zipf's Law 
\end_layout

\begin_layout Standard
These are interesting:
\end_layout

\begin_layout Itemize
\begin_inset Quotes eld
\end_inset

Zipf’s law holds for phrases, not words
\begin_inset Quotes erd
\end_inset

, Jake Ryland Williams, Paul R.
 Lessard, Suma Desu, Eric M.
 Clark, James P.
 Bagrow, Christopher M.
 Danforth & Peter Sheridan Dodds, Scientific Reports volume 5, Article number:
 12209 (2015)
\end_layout

\begin_layout Standard
Based on my experience, it holds for both ...
 it holds for everything.
\end_layout

\begin_layout Section*
Grammatical Class Redux
\end_layout

\begin_layout Standard
The word-disjunct dataset, containing counts 
\begin_inset Formula $N\left(w,d\right)$
\end_inset

, is far too large to be used as a practical dictionary for parsing.
 It also fails to provide sufficient coverage to parse most sentences.
 Thus, it needs to be simultaneously be compressed and also generalized.
 This is partly achieved by clustering words 
\begin_inset Formula $w$
\end_inset

 into word-classes 
\begin_inset Formula $g$
\end_inset

.
 More precisely, this is a partitioning of the observation counts 
\begin_inset Formula $N\left(w,d\right)$
\end_inset

 into count-classes 
\begin_inset Formula $N\left(g,d\right)$
\end_inset

 where 
\begin_inset Formula $g$
\end_inset

 is a set of words that behave in a grammatically similar fashion.
\end_layout

\begin_layout Standard
One naive (but approximate and incorrect) way of doing this is to come up
 with a membership function – a membership matrix 
\begin_inset Formula $p\left(g|w\right)$
\end_inset

, behaving like a normalized conditional probability
\begin_inset Formula 
\[
\sum_{g}p\left(g|w\right)=1
\]

\end_inset

which says that every word 
\begin_inset Formula $w$
\end_inset

 belongs to one or more clases 
\begin_inset Formula $g$
\end_inset

.
 A partitioning of counts is then
\end_layout

\begin_layout Standard
\begin_inset Formula 
\[
N\left(g,d\right)=\sum_{w}p\left(g|w\right)N\left(w,d\right)
\]

\end_inset

The intended interpretation is that 
\begin_inset Formula $g$
\end_inset

 is a set of words that behave in a similar fashion, syntactically, so that
 
\begin_inset Formula $p\left(g|w\right)$
\end_inset

 is the conditional probability that a word 
\begin_inset Formula $w$
\end_inset

 belongs to 
\begin_inset Formula $g$
\end_inset

.
 Typically, 
\begin_inset Formula $w$
\end_inset

 will belong to no more than half-a-dozen grammatical classes, with each
 grammatical class corresponding loosely to each word-sense that the word
 has.
 
\end_layout

\begin_layout Standard
The problem with the above is that it is incorrect – a too-strong approximation.
 The problem is that different word-senses behave in grammatically different
 fashion – 
\emph on
e.g.

\emph default
 a word can be both a verb and a noun.
 The correct partitioning is then necessarily given by 
\begin_inset Formula $p\left(g|w,d\right)$
\end_inset

 so that, for example, if 
\begin_inset Formula $g$
\end_inset

 is a verb-class, then 
\begin_inset Formula $p\left(g|w,d\right)\ne0$
\end_inset

 only when 
\begin_inset Formula $d$
\end_inset

 is a verb-style disjunct, otherwise, 
\begin_inset Formula $p\left(g|w,d\right)=0$
\end_inset

 whenever 
\begin_inset Formula $d$
\end_inset

 is a noun-style disjunct.
 That is, the correct partitioning is given by 
\begin_inset Formula 
\[
N\left(g,d\right)=\sum_{w}p\left(g|w,d\right)N\left(w,d\right)
\]

\end_inset

and word-sense disambiguation fundamentally disallow the assumption that
 
\begin_inset Formula $p\left(g|w,d\right)\sim p\left(g|w\right)$
\end_inset

 since this assumption erases the grammatical differences between different
 word-senses.
\end_layout

\begin_layout Section*
Word-Sense Disambiguation
\end_layout

\begin_layout Standard
How does this work with entropic similarity? Let
\begin_inset Formula 
\[
\vec{w}=\sum_{d}N\left(w,d\right)\widehat{e}_{d}
\]

\end_inset

be the vector defined by the observed counts of disjuncts.
 Suppose it is a linear combination of two distinct word-senses 
\begin_inset Formula $\vec{s}$
\end_inset

 and 
\begin_inset Formula $\vec{t}$
\end_inset

, so that 
\begin_inset Formula 
\[
\vec{w}=\vec{s}+\vec{t}
\]

\end_inset

but we don't know what the counts for these are, and are so faced with the
 challenge of determining the linear decomposition.
 
\end_layout

\begin_layout Standard
Suppose that there is some grammatical class 
\begin_inset Formula $\vec{g}$
\end_inset

 and we've noticed that 
\begin_inset Formula $\vec{w}$
\end_inset

 is similar to 
\begin_inset Formula $\vec{g}$
\end_inset

 and this similarity is probably due to 
\begin_inset Formula $\vec{s}$
\end_inset

 being similar to 
\begin_inset Formula $\vec{g}$
\end_inset

.
 How can we use this to determine 
\begin_inset Formula $\vec{s}$
\end_inset

? A classical strategy is orthogonalization: assume that 
\begin_inset Formula $\vec{s}$
\end_inset

 is parallel to 
\begin_inset Formula $\vec{g}$
\end_inset

, and take the orthogonal complement.
 The primary problem here is that the space is not Euclidean: it does not
 have sphere-symmetry that preserves the dot product.
 The vectors 
\begin_inset Formula $\vec{w}$
\end_inset

 inhabit a simplex; the vector coefficients must always remain zero or positive,
 and all decompositions must preserve the total count 
\begin_inset Formula $N\left(*,*\right)$
\end_inset

 as an invariant.
\end_layout

\begin_layout Standard
Recall the definition of 
\begin_inset Formula $i\left(u,v\right)$
\end_inset

 above, and note that 
\begin_inset Formula $i\left(u,v\right)=\vec{u}\cdot\vec{v}$
\end_inset

.
 Note that 
\begin_inset Formula $*$
\end_inset

 behaves like a constant vector; 
\emph on
i.e.

\emph default
 
\begin_inset Formula $\vec{*}=\sum_{d}N\left(*,d\right)\widehat{e}_{d}=\sum_{d}\sum_{w}N\left(w,d\right)\widehat{e}_{d}$
\end_inset

 is a perfectly well-defined vector.
\end_layout

\begin_layout Standard
Thus, given the WSD problem before us, it seems we want to find a vector
 
\begin_inset Formula $\vec{s}$
\end_inset

 such that 
\begin_inset Formula $MI\left(\vec{g},\vec{s}\right)$
\end_inset

 is maximized, and so that 
\begin_inset Formula $MI\left(\vec{g},\vec{t}\right)$
\end_inset

 is minimized, subject to the constraint 
\begin_inset Formula $\vec{w}=\vec{s}+\vec{t}$
\end_inset

.
 Lets work that out.
 By definition,
\begin_inset Formula 
\[
MI\left(\vec{g},\vec{s}\right)=\log_{2}\frac{i\left(g,s\right)i\left(*,*\right)}{i\left(g,*\right)i\left(*,s\right)}
\]

\end_inset

The extrema of this would be given by those values of 
\begin_inset Formula $\vec{s}$
\end_inset

 that are stationary under perturbations 
\begin_inset Formula $\vec{s}+\delta\vec{s}$
\end_inset

, where 
\begin_inset Formula 
\[
\delta\vec{s}=\sum_{d}\delta N\left(s,d\right)\widehat{e}_{d}
\]

\end_inset

so that we must solve 
\begin_inset Formula $\left|d\right|$
\end_inset

 different differential equations, one for each disjunct 
\begin_inset Formula $d$
\end_inset

:
\begin_inset Formula 
\[
0=\left.\frac{\partial MI\left(\vec{g},\vec{s}\right)}{\partial N\left(s,d\right)}\right|_{d\mbox{ held fixed}}
\]

\end_inset

Its a mess.
 Lets do it.
 We have
\begin_inset Formula 
\[
0=\frac{\partial}{\partial N\left(s,d\right)}\,\frac{\sum_{\beta}N\left(s,\beta\right)N\left(g,\beta\right)i\left(*,*\right)}{\sum_{\alpha}N\left(s,\alpha\right)N\left(*,\alpha\right)i\left(g,*\right)}
\]

\end_inset

The two 
\begin_inset Formula $i$
\end_inset

 terms are constants; drop them.
 What is left is
\begin_inset Formula 
\[
0=\frac{N\left(g,d\right)}{i\left(s,*\right)}-\frac{i\left(s,g\right)}{\left(i\left(s,*\right)\right)^{2}}N\left(*,d\right)
\]

\end_inset

Reorganizing:
\begin_inset Formula 
\[
0=N\left(g,d\right)i\left(s,*\right)-N\left(*,d\right)i\left(s,g\right)
\]

\end_inset

or
\begin_inset Formula 
\[
\frac{i\left(s,g\right)}{i\left(s,*\right)}=\frac{N\left(g,d\right)}{N\left(*,d\right)}
\]

\end_inset

This is mis-constrained, in several ways.
 First, the RHS depends on 
\begin_inset Formula $d$
\end_inset

 while the LHS does not.
 Not only is it impossible to satisfy, but even if it were, it tells us
 nothing about the components of 
\begin_inset Formula $\vec{s}$
\end_inset

, a formula for which we were hoping to find.
 
\end_layout

\begin_layout Standard
Compute again, to second order.
 Up to a constant,
\begin_inset Formula 
\[
MI\left(\vec{g},\vec{s}\right)=\log_{2}i\left(g,s\right)-\log_{2}i\left(s,*\right)+C
\]

\end_inset

so
\begin_inset Formula 
\[
\frac{\partial}{\partial N\left(s,d\right)}\,\log_{2}i\left(s,g\right)=\frac{N\left(g,d\right)}{i\left(s,g\right)\log2}
\]

\end_inset

so
\begin_inset Formula 
\[
\frac{\partial}{\partial N\left(s,d\right)}\frac{\partial}{\partial N\left(s,d^{\prime}\right)}\,\log_{2}i\left(s,g\right)=-\frac{N\left(g,d\right)N\left(g,d^{\prime}\right)}{\left(i\left(s,g\right)\right)^{2}\log2}
\]

\end_inset

so
\begin_inset Formula 
\[
0=\frac{\partial}{\partial N\left(s,d\right)}\frac{\partial}{\partial N\left(s,d^{\prime}\right)}\,MI\left(\vec{g},\vec{s}\right)=-\frac{N\left(g,d\right)N\left(g,d^{\prime}\right)}{\left(i\left(s,g\right)\right)^{2}}+\frac{N\left(*,d\right)N\left(*,d^{\prime}\right)}{\left(i\left(s,*\right)\right)^{2}}
\]

\end_inset

or
\begin_inset Formula 
\[
\frac{\left(i\left(s,g\right)\right)^{2}}{\left(i\left(s,*\right)\right)^{2}}=\frac{N\left(g,d\right)N\left(g,d^{\prime}\right)}{N\left(*,d\right)N\left(*,d^{\prime}\right)}
\]

\end_inset

which is equally absurd; its just the square of the earlier result.
 WTF.
 But of course, that's obvious, in a way.
 But what to do? Why is this not working?
\end_layout

\begin_layout Standard
Repeating the first-order variation calculation for 
\begin_inset Formula $\cos\theta=i\left(s,g\right)/\sqrt{i\left(s,s\right)i\left(g,g\right)}$
\end_inset

 promptly gives the standard orthogonalization: 
\emph on
viz
\emph default
 that 
\begin_inset Formula $N\left(s,d\right)\sim N\left(g,d\right)$
\end_inset

 is the condition for parallel vectors.
 This is driven not by the numerator 
\begin_inset Formula $i\left(s,g\right)$
\end_inset

 which is shared in common by both 
\begin_inset Formula $\cos\theta$
\end_inset

 and by 
\begin_inset Formula $MI\left(s,g\right)$
\end_inset

 but is rather due to the quadratic 
\begin_inset Formula $i\left(s,s\right)$
\end_inset

 in the denominator.
 In a sense, it is a 
\begin_inset Quotes eld
\end_inset

second order effect
\begin_inset Quotes erd
\end_inset

.
\begin_inset Foot
status open

\begin_layout Plain Layout
A quick sketch might help.
 The quadratic term contributes 
\begin_inset Formula 
\[
\frac{\partial}{\partial N\left(s,d\right)}i\left(s,s\right)=2N\left(s,d\right)
\]

\end_inset

 and so 
\begin_inset Formula 
\[
0=\frac{\partial}{\partial N\left(s,d\right)}\cos\theta=i\left(s,s\right)N\left(g,d\right)-i\left(s,g\right)N\left(s,d\right)
\]

\end_inset

after dropping a constant factor of 
\begin_inset Formula $\sqrt{i\left(g,g\right)}$
\end_inset

 and multiplying by 
\begin_inset Formula $i\left(s,s\right)$
\end_inset

.
 Then, up to scale factor, one gets 
\begin_inset Formula $N\left(s,d\right)\sim N\left(g,d\right)$
\end_inset

 which is the desired result: 
\begin_inset Formula $\vec{s}$
\end_inset

 is collinear with 
\begin_inset Formula $\vec{g}$
\end_inset

.
 To maintain the constraint that 
\begin_inset Formula $\vec{w}=\vec{s}+\vec{t}$
\end_inset

 one must set 
\begin_inset Formula $N\left(s,d\right)=0$
\end_inset

 whenever 
\begin_inset Formula $N\left(w,d\right)=0$
\end_inset

; thus, in the end, 
\begin_inset Formula $\vec{s}$
\end_inset

 might not be completely colinear with 
\begin_inset Formula $\vec{g}$
\end_inset

, but only close.
 The MI formulation lacks the quadratic 
\begin_inset Formula $i\left(s,s\right)$
\end_inset

 term and thus is unable to generate some analog to 
\begin_inset Formula $N\left(s,d\right)\sim N\left(g,d\right)$
\end_inset

.
 
\end_layout

\end_inset

 
\end_layout

\begin_layout Standard
OK, so what is the source of the flawed thinking above? What is the root
 cause of the failure of variational principles on MI? Geometrically, the
 answer is simple: it means that 
\begin_inset Formula $\cos\theta$
\end_inset

 is symptomatic of a sphere: smooth, differentiable, having will-defined
 derivatives.
 By contrast, surfaces of minimal (or maximal) MI are polytopes of some
 sort, consisting of smooth facets, together with non-differentiable edges
 where faces meet.
 Variational principles do not apply.
 We'll have to use some kind of linear programming techniques to find a
 solution.
\end_layout

\begin_layout Standard
So with that in mind, lets start again:
\begin_inset Formula 
\[
MI\left(\vec{g},\vec{s}\right)=\log_{2}\frac{\vec{s}\cdot\vec{g}}{\vec{s}\cdot\vec{*}}+C
\]

\end_inset

and so we're dealing with a ratio of two dot products.
 This is now easy to minimize and maximize, conceptually.
 In one case, any 
\begin_inset Formula $\vec{s}$
\end_inset

 that satisfies 
\begin_inset Formula $\vec{s}\cdot\vec{g}=0$
\end_inset

 will do the trick.
 More appropriately, this would be 
\begin_inset Formula $\vec{t}$
\end_inset

 the orthogonal complement, and the question becomes: can we find such a
 
\begin_inset Formula $\vec{t}$
\end_inset

 such that no coefficients are negative, and such that the coefficients
 of 
\begin_inset Formula $\vec{s}=\vec{w}-\vec{t}$
\end_inset

 are also non-negative? Given that we already know that 
\begin_inset Formula $MI\left(\vec{g},\vec{w}\right)$
\end_inset

 is large, it is clear that bot share a lot of disjuncts in common; the
 handful that are in 
\begin_inset Formula $\vec{w}$
\end_inset

 that are not also in 
\begin_inset Formula $\vec{g}$
\end_inset

 could serve as a basis for 
\begin_inset Formula $\vec{t}$
\end_inset

.
 That's a start, but not a terribly good one.
\end_layout

\begin_layout Standard
A different extremum would come from finding some 
\begin_inset Formula $\vec{s}$
\end_inset

 that satisfies 
\begin_inset Formula $\vec{s}\cdot\vec{*}=0$
\end_inset

.
 It is fairly straight-forward to see that such an 
\begin_inset Formula $\vec{s}$
\end_inset

, with all non-negative coefficients, does not exist: By definition, 
\begin_inset Formula $\vec{*}$
\end_inset

 has all-positive coefficients, 
\emph on
i.e.

\emph default
 
\begin_inset Formula $N\left(*,d\right)>0$
\end_inset

 for all 
\begin_inset Formula $d$
\end_inset

; none are zero.
 Any vector orthogonal to this must have at least one negative coefficient
 (excluding the trivial zero-vector).
 This also suggests what the almost-orthogonal vectors will be: the almost-ortho
gonal vectors will be those 
\begin_inset Formula $\vec{s}$
\end_inset

 for which most 
\begin_inset Formula $N\left(s,d\right)$
\end_inset

 are zero, except for a few 
\begin_inset Formula $d$
\end_inset

 where 
\begin_inset Formula $N\left(*,d\right)$
\end_inset

 is small.
 That is, sort the set of 
\begin_inset Formula $\left\{ d\right\} $
\end_inset

 according to smallest 
\begin_inset Formula $N\left(*,d\right)$
\end_inset

 and then make linear combinations of the the first few in that sorted set.
 More specifically, the only allowed combinations are those for which 
\begin_inset Formula $N\left(w,d\right)>0$
\end_inset

, and so then search for that 
\begin_inset Formula $d$
\end_inset

 in that set with the smallest 
\begin_inset Formula $N\left(*,d\right)$
\end_inset

.
 Of course, this only works if that same 
\begin_inset Formula $d$
\end_inset

 has appreciable overlap with 
\begin_inset Formula $\vec{g}$
\end_inset

 i.e.
 if 
\begin_inset Formula $N\left(g,d\right)$
\end_inset

 is large or at least non-zero.
\end_layout

\begin_layout Standard
This is now gelling towards an algorithm: Obtain the set 
\begin_inset Formula $\left\{ d\right\} =\left\{ d|N\left(w,d\right)N\left(g,d\right)>0\right\} $
\end_inset

 and rank that set, from highest to lowest, according to 
\begin_inset Formula $N\left(g,d\right)/N\left(*,d\right)$
\end_inset

.
 Pick that 
\begin_inset Formula $d$
\end_inset

 and set 
\begin_inset Formula $N\left(s,d\right)=1$
\end_inset

.
 Now pick the second disjunct in that list, call it 
\begin_inset Formula $d^{\prime}$
\end_inset

 and consider fractions of the form 
\begin_inset Formula 
\[
F=\frac{N\left(g,d\right)+\alpha N\left(g,d^{\prime}\right)}{N\left(*,d\right)+\alpha N\left(*,d^{\prime}\right)}
\]

\end_inset

and solve for the non-negative 
\begin_inset Formula $\alpha$
\end_inset

 that maximizes this fraction.
 It appears that the only possible solution to this is 
\begin_inset Formula $\alpha=0$
\end_inset

; this is forced, since by definition 
\begin_inset Formula $N\left(g,d\right)/N\left(*,d\right)>N\left(g,d^{\prime}\right)/N\left(*,d^{\prime}\right)$
\end_inset

.
 Thus, the way to maximize 
\begin_inset Formula $MI\left(\vec{g},\vec{s}\right)$
\end_inset

 is to set 
\begin_inset Formula $N\left(s,d\right)=N\left(w,d\right)$
\end_inset

 and set all other 
\begin_inset Formula $N\left(s,d^{\prime}\right)=0$
\end_inset

.
 This leaves behind 
\begin_inset Formula $N\left(t,d\right)=0$
\end_inset

 and 
\begin_inset Formula $N\left(t,d^{\prime}\right)=N\left(w,d^{\prime}\right)$
\end_inset

 for all other 
\begin_inset Formula $d^{\prime}$
\end_inset

.
 However, it is likely that the result 
\begin_inset Formula $t$
\end_inset

 still has a large 
\begin_inset Formula $MI\left(\vec{g},\vec{t}\right)$
\end_inset

 and so this has not been minimized.
 Recall, we want to maximize 
\begin_inset Formula $MI\left(\vec{g},\vec{s}\right)$
\end_inset

 and also simultaneously minimize 
\begin_inset Formula $MI\left(\vec{g},\vec{t}\right)$
\end_inset

.
 It seems that there is a trade-off between the two.
 
\end_layout

\begin_layout Standard
In fact, we want to maximize some linear combination of 
\begin_inset Formula $MI\left(\vec{g},\vec{s}\right)$
\end_inset

 and 
\begin_inset Formula $MI\left(\vec{g},\vec{t}\right)$
\end_inset

.
 Since the MI are being interpreted as entropies, it seems like straight
 addition is the thing to do, and so perhaps the thing to maximize is 
\begin_inset Formula $S=MI\left(\vec{g},\vec{s}\right)-MI\left(\vec{g},\vec{t}\right)$
\end_inset

.
 Another possibility might be 
\begin_inset Formula $H=p\left(\vec{s}\right)MI\left(\vec{g},\vec{s}\right)-p\left(\vec{t}\right)MI\left(\vec{g},\vec{t}\right)$
\end_inset

 where 
\begin_inset Formula $p\left(\vec{w}\right)$
\end_inset

 is some probability; possibly, for example, 
\begin_inset Formula $p\left(\vec{w}\right)=i\left(w,*\right)/i\left(*,*\right)$
\end_inset

 seems to be 
\begin_inset Quotes eld
\end_inset

natural
\begin_inset Quotes erd
\end_inset

.
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Is there any chance that we got lucky, that by adding enough complexity
 to the problem, we arrived where, maybe this time, the variational arguments
 will work? Let's try them on for size, proceeding naively, as before.
 Just as before, but in altered notation, 
\begin_inset Formula 
\[
\frac{\partial}{\partial N\left(s,d\right)}\vec{s}\cdot\vec{w}=\frac{\partial}{\partial N\left(s,d\right)}\sum_{d^{\prime}}N\left(s,d^{\prime}\right)N\left(w,d^{\prime}\right)=N\left(w,d\right)
\]

\end_inset

and so
\begin_inset Formula 
\begin{align*}
0=\frac{\partial S}{\partial N\left(s,d\right)} & =\frac{\partial}{\partial N\left(s,d\right)}\left[\log_{2}\frac{\vec{s}\cdot\vec{g}}{\vec{s}\cdot\vec{*}}-\log_{2}\frac{\left(\vec{w}-\vec{s}\right)\cdot\vec{g}}{\left(\vec{w}-\vec{s}\right)\cdot\vec{*}}\right]\\
 & =\frac{\vec{s}\cdot\vec{*}}{\vec{s}\cdot\vec{g}}\left(\frac{N\left(g,d\right)}{\vec{s}\cdot\vec{*}}-\frac{\vec{s}\cdot\vec{g}\,N\left(*,d\right)}{\left(\vec{s}\cdot\vec{*}\right)^{2}}\right)-\frac{\left(\vec{w}-\vec{s}\right)\cdot\vec{*}}{\left(\vec{w}-\vec{s}\right)\cdot\vec{g}}\left(\frac{-N\left(g,d\right)}{\left(\vec{w}-\vec{s}\right)\cdot\vec{*}}+\frac{\left(\vec{w}-\vec{s}\right)\cdot\vec{g}\,N\left(*,d\right)}{\left(\left(\vec{w}-\vec{s}\right)\cdot\vec{*}\right)^{2}}\right)
\end{align*}

\end_inset

What a mess.
 Continuing as before, we run into the same problem as before: an over-constrain
ed/mis-constrained system of equations:
\begin_inset Formula 
\[
0=N\left(*,d\right)\vec{s}\cdot\vec{g}\vec{w}\cdot\vec{*}\left(\vec{s}-\vec{w}\right)\cdot\vec{g}-N\left(g,d\right)\vec{s}\cdot\vec{*}\vec{w}\cdot\vec{g}\left(\vec{s}-\vec{w}\right)\cdot\vec{*}
\]

\end_inset

 or
\begin_inset Formula 
\[
\frac{N\left(g,d\right)}{N\left(*,d\right)}=\frac{\vec{s}\cdot\vec{g}\,\vec{w}\cdot\vec{*}\,\left(\vec{s}-\vec{w}\right)\cdot\vec{g}}{\vec{s}\cdot\vec{*}\,\vec{w}\cdot\vec{g}\,\left(\vec{s}-\vec{w}\right)\cdot\vec{*}}
\]

\end_inset

so that, again, the LHS depends on 
\begin_inset Formula $d$
\end_inset

 and the RHS does not, making it impossible to satisfy in general.
 Lets try again with 
\begin_inset Formula $H$
\end_inset

, making use of 
\begin_inset Formula 
\[
\frac{\partial}{\partial N\left(s,d\right)}p\left(\vec{s}\right)=\frac{1}{i\left(*,*\right)}\,\frac{\partial}{\partial N\left(s,d\right)}i\left(s,*\right)=\frac{N\left(*,d\right)}{\vec{*}\cdot\vec{*}}=-\frac{\partial}{\partial N\left(s,d\right)}p\left(\vec{t}\right)
\]

\end_inset

It is convenient to write 
\begin_inset Formula $p\left(\vec{s}\right)i\left(*,*\right)=\vec{s}\cdot\vec{*}$
\end_inset

 and drop the 
\begin_inset Formula $i\left(*,*\right)$
\end_inset

 as a constant.
\begin_inset Formula 
\begin{align*}
0=\frac{\partial H}{\partial N\left(s,d\right)}= & \frac{\left(\vec{s}\cdot\vec{*}\right)^{2}}{\vec{s}\cdot\vec{g}}\left(\frac{N\left(g,d\right)}{\vec{s}\cdot\vec{*}}-\frac{\vec{s}\cdot\vec{g}\,N\left(*,d\right)}{\left(\vec{s}\cdot\vec{*}\right)^{2}}\right)+N\left(*,d\right)\log_{2}\frac{\vec{s}\cdot\vec{g}}{\vec{s}\cdot\vec{*}}\\
 & -\frac{\left(\left(\vec{w}-\vec{s}\right)\cdot\vec{*}\right)^{2}}{\left(\vec{w}-\vec{s}\right)\cdot\vec{g}}\left(\frac{-N\left(g,d\right)}{\left(\vec{w}-\vec{s}\right)\cdot\vec{*}}+\frac{\left(\vec{w}-\vec{s}\right)\cdot\vec{g}\,N\left(*,d\right)}{\left(\left(\vec{w}-\vec{s}\right)\cdot\vec{*}\right)^{2}}\right)+N\left(*,d\right)\log_{2}\frac{\left(\vec{w}-\vec{s}\right)\cdot\vec{g}}{\left(\vec{w}-\vec{s}\right)\cdot\vec{*}}
\end{align*}

\end_inset

Again, quite the mess.
 Grouping terms, we see that this too has the same failures as before: It
 lacks any terms that are linear in 
\begin_inset Formula $N\left(s,d\right)$
\end_inset

 which are what's needed to make the equations solvable.
 So we conclude, wtf, what were we thinking?
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The problem is of linear-programming-type, but not linear.
 Maximizing 
\begin_inset Formula $S$
\end_inset

 means maximizing 
\begin_inset Formula 
\[
\frac{\vec{s}\cdot\vec{g}}{\vec{s}\cdot\vec{*}}\frac{\left(\vec{w}-\vec{s}\right)\cdot\vec{*}}{\left(\vec{w}-\vec{s}\right)\cdot\vec{g}}=\frac{\vec{s}\cdot\vec{g}}{\vec{s}\cdot\vec{*}}\frac{\vec{t}\cdot\vec{*}}{\vec{t}\cdot\vec{g}}
\]

\end_inset

subject to 
\begin_inset Formula $0<\vec{s}\le\vec{w}$
\end_inset

.
 This is certainly not linear; so it's actually a 
\begin_inset Quotes eld
\end_inset

convex programming
\begin_inset Quotes erd
\end_inset

 problem, I guess.
 Its not that bad (I think) because there are only four dot products.
 It is slightly confusing because it is completely projective in all four
 vectors: the magnitudes all cancel.
\end_layout

\begin_layout Standard
Define sets that define the support of the vectors.
 Let 
\begin_inset Formula $G=\left\{ d|N\left(g,d\right)>0\right\} $
\end_inset

 and likewise for 
\begin_inset Formula $W$
\end_inset

 and 
\begin_inset Formula $S,T$
\end_inset

.
 By definition 
\begin_inset Formula $N\left(*,d\right)>0$
\end_inset

 for all 
\begin_inset Formula $d$
\end_inset

, so this is the universe 
\begin_inset Formula $U$
\end_inset

.
 The constraint is that 
\begin_inset Formula $W=S\cup T$
\end_inset

.
 By definition, 
\begin_inset Formula $G\cap W\ne\varnothing$
\end_inset

.
 By presumption, 
\begin_inset Formula $G\subset U$
\end_inset

 and 
\begin_inset Formula $W\subset U$
\end_inset

 are proper subsets.
\end_layout

\begin_layout Standard
To proceed, turn this into a decision problem: for and fixed disjunct 
\begin_inset Formula $d$
\end_inset

, select some 
\begin_inset Formula $\alpha$
\end_inset

 such that 
\begin_inset Formula $N\left(s,d\right)=\alpha N\left(w,d\right)$
\end_inset

 and 
\begin_inset Formula $N\left(t,d\right)=\left(1-\alpha\right)N\left(w,d\right)$
\end_inset

.
 To keep things easy for now, make this a binary decision problem: 
\begin_inset Formula $\alpha$
\end_inset

 is zero or one.
 Holding all other 
\begin_inset Formula $N$
\end_inset

 fixed, the optimization problem is to maximize
\begin_inset Formula 
\[
\frac{j_{d}\left(s,g\right)+\alpha N\left(w,d\right)N\left(g,d\right)}{j_{d}\left(s,*\right)+\alpha N\left(w,d\right)N\left(*,d\right)}\;\frac{j_{d}\left(t,*\right)+\left(1-\alpha\right)N\left(w,d\right)N\left(*,d\right)}{j_{d}\left(t,g\right)+\left(1-\alpha\right)N\left(w,d\right)N\left(g,d\right)}
\]

\end_inset

where 
\begin_inset Formula $j_{d}\left(u,w\right)=i\left(u,w\right)-N\left(u,d\right)N\left(w,d\right)$
\end_inset

, that is, the dot product, excluding the 
\begin_inset Formula $d$
\end_inset

 term.
 This really is a binary decision problem: one need only look at the two
 endpoints 
\begin_inset Formula $\alpha=0$
\end_inset

 and 
\begin_inset Formula $\alpha=1$
\end_inset

.
 It is either true that 
\begin_inset Formula 
\[
\frac{j_{d}\left(s,g\right)}{j_{d}\left(s,*\right)}\;\frac{j_{d}\left(t,*\right)+N\left(w,d\right)N\left(*,d\right)}{j_{d}\left(t,g\right)+N\left(w,d\right)N\left(g,d\right)}<\frac{j_{d}\left(s,g\right)+N\left(w,d\right)N\left(g,d\right)}{j_{d}\left(s,*\right)+N\left(w,d\right)N\left(*,d\right)}\;\frac{j_{d}\left(t,*\right)}{j_{d}\left(t,g\right)}
\]

\end_inset

or it is false.
 That is, if LHS < RHS holds true, then 
\begin_inset Formula $\alpha=1$
\end_inset

 maximizes, and the disjunct 
\begin_inset Formula $d$
\end_inset

 belongs to the set 
\begin_inset Formula $S$
\end_inset

; otherwise 
\begin_inset Formula $d$
\end_inset

 belongs in set 
\begin_inset Formula $T$
\end_inset

.
\end_layout

\begin_layout Standard
This is readily computed numerically.
\end_layout

\begin_layout Standard
So this is actually a binary decision problem, more narrowly, an binary-integer
 programming problem.
 It's supposed NP-hard, but I'm thinking that this is a special case that
 might not be.
 A plausible algo is to rank the disjuncts by size of 
\begin_inset Formula $N\left(*,d\right)$
\end_inset

 from greatest to least, and assign the disjuncts, one by one, to either
 of the two classes.
 One can do likewise for 
\begin_inset Formula $H$
\end_inset

 instead of 
\begin_inset Formula $S$
\end_inset

, just using a different, more complex decision.
 Both run in linear time according to size of 
\begin_inset Formula $W\cap G$
\end_inset

.
\end_layout

\begin_layout Standard
As before, the niput data, coming from disjuncts from WSD parses, is noisey,
 and so a binary assignment might be incorrect.
 It might be better to assign only a fraction of the passing disjuncts to
 
\begin_inset Formula $\vec{s}$
\end_inset

 and place the remainder in 
\begin_inset Formula $\vec{t}$
\end_inset

.
 This requires yet more experimentation.
\end_layout

\begin_layout Section*
Link Merge Redux
\end_layout

\begin_layout Standard
The word-disjunct dataset, containing counts 
\begin_inset Formula $N\left(w,d\right)$
\end_inset

, is far too large to be used as a practical dictionary for parsing.
 It also fails to provide sufficient coverage to parse most sentences.
 Thus, it needs to be simultaneously be compressed and also generalized.
 This is partly achieved by clustering words into word-classes.
 However, the current system leaves the connectors un-clustered.
 That is, each disjunct is an ordered sequence of (pseudo-)connectors: 
\begin_inset Formula 
\[
d=\left(c_{1},c_{2},\cdots,c_{k}\right)
\]

\end_inset

with each (pseudo-)connector being a word plus direction (polarity) indicator:
\begin_inset Formula 
\[
c_{j}=\left[w_{j},p_{j}\right]
\]

\end_inset

with 
\begin_inset Formula $p_{j}\in\left\{ +,-\right\} $
\end_inset

.
 We wish to cluster these, replacing the 
\begin_inset Formula $c_{j}$
\end_inset

 with a 
\begin_inset Formula $\gamma_{j}=\left[g_{j},p_{j}\right]$
\end_inset

 where each 
\begin_inset Formula $g_{j}$
\end_inset

 is a grammatical class, with 
\begin_inset Formula $w_{j}\in g_{j}$
\end_inset

.
 A naive (but wrong) conception of self-consistency it to have the 
\begin_inset Formula $g_{j}$
\end_inset

's appearing in connectors be the same grammatical classes as those used
 to categorize words.
 The notion of self-consistent connector classes is different (given below).
 At any rate, how does one cluster connectors? 
\end_layout

\begin_layout Standard
To proceed, a notation that breaks out the spider diagram is needed.
 Instead of writing 
\begin_inset Formula $N\left(w,d\right)$
\end_inset

, one actually has a collection of observation counts 
\begin_inset Formula 
\[
N\left(w,\left(\left[w_{1},p_{1}\right],\left[w_{2},p_{2}\right],\cdots,\left[w_{k},p_{k}\right]\right)\right)
\]

\end_inset

with 
\begin_inset Formula $k$
\end_inset

 being the arity of the disjunct.
 Square brackets are used for easy reading.
 Prior to clustering, one can convert pseudo-connectors to real connectors
 (and real links) by observing that word 
\begin_inset Formula $w_{a}$
\end_inset

 (on the left) can link to word 
\begin_inset Formula $w_{b}$
\end_inset

 (on the right) if an only if exists two word-disjunct pairs 
\begin_inset Formula $\left(w_{a},d_{a}\right)$
\end_inset

 and 
\begin_inset Formula $\left(w_{b},d_{b}\right)$
\end_inset

 such that 
\begin_inset Formula $d_{a}=\left(\left[w_{1},p_{1}\right],\cdots,\left[w_{j},p_{j}\right],\cdots,\left[w_{k},p_{k}\right]\right)$
\end_inset

 and 
\begin_inset Formula $d_{b}=\left(\left[w_{1},p_{1}\right],\cdots,\left[w_{m},p_{m}\right],\cdots,\left[w_{n},p_{n}\right]\right)$
\end_inset

 and 
\begin_inset Formula $w_{j}=w_{b}$
\end_inset

 and 
\begin_inset Formula $p_{j}=+$
\end_inset

 and 
\begin_inset Formula $w_{m}=w_{a}$
\end_inset

 and 
\begin_inset Formula $p_{m}=-$
\end_inset

.
 A link can then be defined as any function that provides a unique label
 
\begin_inset Formula $l_{ab}$
\end_inset

 to the word-pair 
\begin_inset Formula $(w_{a},w_{b})$
\end_inset

.
 Given a 
\begin_inset Quotes eld
\end_inset

true link
\begin_inset Quotes erd
\end_inset

 type 
\begin_inset Formula $l$
\end_inset

, the corresponding 
\begin_inset Quotes eld
\end_inset

true connectors
\begin_inset Quotes erd
\end_inset

 are then 
\begin_inset Formula $l+$
\end_inset

 and 
\begin_inset Formula $l-$
\end_inset

.
 Thus, for the above example, 
\begin_inset Formula $\left[w_{j},p_{j}\right]\to l_{ab}+$
\end_inset

 and 
\begin_inset Formula $\left[w_{m},p_{m}\right]\to l_{ab}-$
\end_inset

.
\end_layout

\begin_layout Standard
The clustering task is now to find all other links that can be argued to
 be 
\begin_inset Quotes eld
\end_inset

similar
\begin_inset Quotes erd
\end_inset

.
 One very simple ansatz is to create a link 
\begin_inset Formula $l_{gg^{\prime}}$
\end_inset

 that corresponds to the gram-class pair 
\begin_inset Formula $\left(g,g^{\prime}\right)$
\end_inset

 and is used to connect any word-pair 
\begin_inset Formula $(w_{a},w_{b})$
\end_inset

 whenever 
\begin_inset Formula $w_{a}\in g$
\end_inset

 and 
\begin_inset Formula $w_{b}\in g^{\prime}.$
\end_inset

 This ansataz has a strong 
\begin_inset Quotes eld
\end_inset

broadening
\begin_inset Quotes erd
\end_inset

 effect, as the generic link type allows words in 
\begin_inset Formula $g$
\end_inset

 and 
\begin_inset Formula $g^{\prime}$
\end_inset

 to connect, for which actual links were never observed at earlier stages.
 (This is the variant used in the 
\begin_inset Quotes eld
\end_inset

baseline
\begin_inset Quotes erd
\end_inset

 datasets measured in the grammar baseline report.) 
\end_layout

\begin_layout Standard
How might one determine if two links are similar? The same way that one
 determines if any other vectors are similar.
 In this case, one starts with the double-spider diagram:
\end_layout

\begin_layout Standard
\align center
\begin_inset Graphics
	filename images/seeds-two.eps
	width 60col%

\end_inset


\end_layout

\begin_layout Standard
The disjoint union of the open circles (the unconnected connectors) define
 a defacto basis element for the two linked words.
 Associated with this is a product of observation counts, which can effectively
 be used to define any distance metric previously considered.
 Notationally, write 
\begin_inset Formula 
\[
d_{a}=\left(c_{1}^{\left(a\right)},\cdots,c_{j}^{\left(a\right)},\cdots,c_{k}^{\left(a\right)}\right)
\]

\end_inset

just as before, but now with a superscript 
\begin_inset Formula $\left(a\right)$
\end_inset

 to tell the connectors apart from hos on 
\begin_inset Formula $d_{b}$
\end_inset

.
 The connector set on 
\begin_inset Formula $l_{ab}$
\end_inset

 is then 
\begin_inset Formula 
\[
d_{ab}=\left(c_{1}^{\left(a\right)},\cdots,\widehat{c_{j}^{\left(a\right)}},\cdots,c_{k}^{\left(a\right)},c_{1}^{\left(b\right)},\cdots,\widehat{c_{m}^{\left(b\right)}},\cdots,c_{n}^{\left(b\right)}\right)
\]

\end_inset

with the widehat notation 
\begin_inset Formula $\widehat{c_{i}}$
\end_inset

 indicating that connector 
\begin_inset Formula $c_{i}$
\end_inset

 is absent from the list.
 Both 
\begin_inset Formula $c_{j}^{\left(a\right)}$
\end_inset

 and 
\begin_inset Formula $c_{m}^{\left(b\right)}$
\end_inset

 need to be absent, as those are the two connectors forming the link.
 Thus, associated with every 
\begin_inset Formula $l_{ab}$
\end_inset

 is a (fairly large) set of 
\begin_inset Formula $d_{ab}$
\end_inset

.
 To turn this into a bona-fide vector, a count is needed.
 It seems reasonable to define 
\begin_inset Formula 
\[
N\left(l_{ab},d_{ab}\right)=N\left(w_{a},d_{a}\right)N\left(w_{b},d_{b}\right)
\]

\end_inset

One might consider adjusting the above by counts on connectors, or ..
 something...
 but this seems like un-needed complexity, at this point.
\end_layout

\begin_layout Standard
How practical is this? Lets look at the number of potential links
\begin_inset Foot
status collapsed

\begin_layout Plain Layout
Counted with the 
\family typewriter
count-quarter-links
\family default
, 
\family typewriter
count-half-links
\family default
 and 
\family typewriter
count-links
\family default
 functions in 
\family typewriter
count-links.scm.
\end_layout

\end_inset

 in various datasets:
\end_layout

\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="7">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Dimensions
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
qrt-links
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
half-links
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
pairs
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
full-links
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Secs
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Name
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1610 x 67K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
146K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
309K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
30.4K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2.45M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
184K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_micro_marg
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7385 x 270K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
864K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.33M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
88.6K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.8M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
608K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_mini_marg
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
17K x 947K
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3.78M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
4.64M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
1.56M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_large_marg
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
55K x 6.03M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
21.1M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
23.9M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
7.91M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_huge_marg
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
438K x 23.3M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
76.9M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
90.1M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
31.7M
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
en_full_marg
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Standard
The columns in the table are:
\end_layout

\begin_layout Itemize

\series bold
Dimensions
\series default
 – Number of words x Number of disjuncts
\end_layout

\begin_layout Itemize

\series bold
qrt-links
\series default
 – Total number of unique links that can be made between some word in the
 dataset, and some connector in some disjunct.
 This is a 
\begin_inset Quotes eld
\end_inset

quarter-link
\begin_inset Quotes erd
\end_inset

, as any given disjunct might be shared by multiple (word,disjunct) pairs.
 The counting is without multiplicity (
\emph on
i.e.

\emph default
 if the connector appears twice in the sequence, it is counted only once.)
\end_layout

\begin_layout Itemize

\series bold
half-links
\series default
 – Total number of unique links that can be made between some word in the
 dataset, and some connector in any (word, disjunct) pair.
 This is a 
\begin_inset Quotes eld
\end_inset

half-link
\begin_inset Quotes erd
\end_inset

, as the relation is not symmetric; to be symmetric, i.e.
 a full-link, there needs to be another matching half-link going in the
 opposite direction.
 The counting is without multiplicity (
\emph on
i.e.

\emph default
 if the connector appears twice in the sequence, it is counted only once.)
\end_layout

\begin_layout Itemize

\series bold
pairs
\series default
 – Total number of word-pairs having full-links between them.
\end_layout

\begin_layout Itemize

\series bold
full-links
\series default
 – Total number of unique full-links in the dataset, as defined above.
 
\end_layout

\begin_layout Itemize

\series bold
Secs
\series default
 – Number of sections in the dataset.
\end_layout

\begin_layout Itemize

\series bold
Name
\series default
 – Dataset name
\end_layout

\begin_layout Section*
Disjunct dataset redo
\end_layout

\begin_layout Standard
The code that converted MST oparses to disjuncts had a bug in it – sentences
 with multiple instances of the same word created disjuncts that were incorrect.
 Fixed in pull req opencog/atomspace#2252.
 However, need to redo the DB's.
\end_layout

\begin_layout Standard
While we are at it, here's a new processing pipeline:
\end_layout

\begin_layout Itemize
Filter the word-pair database
\emph on
 en_pairs_cfive
\emph default
 so that all word-pairs with MI<1.8 are removed.
 The value of 1.8 seems reasonable, as this is the mean MI of that database.
\end_layout

\begin_layout Itemize
Instead of creating MST parses, create MPG parses: 
\begin_inset Quotes eld
\end_inset

Maximum Planar Graph
\begin_inset Quotes erd
\end_inset

 parses.
 These start with the MST parse, and add additional high-MI edges, creating
 a planar graph with loops.
 The resulting disjuncts are larger.
\end_layout

\begin_layout Itemize
At a later time, e.g.
 during clustering, the disjuncts can be trimmed down (if desired) by looking
 up the MI between germ and connector, and discarding it if it falls below
 some threshold.
 Thus, MPG disjuncts are both 
\begin_inset Quotes eld
\end_inset

fuller
\begin_inset Quotes erd
\end_inset

 than MST disjuncts, but can also be trimmed in a fairly coherent fashion,
 offering yet another way to limit dataset size (in exchange for additional
 processing :-o).
\end_layout

\begin_layout Section*
Reading list
\end_layout

\begin_layout Standard
Papers that people want me to read and have an opinion on:
\end_layout

\begin_layout Itemize
https://arxiv.org/abs/1904.03746
\end_layout

\begin_layout Section*
TODO
\end_layout

\begin_layout Standard
Explain how mutual exclusion of concepts as performed by humans when learning
 new concepts, resembles optimal strategies for the channel coding theorem,
 by minimizing confusion between similar concepts.
 This is the 
\begin_inset Quotes eld
\end_inset

mutual exclusion
\begin_inset Quotes erd
\end_inset

 principle.
 Well, MI already provides a certain measure of exclusivity.
\end_layout

\begin_layout Section*
The End of Part One
\end_layout

\begin_layout Standard
This diary became too long to be manageable; it was closed out on 26 February
 2021.
 This diary resumes at learn-lang-diary-2.
\end_layout

\begin_layout Standard
\begin_inset CommandInset bibtex
LatexCommand bibtex
bibfiles "lang"
options "alpha"

\end_inset


\end_layout

\end_body
\end_document