Skip to content

Latest commit

 

History

History
492 lines (391 loc) · 9.85 KB

Usage.md

File metadata and controls

492 lines (391 loc) · 9.85 KB

Lexd Syntax

Invocation

The lexd binary generates AT&T format transducers.

Sample, save to verb.lexd:

PATTERNS
VerbRoot VerbInfl

LEXICON VerbRoot
sing
walk
dance

LEXICON VerbInfl
<v><pres>:
<v><pres><p3><sg>:s

Compile it (without flag diacritics) to ATT transducer format:

$ lexd verb.lexd > verb-generator.att

To compile to an lttoolbox transducer binary dictionary, use lt-comp; this can be used for lookup with lt-proc:

$ lt-comp rl verb-generator.att verb-analyser.bin
main@standard 17 19
$ echo 'sings' | lt-proc verb-analyser.bin
^sings/sing<v><pres><p3><sg>$

To extract forms, use the HFST to first compile to hfst binary format:

$ hfst-txt2fst verb-generator.att -o verb-generator.hfst

Then you can use hfst-fst2strings:

$ hfst-fst2strings verb-generator.hfst
sing<v><pres>:sing
sing<v><pres><p3><sg>:sings
walk<v><pres>:walk
walk<v><pres><p3><sg>:walks
dance<v><pres>:dance
dance<v><pres><p3><sg>:dances

Basic Syntax

A Lexd rule file defines lexicons and patterns. Each lexicon consists of a list of entries which have an analysis side and a generation side, similar to lexicons in HFST Lexc. Patterns, meanwhile, replace Lexc's continuation lexicons. Each pattern consists of a list of lexicons or named patterns which the compiler concatenates in that order.

PATTERNS
VerbRoot VerbInfl

LEXICON VerbRoot
sing
walk
dance

LEXICON VerbInfl
<v><pres>:
<v><pres><p3><sg>:s

forms generated:

sing/sing<v><pres>
sings/sing<v><pres><p3><sg>
walk/walk<v><pres>
walks/walk<v><pres><p3><sg>
dance/dance<v><pres>
dances/dance<v><pres><p3><sg>

Symbols enclosed in angle brackets or braces will be automatically interpreted as multicharacter symbols (presumably tags and archiphonemes, respectively):

PATTERNS
X

LEXICON X
x<ij>:x{i}

resulting ATT file:

0	1	x	x	0.000000	
1	2	<ij>	{i}	0.000000	
2	0.000000

Any character can be escaped with a backslash:

PATTERNS
X

LEXICON X
x\<ij>:x{i}

resulting ATT file:

0	1	x	x	0.000000	
1	2	<	{i}	0.000000	
2	3	i	@0@	0.000000	
3	4	j	@0@	0.000000	
4	5	>	@0@	0.000000	
5	0.000000

And comments begin with #.

Alignment

Patterns can list different sides of each lexicon in different places. When the compiler encounters a one-sided lexicon reference in a pattern, it attaches all entries from that side of that lexicon to the transducer and then builds the rest of the pattern, attaching a separate copy for each entry. However, in these copies, for any subsequent mentions of that lexicon, only the corresponding segment of that entry will be attached, thus avoiding over-generation. The same lexicon can be mentioned arbitrarily many times, making it straightforward to write rules for phenomena such as reduplication.

PATTERNS
:VerbInfl VerbRoot VerbInfl:
:VerbInfl :VerbRoot VerbRoot VerbInfl: Redup:

LEXICON VerbRoot
bloop
vroom

LEXICON VerbInfl
<v><pres>:en

LEXICON Redup
<redup>:

forms generated:

enbloop/bloop<v><pres>
envroom/vroom<v><pres>
enbloopbloop/bloop<v><pres><redup>
envroomvroom/vroom<v><pres><redup>

To handle more complex cases, such as infixation and Semitic triliteral roots, lexicon entries can have multiple segments which patterns can refer to independently.

PATTERNS
C(1) :V(1) C(2) :V(2) C(3) V(2):

LEXICON C(3)
sh m r
y sh v

LEXICON V(2)
:a <v><p3><sg>:a
:o <v><pprs>:e

forms generated:

shamar/shmr<v><p3><sg>
shomer/shmr<v><pprs>
yashav/yshv<v><p3><sg>
yoshev/yshv<v><pprs>

It is also possible to give lexicons multiple names using the ALIAS command, which allows patterns to refer to multiple independent copies, which can then be used for productive compounding.

PATTERNS
NounStem NounInfl
NounStem NounInflComp Comp NounStem2 NounInfl

LEXICON Comp
<comp>+:

LEXICON NounStem
shoop
blarg

ALIAS NounStem NounStem2

LEXICON NounInfl
<n><sg>:
<n><pl>:ah

LEXICON NounInflComp
<n>:a

forms generated:

shoop/shoop<n><sg>
shoopah/shoop<n><pl>
shoopashoop/shoop<n><comp>+shoop<n><sg>
shoopashoopah/shoop<n><comp>+shoop<n><pl>
shoopablarg/shoop<n><comp>+blarg<n><sg>
shoopablargah/shoop<n><comp>+blarg<n><pl>
blarg/blarg<n><sg>
blargah/blarg<n><pl>
blargashoop/blarg<n><comp>+shoop<n><sg>
blargashoopah/blarg<n><comp>+shoop<n><pl>
blargablarg/blarg<n><comp>+blarg<n><sg>
blargablargah/blarg<n><comp>+blarg<n><pl>

Patterns can be named and included in other patterns. In addition to being less repetitive to write, it also compiles faster.

PATTERN VerbStem
VerbRoot
VerbRoot Causative
AuxRoot

PATTERNS
VerbStem Tense PersonNumber

This is equivalent to

PATTERNS
VerbRoot Tense PersonNumber
VerbRoot Causative Tense PersonNumber
AuxRoot Tense PersonNumber

Pattern Operators

Some simple operators are supported to help write patterns concisely:

  • the option quantifier ? can be applied to a single token
PATTERNS
Negation? Adjective
# equivalent to:
# Negation Adjective
# Adjective

Placing the ? quantifier between a lexicon name and the segment number will make that lexicon as a whole optional in that pattern:

PATTERNS
OptionalCircumfix?(1) Stem OptionalCircumfix?(2)
# equivalent to:
# OptionalCircumfix(1) Stem OptionalCircumfix(2)
# Stem

Note that in this case the column specification after the ? is required.

PATTERNS
:Prefix?(1) Stem Prefix?(1):
# equivalent to
# :Prefix(1) Stem Prefix(1): => :Prefix Stem Prefix:
# Stem
:Prefix? Stem Prefix:?
# equivalent to
# :Prefix Stem Prefix:
# :Prefix Stem
# Stem Prefix:
# Stem

The quantifiers * (repeat 0 or more times) and + (repeat 1 or more times) function similarly, though they only support modification of a single token and not distributed modification of a lexicon across an entire line.

  • the alternation operator | between two tokens causes one pattern for each alternate
PATTERNS
VerbStem Case

PATTERN Case
Absolutive
Oblique Ergative|Genitive
# equivalent to:
# Oblique Ergative
# Oblique Genitive
  • the sieve operators < and > allow left and right extensions
PATTERNS
VerbStem > Nominalisation > Case
# equivalent to:
# VerbStem
# VerbStem Nominalisation
# VerbStem Nominalisation Case

Anonymous Lexicons and Patterns

Patterns can contain anonymous lexicons to avoid needing to explicitly declare lexicons for very simple things.

PATTERNS
NounStem [<n>:] NounNumber

LEXICON NounStem
sock
ninja

LEXICON NounNumber
<sg>:
<pl>:s

forms generated:

ninja/ninja<n><sg>
ninjas/ninja<n><pl>
sock/sock<n><sg>
socks/sock<n><pl>

Anonymous patterns function similarly:

PATTERNS
(VerbRoot Causative?) | AuxRoot Tense PersonNumber
# equivalent to:
# PATTERN VerbStem
# VerbRoot Causative?
# PATTERNS
# VerbStem|AuxRoot Tense PersonNumber

Anonymous patterns can be nested and both patterns and lexicons can be quantified:

PATTERNS
NounRoot ([<n>:] (Number Case)?) | (Verbalizer Tense)

Tags

Lexicon entries can be tagged using square brackets:

LEXICON NounRoot
sock[count]
rice[mass]
sand[count,mass]

Or tags can be applied-by-default to an entire block:

LEXICON NounRoot[count]
sock
rice[mass,-count]
sand[mass]

When referring the lexicon, these tags can then be selected for:

PATTERNS
NounRoot[count] [<n>:] Number # 'sock' and 'sand', but not 'rice'
NounRoot[mass] [<n>:]         # 'rice' and 'sand', but not 'sock'

The absense of a tag can also be selected for:

PATTERNS
NounRoot[-count] [<n>:]       # 'rice' only

Tag selectors can also be applied to patterns:

PATTERN NounStem
NounRoot [<n>:]

PATTERNS
NounStem[count] Number
NounStem[mass]

Distribution rules are as follows:

(A B)[x]  = (A[x] B) | (A B[x])
(A B)[-x] = A[-x] B[-x]

Union and symmetric difference are implemented with the following syntax:

A[|[x,y]] = A[x]    | A[y]    # union / logical or
A[^[x,y]] = A[x,-y] | A[-x,y] # symmetric difference / exclusive-or

These can be useful with lexically conditioned patterns. Here's an example showing declension paradigms and noun class:

PATTERNS
(NounStem CaseEnding)[^[Decl1,Decl2],^[N,M,F]]

LEXICON NounStem
mensa:mens[Decl1,F]     # table
poeta:poet[Decl1,M]     # poet
dominus:domin[Decl2,M]  # master
bellum:bell[Decl2,N]    # war

LEXICON CaseEnding[Decl2]
<nom>:>us[M]
<nom>:>um[N]
<acc>:>um    # M or N

LEXICON CaseEnding[Decl1]
<nom>:>a     # any gender
<acc>:>am    # any gender

produces the forms (through lexd | hfst-txt2fst | hfst-fst2strings)

poeta<nom>:poet>a
poeta<acc>:poet>am
mensa<nom>:mens>a
mensa<acc>:mens>am
bellum<nom>:bell>um
bellum<acc>:bell>um
dominus<nom>:domin>us
dominus<acc>:domin>um

The exclusive or filter in this example will produce pairs of stems and case endings such that between them, one has a declension tag and one (possibly the same one) has a gender tag, and neither has any other declension or gender tag. Thus, if we were to add a third declension such as

arbor:arbor[Decl3,F]

but we then forgot to add Decl3 to the filter in the pattern, then from the perspective of the filter, it wouldn't have a declension tag, so it would get paired with every case ending that does have a declension tag (which in this example is all of them).

Regular Expressions

If a lexicon entry begins with a forward slash, it is interpreted as a regular expression.

PATTERNS
SomeLexicon

LEXICON SomeLexicon
/x(y|zz)?[n-p]/

produces the forms

xn
xo
xp
xyn
xyo
xyp
xzzn
xzzo
xzzp

Currently Supported Syntax

  • Grouping with ()
  • Quantification with ?, *, and +
    • Currently, quantifiers may only be applied to fully parenthesized groups, so x+ is an error and must be written as (x)+.
  • Alternation with |
  • Character classes with []
  • Character ranges such as [a-z]
  • Multichar symbols, following the same rules are normal lexicon entries
  • Two-sided strings using :
    • a:b is the same as in a normal entry
    • [ab]:c is equivalent to (a:c)|(b:c)