On 18 November 2015 at 02:22, Alex Fink <[log in to unmask]> wrote:
[...]
>>Jim mentioned that one in the other thread on Word Building. It is a
>>pretty neat little tool, but it doesn't do quite exactly the same
>>things as Logopoeist does / is intended to do once I finish all the
>>pending feature ideas. Nevertheless, yes, there is a lot of
>>wheel-reinvention in this space, which is why I'd like to get a bunch
>>of existing word generators collected in one place for easy
>>comparison.
>
> This, a start has been made on on Frath:
>   http://www.frathwiki.com/Software_tools_for_conlanging#Word_generators
>
> It could do with more comparative discussion of the features of each, certainly...

That is a reasonably good list to start with. Here's what I have
managed to glean from spending a few hours investigating
general-purpose word generation options:

Awkwords:
On-line, GUI-based (source code available for download). Outputs a
simple unsorted list of words. Unicode support is unknown.
Awkwords allows up to 4 named "subpatterns", which encompass both
classes of phonemes and named re-usable subsections of phonotactic
patterns. Phonemes can be represented by multiple characters (i.e., it
permits digraphs, trigraphs, n-graphs). The symbols '/', '^', '*',
'(', ')'. '[', and ']' are reserved for the pattern syntax, but can be
included in output by enclosing them in double-quotes ("); empty
quotes are used to put an actual quote character in the output, but
unbalanced quotes cause Awkwords to crash.

Awkwords makes use of traditional "CV(C)" style syllable-structure
notation to describe words structures, with '/' used to indicate
alternative options; square brackets ([]) are used to delimit sections
where one of one or more options must be chosen, while parentheses
indicate an optional section. There are no mechanisms for arbitrary
repetition or inter-syllable dependencies. Words are described
entirely in terms of a single fixed, maximum-length (but not
fixed-length, due to optional sections) pattern.

Optional sections are included or not with equal probability, with no
way to alter the distribution. The distribution for a set of
alternatives, however, can be altered by specifying a weight for each
alternative, by following the alternative with an asterisk and an
integer between 1 and 128 (much like Logopoeist).

Awkwords also allows specifying "filters": specific strings that
cannot be generated by a particular pattern section which might
otherwise randomly produce them. These are satisfied by producing
random words, and then removing those that match filters from the
output. New words are not generated to replace those that are filtered
out, however, which means that, if you request 100 random words, you
are not guaranteed to get 100- there may be fewer. Awkwords also
allows optionally removing duplicates, which has the same effect-
replacements are not generated, reducing the size of the output from
what was requested.

Gen:
Online, GUI-based, like Awkwords. Can output a list/table of words, a
list of all possible syllables, or a random-pseudo text (like a lorem
ipsum) to demonstrate the overall "feel" of a phonology. Gen
explicitly supports unicode, and provides a GUI option for
copy-pasting IPA characters.

Gen allows defining an arbitrary number of one-character class /
category variables for phonemes. Phonemes can only be represented by a
single character- no di-, tri-, or n-graphs allowed.

Words are described by defining syllable templates consisting of
category variables and literal characters (anything that isn't
recognized as a defined variable name). Syllables are then combined
randomly- no control is available over allowed syllable combinations.
There is also no mechanism for specifying optional sections- all
possible syllable shapes, or allowed sequences of syllables, must be
explicitly listed.

Characters within a class are selected according to a power-law
dropoff (with characters listed earlier in a class definition
appearing more frequently). There is no way to manually edit the
distributions, but five intuitive options are available for the
steepness of the dropoff. Syllables are also selected according to a
power law distribution, more frequent syllable types listed first, but
there are only two options in this case.

The means of selecting word lengths is not clear, but one can control
the frequency of monosyllables vs. everything else in broad terms.

Gen also allows post-processing the randomly generated words with
rewrite rules; these are regular-expression matching rules which
replace any matched strings with a fixed replacement, and are applied
to all words in the order in which they are defined.

Lexifer:
Offline (Python), command-line /  DSL-based. Full unicode support. Can
output pseudo-text like Gen (default), or a simple list of specified
length.
Lexifer generates words based on instructions from a simple
declarative domain-specific programming language, stored in "phonology
definition files". Lexifer supports arbitrary n-graphs and custom sort
orders for its output (which can be optionally sorted or unsorted). Up
to 26 different phoneme classes can be defined (one for each capital
letter). By default, phonemes from each class are selected according
to the Gusein-Zade distribution (first listed being most frequent),
but you can specify custom weights (similar to Awkwards, Boris, and
Logopoeist) with a colon followed by a number (presumably, an
integer). This is all-or-nothing, though; if any weights are specified
for a class, all members of that class must have explicit weights.
Also like Gen, Lexifer uses syllable structure descriptions, which are
selected according to a Zipf distribution, with no inter-syllable
restrictions. Lexifer does not allow traditional "CV(C)"-style
parentheses-for-optional syllable structure descriptions or grouping
subsections of a structure like Awkwards, but does use a question mark
to indicate that a single phoneme slot is optional. A single global
setting can be used to manipulate the frequency of all optional
elements, but they cannot be set individually.

Lexifer allows you to specify disallowed patterns as regexes, which
prevent any words that would otherwise match them from being produced.
It also has a special case mechanism to disallow sequences of two
identical phonemes in a particular position, involving placing an
exclamation mark at the place in a syllable structure pattern where
the repetition should be disallowed.

Like Gen, Lexifer also allows you to specify rewrite rules that
replace specific sequences of phonemes wherever they would otherwise
occur in the generated words.

Uniquely, Lexifer comes with several built-in special-purpose
post-processing algorithms that can be selectively opted into, which
perform common operations like metathesis and assimilation; these
assume that you use IPA symbols (or built-in alternative n-graphs)
with their normal IPA values.

For proper operation, Lexifer requires specifying the complete
phonetic inventory apart from the individual phoneme classes; this is
combined with the specification of sort order.

Wordo / WordBuilder:
Online / Offline (C#), GUI + DSL-based. Unicode support is unknown.
Outputs a lot things.
WordBuilder uses a fairly complex imperative language to describe the
step-by-step construction of words by manipulating strings, including
loops, branches, and random selections from custom distributions. This
can do a whole lot of stuff, which it is difficult to summarize, but
at the expense of being more difficult to understand and analyze than
some of the other methods. Playing with it feels (easier than, but)
like you might as well just be writing another bespoke word generator
in Python or Perl.

Gleb:
Gleb is a marginal entry to this list, being primarily a phonology
generator than a word generator, but has a word generator built in in
order to demonstrate the phonologies that it generates. Not much more
I can say about it without reading the source.

Boris:
Offline (Perl), interactive command-line / DSL-based. Unicode support
is unknown. Outputs a simple word list.
Like Gleb, Boris can also generate random phonologies, and mutations
of existing phonologies, but focuses on word generation. The
interactive aspect of Boris is unique among word generators I've
looked at so far.
The Boris DSL consists of probabilistic context-free grammar rules and
rewrite rules. Grammar rules consist of a series of patterns that can
be used to replace a given syntax variable and the frequency of each
choice (specified with a colon and following integer, like Lexifer).
Patterns can consist of other syntax variables or literal characters.
The word-grammar approach allows encoding long-range dependencies
between syllables (e.g., vowel harmony rules) that are not possible in
systems that just use syllable structure patterns, with manual
specification of arbitrary distributions for both patterns and phoneme
selection. The notation is, however, a little harder to read than
traditional "CV(C)"-style syllable structure rules. Rewrite rules are
based on regex substitution, like Gen.

Everyword:
Offline (Perl), command-line / DSL-based. Unicode support is unknown.
Not technically a random word generator. Everyword generates all
possible words for a finite, fixed-length phonology (no repetitions or
recursive grammar productions such as are possible in Boris). The DSL
allows specifying disallowed phoneme sequences (like Awkwords's
filters), and defining an arbitrary number of phoneme classes which
can then be combined into arbitrary syllable structure patterns
similar to Gen. Statistical distributions are irrelevant, since it
just generates everything once.

Logopoeist:
Offline (Go, originally Python), command-line / DSL-based. Full
unicode support. Outputs a simple word list.
Like Boris, Logopoeist's configuration is based on probabilistic
context-free grammar rules. Unlike Boris (and like most other system),
however, Logopoeist distinguishes between syntax variables and phoneme
classes (it is possible to get by with just syntax rules that directly
produce phonemes as terminal symbols in Boris-style, but it's not
meant to be used that way). In my opinion, this makes the notation
easier to follow than Boris's, though it is still less familiar than
the traditional "CV(C)"-style patterns. In addition to specifying
arbitrary distributions for syntax rules and phoneme classes,
Logopoeist also allows specifying conditional distributions that alter
phoneme selections in certain contexts (based on preceding n-grams),
and disallowing specific phonemes in certain contexts (i.e., removing
them from the pre-existing distribution). Phoneme and syntax rule
frequencies are specified with asterisks, like Awkwords, but allow
arbitrary decimal values, not just integers. There are several failure
modes for unwary users: currently, there are no checks for
non-terminating grammars, which can cause Logopoeist to freeze up in
an infinite loop (until it crashes from lack of memory); additionally,
Logopoeist filters duplicates from its output, but also creates
replacements; if you request more unique words than can be generated
from a finite phonology, then it will also get stuck in an infinite
loop.
(It is tempting to describe all the cool features that Logopoeist is
*planned* to have, but for fairness I'm sticking with the state of the
currently-available release version).

There are a few other special-purpose generators I found as well, but
I'm only considering here stuff that is intended to be generally
applicable to any conlang.

Fixup-rewrite rules seem extremely popular, and it has been suggested
that somebody create a generator that allows *probabilistic*
application of such rewrite rules. In my opinion, however, this would
be more properly implemented as part of a sound-change applier, or
some other separate program that operates on lexicons after the word
generator is finished.

I'm pretty sure this covers all the tools that were mentioned in the
"Word Building" thread, and it seems the power of context-free
word-syntaxes has not yet been combined with "CV(C)"-style syllable
patterns. Additionally, nobody has created a way to specify arbitrary
distributions for the optional sections of those kinds of patterns
(Lexifer comes close, but it's a global setting, not
pattern-specific). Additionally, I've found no tools that let you
specify different pre-defined distributions (i.e., Zipf vs.
Gusein-Zade), or tweak a pre-existing distribution- it's all either
100% manually-specified weights, or one built-in distribution. These
are both features that I hope to add to Logopoeist.

Another possible approach that seems completely untapped is generating
phoneme sequences from a Markov-chain n-gram model and then filtering
the results based on syllable structure rules (to eliminate the
occasional accidental sequence of 7 consonants because you allow
clusters of three and the Markov model doesn't know those n-grams
weren't supposed to overlap).

Several of the authors of some of these programs are on the list, so
if any of you see something that is inaccurate, or would like to add
more detail to the comparative description of your tool, please do so
note. I shall eventually probably add this to FrathWiki, as well as
create a document to be hosted on the conlang-software-dev GitHub
organization.

-l.