Misspelled/WordDict.lua

--[[
Misspelled utilizes a slightly Lua'ified version of the Hunspell Dictionary format.
Note: Not all Dictionary sets tags are supported.

Phonetic information has been appeneded, after the affix codes, to each base word
example: "hello/GMS/HL"
A phonetic lookup set "Phonetic" exists before the base words.

Here is some rather generic documentation on that format.
From: http://sourceforge.net/docman/display_doc.php?docid=29374&group_id=143754

TRY
sets the change characters for suggestions.

REP (Replace)
sets a replacement table for multiple character corrections in suggestion mode.

The following affix file example:. 'TRY' suggestions differ from the bad word with an
English letter or an apostrophe. With these REP definitions, Hunspell can suggest the
right word form, when the misspelled word contains f instead of ph and vice versa.


	TRY esianrtolcdugmphbyfvkwzESIANRTOLCDUGMPHBYFVKWZ

	REP 2
	REP f ph
	REP ph f

	PFX A Y 1
	PFX A 0 re .

	SFX B Y 2
	SFX B 0 ed [^y]
	SFX B y ied y

There are two affix classes in the dictionary.
Class A defines a 're-' prefix. Class B defines two '-ed' suffixes.
First suffix can be added to a word if the last character of the word isn't 'y'.
Second suffix can be added to the words terminated with an 'y'. (See later.)

The following dictionary file uses these affix classes.

	3
	hello
	try/B
	work/AB

All accepted words with this dictionary: "hello", "try", "tried", "work", "worked", "rework", "reworked".


Affix Compression
--------------------------------------------------------------------------
An affix is either a  prefix or a suffix attached to root words to make
other words.  For example supply -> supplied by dropping the "y" and
adding an "ied" (the suffix).

Here is an example of how to define one specific suffix borrowed
from the en_US.aff file used by the OpenOffice org spellchecker

SFX D Y 4
SFX D   0     d          e
SFX D   y     ied        [^aeiou]y
SFX D   0     ed         [^ey]
SFX D   0     ed         [aeiou]y

This file is space delimited and case sensitive.
So this information can be interpreted as follows:

The first line has 4 fields:

Field
-----
1     SFX - indicates this is a suffix
2     D   - is the name of the character which represents this suffix
3     Y   - indicates it can be combined with prefixes (cross product)
4     4   - indicates that sequence of 4 affix entries are needed to
               properly store the affix information

The remaining lines describe the unique information for the 4 affix
entries that make up this affix.  Each line can be interpreted
as follows: (note fields 1 and 2 are used as a check against line 1 info)

Field
-----
1     SFX         - indicates this is a suffix
2     D           - is the name of the character which represents this affix
3     y           - the string of chars to strip off before adding affix
                         (a 0 here indicates the NULL string)
4     ied         - the string of affix characters to add
                         (a 0 here indicates the NULL string)
5     [^aeiou]y   - the conditions which must be met before the affix
                    can be applied

Field 5 is interesting.  Since this is a suffix, field 5 tells us that
there are 2 conditions that must be met.  The first condition is that
the next to the last character in the word must *NOT* be any of the
following "a", "e", "i", "o" or "u".  The second condition is that
the last character of the word must end in "y".

So how can we encode this information concisely and be able to
test for both conditions in a fast manner?  The answer is found
but studying the wonderful ispell code of Geoff Kuenning, et.al.
(now available under a normal BSD license).

If we set up a conds array of 256 bytes indexed (0 to 255) and access it
using a character (cast to an unsigned char) of a string, we have 8 bits
of information we can store about that character.  Specifically we
could use each bit to say if that character is allowed in any of the
last (or first for prefixes) 8 characters of the word.

Basically, each character at one end of the word (up to the number
of conditions) is used to index into the conds array and the resulting
value found there says whether the that character is valid for a
specific character position in the word.

For prefixes, it does this by setting bit 0 if that char is valid
in the first position, bit 1 if valid in the second position, and so on.

If a bit is not set, then that char is not valid for that postion in the
word.

If working with suffixes bit 0 is used for the character closest
to the front, bit 1 for the next character towards the end, ...,
with bit numconds-1 representing the last char at the end of the string.

Note: since entries in the conds[] are 8 bits, only 8 conditions
(read that only 8 character positions) can be examined at one
end of a word (the beginning for prefixes and the end for suffixes.

So to make this clearer, lets encode the conds array values for the
first two affentries for the suffix D described earlier.


  For the first affentry:
     numconds = 1             (only examine the last character)

     conds['e'] =  (1 << 0)   (the word must end in an E)
     all others are all 0

  For the second affentry:
     numconds = 2             (only examine the last two characters)

     conds[X] = conds[X] | (1 << 0)     (aeiou are not allowed)
         where X is all characters *but* a, e, i, o, or u


     conds['y'] = (1 << 1)     (the last char must be a y)
     all other bits for all other entries in the conds array are zero


--]]

if bit == nil then
	require("bit")
end

if string.utf8len == nil then
	require("UTF8\\utf8data")
	require("UTF8\\utf8")
end


--Local reference to Global functions for speed
local ipairs = ipairs
local pairs = pairs
local string_byte = string.byte
local string_utf8byte = string.utf8byte
local string_find = string.find
local string_gsub = string.gsub
local string_len = string.len
local string_utf8len = string.utf8len
local string_lower = string.lower
local string_utf8lower = string.utf8lower
local string_match = string.match
local string_rep = string.rep
local string_sub = string.sub
local string_utf8sub = string.utf8sub
local string_upper = string.upper
local string_utf8upper = string.utf8upper

local table_concat = table.concat
local table_insert = table.insert
local table_sort = table.sort

local bit_bor = bit.bor
local bit_lshift = bit.lshift
local bit_rshift = bit.rshift
local bit_band = bit.band
local bit_bnot = bit.bnot
local math_min = math.min

local utf8 = false

--WordDict_Dictionary = {}

local DICTIONARY_TO_LOAD_FROM

--split a string, at delimiter into a table
-- Compatibility: Lua-5.1
local split_t
local split_fpat
local split_last_end
local split_s, split_e, split_cap
local function split(str, pat)
   split_t = {}  -- NOTE: use {n = 0} in Lua-5.0
   split_fpat = "(.-)" .. pat
   split_last_end = 1
   split_s, split_e, split_cap = string_find(str, split_fpat, 1)
   while split_s do
      if split_s ~= 1 or split_cap ~= "" then
	 table_insert(split_t, split_cap)
      end
      split_last_end = split_e+1
      split_s, split_e, split_cap = string_find(str, split_fpat, split_last_end)
   end
	 if split_last_end <= #str then
		split_cap = string_sub(str, split_last_end)
		table_insert(split_t, split_cap)
	 end
   return split_t
end


--WordDict Class
WordDict = {}

WordDict = {
			locale = "",
			baseWords = {},
			replaceChars = {},
			prefixRules = {},
			suffixRules = {},
			phoneticRules= {},
			tryChars = {},
			lastContainsWord = "",
			possibleBaseWords = {},
			suggestions= {},
			soundslike = "Generic",
			maxSuggestions = 8,
			maxEditDistance = 4
			}

WordDict.Const = {
	SoundslikeAlgorithms = {GENERIC="Generic", PHONETIC="Phonetic"} --Enum for the type of sounds like algorithm used in this dictionary.
}

WordDict.AffixUtility = {}

--Takes aff file condition string and creates the condition array
--Rather similar to hunspell affixmgr.cpp AffixMgr::encodeit()
--http://www.google.com/codesearch/p?hl=en#e_ObwTAVPyo/extensions/spellcheck/hunspell/src/affixmgr.cpp&q=affix encode condition&l=1004
function WordDict.AffixUtility:EncodeConditions(conditionText)
	local condition = {}
	local utf8Condition = {}
	utf8Condition["ascii"] = {}
	utf8Condition["all"] = {}
	utf8Condition["neg"] = {}
	utf8Condition["utf8Chars"] = {}

	local group = false  	--Group indicator
	local groupend = false	--end contition indicator
	local neg = false -- complement indicator
	local conditionCount = 0 -- number of conditions

	local memberChars = {}
	local numMember = 0		-- number of member in group

	local str_sub = string_sub
	local str_len = string_len

	if utf8 == true then
		str_len = string.utf8len
		str_sub = string.utf8sub
 	end

	--local BeginningOnly = false
	--local EndOnly = false

	local ReplaceMode = false


	--print("conditionText", conditionText)

	--Clear the condition array
	for j=0,255 do
		condition[j] = 0
	end

	--Clear the utf8Condition array
	for j=0, 127 do
		utf8Condition.ascii[j] = 0
	end

	--If no condition just return
	if conditionText == "." then
		conditionCount = 0
		return conditionCount, condition
	end

	for i = 1, string_len(conditionText) do
		local cond = str_sub(conditionText, i, i)
		--	print("i", i, "cond", cond)

		--start group indicator
		if cond == "[" then
			group = true

		--complement flag (negative group)
		elseif cond == "^" and group == true then
			neg = true

		--end group indicator
		elseif cond == "]" then
			endgroup = true  -- end of a group

		--add character of group to list
		elseif group == true then
			memberChars[numMember] = cond
			numMember = numMember + 1
		else
			endgroup = true --no group
		end

		--set condition
		if endgroup == true then
			if utf8 == false then
				if group == true then
					if neg == false then
						--turn on chars in member group
						--set the proper bits in the condition array vals for those chars
						for j = 0, numMember -1 do
							local charCode = string_byte(memberChars[j])
							condition[charCode] = bit_bor(condition[charCode], bit_lshift(1, conditionCount))
						end
					else
						--complement so set all of chars on, then uset indicated ones.
						for j=0,255 do
							condition[j] = bit_bor(condition[j], bit_lshift(1, conditionCount))
						end
						--turn off chars in member groupend
						for j = 0, numMember -1 do
							local charCode = string_byte(memberChars[j])
							condition[charCode] = bit_band(condition[charCode], bit_bnot(bit_lshift(1, conditionCount)))
						end
					end
					neg = false
					group = false
					numMember = 0
				else
					--not in a group, so just set the proper bit for this char
					--but first handle special case of . inside condition
					if cond == "." then
						--wild char character, turn all chars on
						for j=0, 255 do
							condition[j] = bit_bor(condition[j], bit_lshift(1, conditionCount))
						end
					else
						--turn on char
						local charCode = string_byte(cond)
						condition[charCode] = bit_bor(condition[charCode], bit_lshift(1, conditionCount))
					end
				end

			else	-- utf8 character set
				--Use two storage methods to store the conditions.
				--First, for simple one byte utf8 characters (ascii) use the storage method above
				--with the understanding tha we know an 1 byte ACSII utf char has to be less than 128,
				--and this requires fewer bits of storage
				--
				--For multi-byte utf8 condition chars, store, up to 8 of them, in and character array,
				--along with the flag information about negative condition groups and the all char
				--wildcard use
				--struct {
				--	char ascii[SETSIZE/2];
				--	char neg[8];
				--	char all[8];
				--	w_char * wchars[8];
				--	int wlen[8];
				--} utf8condition;
				--
				--use the utf8 safe version of byte


				if group == true then
					utf8Condition.neg[conditionCount] = neg
					--
					--First handel the Ascii one byte utf8 chars
					--
					if neg == false then
						--set the proper bits in the condition array vals for those chars
						for j=0, numMember -1 do
							local charCode = string_utf8byte(memberChars[j])
							if charCode > 0 and charCode < 128 then
								utf8Condition.ascii[charCode] = bit_bor(utf8Condition.ascii[charCode], bit_lshift(1, conditionCount))
							end
						end
					else --neg = true
						--complement so set all of chars on, then uset indicated ones.
						for j=0,127 do
							utf8Condition.ascii[j] = bit_bor(utf8Condition.ascii[j], bit_lshift(1, conditionCount))
						end

						--turn off chars in member groupend
						for j = 0, numMember -1 do
							local charCode = string_utf8byte(memberChars[j])
							if charCode < 128 then
								utf8Condition.ascii[charCode] = bit_band(utf8Condition.ascii[charCode], bit_bnot(bit_lshift(1, conditionCount)))
							end
						end
					end

					--Store utf8 multi-byte conditions
					for j = 0, numMember - 1 do
						if string_byte(memberChars[j]) >= 128 then
							if utf8Condition.utf8Chars[conditionCount] == nil then
								utf8Condition.utf8Chars[conditionCount] = ""
							end
							utf8Condition.utf8Chars[conditionCount] = utf8Condition.utf8Chars[conditionCount] .. memberChars[j]
						end
					end

					neg = false
					group = false
					numMember = 0

				else  --group == false
					local condCode = string_utf8byte(cond)
					--Is cond a utf8 multi-byte character
					if  condCode >= 128 then
							if utf8Condition.utf8Chars[conditionCount] == nil then
								utf8Condition.utf8Chars[conditionCount] = ""
							end
							utf8Condition.utf8Chars[conditionCount] = utf8Condition.utf8Chars[conditionCount] .. cond
					else
						--not a group so just set the proper bit for this char
						--by first handle special case of . inside condition
						if cond == "." then
							utf8Condition.all[conditionCount] = true
							for j = 0, 127 do
								utf8Condition.ascii[j] = bit_bor(utf8Condition.ascii[j], bit_lshift(1, conditionCount))
							end
						else
							utf8Condition.all[conditionCount] = false
							utf8Condition.ascii[condCode] = bit_bor(utf8Condition.ascii[condCode], bit_lshift(1, conditionCount))
						end
					end
					neg = false
				end
				conditionCount = conditionCount + 1
				endgroup = false
				neg = false
			end
		end
	end

	--Condition["conditionCount"] = conditionCount
	return conditionCount, condition, utf8Condition
end

--Removes the affix suffix rule entry for the word if valid
function WordDict.AffixUtility:RemoveSuffix(word, entry)
	local tempLength
	local tempWord
	local c
	local charCode
	local passCount

	if utf8 == false then
		tempLength = #word - #entry.AddChars
		if (tempLength > 0)
			and (tempLength + #entry.StripChars >= entry.ConditionCount)
			and (string_match(word, entry.AddChars .. "$") ~= nil) then  --word ends with suffix

			--word without affix
			tempWord = string_sub(word, 1, tempLength)
			--add back strip chars
			tempWord = tempWord .. entry.StripChars
			--check that this is valid
			passCount = 0
			for i = 0, entry.ConditionCount - 1 do
				charCode = string_byte(string_sub(tempWord, #tempWord - (entry.ConditionCount -i) + 1))

				if bit_band(entry.Conditions[charCode], bit_lshift(1, i)) == bit_lshift(1 , i) then
					passCount = passCount + 1
				end
			end
			if passCount == entry.ConditionCount then
				return tempWord
			end
		end
	else
		tempLength = string_utf8len(word) - string_utf8len(entry.AddChars)
 		if (tempLength > 0)
			and (tempLength + string_utf8len(entry.StripChars) >= entry.ConditionCount)
			and (string_match(word, entry.AddChars .. "$") ~= nil) then --word ends with suffix

			--word without suffix
			tempWord = string_utf8sub(word, 1, tempLength)
			--add back strip chars
			tempWord = tempWord .. entry.StripChars

			--check if tempWord is valid
			passCount = 0
			for i = 0, entry.ConditionCount - 1 do
				c = string_utf8sub(tempWord, string_utf8len(tempWord) - (entry.ConditionCount - i) + 1)
				charCode = string_utf8byte(c)

				--if this charCode is 1 byte, check the utf8Condition.Ascii table
				if charCode > 0 and charCode < 128 then
					if bit_band(entry.Utf8Conditions.ascii[charCode], bit_lshift(1, i)) == bit_lshift(1, i) then
						passCount = passCount + 1
					end
				else
					--this is a multi byte utf8 char
					--check if it's stored in the utf8 specific part of the rule.
					local all
					local neg
					local utf8Chars

					all = entry.Utf8Conditions.all[i]
					neg = entry.Utf8Conditions.neg[i]
					utf8Chars = entry.Utf8Conditions.utf8Chars[i]

					--check if this condition has any utf8 multibyte chars.
					if utf8Chars ~= nil then
						if all == true then	-- "." match all characters
							passCount = passCount + 1
						elseif neg == false then
							--check if this char is present
							if string_find(utf8Chars, c) ~= nil then
								passCount = passCount + 1
							end
						else  -- neg == true.  Make sure we can't find the char.
							if string_find(utf8Chars, c) == nil then
								passCount = passCount + 1
							end
						end
					end
				end
			end
			if passCount == entry.ConditionCount then
				return tempWord
			end
		end
	end
	return word
end

--Removes the affix prefix rule entry for the word if valid
function WordDict.AffixUtility:RemovePrefix(word, entry)
	local tempLength
	local tempWord
	local c
	local charCode
	local passCount
	if utf8 == false then
		tempLength = #word - #entry.AddChars
		if (tempLength > 0)
			and (tempLength + #entry.StripChars >= entry.ConditionCount)
			and (string_match(word, "^" .. entry.AddChars) ~= nil) then  --word starts with

			--word with out affix
			tempWord = string_sub(word, #entry.AddChars+1)
			--add back strip chars
			tempWord = entry.StripChars .. tempWord
			--check that this is valid
			passCount = 0
			for i = 0, entry.ConditionCount - 1 do
				charCode = string_byte(string_sub(tempWord, i+1, i+1))

				if bit_band(entry.Conditions[charCode], bit_lshift(1, i)) == bit_lshift(1 , i) then
					passCount = passCount + 1
				end
			end
			if passCount == entry.ConditionCount then
				return tempWord
			end
		end
	else
		tempLength = string_utf8len(word) - string_utf8len(entry.AddChars)
 		if (tempLength > 0)
 			and (tempLength + string_utf8len(entry.StripChars) >= entry.ConditionCount)
			and (string_match(word, "^" .. entry.AddChars) ~= nil) then --word starts with

			--word with out prefix
			tempWord = string_sub(word, string_utf8len(entry.AddChars) + 1)
			--add back strip chars
			tempWord = entry.StripChars .. tempWord

			--check if tempWord is valid
			passCount = 0
			for i = 0, entry.ConditionCount - 1 do
				c = string_utf8sub(tempWord, i+1, i+1)
				charCode = string_utf8byte(c)

				--if this charCode is 1 byte, check the utf8Condition.Ascii table
				if charCode > 0 and charCode < 128 then
					if bit_band(entry.Utf8Conditions.ascii[charCode], bit_lshift(1, i)) == bit_lshift(1, i) then
						passCount = passCount + 1
					end
				else
					--this is a multi byte utf8 char
					local all
					local neg
					local utf8Chars

					all = entry.Utf8Conditions.all[i]
					neg = entry.Utf8Conditions.neg[i]
					utf8Chars = entry.Utf8Conditions.utf8Chars[i]
					--check if this condition has any utf8 multibyte chars.
					if utf8Chars ~= nil then
						if all == true then	-- "." match all characters
							passCount = passCount + 1
						elseif neg == false then
							--check if this char is present
							if string_find(utf8Chars, c) ~= nil then
								passCount = passCount + 1
							end
						else
							if string_find(utf8Chars, c) == nil then
								passCount = passCount + 1
							end
						end
					end
				end
			end
			if passCount == entry.ConditionCount then
				return tempWord
			end
		end
	end
	return word
end

--Adds a suffix to a word
--Returns the word with the suffix added
function WordDict.AffixUtility:AddSuffix(word, rule)
	local passCount
	local charCode
	local tempLen
	local tempWord

	for i, entry in ipairs(rule.AffixEntries) do
		--check if this entry is valid
		if utf8 == false then
			if #word >= entry.ConditionCount then
				passCount = 0
				for i=0, entry.ConditionCount - 1 do
					charCode = string_byte(string_sub(word, #word - (entry.ConditionCount -i)+1,#word - (entry.ConditionCount -i)+1))

					if bit_band(entry.Conditions[charCode], bit_lshift(1, i)) == bit_lshift(1 , i) then
						passCount = passCount + 1
					else
						break
					end
				end
				if passCount == entry.ConditionCount then
					tempLen = #word - #entry.StripChars
					tempWord = string_sub(word, 1, tempLen)
					tempWord = tempWord .. entry.AddChars
					return tempWord
				end
			end
		else  --utf8
			if string_utf8len(word) >= entry.ConditionCount then
				passCount = 0
				for i = 0, entry.ConditionCount - 1 do
					--For each condition, we need to check utf8 characters at the end of the word
					local conditionPos = string_utf8len(word) - (entry.ConditionCount - i) + 1
					local c = string_utf8sub(word, conditionPos, conditionPos)
					charCode = string_utf8byte(c)

					--if this charCode is 1 byte, check the utf8Condition.Ascii table
					if charCode > 0 and charCode < 128 then
						if bit_band(entry.Utf8Conditions.ascii[charCode], bit_lshift(1, i)) == bit_lshift(1, i) then
							passCount = passCount + 1
						else
							break
						end
					else
						--this is a multi-byte utf8 character, check
						local all
						local neg
						local utf8Chars

						all = entry.Utf8Conditions.all[i]
						neg = entry.Utf8Conditions.neg[i]
						utf8Chars = entry.Utf8Conditions.utf8Chars[i]
						--check if this condition has any utf8 multibyte chars.
						if utf8Chars ~= nil then
							if all == true then	-- "." match all characters
								passCount = passCount + 1
							elseif neg == false then
								--check if this char is present
								if string_find(utf8Chars, c) ~= nil then
									passCount = passCount + 1
								else
									break
								end
							else	-- neg == true
								-- check complement
								if string_find(utf8Chars, c) == nil then
									passCount = passCount + 1
								else
									break
								end
							end
						end
					end
				end
				if passCount == entry.ConditionCount then
					tempLen = string_len(word) - #entry.StripChars
					tempWord = string_sub(word, 1 , tempLen)
					tempWord = tempWord .. entry.AddChars
					return tempWord
				end
			end
		end
	end
	return word
end

--Adds a prefix to a word
--Returns the word with the prefix added
function WordDict.AffixUtility:AddPrefix(word, rule)
	local passCount
	local charCode
	local tempLen
	local tempWord
	for i, entry in ipairs(rule.AffixEntries) do
		--check if this entry is valid
		if utf8 == false then
			if #word >= entry.ConditionCount then
				passCount = 0
				for i=0, entry.ConditionCount -1 do
					charCode = string_byte(string_sub(word, i+1, i+1))

					if bit_band(entry.Conditions[charCode], bit_lshift(1, i)) == bit_lshift(1 , i) then
						passCount = passCount + 1
					else
						break
					end
				end
				if passCount == entry.ConditionCount then
					tempWord = string_sub(word, #entry.StripChars)
					tempWord = entry.AddChars .. tempWord
					return tempWord
				end
			end
		else  --utf8
			if string_utf8len(word) >= entry.ConditionCount then
				passCount = 0
				for i = 0, entry.ConditionCount - 1 do
					--For each condition, we need to check utf8 characters at the end of the word
					local c = string_utf8sub(word, i + 1, i + 1)
					charCode = string_utf8byte(c)

					--if this charCode is 1 byte, check the utf8Condition.Ascii table
					if charCode > 0 and charCode < 128 then
						if bit_band(entry.Utf8Conditions.ascii[charCode], bit_lshift(1, i)) == bit_lshift(1, i) then
							passCount = passCount + 1
						else
							break
						end
					else
						--this is a multi-byte utf8 character, check
						local all
						local neg
						local utf8Chars

						all = entry.Utf8Conditions.all[i]
						neg = entry.Utf8Conditions.neg[i]
						utf8Chars = entry.Utf8Conditions.utf8Chars[i]

						--check if this condition has any utf8 multibyte chars.
						if utf8Chars ~= nil then
							if all == true then	-- "." match all characters
								passCount = passCount + 1
							elseif neg == false then
								--check if this char is present
								if string_find(utf8Chars, c) ~= nil then
									passCount = passCount + 1
								else
									break
								end
							else  -- neg == true
								-- check for the complement
								if string_find(utf8Chars, c) == nil then
									passCount = passCount + 1
								else
									break
								end
							end
						end
					end
				end
				if passCount == entry.ConditionCount then
					tempWord = string_sub(word, #entry.StripChars)
					tempWord = entry.AddChars .. tempWord
					return tempWord
				end

			end
		end
	end
	return word
end


function WordDict:LoadPrefixRules()
	local row = 1 --row pointer
	local rowCount = #DICTIONARY_TO_LOAD_FROM.Prefix

	local header
	local hparts
	local hpart1
	local hpart2
	local hpart3
	local rulecount
	local rule1
	local rule2
	local rule3
	local rule4
	local AllowCombine = false
	local entry
	local parsedEntry
	local condCount
	local conditions
	local utf8Conditions

	local p = DICTIONARY_TO_LOAD_FROM.Prefix

	while row < rowCount do
			header = p[row]

			-- headers are in 3 parts
			hparts = split(header, "%s+")
			hpart1 = hparts[1] -- Part 1 = affix key
			hpart2 = hparts[2] -- Part 2 = Allow combine flag
			hpart3 = hparts[3] -- Part 3 = rule count

 			--determin how many parts there are to this prefix
			rulecount = tonumber(hpart3 )
			--print("header", hpart1, hpart2, hpart3)

			--Add header parts 1 & 2 to PrefixRules collection
			self.prefixRules[hpart1] = {["Combine"]=hpart2, ["Rules"]={}}

			--Rules have 4 parts
			--Part1: affix(prefix/suffix) key
			--Part2: strip char
			--Part3: add chars
			--Part4: conditions

			AllowCombine = false
			if hpart2 == "Y" then AllowCombine = true end

			--Add header parts 1 & 2 to PrefixRules collection
			self.prefixRules[hpart1] = {["AllowCombine"]=AllowCombine, ["AffixEntries"]={}}

			for x = 1, rulecount do
				row = row + 1
				entry = p[row]
				parsedEntry = split(entry, "%s+")
				self.prefixRules[hpart1]["AffixEntries"][x] = {}
				--part 2 - strip char
				if parsedEntry[2] == "0" then
					self.prefixRules[hpart1]["AffixEntries"][x]["StripChars"] = ""
				else
				    self.prefixRules[hpart1]["AffixEntries"][x]["StripChars"] = parsedEntry[2]
				end
				--part 3 - add chars
				self.prefixRules[hpart1]["AffixEntries"][x]["AddChars"] = parsedEntry[3]
				--part 4 - conditions
				condCount = nil
				conditions = nil
				utf8Conditions = nil
				condCount, conditions, utf8Conditions = WordDict.AffixUtility:EncodeConditions(parsedEntry[4])
				self.prefixRules[hpart1]["AffixEntries"][x]["ConditionCount"] = condCount
				self.prefixRules[hpart1]["AffixEntries"][x]["Conditions"] = conditions
				self.prefixRules[hpart1]["AffixEntries"][x]["Utf8Conditions"] = utf8Conditions
			end
			row = row + 1
	end

end

function WordDict:LoadSuffixRules()
	local row = 1 --row pointer
	local rowCount = #DICTIONARY_TO_LOAD_FROM.Suffix

	local header
	local hparts
	local hpart1
	local hpart2
	local hpart3
	local rulecount
	local rule1
	local rule2
	local rule3
	local rule4
	local AllowCombine = false
	local entry
	local parsedEntry
	local condCount
	local conditions
	local utf8Conditions

	local s = DICTIONARY_TO_LOAD_FROM.Suffix

	while row < rowCount do
		header = s[row]
		-- headers are in 3 parts
		hparts = split(header, "%s+")
		hpart1 = hparts[1] -- Part 1 = affix key
		hpart2 = hparts[2] -- Part 2 = combine flag
		hpart3 = hparts[3] -- Part 3 = rule count

		--determin how many parts there are to this prefix
		rulecount = tonumber(hpart3)
		--print("Suffix Header", hpart1, hpart2, hpart3, "##", header, "##")


		AllowCombine = false
		if hpart2 == "Y" then AllowCombine = true end

		--Add header parts 1 & 2 to SuffixRules collection
		self.suffixRules[hpart1] = {["AllowCombine"]=AllowCombine, ["AffixEntries"]={}}

		--Rules have 4 parts
		--Part1: affix(prefix/suffix) key
		--Part2: strip char
		--Part3: add chars
		--Part4: conditions

		for x = 1, rulecount do
			row = row + 1
			entry = s[row]
			parsedEntry = split(entry, "%s+")

			--print("Suffix Entry:", entry, "##")
			--tprint(parsedEntry)

			self.suffixRules[hpart1]["AffixEntries"][x] = {}
			--part 2 - strip char
			if parsedEntry[2] == "0" then
				self.suffixRules[hpart1]["AffixEntries"][x]["StripChars"] = ""
			else
				self.suffixRules[hpart1]["AffixEntries"][x]["StripChars"] = parsedEntry[2]
			end
			--part 3 - add chars
			self.suffixRules[hpart1]["AffixEntries"][x]["AddChars"] = parsedEntry[3]
			--part 4 - conditions
			condCount = nil
			conditions = nil
			utf8Conditions = nil
			condCount, conditions, utf8Conditions = WordDict.AffixUtility:EncodeConditions(parsedEntry[4])
			self.suffixRules[hpart1]["AffixEntries"][x]["ConditionCount"] = condCount
			self.suffixRules[hpart1]["AffixEntries"][x]["Conditions"] = conditions
			self.suffixRules[hpart1]["AffixEntries"][x]["Utf8Conditions"] = utf8Conditions
		end
		row = row + 1
	end

end

WordDict.PhoneticUtility = {}

function WordDict.PhoneticUtility:EncodeRule(ruleText)

	local string_byte = string_byte
	local string_len = string_len
	local string_sub = string_sub

	if utf8 then
		string_byte = string_utf8byte
		string_len = string_utf8len
		string_sub = string_utf8sub
	end

	local group = false  	--Group indicator
	local groupend = false	--end contition indicator

	local memberChars = {}
	local numMember = 0		-- number of member in groupend

	local BeginningOnly = false
	local EndOnly = false
	local ConsumeCount = 0
	local ReplaceMode = false
	local Priority = 0
	local ConditionCount = 0
	local cond
	local charCode

	local Rule = {}
	Rule["Condition"] = {}
	Rule["ConsumeCount"] = 0

	--for j = 0, 255 do
	--	RuleCondition[j] = 0
	--end

	--print("ruletext", ruleText)

	for i = 1, string_len(ruleText) do
		cond = string_sub(ruleText, i, i)
		--print("i", i, "cond", cond)

		if cond == "(" then group = true
		elseif cond == ")" then endgroup = true
		elseif cond == "^" then
			BeginningOnly = true
			Rule["BeginningOnly"] = true
		elseif cond == "$" then
			EndOnly = true
			Rule["EndOnly"] = true
		elseif cond == "-" then
			ConsumeCount = ConsumeCount + 1
			Rule["ConsumeCount"] = ConsumeCount
		elseif cond == "<" then
			ReplaceMode = true
			Rule["ReplaceMode"] = true
		--local p = tonumber(cond)
		elseif tonumber(cond) ~= nil then
			Priority = tonumber(cond)
			Rule["Priority"] = Priority
		else
			if group == true then
				--add chars to group
				memberChars[numMember] = cond
				numMember = numMember + 1
			else
				endgroup = true
			end
		end
		if endgroup == true then
			if group == true then
				--turn on chars in member group
				for j = 0, numMember - 1 do
					charCode = string_byte(memberChars[j])
					Rule["Condition"][charCode] = bit_bor(Rule["Condition"][charCode] or 0, bit_lshift(1, ConditionCount))
				end
				group = false
				numMember = 0
			else
				--turn on char
				charCode = string_byte(cond)
				Rule["Condition"][charCode] = bit_bor(Rule["Condition"][charCode] or 0, bit_lshift(1, ConditionCount))
			end
			endgroup = false
			ConditionCount = ConditionCount + 1
			Rule["ConditionCount"] = ConditionCount
		end
	end
	return Rule
end

function WordDict:LoadPhoneticRules()
	local pr = DICTIONARY_TO_LOAD_FROM.Phonetic

	--Phonetic rules are in 2 parts
	--Part1: Encoded Rule
	--Part2: ReplaceString
	for i, v in ipairs(pr) do
		local parsedrule = split(v, "%s+")
		--self.phoneticRules[i]= {parsedrule[1], parsedrule[2]}
		self.phoneticRules[i] = WordDict.PhoneticUtility:EncodeRule(parsedrule[1])
		self.phoneticRules[i]["ReplaceString"] = parsedrule[2]

	end
end

function WordDict:LoadWords()
	local w = DICTIONARY_TO_LOAD_FROM.Words

	local baseWord
	local affixKeys
	local phoneticCode
	local parse = {}

	for i, v in ipairs(w) do
		-- split word input line into its parts
		--Part1: base word
		--Part2: affix keys
		--Part3: phonetic code
		--       Note: Some languanges don't have phonetic info,
		--		 and just use the Generic Phonetic method.
		parse = split(v, "/")
		baseWord = parse[1]
		if parse[2] ~= nil then
			affixKeys = parse[2]
		else
			affixKeys = ""
		end
		if parse[3] ~= nil then
			phoneticCode = parse[3]
		elseif self.soundslike == "Generic" then
			phoneticCode = self:GenericSoundsLike(baseWord)
		else
			phoneticCode = ""
		end


		--(r18) Don't store the affixKeys and phoneticCode in a sub table
		--Saves 6MB of memory.
		--self.baseWords[baseWord] = {affixKeys, phoneticCode}
		self.baseWords[baseWord] = affixKeys .."/" .. phoneticCode
	end
end

--Generates a phonetic code of how the word sounds based on phonetic
--ruless loaded from the dictionary
--Input: word (string)
--Return: a code of how the word sounds
function WordDict:PhoneticCode(word)
	local string_byte = string_byte
	local string_len = string_len
	local string_upper = string_upper
	local string_sub = string_sub

	if utf8 then
		string_byte = string_utf8byte
		string_len = string_utf8len
		string_sub = string_utf8sub
		string_upper = string_utf8upper
	end

	local tempWord = string_upper(word)
	local prevWord = ""
	local code = ""

	local begining
	local ending
	local passCount
	local charCode
	local ruleCondition

	while #tempWord > 0 do
		--save previous word
		prevWord = tempWord

		for i,v in ipairs(self.phoneticRules) do
			begining = false
			ending = false
			if #tempWord == #word then
				begining = true
			end
			if v.ConditionCount == string_len(tempWord) then
				ending = true
			end

			if ((v.BeginningOnly == begining or not v.BeginningOnly)
				and (v.EndOnly == ending or not v.EndOnly)
				and (v.ConditionCount <= string_len(tempWord))) then

				passCount = 0
				-- loop through conditions
				for i = 1, v.ConditionCount do
					charCode = string_byte(string_sub(tempWord, i, i))
					ruleCondition = v.Condition[charCode] or 0

					if bit_band(ruleCondition, bit_lshift(1, i-1)) == bit_lshift(1, i-1) then
						passCount = passCount + 1 -- condition passed
					else
						break -- rule fails if on condition fails
					end
				end
				-- if all conditions passed
				if passCount == v.ConditionCount then
					if v.ReplaceMode == true then
						tempWord = v.ReplaceString .. string_sub(tempWord, v.ConditionCount - v.ConsumeCount + 1, -1)
					else
						if v.ReplaceString ~= "_" then
							code = code .. v.ReplaceString
						end
						tempWord = string_sub(tempWord, v.ConditionCount - v.ConsumeCount + 1)
					end
					break
				end
			end
		end

		-- if no match consume one char
		if #prevWord == #tempWord then
			tempWord = string_sub(tempWord, 2)
		end
	end
	return code
end

local AccentTransformations={
	["á"]="a",["à"]="a",["ã"]="a",["â"]="a",["å"]="a",["ä"]="a",
	["ç"]="c",
	["é"]="e",["è"]="e",["ë"]="e",["ê"]="e",
	["í"]="i",["ì"]="i",["ï"]="i",["î"]="i",
	["ñ"]="n",
	["ó"]="o",["ò"]="o",["õ"]="o",["ô"]="o",["ö"]="o",["ø"]="o",
	["ù"]="u",["û"]="u",["ú"]="u",["ü"]="u",
	["ÿ"]="y",["ý"]="y",
	["Á"]="A",["À"]="A",["Ã"]="A",["Â"]="A",["Å"]="A",["Ä"]="A",
	["Ç"]="C",
	["É"]="E",["È"]="E",["Ë"]="E",["Ê"]="E",
	["Í"]="I",["Ì"]="I",["Ï"]="I",["Î"]="I",
	["Ñ"]="N",
	["Ó"]="O",["Ò"]="O",["Õ"]="O",["Ô"]="O",["Ö"]="O",["Ø"]="0",
	["Ù"]="U",["Û"]="U",["Ú"]="U",["Ü"]="U",
	["Ý"]="Y",["Ÿ"]="Y",
	["Ð"]="D",
	["Œ"]="OE",
	["Þ"]="TH",
	["æ"]="AE",
	["ð"]="d",
	["œ"]="oe",
	["ß"]="ss",
	["þ"]="th"
}
local function removeAccents(text)
	return string_gsub(text, ".", AccentTransformations)
end

--Returns a generic sounds like code, for languanges that don't specify phonetic codes in their dirctionaries,
--and choose to use this "Generic" function in it's place.  Should work for Spanish and French.
--Remove all accents, vowels, and numbers.  Then uppercase the string, and return non repeating characters.
--Example: "herrmanóÞ" returns "HRMNTH"
--Hopefully this code doesn't need to worry about UTF8 because we remove all Accents
function WordDict:GenericSoundsLike(text)
	if text == nil then return "" end
	if #text == 0 then return "" end

	local s = string.upper(removeAccents(text))
	local sRet = ""
	local cPrev
	local cCurr

	s = string.gsub(s, "[AEIOUY0-9]", "")

	if #s >0 then
		cPrev = string.sub(s,1,1)
		sRet = cPrev

		for i = 1, #s do
			cCurr = string.sub(s,i,i)
			if cCurr ~= cPrev then
				sRet = sRet .. cCurr
				cPrev = cCurr
			end
		end
	end
	return sRet
end


--Searched all containted word lists for word
function WordDict:Contains(word)
	--save the word we're checking, so if someone calls Suggest(word)
	--without first calling Contains, we can, call Contains() to refresh the
	--possibleBaseWords list.
	self.lastContainsWord = word

	--clean up possible base word lists
	self.possibleBaseWords = {}

	--Step1: If word length is 1 char, return true
	if #word == 1 then
		return true
	end

	--Step2: search BaseWords
	if self.baseWords[word] ~= nil then
		return true
	end

	--Step3: Remove suffix, then search basewords
	-- save suffixed words for use when removing prefix
	local suffixWords = {}

	suffixWords[word] = 1

	--tprint(self.suffixRules)
	local tempWord
	local baseWordTemp
	local affixKeys
	for key, rule in pairs(self.suffixRules) do
		--print("suffix rule: ", key)
		for x, entry in ipairs(rule.AffixEntries) do
			--print("word/AffixEntry addchars", word, entry.AddChars)
			--tprint(entry)

			tempWord = WordDict.AffixUtility:RemoveSuffix(word, entry)
			if tempWord ~= word then
				baseWordTemp = self.baseWords[tempWord]
				if baseWordTemp ~= nil then
					affixKeys = string_sub(baseWordTemp, 1, (string_find(baseWordTemp, "/") -1))
					--make sure base word has this affix key
					if string_find(affixKeys, key) ~= nil then
						return true --word found
					end
				end

				if rule.AllowCombine == true then
					-- saving word to check if it is a word after prefix is removed
					suffixWords[tempWord] = 1
				else
					-- saving possible base words for use in generating suggestions
					self.possibleBaseWords[tempWord] = 1
				end
			end
		end
	end
	--save possible base words for use in generating suggestions
	for k, v in pairs(suffixWords) do
		self.possibleBaseWords[k] = 1
	end

	--Step4: Remove prefix, then search basewords
	for key, rule in pairs(self.prefixRules) do
		for x, entry in ipairs(rule.AffixEntries) do
			for suffixWord, y in pairs(suffixWords) do
				tempWord = WordDict.AffixUtility:RemovePrefix(suffixWord, entry)
				if tempWord ~= suffixWord then
					baseWordTemp = self.baseWords[tempWord]
					if baseWordTemp ~= nil then
						--make sure base word has this affix keys
						affixKeys = string_sub(baseWordTemp, 1, (string_find(baseWordTemp, "/") -1))

						if string_find(affixKeys, key) ~= nil then
							return true --word found
						end
					end

					--save possible base workd for use in generating suggestions
					self.possibleBaseWords[tempWord] = 1
				end
			end
		end
	end

	return false
end

--Expands an affix compressed base word
--takes in a base word entry with coresponsind affix & phonetic info
--returns a table of the expanded words
function WordDict:ExpandWord(word, affixKeys)
	local suffixWords = {}
	local words = {}

	local string_len = string_len
	local string_sub = string_sub

	if utf8 then
		string_len = string_utf8len
		string_sub = string_utf8sub
	end

	--add word to the suffixWords table
	suffixWords[word] = 1
	local prefixKeys = ""

	-- check suffix keys first
	local key
	local rule
	local tempWord
	for x=1, string_len(affixKeys) do
		key = string_sub(affixKeys, x, x)
		if WordDict.suffixRules[key] ~= nil then
			rule = WordDict.suffixRules[key]
			tempWord = WordDict.AffixUtility:AddSuffix(word, rule)
			if tempWord ~= word then
				if rule.AllowCombine == true then
					suffixWords[tempWord] = 1
				else
					words[tempWord] = 1
				end
			end
		elseif WordDict.prefixRules[key] ~= nil then
			prefixKeys = prefixKeys..key
		end
	end

	--apply prefixes
	for x = 1, string_len(prefixKeys) do
		key = string_sub(prefixKeys, x, x)
		rule = WordDict.prefixRules[key]
		for suffixWord,v in pairs(suffixWords) do
			tempWord = WordDict.AffixUtility:AddPrefix(suffixWord, rule)
			if tempWord ~= suffixWord then
				words[tempWord] = 1
			end
		end
	end

	for key,v in pairs(suffixWords) do
		words[key] = 1
	end
	return words
end

--Calculated the minimum number of changes, inserts of deletes
--required to change the firstWord into the secondWord
--PositionPriority: set to true if the first and last char should have priority
function WordDict:EditDistance(source, target)
	local string_len = string_len
	local string_lower = string_lower
	local string_sub = string_sub

	if utf8 == true then
		string_len = string_utf8len
		string_lower = string_utf8lower
		string_sub = string_utf8sub
	end

	local m = string_len(source)
	local n = string_len(target)

	if n == 0 then return m end
	if m == 0 then return n end

	if source == "teh" and target == "the" then return 0.1 end
	
	local rows = {}
	rows[0] = {[0]=0}

	for x=0, m+1 do
		rows[x] = {[0]=0}
		for y=0,n do
			rows[x][y] = 0
		end
	end

	--tprint(rows)

	for i=0,m do
		rows[i][0] = i
	end

	for j=0,n do
		rows[0][j] = j
	end

	--Ignore if the strings that only differ by case
	source = string_lower(source)
	target = string_lower(target)

	--tprint(rows)
	local subsitutionCost

	for i=1, m do
		for j=1,n do
			if string_sub(source, i, i) == string_sub(target, j, j) then
				subsitutionCost = 0
			else
				subsitutionCost = 1
			end

			rows[i][j] = math_min(	rows[i-1][j] + 1, 					--Insertion
									rows[i][j-1] + 1, 					--Deletion
									rows[i-1][j-1] + subsitutionCost)	--Substitution
		end
	end

	local dist = rows[m][n]

	-- extra edit on first and last chars if position Priority is important
	if string_sub(source, 1, 1) ~= string_sub(target, 1, 1) then
		dist = dist + 1
	end

	if string_sub(source, -1) ~= string_sub(target, -1) then
		dist = dist + 1
	end

	--potential additional
	-- If every character in sounce is in target, give it priority
	--and if every character in target is also in source, give it even more priority.
--~ 	public int EditDistance(string source, string target, bool positionPriority)
--~ 	{
--~ 	string testasciiSource = source.ToLower();
--~ 	string testasciiTarget = target.ToLower();
--~ 	int dist = -1;

--~ 	// If every character in source is also in target, give it priority

--~ 	bool subset = false;
--~		if (Math.Abs(testasciiTarget.Length - testasciiSource.Length) <= 1)
--~		{
--~			subset = true;
--~ 		for (int i = 0; i < testasciiSource.Length; i++)
--~ 		if (testasciiTarget.IndexOf(testasciiSource[i]) == -1)
--~ 		{
--~ 			subset = false;
--~ 			break;
--~ 		}
--~		}

--~ 	// If every character in target is also in source, give it even more priority

--~ 	if (subset)
--~ 	{
--~ 	dist--;
--~ 	for (int i = 0; i < testasciiTarget.Length; i++)
--~ 	if (testasciiSource.IndexOf(testasciiTarget[i]) == -1)
--~ 	{
--~ 	dist++;
--~ 	break;
--~ 	}
--~ 	}

--~ 	if (!subset)
--~ 	{
--~ 	// i.e. 2-D array
--~ 	Array matrix = Array.CreateInstance(typeof(int), source.Length+1, target.Length+1);

--~ 	// boundary conditions
--~ 	matrix.SetValue(0, 0, 0);

--~ 	for(int j=1; j <= target.Length; j++)
--~ 	{
--~ 	// boundary conditions
--~ 	int val = (int)matrix.GetValue(0,j-1);
--~ 	matrix.SetValue(val+1, 0, j);
--~ 	}

--~ 	// outer loop
--~ 	for(int i=1; i <= source.Length; i++)
--~ 	{
--~ 	// boundary conditions
--~ 	int val = (int)matrix.GetValue(i-1, 0);
--~ 	matrix.SetValue(val+1, i, 0);

--~ 	// inner loop
--~ 	for(int j=1; j <= target.Length; j++)
--~ 	{
--~ 	int diag = (int)matrix.GetValue(i-1, j-1);

--~ 	if(source.Substring(i-1, 1) != target.Substring(j-1, 1))
--~ 	{
--~ 	diag++;
--~ 	}

--~ 	int deletion = (int)matrix.GetValue(i-1, j);
--~ 	int insertion = (int)matrix.GetValue(i, j-1);
--~ 	int match = Math.Min(deletion+1, insertion+1);
--~ 	matrix.SetValue(Math.Min(diag, match), i, j);
--~ 	}//for j
--~ 	}//for i

--~ 	dist = (int)matrix.GetValue(source.Length, target.Length);
--~ 	}

--~ 	// extra edit on first and last chars
--~ 	if (positionPriority)
--~ 	{
--~ 	if (source[0] != target[0]) dist++;
--~ 	if (source[source.Length-1] != target[target.Length-1]) dist++;
--~ 	}
--~ 	return dist;
--~ 	}

	--tprint(rows)

	return dist
end



--returns true if word or lower(word) are in baseWords
function WordDict:TestWord(word)
	if WordDict:Contains(word) == true then
		return true
	end

	--check the lower case version of the word, if it contains a capitol
	local w
	if utf8 == false then
		w = string_lower(word)
		if w ~= word then
			if WordDict:Contains(w) == true  then
				return true
			end
		end
	else
		w = string_utf8lower(word)
		if w ~= word then
			if WordDict:Contains(w) == true  then
				return true
			end
		end
	end

	return false
end


function WordDict:FirstCapital(currentWord)
	local suggestions = {}
	local suggestionsCount = 0

	local string_upper = string_upper
	local string_sub = string_sub

	if utf8 == true then
		string_upper = string_utf8upper
		string_sub = string_utf8sub
	end

	local testWord = string_upper(string_sub(currentWord, 1, 1)) .. string_lower(string_sub(currentWord, 2))

	if self:TestWord(testWord) == true then
		suggestions[testWord] = self:EditDistance(currentWord, testWord)
		suggestionsCount = 1
	end

	return suggestions, suggestionsCount
end

--suggestions for a typical fault of spelling, that
--differs with more, than 1 letter from the right form.
function WordDict:ReplaceChars(CurrentWord)
	local suggestions = {}
	local suggestionsCount = 0

	local string_lower = string_lower
	--local string_sub = string_sub

	if utf8 then
		string_lower = string_utf8lower
		--string_sub = string_utf8sub
	end

	--Test each set of replacement (key, replacement) pairs.
	for i = 1, #self.replaceChars do
		local key
		local replacement
		local testWord = CurrentWord
		local keyMatches = - 1
		local pos1, pos2
		local atEnd = false

		key, replacement = unpack(split(self.replaceChars[i], "%s+"))

		pos1, pos2 = string_find(CurrentWord, key, 1, true)
		if pos2 == #CurrentWord then
			atEnd = true
		end
		while pos1 ~= nil do
			testWord = string_sub(testWord, 1, pos1 -1) .. replacement .. string_sub(testWord, pos2 + 1)

			if self:TestWord(testWord) == true then
				suggestions[string_lower(testWord)] = self:EditDistance(CurrentWord, testWord)
				suggestionsCount = suggestionsCount + 1
			end

			if atEnd == true then
				break
			end

			pos1, pos2 = string_find(CurrentWord, key, pos1 + 1, true)
			if pos2 == #testWord then
				atEnd = true
			end
		end
	end

	return suggestions, suggestionsCount
end


--swap out each char one by one and try all the tryme chars
--in its place to see if that makes a good word
function WordDict:BadChar(CurrentWord)
	local suggestions = {}
	local suggestionsCount = 0

	local tempWord = {}
	local testWord
	local saveChar
	local tryChar
	local string_len = string_len
	local string_sub = string_sub
	local string_lower = string_lower

	if utf8 then
		string_len = string_utf8len
		string_sub = string_utf8sub
		string_lower = string.utf8lower
	end

	for i=1, string_len(CurrentWord) do
		tempWord[i] = string_sub(CurrentWord, i, i)
	end

	for i=1, string_len(CurrentWord) do
		saveChar = tempWord[i]

		for x=1, string_len(self.tryChars) do
			tryChar = string_sub(self.tryChars, x, x)
			if tempWord[i] ~= tryChar then
				tempWord[i] = tryChar

				testWord = table_concat(tempWord)

				if self:TestWord(testWord) == true then
					suggestions[string_lower(testWord)] = self:EditDistance(CurrentWord, testWord)
					suggestionsCount = suggestionsCount + 1
				end
			end
		end

		tempWord[i] = saveChar
	end

	return suggestions, suggestionsCount
end

--try omitting one char of word at a time
function WordDict:ExtraChar(CurrentWord)
	local suggestions = {}
	local suggestionsCount = 0
	local tempWord

	local string_len = string_len
	local string_sub = string_sub
	local string_lower = string_lower

	if utf8 == true then
		string_len = string_utf8len
		string_sub = string_utf8sub
		string_lower = string_utf8lower
	end

	if string_len(CurrentWord) > 1 then
		for i=1, string_len(CurrentWord) do
			tempWord = string_sub(CurrentWord, 0, i-1) .. string_sub(CurrentWord, i+1)

			if self:TestWord(tempWord) == true then
				suggestions[string_lower(tempWord)] = self:EditDistance(CurrentWord, tempWord)
				suggestionsCount = suggestionsCount + 1
			end
		end
	end

	return suggestions, suggestionsCount
end


--try inserting a tryme character before every letter
function WordDict:ForgotChar(CurrentWord)
	local suggestions = {}
	local suggestionsCount = 0
	local tempWord
	local tryme = self.tryChars

	local string_len = string_len
	local string_sub = string_sub
	local string_lower = string_lower

	if utf8 == true then
		string_len = string_utf8len
		string_sub = string_utf8sub
		string_lower = string_utf8lower
	end

	for i=1, string_len(CurrentWord) do
		for x=1, string_len(tryme) do
			tempWord = string_sub(CurrentWord, 0, i) .. string_sub(tryme, x, x) .. string_sub(CurrentWord, i+1)
			if self:TestWord(tempWord) == true then
				suggestions[string_lower(tempWord)] = self:EditDistance(CurrentWord, tempWord)
				local suggestionsCount = suggestionsCount + 1
			end
		end
	end

	return suggestions, suggestionsCount
end

--try swapping adjacent chars one by one
function WordDict:SwapChar(CurrentWord)
	local suggestions = {}
	local suggestionsCount = 0
	local tempWord = {}
	local swap
	local testWord

	local string_len = string_len
	local string_sub = string_sub
	local string_lowr = string_lower

	if utf8 == true then
		string_len = string_utf8len
		string_sub = string_utf8sub
		string_lower = string_utf8lower
	end

	for i=1, string_len(CurrentWord) - 1 do
		tempWord = {}
		for x=1, string_len(CurrentWord) do
			tempWord[x] = string_sub(CurrentWord, x, x)
		end

		swap = tempWord[i]
		tempWord[i] = tempWord[i+1]
		tempWord[i+1] = swap

		testWord = table_concat(tempWord)

		if self:TestWord(testWord) == true then
			suggestions[string_lower(testWord)] = self:EditDistance(CurrentWord, testWord)
			suggestionsCount = suggestionsCount + 1
		end
	end

	return suggestions, suggestionsCount
end


--split the string into two pieces after every char
--if both pieces are good words make them a suggestion
function WordDict:TwoWords(CurrentWord)
	local suggestions = {}
	local suggestionsCount = 0

	local string_len = string_len
	local string_sub = string_sub
	local string_lower = string_lower

	if utf8 == true then
		string_len = string_utf8len
		string_sub = string_utf8sub
		string_lower = string_utf8lower
	end

	if #CurrentWord < 3 then
		return suggestions, suggestionsCount
	end

	local firstWord
	local secondWord
	local tempWord

	for i=1, string_len(CurrentWord) -2 do
		firstWord = string_sub(CurrentWord, 1, i)
		secondWord = string_sub(CurrentWord, i+1)

		if self:TestWord(firstWord) == true then
			if self:TestWord(secondWord) == true then
				tempWord = firstWord .. " " .. secondWord
				suggestions[tempWord] = self:EditDistance(CurrentWord, tempWord) + 0.5
				suggestionsCount = suggestionsCount + 1
			end
		end
	end

	return suggestions, suggestionsCount
end


function WordDict:Suggest(currentWord)
	--Reload the possibleBaseWords if Suggest() isn't being called right after Contains()
	if self.lastContainsWord ~= currentWord then
		WordDict:Contains(currentWord)
	end

	local tempSugCount = 0

	tempSuggestion = {}

	-- generate phonetic code for possible root word
	local codes = {}
	local tempCode

	for tempWord, v in pairs(WordDict.possibleBaseWords) do
		if tempWord ~= nil and #tempWord ~= 1 then
			if self.soundslike == WordDict.Const.SoundslikeAlgorithms.PHONETIC then
				tempCode = WordDict:PhoneticCode(tempWord)
			elseif self.soundslike == WordDict.Const.SoundslikeAlgorithms.GENERIC then
				tempCode = WordDict:GenericSoundsLike(tempWord)
			else
				tempCode = ""
			end
			if #tempCode > 0 and codes[tempCode] == nil then
				codes[tempCode] = 1
			end
		end
	end

	local codeCount = 0
	for k,v in pairs(codes) do
		codeCount = codeCount + 1
	end

	local words = {}
	local editDistance
	local affixKeys
	local pCode

	if codeCount > 0 then
		--search root words for phonetic codes
		for word, v in pairs(WordDict.baseWords) do
			--check if this word's phonetic code matches a phonetic stored in codes
--~ 		affixKeys = string_sub(v, 1, (string_find(v, "/") -1))
--~ 		pCode = string_sub(v, (string_find(v, "/") + 1))
			--New version of above
			affixKeys, pCode = string_match(v, "(.*)/(.*)")
			if codes[pCode] ~= nil then
				words = {}
				words = WordDict:ExpandWord(word, affixKeys)
				--add expanded words
				for expandedWord,v in pairs(words) do
					editDistance = WordDict:EditDistance(currentWord, expandedWord)
					tempSuggestion[expandedWord] = editDistance
					tempSugCount = tempSugCount + 1
				end
			end
		end
	end

	--Obtain additional suggestions

	--check if the first letter should have been capitalized
	--print("Call: FirstCapital()", tempSugCount, "Mem:", collectgarbage("count"))
	for sWord, editDistance in pairs(self:FirstCapital(currentWord)) do
		tempSuggestion[sWord] = editDistance
		tempSugCount = tempSugCount + 1
	end

	--check if a char was typed in place of a replacementChar
	--print("Call: ReplaceChars()", tempSugCount, "Mem:", collectgarbage("count"))
	for sWord, editDistance in pairs(self:ReplaceChars(currentWord)) do
		tempSuggestion[sWord] = editDistance
		tempSugCount = tempSugCount + 1
	end

	--swap out each char on by one and try all the tryme
	--chars in its place to see if that makes a good word
	--print("Call: BadChar()", "", tempSugCount, "Mem:", collectgarbage("count"))
	if utf8 == false then
		for sWord, editDistance in pairs(self:BadChar(currentWord)) do
			tempSuggestion[sWord] = editDistance
			tempSugCount = tempSugCount + 1
		end
	end

	--try swapping adjcent chars on by one
	--print("Call: SwapChar()", tempSugCount, "Mem:", collectgarbage("count"))
	for sWord, editDistance in pairs(self:SwapChar(currentWord)) do
		tempSuggestion[sWord] = editDistance
		tempSugCount = tempSugCount + 1
	end

	--try inserting a tryme character before every letter
	--print("Call: ForgotChar()", tempSugCount, "Mem:", collectgarbage("count"))
	if utf8 == false then
		for sWord, editDistance in pairs(self:ForgotChar(currentWord)) do
			tempSuggestion[sWord] = editDistance
			tempSugCount = tempSugCount + 1
		end
	end

	--try ommitting one char of word at a time
	--print("Call: ExtraChar()", tempSugCount, "Mem:", collectgarbage("count"))
	for sWord, editDistance in pairs(self:ExtraChar(currentWord)) do
		tempSuggestion[sWord] = editDistance
		tempSugCount = tempSugCount + 1
	end

	--split the string into two pieces after every char
	--if both pieces are good words mak them a suggestions
	--print("Call: TwoWords()", tempSugCount, "Mem:", collectgarbage("count"))
	for sWord, editDistance in pairs(self:TwoWords(currentWord)) do
		tempSuggestion[sWord] = editDistance
		tempSugCount = tempSugCount + 1
	end


	--Sort the associative table holding suggestions, by EditDistance.
	local sorted = {}
	for w, d in pairs(tempSuggestion) do
		table_insert(sorted, {Word = w, EditDistance = d})
	end
	table_sort(sorted, function(A, B) return A.EditDistance < B.EditDistance end)

	--print("Sorted Table:")
	--tprint(sorted)

	WordDict.suggestions = {}


	for i, v in ipairs(sorted) do
		if v.EditDistance < self.maxEditDistance then
			table_insert(WordDict.suggestions, {Word = v.Word, EditDistance = v.EditDistance})

			if #self.suggestions >= self.maxSuggestions then
				break
			end
		end
	end

	return self.suggestions
end

function WordDict:Init(locale)

	if locale == nil then
		self.locale = GetLocale()
	else
		self.locale = locale
	end

	--Load the dictionary

	--If we don't have a languange specific dictionary, load the US English dictionary
	if self.locale == "ruRU" then
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_ruRU()
		--utf8 = true
	elseif self.locale == "itIT" then		
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_itIT()
		--utf8 = true
	elseif self.locale == "enGB" then		
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_enGB()
		--utf8 = true
	elseif self.locale == "deDE" then
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_deDE()
		--utf8 = false
	elseif self.locale == "esES" then
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_esES()
		--utf8 = true
	elseif self.locale == "frFR" then
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_frFR()
		--utf8 = true
	elseif self.locale == "enUS" then
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_enUS()
		--utf8 = false
	else
		DICTIONARY_TO_LOAD_FROM = WordDict_GetDictionary_enUS()
		--utf8 = false
	end

	--set the dictionary functions to nil to free memory
	WordDict_GetDictionary_ruRU = nil
	WordDict_GetDictionary_itIT = nil
	WordDict_GetDictionary_enGB = nil
	WordDict_GetDictionary_deDE = nil
	WordDict_GetDictionary_esES = nil
	WordDict_GetDictionary_frFR = nil
	WordDict_GetDictionary_enUS = nil

	--Set the UTF8 flag
	utf8 = DICTIONARY_TO_LOAD_FROM.UTF8

	--Load the tryChars
	self.tryChars = DICTIONARY_TO_LOAD_FROM.Try[1]

	--Load the replaceChars
	for i, v in ipairs(DICTIONARY_TO_LOAD_FROM.Replace) do
		table_insert(self.replaceChars, v)
	end

	--Load Affix Rules
	self:LoadPrefixRules()
	self:LoadSuffixRules()

	--Some languanges just use the generic sounds like routine, while most have a phonetic code defined for each baseword.
	self.soundslike = DICTIONARY_TO_LOAD_FROM.Soundslike
	if self.soundslike == "Phonetic" then
		--Load Phonetic Rules
		self:LoadPhoneticRules()
	end

	--Load the Base Words
	self:LoadWords()

	--Free the table we loaded the dictionary from
	DICTIONARY_TO_LOAD_FROM = nil

	--Collect Garbage to free up memory
	collectgarbage("collect")

	return self.locale
end

--[[
--testing

function tprint (t, indent, done)
  -- in case we run it standalone
  local Note = Note or print
  local Tell = Tell or io.write

  -- show strings differently to distinguish them from numbers
  local function show (val)
    if type (val) == "string" then
      return '"' .. val .. '"'
    else
      return tostring (val)
    end -- if
  end -- show
  -- entry point here
  done = done or {}
  indent = indent or 1
  for key, value in pairs (t) do
    Tell(string_rep (" ", indent)) -- indent it
    if type (value) == "table" and not done [value] then
      done [value] = true
      Note (show (key), ":");
      tprint (value, indent + 2, done)
    else
      Tell (show (key), "=")
      print (show (value))
    end
  end
end

function GetLocale() return "frFR" end
--function GetLocale() return "deDE" end
--function GetLocale() return "esES" end
--function GetLocale() return "enUS" end



--require("bit")

require("Dict\\Dic_" .. GetLocale())

WordDict:Init()

function test1()
	print("UTF8", utf8)

	--testword = "applepie"
	--testword = "hermonóo"
	--testword = "helpin"
	testword = "procrea"
	testword = "contraórdenes"
	testword = "holla"
	testword = "hóla"
	testword = "Bahammas"
	testword = "mehrstelling"
	testword = "abbürtsen"
	testword = "angstrœm"
	testword = "d'angstrœm"
	testword = "angstrœms"
	testword = "qu'a"
	print("Contains")
	print("Testing: " ..testword, WordDict:Contains(testword), "Len", string_utf8len(testword))
    tprint(WordDict:Suggest(testword))

	--s, sc = WordDict:ReplaceChars("holla")
	--tprint(s)

	tprint(WordDict:ExpandWord("a", "Q"))

	print("Phonetic Code:", WordDict:PhoneticCode("angstrœm"))
--	print(WordDict.AffixUtility:RemovePrefix("procrea", WordDict.prefixRules["o"]))
end


test1()

--~ print(string.utf8byte("Đ"))

--~ cnt,cc,ccc = WordDict.AffixUtility:EncodeConditions("[^eioĐ]ño")
--~ print("Condition Count:", cnt)
--~ tprint(ccc, 4)



--tprint(WordDict:Suggest(testword))


--~ print("len:", #"hermanó")
--~ print(WordDict:Contains("hermanos"))
--~ print(WordDict:Contains("hermonóo"))
--~ tprint(WordDict:Suggest("hermonó0"))

--print("Memory:", collectgarbage("count"))
--Timmer now = os.clock(); test(); print(os.clock() - now)
---]]