ChangeLog.old

77c845d (HEAD -> master, origin/master, origin/HEAD) Fix flag elimination bug (reported in issue #342).
c827bb6 Do not call fsm_destroy until issue with 'double free or corruption' has been fixed.
3b0e58a Provide a get_current_dir_name() for platforms without one
b0fb7a6 Merge branch 'master' of https://github.com/hfst/hfst
65a4d79 Search for included files under scriptdir, not working dir. If an included filename is non-absolute, prefix it with the directory the script was in when trying to read it. This only happens in non-Windows environments for now.
f5e94d7 Free allocated memory.
6144e75 tokenise -g: check for eof
cf46e69 Apply more changes from latest release of foma.
a9f8397 Apply some changes from newest release of foma.
cc8c96b Make a separate directory python/pypi for creating pypi packages.
479789e Rename the pypi package to 'hfst' and use a four-number versioning scheme. Also define '__version__'. Add an empty config file as hfst c++ headers include it.
eb304f0 tests: trying sed instead of awk for exit on <STREAMCMD:FLUSH>
798383d tokenize: handle Apertium-style superblanks in --giella-cg format
7922399 Provide a way to scale the vector similarity calculations via variable
166d418 Merge branch 'master' of https://github.com/hfst/hfst
9bc5315 parse word2vec -style text format and assume first line is info
5d3f6b8 Do not allow unescaped dots in regular expressions.
ce5e2a5 Merge branch 'master' of https://github.com/hfst/hfst
a8f21f0 Observe HAVE_SFST and HAVE_FOMA when including back-end header files.
8a9c0de nwords catenate-style argument for Like() function
1433e45 Merge branch 'master' of https://github.com/hfst/hfst
1ec8e03 Add experimental two-vector model for word sense
72d4f72 (tag: v3.12.0) Ready for release 3.12.0.
da86647 Read vector models and do simple ranking
42df833 Small fixes to memory leaks and efficiency.
e2abd89 Fix some memory leaks.
da54acc Fix leaks reported by valgrind.
92cf4dc Fix more memory leaks.
9bd9749 Fix some lost memory issues reported by valgrind.
6e129a4 Remove unnecessary variable. Delete transducer definitions and latest regexp when destructor is called.
d700309 flush given outstream, not necessarily cout
8d4f824 Add a destructor to XfstCompiler.
6b10aab Update README.
863e94a Add missing files to dist and remove generated ones.
e4c1304 Rename hfst.rules into hfst.sfst_rules. Fix hfst.types into hfst.ImplementationType in tests.
80f18a8 Use single quotes instead of double quotes in docstrings.
52a9d75 Add test for hfst.xerox_rules.
00b020c Rename xerox_replace_left and xerox_restriction back to replace_left and restriction in package hfst.xerox_rules.
d1556dd Fix documentation of parameter 'output' of function 'extract_paths'.
3df4a48 Sentence separation for finnpos-mode
d3058c7 Issue a warning if automake is too old, which can make build fail if pre-generated yacc files are modified.
9166cdc Move ImplementationTypes and ReplaceTypes to their own classes and get rid of package hfst.types.
6d6c631 Fix datatypes used in for loops.
c6aa5e1 Fix some typos in documentation.
8073675 Use renamed versions of xerox rule functions.
ae70056 Tentatively add HfstXeroxRules to python API in package hfst.xerox_rules.
dee0176 Fix some more warnings.
37edbbb Fix minor warnings generated by clang.
faa7647 Fix typos in ifdef guards.
41c5a32 Add a simple script for fetching statistics about hfst downloads.
d9d8077 Update pypi setup.
822f4bb doh, forgot to add test expected output
cccba22 hfst-tokenise giella-cg: fix indentation issue with backtracking
5a26269 Fix still some warnings.
32c82b3 giella-cg backtracking: ensure ignored spaces are still output
03d6a23 Do not warn about pragma push/pop issues or unsafe cstring functions on windows
9c12094 Use hfst::hfst_fopen instead of fopen. Fix some warnings related to datatype conversions.
d64cb49 Add function hfst::hfst_fopen that uses fopen_s when compiling with msvc.
5ed27ea Fix some warnings caused by conversions (possible loss of data).
b93a144 Do not use extern templates when compiling with msvc.
73f7e18 Remove files generated by the tests.
61ac4f8 Add a batch file version of test.sh for testing the python bindings on windows.
b53a128 Fix warnings.
7cb8740 Update pypi setup script.
cc9cca4 Remove unnecessary back-end include directories as well as commented code from Makefiles.
fd5f007 Fix some minor warnings.
ec114ea --giella-cg support for @PMATCH_BACKTRACK@ symbol (#337)
15fbb1f Update windows script.
097826c Remove extra comma from initialization.
d6b001d Add missing header file.
9b3c122 Fix warnings generated when compiling with -Wall.
a0dee64 Fix most of the warnings generated when compiling with -Wall.
89b2061 rename --gtd to --giella-cg, and more descriptive help text
335df14 Add support for compiling on windows.
85585c4 Fix a typo.
e628f55 Add a manifest file for pypi packaging.
48c34ef Tentatively add a setup file for creating pypi packages.
6abde49 update to proc to make it possible to weight surface forms
5e379fc Update windows compilation script.
1f55a51 Define int64 for msvc.
21f52d0 Implement snprintf for msvc compilation.
c05469e Replace definition _Bool with Boolean, as _Bool conflicts with typedefs in msvc header files.
00bbb1d Get rid of unnecessary typenames and typos noticed during compiling HFST on windows.
b160c07 Forward declare back-end implementation datatypes and wrap their destructors.
067c3c9 Add missing <iterator> header.
2f0c8df Add missing std:: to vector.
34e99ad Issue an error message if glib unicode handler is requested but not found.
d28075c Declare all functions that are needed in start_xfst().
ec34dea Add math.h
4a3be6b Make preparations for supporting OpenFst version 1.5.4.
c9940f6 Include HfstBasicTransducer.h instead of HfstTransitionGraph.h.
8560638 Add extern templates.
dc96953 Remove unnecessary OpenFst headers and use forward declarations instead.
699c098 Wrap StdVectorFst and LogFst destructors as both classes are forward declared.
7f81f41 Forward declare as much as possible when using OpenFst library. Also remove unnecessary include directives and add missing std namespaces.
a0b2fb3 Prepend missing 'std::'.
c8137fa Make HfstBasicTransducer and HfstBasicTransition fully implemented classes instead of generated template specializations.
eb5b877 Improve the way error messages are handled.
b8278ec Change 'import libhfst' to 'import hfst'.
2b9e44b Move implementations of HfstTropicalTransducerTransitionData away from header file.
ce05c67 Merge branch 'master' of https://github.com/hfst/hfst
3ec0d09 Alter expected result to conform with newly expected behaviour
759605c Do not allow newline characters inside quoted literals.
f296b6b Merge branch 'master' of https://github.com/hfst/hfst
63d7544 Allocate enough memory for XRE error message.
f968ebf Trying out facility for communicating sentence boundaries to plain tokenization mode
6a78c4c Merge branch 'master' of https://github.com/hfst/hfst
bf2ded3 Don't print size info when unverbose
5a33841 Don't separate tokens with empty lines in plain tokenization mode
d0a1b55 Warn about undefined multichar symbols when compiling lexc in verbose mode.
5d2c67d Fix a typo in option handling.
e2fee81 Tentatively add an option --do-not-minimize to hfst-regexp2fst which performs determinization instead of minimization. Also make hfst-xfst use determinization if variable 'minimal' is set to 'OFF'. Implement this by adding function hfst::set_minimization(bool) to HFST API and a function HfstTransducer::optimize() which performs minimization, or determinization if hfst::set_minimization(false) has been called. Replace calls to minimize() with optimize() in parts of HFST code relevant to the tools mentioned.
82aa26e Get rid of CR-LF newlines.
44f2e21 Remove commented code.
94b7607 Print an error message if output type is defined several times.
966a895 Fix mismatched free/delete reported by valgrind.
6dc1d8f Do not use strdup with -std=c99.
407a517 Use delete_foma instead of delete/free when deallocating memory used by a foma transducer.
07508b5 Limit to C++11 for now
d15b4be Use free with malloc, not delete.
b693b3a Add a function 'void LexcCompiler::reset()' and use it in XfstCompiler when reading lexc input again.
bc5ac27 Remove revision info from output of hfst-info as it doesn't work correctly in git.
ace4801 Pass cascade argument by reference in lookup_cascading functions.
e2ea4cb Fix a typo in function 'strip_percents' which resulted in incorrectly escaped zeros.
92a7152 delete more lines
9f940a3 remove osx lines from yml
f7bd085 Unobservable change - nothing happened here
d1bf79b close #336: off-by-one segfaults in OlLetterTrie
2574e27 Avoid using projections in casing functions (which cause minimization issues with large weighted transducers).
73a3ff0 Parametrize optionality in casing functions
0adc1f9 Make sure there's no case fallthrough
58eb35d Fix other casing functions too
c366c4c Another fix to Cap()
704c1c7 Merge branch 'master' of https://github.com/hfst/hfst
5a5df2d Change operation of Cap() and make timing per-object
44f0ae7 Fix missing empty lines in output when cascading composition is used.
284f9d0 Add an option --max-number=N for OL transducers which limits the number of results. If not defined, defaults to 5 for infinitely ambiguous input and to infinity for finitely ambiguous input.
5686864 There are some problems in using the bindings with python version 2, so refer only to python3 in README.
5a17fb0 Add a missing 'break' to switch block, making variable infinite_cutoff work correctly.
4c2f26f If no results are found, print it only once.
d703776 Tentatively add support for --xfst=print-pairs for OL transducers.
b7fcab9 Remember to flush the streams also when no results are found.
7ba32cf Merge branch 'master' of https://github.com/hfst/hfst
bc7a80a Implement a pair-lookup for optimized-lookup transducers.
ce3133f Add a fourth argument of type hfst::ImplementationType to the constructor and a function get_type() to ImplementationTypeNotAvailableException.
4ffe105 Print to console on windows if print-pairs is specified.
587f3c0 Do not print extra newlines in cascading composition.
4353894 Fix a bug/typo in string comparison.
72de782 Improve documentation in HfstTransducer.lookup.
0c888e9 Rename HfstBasicTransducer.lookup_fd to HfstBasicTransducer.lookup and add an argument 'obey_flags'.
d5493ac Rename HfstTransitionGraph's lookup_fd to lookup and change some parameter names to be more descriptive.
56a8d2f Add instructions for linking hfst c++ library statically in comments.
67e25ff Add missing file to windows scripts.
8db14ce (tag: v3.11.0) Ready for release 3.11.0.
fe37aa0 Update morphology checks.
8d3f162 Update EXTRA_DIST.
6dac246 Do not use binary transducers in tests.
ee62c44 Add missing doc file.
a367645 Mention that python tests are no longer available.
da2678d Skip Python tests as they can now be performed in the folder python/test.
ea47f0b One more stray memory leaking cache
95e3480 Memory conservation: only cache named objects and always minimize them.
6b113b2 Merge branch 'master' of https://github.com/hfst/hfst
974d240 Turn off automatic delimiter-contexts ("set need-separators off") in the presence of user-defined contexts
01fccfd Use docstrings.i instead of swig's autodoc. Document the contents of hfst and hfst.exceptions modules.
4322ca7 Add a docstrings file created semiautomatically from doxygen documentation.
816ce6c Add at least a simple docstring for all functions.
82c8f6a Merge branch 'master' of https://github.com/hfst/hfst
1b11fbf Obey -i option for giving the pmatch archive filename
4620b1b std=c++ works on OS X but not Linux, so disable Linux+Clang
d245a74 Test if gnu++ appeases Travis' clang
3d9c66b Include needed m4
5e73db3 Copy hfst-ospell's C++11+ detection
9af66d0 short int -> short; long int -> long
daa457e Yet more documentation..
e1b01ff More docstrings.
711a089 Add docstrings.
acf5696 tokenise: don't skip output-empties
b5b4434 tokenise: don't let empty analyses affect weight classes
18f5a2b travis: fix indentation
62d216a getopt_long() returns an int, not char; this is important because char is unsigned on many platforms which means char(-1) is 255
39d28d1 Do not check if lookup is infinitely ambiguous if --time-cutoff is defined.
c42af37 Remove redundant flag diacritic check. It was once removed, but got pushed there again.
b7b5882 Add support for --cascade also for optimized-lookup transducers.
9169e38 Support combination --cascade=composition --xfst=print-pairs.
51a0b61 Merge branch 'master' of https://github.com/hfst/hfst
ef6ed81 Fix the way cycle limit argument is passed.
9ad0950 Merge branch 'master' of https://github.com/hfst/hfst
6d94c90 Add a preliminary omorfi-friendly conll-u mode
45a0bf2 Remove redundant filtering of paths containing flag diacritics. Flags are handled already in function HfstBasicTransducer::lookup_fd.
5c7135a Implement option --cascade.
18d5e66 Tentatively add option --cascade to hfst-lookup.
1dcad09 Print error message if --xfst=obey-flags is used with optimized-lookup transducers. They always obey flag diacritics.
670ee39 Merge branch 'master' of https://github.com/hfst/hfst
f903d9c When doing apply up or down in hfst-xfst's, check if 'obey-flags' is ON or OFF.
8e9c6e6 travis: --enable-silent-rules
a253b8d Add an argument obey_flags=false to is_lookup_infinitely_ambiguous functions.
14bdd9c Make is_lookup_infinitely_ambiguous aware of flags. Use typedef hfst::StringVector in HfstLookupFlagDiacritics to prevent conflicting definitions in hfst-twolc which defines a class StringVector.
349fb73 Add option obey_flags=false to lookup_fd.
fdf912a Add tentative script for converting from doxygen documentation to docstrings.
7618907 Modify function visibility.
bc01c0b Control visibility of functions in HfstTransducer.
b535aa6 Follow foma and xfst convention for operator ignore '/', i.e. expand any identities and unknowns before inserting the strings to be ignored.
c1b4341 Add an argument 'force' to HfstTransducer::harmonize which makes the function harmonize also FOMA_TYPE transducers.
724b4e9 Obey variable 'print-sigma' when print_net is called.
6bec507 Add brief descriptions for hfst packages.
ca892d1 Document the implementation types.
98764da Fix bugs in options --xfst=show-flags and --xfst=obey-flags when non-optimized lookup is performed.
93b9fe2 Close the outstream when done. Not doing this was causing a bug where some buffered data never got written to the output file.
7d97106 Obey variable 'print-pairs' when performing apply down in non-optimized format.
ac2fc3a Add parentheses in conditionals of function 'flag_build' to ensure correct interpretation. Also remove redundant code.
28867c1 Allow punctuation as context in the "need-separators on" mode
0cd05ee Add "need-separators no" to test script to reflect changed default in pmatch
2597799 exclude gcc from osx builds
7dfa8ec Track used symbol names
81128c2 Allow lowercase define()
52a4ca4 Merge branch 'master' of https://github.com/hfst/hfst
af71662 Add side arguments to casing functions
7c1b388 Revert back to the original extra_link_args in python bindings.
2d5c670 Rename examples.py to test_examples.py. Perform tests in a more logical order.
4c8a1e3 Update documentation about linking to the HFST C++ library.
9f8ca0f If --enable-no-tools is used, skip tests in test/tools/.
41dc9a2 Mention module hfst.types.
a8c3e56 Remember to apply modifications to weight pushing to tests and hfst package.
835f0ef Get rid of TO_FINAL_STATE and TO_INITIAL_STATE and replace function 'push_weights' with 'push_weights_to_start' and 'push_weights_to_end'.
d05b9db Wrap HfstTransducer implementation types in package 'hfst.types'.
5e1a245 Add missing files to dist.
58d4602 Update README. Handle linking to extension library better.
de888b6 Update README. Handle cases where an implementation type is not available. Add missing files to dist.
1a68126 Fix a typo in URL.
7a2296b Update link in README.
861baf6 tokeniser: --weight-classes=N opt, as in hfst-proc
151f1fa Add functions lookup_optimize, remove_optimization and copy to HfstTransducer.
4efc9af Make HfstTransducer() create the empty transducer.
0f237c1 Comment out hfst_ol::set_extract_tags_mode(bool). Improve error messages in att and prolog readers.
57426ca Remove unnecessary commented file.
b7139cd Fix HfstException documentation.
3b13bf5 Merge branch 'master' of https://github.com/hfst/hfst
7e6d27c Fix a typo.
9f4ceb7 Merge branch 'master' of https://github.com/hfst/hfst
829c9e2 Implement remaining variables in the library and support them in hfst-pmatch, with the exception of need-separators which has to be specified during rule compilation, at least for now.
457c5de Document AttReader and PrologReader.
8695c47 Add HfstException::what() to python api.
2cffffa Add an alias what() for operator() to HfstException.
5fdf61a Add class PrologReader and tests for it.
77eda55 Add a new class AttReader and tests for it.
18edac5 Add package hfst to setup. The package is essentially a wrapper around libhfst module which is still available as before.
e41f131 travis: re-encrypt channel to #hfst
6329bb8 Use all implementation formats in tests.
5a5018c Do not use default_fst_type as pmatch transducers are always in ol format.
b38f87b Add stream tests.
ccaea0e Represent empty transducers as empty lines so that read_att_transducer will work correctly.
fe66b18 Remove unnecessary files.
bf5511d Forgot to do git add before committing...
3710f91 Update test examples.py and add it to test.sh. Remove extra files.
7d5064d Add directory tree for package 'hfst' and its subpackages 'exceptions' and 'rules'. This package encapsulates module libhfst.
7818c1e Delete unnecessary file.
91b86a0 Rewrite part of tests. Document the interface file.
657013d Add new tests.
3b4c186 Merge pull request #323 from unhammer/tokenise--gtd-tags-are-multichars
67125ab rm unneeded import
d927d79 pmatch: store input/output_symbol_strings, _parts refer to them
c665f8a Add missing endverbatim tag.
884d1dc Add missing file.
bc87fc5 Get rid of HfstFile wrapper.
30cde02 Try to preserve the names of transducers when converting.
90b3328 Expand prolog tests. Fix a small bug in naming transducers.
8e61537 Forgot to update testfile.
63a0456 Add a new function for reading prolog input and improve the way att input is handled.
9ef38f7 Split part of code related to reading att input into a separate function. Change the way transducer name is set when reading prolog input.
6d12e0d erge branch 'master' of https://github.com/hfst/hfst
22f1c47 Modify documentation, update tests, add new functions.
d14bc2f link to travis; bug tracker link sf→github
e8c7fd6 README is now symlink to README.rst; for more readable github
2258354 travis: no irc notes on success
1d8a1f4 travis: encrypt IRC channel for hfst/hfst
38c9544 Add HfstTokenizer examples as tests.
c635cd4 More documentation to HfstTokenizer.
c1ca3b6 Add more examples for HfstTokenizer.
0dc0cf4 Reorganize doxygen documentation to represent the new module structure of hfst python api.
95d658c Remove hfst.py, a directory structure is needed instead for doxygen.
5025aab Add new documentation for hfst python package.
4749745 Add more tests.
8f8b345 Split python tests to several files.
45aeb38 Skip one more failing pmatch test.
83dee76 Add missing files to dist.
a44b99e erge branch 'master' of https://github.com/hfst/hfst
b5ca1ec Add more documentation and modify functions.
579ee71 Merge branch 'master' of https://github.com/hfst/hfst
8050575 A PmatchContainer constructor taking a std::vector<HfstTransducer>. This should leave the passed vector untouched.
88aa682 Add more documentation.
ab82047 Add a simple test for XreCompiler.
a25eb2b A new test.
caf1ca8 XreCompiler::setOutputToConsole is needed, uncomment it.
f6acf2a Split c++ code to multiple files from libhfst.i. Update URL in setup.py.
780374a Merge branch 'master' of https://github.com/hfst/hfst
4c9ca19 Settle on a way to handle variable names with dashes (which look like minus signs)
967cc78 actually run the tokenize tests
5df87b6 Add more failing tests.
775a499 Fix some typos.
2e77e2d Add missing files to dist.
11ebca0 Merge branch 'master' of https://github.com/hfst/hfst
067e637 Merging master
4c9d43a Variable setting and reading
881b81c Use -Wno-deprecated also for tests.
85cb277 Modify messages that get printed during make check.
0761730 Skip tests that do not exist. Make prints more informative.
573324e Skip failing pmatch tests.
164e621 Add missing tests for ConvertTransducerFormat. Compile them with -Wno-deprecated.
f945e17 travis: irc notifications on build failure
0229baa Merge branch 'master' of https://github.com/hfst/hfst
93ed0db Remove const declarations from openfst back-end that cause compiler warnings.
5df2a2b Run g++ with -j2 (not -j) on travis; o/w runs out of memory
54b8f45 Comment out a function that is not defined.
48c91b8 Specify -Wno-deprecated also for tests in folders hfst-tagger and hfst-twolc.
810c330 https://travis-ci.org test rules
708171e doh, define multichar-symbols (for 2eb6888)
2eb6888 test for #320 / bf26db9
3aecdbc Merge branch 'master' of github.com:hfst/hfst
bf26db9 only change case if sf and lm case differ (#320)
b3782de Remove a rule that cannot be matched since a rule has already been defined for "DownCase(" earlier.
5ad3487 Add the keyword const to arrays used to store xfst commands.
79c7dfe Make compiler happy by using strdup instead of plain const char *.
f6d9953 Handle return values of fread in foma and sfst back-ends to prevent warnings during compilation.
214b7b6 Fix a typo.
862e858 Replace foma binary files that are used for testing results from lexc with equivalent files in prolog format. Then the tests will not depend on the foma back-end.
680389b tokenize tests
2b7cf7e tokenize: new --gtd Giellatekno/Divvun mode
adb37d4 tokenise: fix #318, header parsing for TOP
76e9069 Fix a bug in flag elimination involving multiple flags. Fixes bug #315.
f653780 Evaluate transducers (as opposed to argumentless functions) where declared
e6c6e82 (tag: v3.10.0) Ready for release 3.10.0.
97c1285 Swap implementations of commands 'apply up' and 'apply down' in hfst-xfst, making them work as they do in foma and xfst.
85ef34c Forgot to add before latest commit...
13280ea Edit warning about converting native foma transducers.
b31a6c1 Fix typo: swap left and right.
808d984 Add hfst-flookup to windows scripts.
66db285 Update bug tracker url and copyright year.
b127ab3 Git ignore hfst-flookup.
1b40de6 Add tests for hfst-flookup.
12e4cca Trandsducer operations no longer return a reference to the transducer, fix this in tests and interface operations.
a128c7f Add a tool hfst-flookup which does lookup from right to left, i.e. in the same way as foma's flookup and xfst's lookup. hfst-lookup and hfst-optimized-lookup stay as they are, but mention in their help message that they do lookup in the opposite direction.
cea21b4 Add a comment about the way gzipped input is handled.
da7ebcc Make HfstInputStream constructor throw a FileIsInGZFormatException if input is in gzipped format. In hfst-fst2fst, issue an error message that asks user to unzip the transducer.
5c2755b Merge commit frankier:add-gitignore manually and also add more files to be ignored by git.
99b470d Merge pull request #313 from frankier/escape-dot-labels
f6905a1 Improve handling of argumentless functions and funcalls
d804d3d Remove some memory leaks in compilation
ef740e0 Fix nested funcall issues
61e1e0a Add optional versions of cap functions and AnyCase().
68e078b Corrections and additions to casing functions, and add Implode() and Explode()
2484633 Add a facility for detecting when hfst-tokenize is given a "bare" dictionary and generating a somewhat intelligent tokenizer when that happens. Due to awkward architectural reasons this involves changes and additions in many parts of the library, but they are not major.
089f3ca Add .gitignore
d1d81db Merge branch 'master' of https://github.com/hfst/hfst
485f575 Fix bug that caused exclusionary lists to be missed much of the time.
03ff992 Escape " in dot labels.
9984096 Factor out replace_all into string-utils.cc.
ca10d77 Revert swapping of hfst-xfst's 'apply up' and 'apply down' commands.
7352cf7 Swap implementations of 'apply up' and 'apply down' in hfst-xfst, so that these commands work in the same way as in foma and xfst.
db2d4dc Add an example for HfstTransducer constructor.
f37b8d1 Tentatively implement lookup also for transducers that are not in optimized-lookup format using composition and path extraction.
1cb2992 Add an HfstTransducer constructor that takes a StringVector as argument.
f32c049 Support python 2 unicode strings in lookup.
8d2c6b8 Fix a typo
f442b91 Make HfstInputStream iterable by implementing functions __iter__, __next__ (for Python 3) and next (for Python 2).
2c660ff Add functions in namespace hfst::rules to python API.
947bb62 Add some information about Python interface in README and advise the users to see file python/README.
067b0c2 Remove slashes from headers used when processing the file.
c4b5ce8 Update documentation.
0e4e8a7 Redefine functions which return a reference to themselves as void to avoid problems on the python side.
5f97eac Tentatively add a C++ function 'std::vector<hfst::HfstTransducer> compile_pmatch_expression(const std::string & pmatch)', mostly copied from hfst-pmatch2fst.cc.
21da500 Add a const declaration to string& argument of PmatchContainer::process and PmatchContainer::match.
cd7f2f8 Tentatively add some pmatch functions. Add a missing typedef for HfstTransducerVector.
a9fbd44 Add a function to HfstInputStream for finding out whether hfst headers are included in transducers. When reading native foma transducers in hfst-fst2fst, warn that inversion may be needed for lookup to work as expected.
fa4565e (tag: v3.9.2) Release 3.9.2.
3f7a57e Update hfst binary file used for tests.
5cd8a71 Add option --pythonpath to python tests. Document the test tools better.
927da9c Make fst2txt functionality test silent.
ad8c82c Update installation check scripts.
ddf749d Remove test.py from EXTRA_DIST, the tests have been rewritten.
dd7d0be Merge pull request #309 from unhammer/pmatch-input-marks
97b6e3f Merge pull request #308 from unhammer/tokenise-minor-cg-fix
3781806 Make small modifications to documentation and typedefs used in function declarations/definitions, so that doxygen can generate documentation right.
00d89ce Update Python and C++ API documentation files.
4319b94 Move api documentation file under folder 'doc' too.
67e1e8c Move C++ documentation file under folder 'doc'.
8e2e651 Do not catch exceptions derived from HfstException inside the constructor of HfstInputStream and throw them as HfstException. Instead, throw them as such.
56ffca8 pmatch: special symbol @PMATCH_INPUT_MARK@ for input marks
90aa695 pmatch: on seeing a special symbol, mark the offset in the input string
bc345c6 CG forms always go in "<>", not before the tab
8dceb24 Merge pull request #307 from unhammer/tokenise-short-options-missing
2c207de short option -m missing from getopt_long
a9314ad Add m4 for older distros
6c944ba Bump patch version since revision has decreased
859f8da Add builting function Interpolate()
c6f9908 Fixed condition for including symbols in exclusionary lists Rather than the printability vector, which may not be completely set yet, check the symbol itself for printability.
780b9af Fix bug involving stale caches of function call results
26d3d8f Change python3 into python in README as version 2 works now too.
7af6f3f Move tools/src/hfst-string-conversions.[cc|h] under libhfst/src/.
a1b5bb2 Update version number to 3.9.1.
d23eb93 Do something useful in the case of a missing root rule
9e4e663 Make Makefiles in openfst back-end licensed as LGPL.
03e1949 Candidate for release 3.9.1.
27f80ee Fix names of files whose names were changed.
8e4f324 Set server based search to NO.
5b2c698 Add word boundaries in xerox mode.
6e78e1f Fix copypasted typo
b6970ab Fix ToLower and ToUpper, which had transposed names and flawed implementations
c939413 Use iterator instead of range-based loop on windows.
e5897fb Fix a couple of typos inside 'ifdef _MSC_VER' detected when compiling on windows.
7694ee0 License for the library code is now lgpl 3.
b8a5b98 LGPL license
a031526 Compatibility with python2.
1baee13 Add exclusionary lists (eg. Exc(Whitespace) matches symbols not in Whitespace)
a92e1d0 -X option fixed in hfst-pair-test. Additionally, the user gets an error message if unescaped :-symbols were found.
8d0a028 Fix a bug in HfstTransitionGraph::find_replacements. Implement compile-replace without composition in XfstCompiler to avoid alignment issues in result.
887381e Fix epsilon handling in compile-replace expressions.
2891a27 Allow epsilons inside compile-replace expressions as xfst does.
bcfea4a Warn about '@_EPSILON_SYMBOL_@' not being an ordinary symbol in hfst only in verbose mode.
5fc456b Allow space after operator '^' in regular expressions.
0758730 Fix binding precedence of :
9f4628e Add throw to appease rpmlint no-return-in-nonvoid-function
78efc5d Fix a bug in function 'fst'.
23a5285 Print a blank line even if there were no results or if tokenization failed
73d317b One && too many
5f4a8fc Rename hfst-proc2 to hfst-tokenize, with hfst-proc2 and hfst-tokenise as aliases
732e37e Add tests for python bindings.
51cbc5e Complete rewrite of pmatch compilation This is a cleanup and rewrite of pmatch compilation, hopefully allowing easier extensibility and maintainability in the future. While this has been reasonably well tested, it is likely that there are some regressions; please report any. Various additional features are pending.
31de91e Move scripts under subdirectories and remove scripts that are not needed.
6156215 Move windows scripts under scripts/windows. Get rid of libhfst_win.i.
0bf33d7 Remove HfstFastTransition.h also from dist.
02fa326 Remove HfstException.cc and HfstException.h files, HfstExceptionDefs.cc and HfstExceptionDefs.h are used instead.
9356517 Remove class HfstFastTransitionData and the template HfstFastTransducer that uses it. HfstBasicTransducer is the template of HfstTransitionGraph that is used everywhere and it is fast enough.
dd6873e Remove tentative reentrant version of xre compiler. The normal xre compiler can handle expressions that the reentrant one was originally intended for.
05c78d8 Update python API documentation.
f933525 Remove old commented code. Add a warning about python bindings not being under autotools; advise users to see python/README.
b127d48 Set -std=c++0x unless a higher standard is required with the -std option.
f2738be Do not generate hfst-scanner header file. Make hfst-scanner depend on hfst-compiler header file to make sure they are generated in the right order.
a44eb61 Check automake version in configure to determine whether header files generated by yacc/bison should use extension .h (automake < 1.12) or .hh (automake >= 1.12).
7f206c0 call_counter is a member of the container, not the transducer
2aeda9f Reweight arcs iff all conditions given with --input-symbol, --output-symbol and --symbol are met.
fddc9ea Forgot to increment call counter
a56089b Only call clock() every millionth time
1c4c871 Add missing declarations
a710ddd Only check clock() every million calls to get_analyses() to save time
8cffb74 Move start_clock setting to where the line is read to make the time limit per-analysis
6488877 Add short option for time cutoff (-t)
b6fdeb6 In pmatch, make the time cutoff "soft" to ensure we at least return something (this was an issue for rulesets with RTNs only - all the time could be spent in one leaving nothing for anything else, causing needless failure)
70e025f Ignore previous commit, I had some old library code confusing me
8dfcd66 Add time cutoff options
8a9f0c3 Fix what seems like a typo in hfst-lookup, &infinite_cutoff for infinite_cutoff It seems like the size_t and size_t* can substitute for each other, but the prototype is for size_t.
2134ef4 Fix time cutoff check in ol library code
8ff4aa3 Omit infinitely ambiguous check when we have a time cutoff
969d479 Fix time cutoff handling
dd30c33 Switch from !€ / !$ to !!€ / !!$ as the test case prefixes, to bring it in line with the conventions in the rest of the Giella infrastructure (where this testing setup is used).
fd175f1 Fix option --to-transducer of hfst-substitute.
d213839 Update HfstTransducer::priority_union declaration. It takes only one argument instead of three.
0dec506 Add new test files to dist.
7915ca0 Priority union bug solved.
daee8f7 Update output of morphology tests.
1148413 Add a check for hfst-ospell.
a179a32 Add new replace markup tests to dist.
c56d1f0 Markup rules completely changed
cb9f08a Update installation check.
5a174af Finally ready for release 3.9.0.
0ec3e97 Add missing files to dist.
61d1d14 fix for --xerox bug#328 introduced by weight fix
b6e49fd Revert -r4579 "Revert changes to hfst-proc made in revision 4427. This will (temporarily) fix bug #328."
f0e85c8 Remember to update the man pages, too.
e31c5d9 Ready for release 3.9.0.
64c985c Revert changes to hfst-proc made in revision 4427. This will (temporarily) fix bug #328.
9ca8ab0 Change all pythoncode delimiters from '{' and '}' to '%{' and '%}'.
128331b Add two missing %
7b961c5 Add tests for hfst-substitute.
67f3dce Add tests where one or both inputs are archives.
26aa331 Set name of composition transducer after composition is called.
1e2ed00 Add tests for binary tools that use input streams whose types differ.
4c1e655 Make HfstTransducer::compose throw and exception if transducers have mismatching types.
e21378d Implement conversion of mismatching transducers also in hfst-substitute.
43e107d Make binary command line tools convert mismatching input streams into common format.
573609f Make 'harmonize_flag_diacritics' throw an exception if transducer types differ. Make 'is_safe_conversion' public.
d750847 Fix a bug in hfst-regexp2fst related to comments and remove support for legacy way of defining weights from regexp parser.
ee3b02c Xre parser - added check if contexts in replace rules are automata
d5258af Implement commands 'view net' and 'write dot' as system calls for non-windows platforms.
7e041e8 Update windows scripts so that they will work with the latest changes in dot and pckimmo printing functions.
8cb95f8 Move dot and pckimmo related functions in their own files.
70199c8 Throw an exception in HfstTransducer::eliminate_flag if flag feature does not occur in the transducer or the flag includes a value or operator.
6d723ca Add missing file
aa6325f Move python tests to their own directory.
bf0db23 Rename directory 'swig' to 'old_python' and directory 'new_python_api' to 'python'.
da9b806 Add information about the possible values of option --xerox-composition to the help message.
f068284 Modify infinite_cutoff and time_cutoff arguments in lookup functions.
8645f60 Lexc, removed option -M + cosmetic changes to the alignment
95e18d3 Handle standard and console streams when libhfst is run from IDLE.
1275140 Throw a FunctionNotImplementedException in HfstTransducer::lookup_fd instead of converting HfstTransducer automatically into HfstBasicTransducer.
7d11276 Update python interface.
dc5a1f4 Added option -A to lexc, it aligns same input and output transitions
1d3e8af Fix a minor bug in python interface on windows.
897a7dd Fix LexcCompiler::printConnectedness const declaration.
de59f0e Replace rules support flag diacritics (fixed lexc with -FM flags)
6817b35 Use separate ostringstreams for standard output and standard error on Windows.
f30e6e9 Improvements to error and output stream handling.
1178005 Support printing to console on Windows.
0f0dd44 Update foma back-end composition algorithm to fix a bug noticed in lexc tests.
6e93227 Replace system exit calls with HfstFatalExceptions.
c73aa33 Check for epsilon cycles with negative weights.
4679a3a Optimize function 'has_negative_epsilon_cycles'.
3bd5ce0 Add a test for checking that unknowns are not expanded into flag diacritics.
6d803df Add option 'check-negative-epsilon-cycles' to hfst-txt2fst and tests for it.
a678e81 Add function HfstTransitionGraph::has_negative_epsilon_cycles.
4405b78 Improve performance of transducer harmonization.
12d3d8e Time cutoff for the bundled standalone optimized-lookup tool too
05e7b6b Add parameter to control output in 'compile_lexc_file'.
879d01e Add function unsigned int LexcCompiler::getVerbosity().
acd442e Allow user to control output in xre and xfst compilation.
cf2a1d7 Improve error handling in lexc compiler.
688823f Improve error handling mechanism in xre and xfst parsers.
083f848 Add a missing period to the end of symbol lines when printing in prolog format.
50f7ebf Added warning about very slow lookup when using HfstTransducer::lookup_fd.
0cf9553 Added warning about very slow lookup when using HfstTransducer::lookup_fd.
c8458dc Added possibility to transform individual transition weights in HfstTransitionGraph.
fb50256 Add option to suppress multichar tokenization and default to on in proc2
8e370a6 Replace part of XfstCompiler's FILE pointers with equivalent calls to ostream.
5a773d1 Tentatively move towards o(string)streams instead of FILE pointers when printing output.
b792ca8 Remove debug printing
42585f5 Implement a time-based search cutoff for hfst-lookup in optimized-lookup mode only
2aafec3 Make 'define_function_args' return a bool instead of calling exit on error in xre parser.
18d6b12 Fix error handling in xre and xfst compilers.
d1f2b2a Rewrite regular expression examples to be more informative.
52dc82e Scale weights to be non-negative in function n_best.
21d8577 Update documentation.
5c71e4b Small fixes to read_att_string.
727e74f Update documentation about special symbols. Add a function for reading AT&T strings.
7b2e11d Add function start_xfst() that starts an interactive XFST compiler.
ca3143c Update documentation between HFST and backend conversions and add the examples to tests.
9ae4311 Fix issues related to level of verbosity when compiling xfst and regexps. Add missing documentation.
94e7535 Merging of modified or added functionality from the xre parser
116a076 Remove obsolete comment
9130ba1 Revert HfstXeroxRules files to version 4484.
04d3a64 Rewrite 'compile_lexc_file'.
4988bb6 Improve handling of error and exit conditions in xfst.
4ba4909 Add missing namespace: 'string' -> 'std::string'.
b3c446d Greatly reduce memory leaks when using hfst-ol format
f9b44a4 Update new python API.
3239aa9 Update windows scripts after moving XfstCompiler.
ba65f77 Move XfstCompiler from tools/src/parsers to libhfst/src/parsers.
330447d Add three missing files to dist.
2570d1f Add flushing to allow instrumentation of hfst-lookup
5a1307d Replace rules now should support Flag diacritics.
571878b Fixed lenient composition (it used to be unimplemented).
209e969 Move HfstStrings2FstTokenizer from tools to libhfst directory.
9105ea6 Use the actual transition data object when copying transitions in conversion.
7b06fa5 Fix a warning about comparison of signed and unsigned value.
4d67d87 Throw StateIndexOutOfBounds exception in get_final_weight, if state does not exist.
4c4f56c Add examples used in documentation to the tests.
8a37298 Test examples and fix them, if needed.
12825fe Add missing documentation and functions.
b2aa4b1 Add a function that takes tokenized input and returns a transducer.
b3da97a Define new functions for creating transducers from string, string list and dictionary.
c6ebab5 Fix some issues in the new python interface.
4be6a9d Add more documentation and tests to the new python interface.
3b312a3 Add aliases for different substitution functions.
ca7834b Modify substitution functions to be more pythonic.
21c7d01 Add more stuff to the new python interface.
5c94995 Fix output of hfst-xfst's command 'help'.
fb4d2ef Fix small bugs in hfst-xfst.
21f9fbb Modify also determinization function so that negative weights are handled correctly.
9b42c70 Added more documentation.
833f6c6 Added more documentation.
ceb8de6 fix NUL flush, bug #240
5ed671d Add function __iter__() to libhfst.HfstBasicTransducer.
323778a Modified libhfst files.
6bc7b3f Create directory for new python API.
d929a12 Update Doxygen HFST version number.
39b7bd7 Update README and Doxygen documentation.
a7efde8 Add some tests with negative weights for contain, merge and parallel rule operators.
8bda879 Modify minimization of weighted transducers so that negative weights are handled correctly.
e7f3b91 Always include 'generate-cc-files.sh' script in dist package.
be9d1ad Do not allow omitting the question mark on either or both sides of colon in regular expressions. Xerox has also removed this feature from their regexp syntax. Interpret expressions such as [foo : bar], [foo: bar] and [foo :bar] as [foo:bar]. Expressions of type [foo:], [:bar] and [:] will yield an error.
98c8423 Test if pmatch-tests.sh.* files exist before removing them to avoid 'rm: cannot remove:' messages.
b8b1039 Remove foma license warning. The current license of foma has no compatibility issues.
13553bd Make some changes for the python interface. Move HfstFile wrapper class to the swig interface file. Make HfstStates and HfstTransitions typedefs public. Add a destructor for HfstTransition. Update version number to 3.8.3.
c64206b Handle a frequently encountered exception
c48b828 Reverted changes done for class HfstFile in revision 4431.
a8cf4ea Removed HfstFile wrapper class and functions using it. A better place for them is in the swig directory.
65b418c Added destructor for class HfstException. It needs to be defined for SWIG.
9965217 tests for --weight-classes 1 and ordering by weights in proc
3e74d14 make tests pass again (only analysis order changed here)
a882fef use std::vector as ProcResult to get output sorted by weight-class
a0bcb70 fix --weight-classes, bug #308
bd9ac6b Now the script generate-cc-files.sh creates also lexer files.
c37b24c Add $(EXEEXT) for Windows
2fa67a4 More Foma fixes for cross-building and clang
aa58fe3 Small modifications to foma code: struct and variable had the same name.
f0b6c85 Updated to foma version 0.9.18alpha.
d566985 GNU address removal
638efc5 Fixed some memory leaks in hfst-compare reported in bug #167.
4a7ea0c Adding the renamed files.
28182f1 Trying to rename README files...
cdd1ad3 Updated scripts.
6906e45 Now all files in this directory are listed in README
ab78a6b Added a README file for the scripts directory.
2374665 Fix help message
445eeb1 Fix containment, implement some additional wanted operations
892fdd8 Ready for release 3.8.3.
4eab0f4 Updated windows scripts.
9d0aba0 Now a warning message gets printed if stack contains transducers whose types differ (can happen if lookup-optimize is called before pushing another transducer onto stack).
f7a8ce8 Added a windows README file for the package that contains eight command line tools.
e4977d4 Added a Windows README file for hfst-xfst.
9e2ef69 Added tests for windows. Made small fixes to hfst-proc and hfst-twolc for windows.
3b3a397 Replaced alternative tokens for logical operators with the standard ones.
65aa949 Checking for existence of config.h and getopt.h.
f2f7fc3 Updated scripts.
aa1f4df Added make scripts for twolc.
e33968b Added twolc to windows compilation scripts.
c9c02b9 Updated windows scripts.
91a1945 Fixing again reading and writing to/from console on windows.
3fd62f3 Using hex values for unicode characters when compiling with msvc to get rid of warnings (and possible segfaults).
023e533 Terminator needs reading, and yes there are partial reads
8575c6d std::string all the way
4ba02e9 Added missing HAVE_CONFIG_H ifdef. Added globals-common.h header as some compilers complain about missing extern variables. Now creating variable size array 'next_u8' dynamically with operator 'new'.
0a36470 Renamed help_message.cc into xfst_help_message.cc and added a header file xfst_help_message.h. Now XfstCompiler includes only the header file.
e25e08d Fixed a typo and added explicit braces to an else statement.
0061f6c Now option --pipe-mode takes an optional argument { input, output, both }, 'both' being the default.
62f26f0 Install hfstdll.h
d09ea9c Now hfst-lookup and hfst-optimized-lookup read from console and print to console by default on windows.
09eaafb Commented out one debug print...
cb24cf2 More improvements to utf-8 handling on windows.
e29bdce Small modifications for better utf-8 support on windows.
75a25d4 Small modifications for some command line tools for windows.
b141560 Added tool compilation scripts for windows. Argument handling on windows is also supported for some tools needed for hfst-xfst testing.
8b73295 Fixed a bug in argument handling.
e516f16 Edited help messages and warnings of hfst-xfst and hfst-lexc on windows about character encodings.
9d31b76 Moved getopt implementation to a separate file.
57f9a81 Added an implementation for long options for windows.
bb564a4 Fixed some issues noticed during msvc compilation. Renamed function 'min' to 'min_' to avoid collision with a macro with the same name. Changed arguments of comparison function given to qsort to (const void *, const void *). Added support for commandline arguments of type '--option=argument' to hfst-xfst for windows. Also changed hfst-xfst's option --print-to-console (default false) into --no-console (default false) on windows.
34d1c54 Updated windows script files.
b46445d Deleted getopt_long cc and h files. Processing command line arguments manually in hfst-xfst.cc on windows.
88c44eb Added some ifdef _MSC_VER directives.
eed630e Applied patch for musl libc compilation.
f3ffa44 Fixed some bugs noticed when compiling on windows.
0029101 Blank line between outputs in --finnpos mode
5c3f801 Updated windows scripts. Added ifdefs for msvc compilation.
1734803 Updated swig scripts for windows.
1169774 Add FinnPos mode and reorganise
d9c6038 Updated and added scripts for compiling python bindings on windows.
5a00166 Fix harmonization issue and speed up the common case of only on top-level matcher
4d8a4a6 Documented the tsv file format, added some linebreaks to make the helpt text easier to read.
2871135 Always skip comment lines. Added verbose output for TSV file reweighting.
3d2e4db Fixed bug #293 by initialising the line variable before using it.
bfd227b Whitespace only.
d850180 Added missing file 'hfstdll.h'.
ed20de2 Edited and added scripts for windows compilation.
25974b1 Added HFSTDLL's for some functions for windows compilation.
9fd87e0 Added option --arcs-only to hfst-reweight. Now weights of all arcs and end states are modified unless --arcs-only or --end-states-only is used. Also fixed '--end-state-only' as '--end-states-only'.
e8c1be4 Added some explicit type conversions.
2a77392 Changed the short form of the xerox mode from -x to -X (upper case), to be more consistent with the other options, and to match the test on line 195.
3ba83d9 Now it is checked if foma back-end and zlib are available before hfst-lexc-wrapper is generated. This also affects the way HfstTransducer::read_lexc_ptr is implemented for FOMA_TYPE: native HFST lexc compiler is used if lexc wrapper is not generated.
8557d61 Updated hfst version number in the python setup file. Fixed the argument type of LexcCompiler::setVerbosity from bool to unsigned int.
4a31565 Changed again some 'not' operators into \!.
e1c0779 Added a missing // in the beginning of a comment line noticed during windows compilation.
f4ecd94 Edited windows batch scripts and added a new one for creating python bindings.
590d56c Changed declaration 'class yy_buffer_state' into 'struct yy_buffer_state'.
1f83336 Fixed declared return type from 'void' into 'int' for functions hlexclex_destroy and pmatcherror.
cfe7852 Xerox twolc test support in hfst-pair-test
48dcdd7 Tentatively fixed missing paths problem in compose-intersect
2f75bbc Moved definition of member 'static const HfstState START' of classes ComposeIntersectFst and ComposeIntersectRulePair from header to cc file.
d4bdfb3 Add [].t() syntax for delimiting tagged (and context-checked!) regions
5eb0069 Added new scripts for testing native windows compilation.
982836a Added missing return values for some functions. Replaced ciso646 keywords for logical operators with standard ones.
ad4273d Added make scripts for windows.
e94301e Updated script.
7fccd3a Removed unnecessary HfstUtf8.h and HfstStrings2FstTokenizer.h inclusions from libhfst/src/ files. Also moved HfstStrings2FstTokenizer into hfst namespace.
d726d57 Now handling all stdbool definitions in foma headers.
7137e4d Now HAVE_CONFIG_H is checked before including config.h. Removed inclusion of zlib.h that had no effect.
b1ba306 Changed ifdef WINDOWS into _MSC_VER. Now zlib.h is included only when requested with -DZLIB.
a16942d Using dynamic arrays instead of static ones to keep cl.exe happy. Also added definition for 'ssize_t' on windows.
7203413 Replaced 'and' with '&&'.
beb57e3 Replaced 'and' with '&&'. Also made a variable defining the size of an array const.
9b26e06 Replaced ciso646 aliases and,or,not with &&,||,\!. Also changed sizeof(variable) into sizeof(variable_type) in ol transducer header because cl.exe complained about that for some reason.
237842a Fixed a bug where operator '==' was used instead of '=' when setting the value of last element in a char array.
dc51b2e Added missing files.
83d7d32 Added scripts and headers for windows compilation.
33409ad Changed operators {not,and,or} into {\!,&&,||} because cl.exe complains about them on windows.
d3779cd Added a script for windows compilation
1d327ca Added support for compiling openfst and foma back-ends with cl.exe on windows.
27b6994 Include zero in signed and unsigned integers
06517c7 Correction to handling trailing nonmatching material in locate_mode
f03b162 Forgot to commit these files too...
da83b10 Added xfsm support for some command line tools.
3097ddf Now HfstTransducer functions throw a FunctionNotImplementedExeption if xfsm implementation is not available.
4008054 Added more xfsm implementations.
818ad51 Added more implementations for XFSM_TYPE for HfstTransducer functions.
664573a Add ignoring
7b7d13f Correct parsing order of PARALLEL_RULES
40066a1 Added support for composition for xfsm transducers.
3c11736 Added a variable to control if (xfsm) transducers that are already minimal are still minimized for profiling purposes.
877c46f Fixed some bugs in escaping special symbols in prolog format. Added support for xfsm format in some command line tools.
3cf0029 Fixed an error noticed when -Werror=format-security was enabled during compilation.
d0ed632 Now xfsm conversion works in hfst-fst2fst and between prolog and binary format in hfst-fst2txt and hfst-txt2fst.
f688567 Added XFSM initializer class.
c803964 Added function minimize to xfsm interface
4c4ca0b Now hfst-compose-intersect does NOT harmonize transducers by default, as harmonizing produces unexpected results with hyperminimized transducers. Option --harmonize turns on harmonization. This is a partial fix to bug #288.
2e99ef0 Added a cross-build Windows patch given in bug report #289.
4cd63b0 Added a missing #if HAVE_XFSM statement.
e000b0c Fixed some bugs in xfsm stream handling.
a478fb2 Added more functions for writing and reading xfsm transducers.
fcc4c5d Tentatively added input and output stream functions for xfsm transducer.
091ca1d Modified and added scripts for creating static binaries.
8d6dd33 Fast compose intersect available using option -f
1915dd1 Reorganized xfsm interface.
ed7f448 Hopefully speed up compose_intersect by doing l .o. (l.2 .o. r) instead of l .o. r directly.
4b965b7 Updated script.
cc3632a Disable hfst-xfst's auto-complet with rl_insert instead of rl_abort, which is not found on Mac.
c41e726 Added a script to generate static binaries.
3a95772 Now xfsm conversion swaps the state numbers to make a more readable state numbering.
b2348d7 Now special symbols should also work in xfsm conversion.
b3af023 Xfsm conversion functions almost work, escaped symbols still need some attention.
b15f184 Conversion functions between xfsm and HfstBasicTransducer formats implemented, alphabets and special symbols still need some work.
5f79106 Autoindent entire file after removing tab characters to appease commit script
68ffd8e Handle single-char ascii symbols shadowing multichar ones
b1bdd27 Modified hfst to support xfsm format.
05da3ff Fix cg quote interpolation
38273d0 Clarify help string
b3450b4 Add segmenting and cg mode; improve handling of special cases and weights
5766bbf Tentatively modified hfst to handle xfsm implementation type.
db747e7 Preserve nonmatching sequances in locate_mode and parse the appropriately downstream
b7156e2 Tentatively added a skeleton implementation for xfsm library that can be included with configure option --with-xfsm (default is no). No linking to the library is performed, this is just a test version.
7427081 Added a preprocessing directive that defines hfst-twols's Alphabet as TwolCAlphabet if HAVE_XFSM is on. This will prevent variable collisions.
f579458 Changed hfst-txt2fst's read_prolog into read_prolog_format to avoid conflict with xfsm library function.
424e9f2 Named tokens 'UPPER' and 'LOWER' in pmatch and xre parsers into 'XRE_UPPER', 'XRE_LOWER', 'PMATCH_UPPER' and 'PMATCH_LOWER' because the xfsm library (that might be added to hfst at some point) has an enumerator with the same name in C namespace.
cc5e2fb Named foma's write_prolog into foma_write_prolog because the xfsm library (that might be added to hfst at some point) has a function with the same name in C namespace.
b7b9a5b Took away unused token 'LOWER' from xfst parser.
7701e12 Added an option --do-not-harmonize to hfst-compose-intersect.
ae06cea Now hfst-compose-intersect harmonizes the rule transducers with the lexicon.
3bfd0e8 Added option --beam to hfst-strings2fst.
b96dd3b More accurately name set_locate_mode as set_extract_tags mode (locate vs. match is determined by function call)
47ef7f0 Fix bug in compiling UNKNOWNs on the left side of pair separators
b97a28c Fix bug affecting locate-mode where ?-matches were being seen as IDENTITIES in input.
ec66bdd Added option --beam also to hfst-lookup. It is mostly untested for this tool.
0057847 Tentatively added option --beam to hfst-optimized-lookup.
e236dd2 Fixed a typo in warning message about missing symbols in input tapes of rule transducers ('output' changed to 'input').
43d9948 In locate mode, zero-length hits could cause infinite loops. Commit fixes it.
f5c1467 Now hfst-lookup checks it it is possible for an input to go through a transducer before calling is_infinitely_ambiguous. Should fix bug #278.
b4e4655 Now reserved symbols are detected in composition if xerox-composition is ON. This will prevent symbol collisions by throwing an error message, giving at least a temporary solution to issues with reserved symbols.
d2ef472 Fixed substitution operator in regexp, now flag diacritics are allowed both as substituting and subtituted symbols. Should fix bug #284.
92d2bc0 Added better support for special symbols in hfst-xfst's substitute command.
7cc650e Added two tests for hfst-xfst's substitute command. They are skipped until substitute can also handle them.
1ff2f40 Fixed a bug in composition when xerox-composition is ON. Flag diacritics @SOMEFLAG@ were earlier transformed into $ and back into @SOMEFLAG@, but symbols of form $...$ are already reserved for lexc. Now diacritics are escaped as %SOMEFLAG% during composition. This creates a new set of reserved symbols which maybe needs more consideration though... Also added debug prints for LexcCompiler.
ef9407e Removed xml checks from configure, since they cause unlinked xml references in hfst-edit-metadata on some platforms.
791aeb4 Fixed flag diacritic recognition, now also flags of form e.g. '@D.FOO@' are accepted and an empty string is returned as their value. Also added full support for flag handling in hfst-xfst.
94b723e Added some conversion functions between StringPairVectors and StringVectors. Added checks for flag diacritics in hfst-xfst non-optimized lookup.
6df280c A small fix to hfst-summarize option handling.
479630d Ready for release 3.8.2.
d5c8545 Added option --print-symbol-pair-statistics(=N) to hfst-summarize.
adff4fc Small fixes to compile-replace. Also restarting the char counter every time a new xre parsing is started. Removed the xml2 dependecy in configure.
b8fd981 Omit unnecessary step in stringification
adb2d91 Another slight speed improvement
b8564f4 Some double free insurance related to previous commit
786a502 Some more pmatch runtime speedup (around 5-10% in most cases)
6c383b4 Remove commented-out lines (same ones I was intending in the previous commit)
8c8be0b Remove extraenous index table fitting test
c150b35 Further improvements to conversion to optimized lookup format
ebb5d63 Large speedup in conversion to optimized-lookup format
5eaf679 Added test cases for compile-replace.
80059f5 Added test cases for merge operation.
e8c709f Fixed an error in hfst-fst2strings --print-separator where two consecutive lines of -- were printed between non-empty transducers in some cases.
9a30760 Now minimizing the merging automaton before merge operation so that epsilons do not cut a succesfull merge path. Also allowing epsilon-to-regexp-marker transitions in the merge filter.
5d89aba Improvements to compile-replace function, now it should work for input and output sides of a transducer.
372d184 Revert bungled change to precedence order
06fa073 Now xre compiler of function merge does not increment the char counter, making it possible to have many merge operators inside one regex.
bc32286 Added a constructor XreCompiler(XreConstructorArguments & args) to facilitate passing xre variables to merge function which needs them in its internal xre compiler.
de47594 Now using internal starptr variables in functions hfst::xre::compile and hfst::xre::compile_first instead of global hfst::xre::startptr. This should fix the strange memory errors which occurred when calling merge operation inside a regular expression.
e7c15a3 Now merge operation filters out non-optimal paths.
e153281 Tentatively added a function 'tokenize_and_align_flag_diacritics' to HfstTokenizer.
fe5cc6d Runtime speed improvements Prereserve table vectors, eliminate special_symbols map
a62fe68 Profiling support with Counter() and --profile & a bunch of smaller changes
09703ef Allowing 1-to-n composition of automata in archives. Fixes bug (or feature request) #277.
035edcd Now passing verbosity to LexcCompiler as an unsigned integer via setVerbosity(uint). Also made small fixes to warning prints in lexc compiler.
5fb6ba7 Added tests for one-sided flag diacritics for hfst-lexc.
602ec80 Now lexc compiler warns about one-sided flag diacritics in verbose mode.
fa95239 Added a tokenizing function that warns about symbol pairs, if needed.
4ad8eb4 Added test case for previous lexc commit (sublexicon defined more than once treated as an error).
4582622 Now multiple definitions of the same lexicon in lexc are treated as an error unless LexcCompiler::setAllowMultipleLexiconDefinitions(true) is called first.
95f58f4 Now lexc parser updates the error status hlexcnerrs when hlexcerror is called. In case of warnings, the error status is nor updated.
946bd68 Speed up list arc processing by replacing some maps and sets with vectors
373721d Changed list to set in merge operation. Removed commented code.
52fe564 Added option --encode-weights to hfst-lexc.
038ebd2 Reinstate undefined symbols as valid tokens as per documentation
212f177 Add defined lists
25e7a02 Add Sigma()
491d771 Added Lst() and support for list arcs in runtime
fe95b0a Sync precedence rules
482bad0 Switch order of precedence of concatenation and other binary operations
93d2887 Added Lit()
a5114b5 Modified the function merge, it now takes as an argument a map of list symbols.
7ef0509 Fixed a bug in xre parser, now definitions and unknowns can be used together in expressions such as 'regex [def:?] ;'.
ed742e9 Now the epsilon symbol is not added as a multichar symbol to hfst-lookup tokenizer if it is the empty string. Should fix bug #275.
0fb0fdb Allow \U00NNNNNN syntax for code points in utf-8 but above U+FFFF
2db8062 Allow \UNNNN as well as \uNNNN
4954a40 Allow \uNNNN in range notation and make some fixes to utf-8 handling
7784091 Added utf-8 character range expressions
f2674b2 An untested implementation of the merge operation added to hfst-xfst.
565d9bb Syntax-level completion of functions (arg placement still not completely free)
9bb6843 Fixed a small bug in hfst tool tester.
83312f7 Ready for release 3.8.1.
bcae2c3 Fixed std::cout into &std::cout in stream pointer comparison.
246258f Now using definitions USE_TR1_UNORDERED_(MAP|SET) when defining what unordered maps and sets to use.
c7c5ee8 Fixed a typo tr2 -> tr1.
97a3e59 Unordered maps and sets are used from std namespace if -std=gnu++11 is requested.
a5a62cb Forgot to update version number in swig bindings.
a578323 Various improvements and additions to function syntax, particularly empty args and string-args
5fb9c8d Added functions for merge operation in HfstTransitionGraph.
1df7d58 Modified weight handling in HfstTransitionGraph::intersect.
1e5bbfc When the alphabet is constructed from a symbol table, set identity to NO_SYM this was supposed to always happen anyway but didn't matter until recently
417bc81 Modified intersection algorithms in HfstTransitionGraph.
67f0a92 Added functions to be used in xerox's merge operation.
05651a4 Small fixes to list definitions in hfst-xfst.
2e1b79d Added an implementation for compile-replace in hfst-xfst, it still needs lot of testing.
244d6bf Yet some more functions added to compile-replace.
e18a645 Added more functions for compile-replace.
39800a1 Tentatively added functions in HfstTransitionGraph to be used in compile-replace.
7b5981f Added function is_well_formed_for_compile_replace to be used in compile-replace command.
c099596 Forgot to comment out debugging prints in tests.
f695bb4 Added brackets around member calls 'Interval.end' and 'Interval.begin' to avoid them getting confused with std::end() and std::begin() templates in C++11.
b0e0fe5 Rolled back earlier revision in interval-set.h
d223f72 Made small modifications for better c++11/c++0x support.
51ff0da Fix bug where identity wasn't being set to NO_SYMBOL when absent from alphabet
c2ee587 Refuse to enter flag loops more than once fixes bug #250 bug - arguably this could be applied to epsilon loops too, pending discussion
15f25c6 Fixed a typo in Makefile.
1faecef Now hfst-xfst gives a warning (or exits) if a binary command tries to access a stack with less than 2 transducers.
a7fe5d6 Now hfst-xfst exits if a command tries to access an empty stack if quit-on-fail is ON and hfst-xfst is not in interactive mode.
25255de Comment explaining usage of hfst-fst2tesseract.xfst.
ed8f040 Added script for converting morphological analyzers to Tesseract word models.
61b43f5 Improvements to loop finding
4d3bab0 Further corrections to loop detection slowdown back to ~10x but may be improved from here
38bb11f Fix some cases of overdetecting infinite ambiguity, there's still some left
f99de3f Forgot to keep adding the repeated states in the loop detection phase
f327e83 Only try to catch infinite ambiguity at epsilon arcs This is the big speed win and presumably correct.
aa0928d This order of comparison is a bit faster since sizes never differ
90b4387 Speed up is_lookup_infinitely_ambiguous() somewhat
3edea32 Try to avoid using negative indexes for arrays
1eedad1 Optimised lookup tests
a1eb3e1 Runtime handling of identity and unknown
dd8a983 Forgot to remove one thing in the last commit
e1db16f Take out our own harmonization hacks now that they're unneeded
6b0a888 Don't use delimiters when they're not necessary, also don't insert everything to RTNs anymore and provide the is_special() function the previous commit required
e6a35c3 Treat special pmatch symbols like flag diacritics for harmonization, also after harmonization add all symbols, including flags, to the alphabets
40feabb Fix bug where delimiters were shadowing the named transducers' names also remove extraenous parsing path
7432ed4 Updated pmatch functionality tests.
6a89ad8 Add string literal syntax for standalone %-escaped chars (they used to be considered symbols which now have to be defined or cause an error)
18b6039 Use minimization guards to keep multiple negative contexts separate in disjunctions
817ec29 Fixed some more spelling errors noticed by lintian.
dcb9e44 Fixed spelling errors found by lintian.
727436d Added again hfst-train-tagger man page which is no more a symlink.
ee79595 Added missing man pages.
3e650c8 Now hfst-reweight-tagger --help returns EXIT_SUCCESS before trying to access uninitialized values.
9665cdc Warn about shadowing definitions
8608975 Alternate syntaxes regex for Define TOP and .#. for #
1874455 Revamp LABEL parsing and introduce curly literal pairs
7b2c44e Updated and added man pages.
cbb4a40 Moved option checking after possible returning from program so that option --help will not generate error messages.
7f69948 Require backslash character to be escaped as \\ in curly literals
832d7b3 Fix bug in unescaping function
6918fd0 Ready for release 3.8.0.
9336e7a Added variable 'lexc-rename-flags' to hfst-xfst.
73c2a33 Desperately fiddle with the way the minus operation expands things
ee6d6fa Add lambda-like anonymous definitions for controlling subexpression boundaries
9ef3676 Changed the flag handling behavior of hfst-xfst and hfst-lexc. Now both tools by default use Xerox's way when composing, i.e. flag diacritics match unknown and identity symbols. This can be controlled with variable 'xerox-composition' (the default is ON) in hfst-xfst and with option '--xerox-composition={ON,OFF}' (the default is also ON) in hfst-lexc. hfst-regexp2fst also has the option '--xerox-composition' which by default is OFF, as it was earlier.
6a57210 Fixed a typo in hfst-regexp2fst option handling.
a64b247 Now one-sided flag diacritics are allowed in composition when flag-is-epsilon is used.
1dfbb23 Added option --log10 for 10-based logarithmic weights in hfst-strings2fst.
0bcf18f Now an error is thrown if flags are not twosided in composition when xerox composition is used.
a959c4b Added exception class FlagDiacriticsAreNotIdentitesException.
6d8c1ab Add pmatch functionality test suite
d5d441e Added option -X flag-is-epsilon to hfst-regexp2fst.
33b17d7 Fix another symbol-leaking issue
91d5ce0 Allow nested logical operations on contexts
d4b01d2 We need to avoid symbol pollution for more than just special symbols (this doesn't completely resolve pollution issues, just some urgent ones)
89149b0 Don't forget to pop the rtn stack when there's nothing matched
6a3552f Minimize after adding delimiters, not before
fdf46b1 Revert bracket-bounding behaviour
fecbf54 Use brackets for extra delimiters to control tag and context boundaries more
51382e2 add_delimiters was happening in the wrong place since recent syntax changes
8467812 Fixed a too strict assertion.
7ac7e49 Added a return value for a case that should never happen to make scan-build happy.
09b1260 Reverted back to the buggy behaviour of function 'getMarkerNumber', since some HfstXeroxRules tests will fail if it works correctly...
c7d12d4 Fixed an error in function 'getMarkerNumber' where istringstream was not properly initialized and returned random values. Also added a print method for class Rule.
7b7f0fb Fixed a typo in filename in EXTRA_DIST.
caad8cb Now alphabets are copied when encoding and decoding flags in composition. Fixes bug #267.
ad47d32 Oops, forgot about restoring scope state afer entry arcs
f325949 Add a scope facility for local context boundaries
591c5b5 Fixed most scan-build issues other than dead store, memory leak and errors from foma back-end.
d207568 Added implementation for twosided flag-diacritics in hfst-xfst.
d3c0506 Now harmonize-flags and flag-is-epsilon are by default OFF in hfst-xfst. Also added a new, mostly untested variable xerox-composition that matches flags with unknowns and identities in composition (default is OFF). All flag-is-epsilon functionalitites are moved under HfstTransducer.
ce348d2 Purge commented-out lines
3752461 Major syntactic changes and additions (AND & OR) to bring in line with Karttunen's documentation - amazingly doesn't break backwards compatibility (brobably / mostly)! Function syntax still lags and is incomplete.
79f7b42 Added missing return value to a function in openfst back-end.
636451e Reverted back to version 3992.
e0442ee Instead of NULL, return a new HfstTransducer in a condition that should never occur. This is for some compilers.
0d7cdee Fixed errors reported in bug #265. Also tentatively added a switch --xfst-harmonization to hfst-xfst that treats flags as ordinary symbols in composition.
08462a7 Resolved another rpmlint issue, hopefully the right way (A HfstTransitionGraph method returning nothing when it seemed to be meant to be returning *this)
68ed253 Resolved some issues and nonissues revealeled by rpmlint
0e2f676 Now xre parser compiles expressions of type foo:bar^{0,N} correctly.
be1e33d A better way see if we have a better location than before
ba31a12 It seems to be generally faster to defer minimization at [] boundaries
17b833b Made 'harmonize-flags == ON' the default for hfst-xfst. Also hfst-xfst's xre parser now harmonizes flags according to variable 'harmonize-flags'.
90cff75 Fixed some issues noticed on c+11 and reported in bug #258: a space between literal and identifier, missing cstdlib header and ostringstream conversions.
9da1a0f hfst-fst2strings now gives an error message if option --nbest or --random is used with transducers in optimized lookup format.
ddea524 Apparently older gcc's libstdc++ require operator< to be const
f6df4d2 Added locate mode on the library side; support for multimatch, weight access and various internal changes
084d1a6 Now all command line tools should give an error message and exit with >0 if they cannot process input in hfst optimized lookup format.
1959537 Now hfst-invert prints an error message if given a transducer in optimized lookup format. This should be fixed in all command line tools.
5a5a3e4 Function set_expand_definitions added to the python interface.
d89d63b Updated HFST version number.
26bad71 Updated HFST version number.
78a7e30 Added option --renameFlags to hfst-lexc for testing purposes.
9e7a3ff Added option --encode-weights to hfst-determinize.
c9ae91f Now hfst-lexc has options --withFlags and --minimizeFlags and hfst-xfst variables lexc-with-flags and lexc-minimize-flags to control if hyperminimization is used when parsing lexc files. Flag minimization can be the default behaviour when it has been properly tested.
57cc2df The Xerox output should have an empty line between each cohort.
01928da Now flag minimization in should work in LexcCompiler. Commented the changes out until all expected results from lexc tests are changed accordingly.
38b3724 Apparently failing to read a stream now throws HfstException so catch that instead
c29a94e Tentatively added a piece of code that filters out multiple flags in lexc result. Currently commented out.
8b0216c A hopefully temporary tool called proc2 for simple tokenization, to become part of something else or be renamed in the future
ae42592 Some fixes to python wrapping of pmatch and showing original input in pmatch
cee590b Make locate() return a simple data structure rather than a string
3ad4e8e Convenience loader function for pmatch
0cc41ab Fix outdated prototype of printConnectedness() and add pmatch prototypes
125ed8a Added option --Werror to hfst-lexc that treats warnings as errors.
2e28445 Added tests for the fixed bug #243 that will pass now.
24e7b46 Now keyword LEXICON is allowed without a preceding newline in lexc parser. Should fix bug #243.
a50dd1a Added a variable encode_epsilons to function priority_union whose default value (true) should fix bug #254. Function lenient composition still calls priority_union with a false value of encode_epsilons.
94dc75b Added support for cross-products such as {foo}:[bar] and foo:[bar] in regexp parser. Tests still needed.
d42799e .NOTPARALLEL for now
85a69f9 Fix serious bug with epsilons in contexts - they were incorrectly moving the input tape.
b04de37 with pipes
941baa9 A super cool progress bar functionality \o/
30cf7bd Fix context-related bug
13c1904 Readability refactoring
7f22c97 Comment fixes and some more deadwood elimination
cf24a3b Remove more unneeded stuff
8d58ce1 Remove some obsolete & commented out bits
dd779dd Locatefy option
93d00f8 Don't make noise about almost always harmless result truncation
9abaa72 Large refactor of almost everything (mainly to better serve tokenization apps w/tape synchronisation)
8f8d846 Undo previous commit, was confused by polluted automake cache
df23ced It seems that newer toolchains don't like c++ classes being defined over multiple .cc files, or something..
1958ec8 Don't count flag diacritics when locatefying
516bf5c Remove obsolete bogus -> rule from when @-> had problems
e5d943a Handle missing @bin files more nicely
04cf7dd Autoconvert @bin arguments if necessary
9ce5490 Remove obsolete dependency readme that hogs the svn trunk page
c427c3d Add constructor option to make diacritic strings blank or not; make pmatch consider them blank
2c2e664 Added file 'test' to EXTRA_DIST..
b1f01a2 Fixed hfst-twolc test Makefile.am
211935e Disabled outdated unit tests.
6b74d9f Tentatively implemented variable 'flag-is-epsilon' in hfst-xfst.
531acff Fixed issue with left arrow conflicts for contexts with impossible word boundaries.
2881c9e Also added tests for regexps with different continuation lexicons in hfst-lexc.
489474a Now regexps with different continuation lexicons are allowed in hfst-lexc. Should fix bug #247
21db219 Fixed list center rules and conflict resolutions.
6942188 Tests for list centers.
cd1193b Tests for list centers.
6e909fd Exit on parse error Once we've printed a parsing error, exit rather that print intimidating hfst exception messages
efdd01b Pattern-locating mode
9061298 Some changes to approach to avoiding stack overflow should fix problems with very long input lines
2cbb071 Add harmonization
1c1007c Add extract tags -option (this commit also includes a secret commented-out facility for profiling activity on the input tape by drawing a funky ascii histogram)
79c629f Fix incorrect check for whether tokenization needs to respect a long symbol
7633686 Allow epsilon-symbol pairs
01eb211 Don't print huge data dumps when parsing fails
2273237 Revert back to the original TropicalWeightTransducer.cc and forget splitting it..
d7755c3 Moved minimize and push functionalities from TropicalWeightTransducer.cc to separate files. This will make compiling TropicalWeightTransducer faster and prevent 'File too big' errors on Windows.
1c74930 While scanning for possible first symbols, don't accidentally go beyond the end of the input tape
c50a51f Ready for release 3.7.1.
f17327e Now removing CR characters from output of hfst-xfst so that tests will pass on windows too.
f374022 Now an error message is thrown in HFST when reading a native SFST transducer that uses the empty symbol.
6596539 Added option --encode-weights (-E) to hfst-minimize.
e7d0c0e Added documentation to parallel replace and removed unnecessary marker insertions.
1d1251a Modified conpilation of weighted parallel rules by inserting markers to keep same mappings with different weights separate. Also added a StringPair variant of insert_symbols_to_alphabet to HfstTransducer and HfstTransitionGraph. Added a test case for hfst-xfst. Updated NSIS installer.
60b45cb Tentatively added a function for substituting weights with markers.
fd29751 Remove spurious obsolete comment
2925fb7 Bounds check accesses to ascii_symbols when reading input too (ascii_symbols used to have a NO_SYMBOL entry for any char, but now without a check some accesses were resulting in bogus keys)
d23d42f Add -D_XOPEN_SOURCE=500 in order to prevent -std=c99 from preventing strdup() prototype being visible
9f7311e Correct bounds check for ascii_symbols alterations
988e259 Now checking that ascii_symbols vector index is not out of bounds when removing shadowing symbols. Should fix bug #235.
7b89f3f Replace rules bug fix
6bbf8bd Parallel rules bug solved
4ad528d Ready for release 3.7.0.
6fd6272 Added missing files to Makefile.
180ebd2 Added option --encode-weights also to hfst-regexp2fst.
04837d3 Added option --encode-weights (default false) to hfst-compose-intersect.
ec55eab replace rules - weights are removed from calculating contexts (they were unnecessary there)
34fcf5f Removed commented code.
d469a2b Modified TropicalWeightTransducer::are_equivalent so that determinization is carried out in a way similar to 'intersect' and 'subtract'.
aee1515 Updated configure.ac
b4fbd3f Reverted hfst-ospell changes.
edb752e Tentatively added hfst-ospell under tools/src, it is only enabled if --enable-ospell is requested.
911b8a0 Use precision 1 in weighted tests.
e45e2fc Reinstate special ascii tokenization while avoiding shadowing longer symbols
5ad406c Disable ascii-lookup when tokenizing
e45fc59 Set encode_weights off in tests to make sure that they work also if it is on.
6f1b3de Fixed a bug in parsing regexps such as '?:{foo}'.
95fcfa3 Make pair notations a bit more consistent; enable RIGHT_ARROW
8b3f312 Now incompatible replace types in parallel rules yield an error.
65884a7 Added variable 'att-epsilon' to hfst-xfst. Also added more weighted parallel replace tests.
4f93231 Now epsilons, identities and unknowns are printed correctly.
4e6f664 Fixed a bug in handling unknowns in hfst-xfst's 'look up'.
b3ac973 Added more weighted parallel replace tests.
2ca06d9 Allow runtime contexts anywhere to facilitate some efficiency strategies
b628ccf Added rest of the replace tests.
d28fef5 More tests...
9cd39cb Added more hfst-xfst replace rule tests.
77d98df Fixed weights in markup rules
f62a2ce Added more tests.
42c8ecb Added more replace tests.
9ad639d Added replace tests, not all of them are yet performed.
095ed97 Add tentatively new replace tests.
8fb7cc9 Added tests for containment operators.
e49b202 A couple of small fixes to containment operator.
06ee31a Sorting arcs before subtraction in TropicalWeightTransducer::subtract.
e2e71f5 Removed unnecessary determinization from TropicalWeightTransducer's intersection function. Added function HfstTransitionGraph::get_transition_pairs. Made some small modifications to xre parsers's containment operator.
e46dcf9 Fix definition of curly literals
0fa1896 More PAIR_SEPARATOR fluff (which should be completely refactored)
8cb51e1 Build input scanner without using a stack guard; also refactor where runtime behaviour is stored
8fb36bb Added tentatively new functions for implementing containment in xre parser. Removed determinization from intersection.
0721534 Changed containment operators so that they also work with transducers that are not automata, i.e. '$.[a:b]'.
c08ec08 Added documentation to TropicalWeightTransducer::intersect.
8455c8a A simple stack guard for first-input -collection (only, for now)
c0f23cf Now all containment operators work, at least with unweighted containment transducers.
562e77f Fixed a bug in '$.' operator in xre parser.
fcb43f5 Use a vector in the prescanner to make it slightly faster
7790325 Add class-identifier to friend declarations; required by some compilers
8623961 Remove extraneous debugging message
be473f1 A very conservative and (for now) computed-at-runtime input prescanner
fb8361c Added a test for empty context in replace rule.
413741c Resorting to global variable 'encode_weights' instead of passing it around as a variable. Allowing empty contexts in replace rules, i.e. 'a -> b || _ '. Fixed a minor printing issue in hfst-xfst.
9c6f3ab It's perhaps a little clearer if is_like_epsilon(epsilon) is false
3c4f82e Fix bug where conversion from ol to basic transducers was missing @I.foo@ arcs in some situations
834a129 Fix bug where weights on the insertion arc were being ignored
0e7e0a6 Log insertions when verbose
3cd1281 Added variables 'max-weight' and 'encode-labels' to hfst-xfst.
981f3cb Tentatively added functions to control label and weight encoding in weighted minimization. Also added max_weight parameter to lookup_fd of HfstBasicTransducer.
c265345 Fixed weighting in containment operators.
ae39ecb Edited xfst tests.
9765b5b Fixed a bug in parsing multiple parallel replace rules.
fd0614c Fixed a bug in the contains one '$.' operator. Improvements to the xfst parser.
700c746 Updated regexp2fst help message.
53585c5 Use "" for special parsing and symbols, {} for tokenizable strings; also allow "define" alongside "Define" Future note: consider adding another syntax for "totally bare" symbols with no special parsing
56d86bc Only print prompt in pipe mode if --verbose is used.
5440fa9 Added yet more tests for weighted rules.
f0abf04 Made xre parser ignore weights in rule contexts. A warning is given in non-silent mode.
b481854 Added more tests for weighted replace rules.
49fd9ba Added optional variants of replace right tests.
17712e1 Forgot to apply weighted_rules -> weighted_replace_right renaming scheme to test.sh.
a0566e4 Renamed weighted_rules_* to weighted_replace_right_*.
54e62ac Addded more weighted rules tests.
92b67e7 Added function HfstTransducer::is_infinitely_ambiguous() and command 'test infinitely-ambiguous' in hfst-xfst.
6633e61 Copy the before-seen states so different paths to the same state don't conflict Reinstates a skipped test
72955a8 Made hfst-summarize print whether transducer is infinitely ambiguous in verbose mode.
946d841 Fixed a couple of typos causing errors when calling is_infinitely_ambiguous..
ec80b66 Tentatively added function is_infinitely_ambiguous() to HfstTransitionGraph. Added variables 'lookup-cutoff', 'print-words-cutoff' and 'precision' to hfst-xfst. Added test cases for weighted rules. test_transducer_functions now returns a skip value until hfst optimized lookup transducer's function is_infinitely_ambiguous is fixed.
9cf83ec A slight followup to the previous commit
b26cd8f Fix stupid bug in tricky control flow part of loop checking
2b0e3b5 Bugfix: distinguish between transitions and transition indexes when determining state equivalence when looking for epsilon loops for a given input.
674b883 Remove extraneous debugging message
9dba5e0 Fix bug where infinite ambiguity checking was polluting the flag diacritic state
daad246 Reorder things so that we should usually run into max_results before running into the stack depth
6bafb97 Respect result limits and implement a recursion depth guard so instead of blowing out the stack we discard overlong results and give a warning message
1431848 Rip out empty main test function so there's only one in the ol compilation unit
bec743d Add string version of is_lookup_infinitely_ambiguous() to HfstTransducer api too
4845339 Added variant is_lookup_infinitely_ambiguous(const std::string &) in HfstTransitionGraph which is used in hfst-xfst to check infinitely ambiguous cases in lookup. Also added a variable precision in hfst-xfst to control the number of decimals shown.
391c4c0 Add a facility for determining whether a given input produces an infinitely ambiguous lookup
7a6fa36 Replace rules - weight now works on both sides, with context.
e6b4af8 Insert special symbols into ?-containing transducers' alphabets right away, fixing some incorrect harmonization behaviour
42dd77c Updated weight management in replace rules
e903072 Finally ready for release 3.6.1.
9db3ddd Better conflict warnings
27a6627 Now it is possible to save and load a stack of transducers in hfst optimized lookup format.
ae84501 Give a more informative error message if tranducer is not in hfst optimized lookup format in hfst-proc.
eb61bef Added hand-written man page for hfst-foma.
49e1c5f Updated man pages.
415d8f0 Now removing symbolic links to command line tools when performing 'make uninstall'.
eda041b A new candidate for release 3.6.1.
6c6081a Updated installation check.
e515805 Edited help messages of lexc and lookup tools.
f2398e2 Made creating and checking flag results less verbose.
dbb84a0 Changed 'return' to 'exit' in shell script.
9dc775e Updated version numbers in NSIS and swig files. Also made lexc wrapper disabled by default.
d76dfcd Fixed a typo.
d2efde2 Corrected syntax error, typos.
2f1296e Ready for release 3.6.1.
672a2b4 Edited help message of hfst-lookup.
86ca382 hfst-lookup now also handles a transducer archive.
0a73c82 Added more options to print (random-)(upper-|lower-)words commands in hfst-xfst.
9ae6bfb Fixed a bug in HfstTransitionGraph::remove_transition and added a test for it.
1f12493 Fixed some reserved words in lexc parser. Added weights for command 'look up' in hfst-xfst.
d98f5c2 Scaled down weights in replace rules.
1d953fc Lex the left paren in function calls along with the function name to make a distinction between a symbol followed by an optionalized transducer and a function call
c17b5be Added a lexc test that contains reserved words in lowercase. The test does not compile in the same way as in foma.
247d365 Lexc weights fixed
8e97a2f small lexc bugfix, tests enabled
e4eb4c0 Allow underscore in symbols (couldn't discover a malignant side effect)
2d0993d A lot more expressiveness involving DefFun()
b251769 Fixed #227 hfst-lexc and multichars with zeros
54bb1e4 Made hfst-xfst use foma's lexc. hfst-lexc-wrapper is enabled by default.
06c35e9 Allow tsv input for reweights
c2006bd Forgot to update -version-info in Makefile.am.
cc52d4b Ready for release 3.6.0.
c4bab38 Edited morphology tests so that they agree with changes in finnish-analyze.sh.
cfbc028 Now checking in configure if tr1::unordered:map/set or std::unordered_map/set is available.
a04fcb3 Replace rule longest match bug fix
5c6d4ae Now hfst-xfst gives better error messages if a file cannot be opened. Removed h-files from SOURCES in tagger Makefile because they confuse some compilers on Mac.
792a4af A preliminary version of DefFun(args...  and Map(fun, @txt...)
6ed03ab Now apply up/down are done using HfstBasicTransducer if the transducer is not in optimized lookup format, which is relatively fast.
9b8c298 Removed unnecessary comments.
b610b50 Added prompt '>' to hfst-lookup, it can be disabled with option --pipe-mode. Added command 'convert net' to hfst-xfst to convert between optimized lookup and standard formats. Now also supporting 'apply up' for optimized lookup transducers.
0f64569 Removed redundant declaration of SfstTransducer::harmonize that caused an error on Mac OS.
501ef0f Added command 'load FILENAME'. Fixes bug #225. Also allowing a ';' after 'define VARNAME'.
d8bb71d Now command 'define foo' is recognized. Fixes bug #224.
dbcdbe9 Set permissions of flag check scripts. Fixed some typos in other tests and in flag diacritic handling (== instead of = in if condition).
52eba53 Added hfst-xfst to the environment, help and version checks.
8d65450 Made minor fixes to tests so that they work correctly on windows and mac too.
0b5d709 Now HfstInputStream constructor throws an exception if file cannot be opened. This exception is also handled in the regexp parser.
980ed3e Added experimental options --enable-no-tools and --with-openfst-log to configure.
15c1ea2 Removed unnecessary warnings from hfst-lexc.
f327366 Fixed a typo in configure file, extension should be 1, not 0.
90a6953 Now creating a variable length array dynamically to avoid compilation errors.
cb847d7 Renamed hfst-lexc2fst to hfst-lexc and hfst-lexc to hfst-lexc-wrapper.
43f5303 Now att format is printed directly to the console in windows environment.
11b0dcb Updated reading and printing functionalities based on tests in MinGW environment.
a57a9b4 A more reasonable parse order for weights and special functions; fixes various bugs concerning weight placement that should be legal
a0ee0c2 Added missing <stack> include to HfstTransitionGrap.h.
f8fd3d1 new lexc2fst, with tests
29d8cbe Fixed a bug in harmonization, now flag diacritics are not harmonized.
9f5f620 Transition to weighted, prioritizing pmatch; recompilation of rulesets required at least for now
b62bfd3 Removed a line from HfstTransitionGraph::substitute that printed debugging info to stderr..
36699e0 Added function HfstTransitionGraph & HfstTransitionGraph::(std::map<HfstSymbol, HfstTransitionGraph> &, bool harmonize)
08d4506 Now standard streams are defined as binary in command line tools when compiling on windows.
03ed696 Updated batch, NSIS and swig scripts for windows.
5b43fe8 Support \uXXXX with a codepoint -> utf-8 generator - this should probably be shared somewhere
e325a0b Know about uppercase ẞ
a207a50 Do not warn user about using foma type in hfst-lexc if they have explicitely requested it with --format foma.
9c19e8f Now hfst-lexc obeys argument --format. Fixes bug #213. Also updated NSIS scripts.
84b99fd Updated NSIS script. Added hfst-twolc and hfst-train-tagger bat files for windows compilation.
882e7f4 Distribution package includes flex/bison-generated cc and hh files. Make check runs tests only for those command line tools that are enabled.
66f323b Allowing tagger tools in mingw.
caa6f26 Fixed a bug in xfst_getline, also added freeing the result from it.
32c6b57 Fixed a memory error in hfst::xre::parse_quoted noticed during make check on MinGW.
8d39597 Fixed some bugs noticed during MinGW compilation: added typedef 'off_t' and compiler definition '__NO_MINGW_LFS' when mings compilation is enabled, and escaped percent sign in fprintf as double percent. Also added tagger tools and hfst-xfst to NSIS installer script.
158e8ef Allow plain input-output -replacement with tokenized quoted literals, like "this":"that"
87a7276 Some refactoring & warn about unused definitions
8ad5fe2 Ready for release 3.5.1.
83d0dc0 Applied lexc patch (bug #208).
4a95514 New lexc, but still incomplete
b8907ea Removed directory fsmbook-examples as fsmbook-tests replaces it.
5c0a757 Now using foma's lexc parser in hfst-xfst by default. Also using verbosity of hfst-xfst and tagger tools in lexc parsing. Also added an option --enable-all-tools in configure.
1213bb0 Use flex's newline count as good-enough parsing error indicator
3ab3a5d Actually fixing that bug this time
6c720e2 Fix typo-bug in previous commit
8a3e52c Much more verbose printing of time spent on harmonization and conversion & a speedup the the process of collecting a unified alphabet
209a12d Small fixes to installation check.
b2730bc Removed redundant command 'rm TMP' from pmatch test. Also removed verbose 'skipping some test cases' print from proc test, returning a skip value 77 is enough.
d902787 Fixed a typo in test script.
d7b4050 Changed temporary transducer name in test-hfst-xfst so that it does not conflict with any other transducers used in the tests.
ce5564e Skipping pmatch_blanks.txt when creating binary transducers from att files. Also made check-tools.sh more verbose. Fixed a typo in make-dist.sh.
3ebd13c Modified the call to 'nodeindexing' in SFST::Transducer::store as it seems to leave its argument unchanged sometimes. Noticed this effect when testing hfst-xfst and getting a segfault when trying to use a stored transducer.
c41db2e Added disjunct_as_tries as a public function of TropicalWeightTransducer.
7c3bd28 Remove ineffectual AM_LDFLAGS line
cb52daa Try setting a toplevel LDFLAGS to standardise to the standards-compliant libstdc++
a1d9ab8 A pmatch functionality test
8fe00ae --flatten and make option passing a bit neater
994689f Add DefIns and make LTR longest the default RIGHTARROW
0b2a9f6 Added argument 'extra_link_args' to Extension constructor. By default it is empty, but when making the debian package it can be replaced by linker flags that make it use /usr/lib/ instead of /usr/local/lib/.
432da48 Print transducer and function definitions as string pairs in the way they are defined by the user.
7fc05b8 Made print paths observe whether obey-flags is set ON or OFF.
0346f8c Now transducer names are saved.
efded06 getline() doesn't return a newline-terminated string if one can't be found; in this case we mustn't erase the final character.
000a25d Updated ChangeLog, maybe after a couple of bug fixes ready to make the actual release..
efb77b7 Added variable 'print-foma-sigma' to the xfst parser.
f279104 Add a pmatch2fst functionality test with a regression test
6ebc08b Fix bug wrt unused networks were getting erased from their container during iteration, causing other networks to be skipped
b911b35 A preliminary "true" negative context facility
39d4db6 Transducer definitions are expanded without calling substitute in xre parser, the new way is faster and less prone to symbol/alphabet errors. Implemented 'print net NET_NAME' in xfst parser. Allowing variable names that can be parsed as numbers in xfst parser.
7078c8f Ready for release 3.5.0.
426e74b Added missing files in fsmbook-tests to distribution package. Edited READLINE conditional in hfst-commandline and lexc-readline-ui.
9024224 Edited help messages of hfst-fst2txt, hfst-txt2fst and hfst-xfst. Added missing files to distribution package.
64a536a Renamed hfst-xfst2fst to hfst-xfst and hfst-xfst to hfst-foma.
3edfd46 Added variable harmonize-flags to xfst parser. A warning message is printed when composing transducers with flag diacritics if verbose in ON and harmonize-flags is OFF.
ce71a8a Added missing variable 'print-weights' to the XfstCompiler constructor with impl argument. Added support for prolog format to hfst-fst2txt and a simple prolog test. Fixed a typo in configure.ac. Removed unnecessary commented text from HfstTransitionGraph.h.
3ba662f Added support for weights when reading of writing in prolog format.
6094847 Edited xre warning messages.
a8a38ce Added a boolean argument 'increment' to function 'set_final_weights' which defines whether existing weights are summed with the new weight or overwritten, the default being overwritten for backwards compatibility. Also added warning messages about hfst-special symbols of form '@_.*_@' to the xre parser that are printed in verbose mode. Also added support for negative weights in xre parser.
fe04740 Fix compilation time output at hundredth-of-a-second precision
998f553 Make --verbose output somewhat neater
7864401 Time individual compilations
dc0dd3e More fine-grained information in --verbose
b782c81 Added option --harmonize-flags to hfst-regex2fst. Also added a function 'set_harmonize_flags' to the xre parser.
763aa83 Using result from foma in DateParser test.
46ace60 Added angle bracket test for hfst-xfst2fst. Omitting it until we decide how symbols enclosed in alngle brackets should be handled.
5efe9b4 Fix some problems wrt ?-expansion and symbol tables in RTN:d networks that have to be aware of each other. Also use dark, forbidden symbol magicks to prevent special markers in RTNs from being clobbered in this process.
9493341 Make harmonize() a public member function (this changes the interface)!
e713b45 Now lexc parser can handle epsilon '0' correctly.
9cc8ad7 Fixed a bug in xfst parser, now semicolon is allowed after 'read lexc <filename>' command and verbosity of the xfst parser is also used when parsing a lexc file. Also added expected results for fsmbook Esperanto tests.
9b7a8d4 Updated fsmbook tests so that they use ready-compiled xfst/foma results given in prolog format. Also fixed a typo in DateParser hfst script. Also checking that fsmbook-tests and xfst2fst are both enabled in configuration.
e22ffb2 Fixed a bug in quoted literals, now ["foo""bar"] is parsed as in xfst: ["foo" "bar"]. Also using foma instead of xfst in fsmbook-tests when compiling Lingala because different interpretations of symbols enclosed in square brackets.
f23f9c8 Now xre parser handles "[.#." and ".#.]", interpreting them as "[ .#." and ".#. ]".
ef252ec Removed unnecessary zlib dependencies.
15ac316 Updated cross product and apply boundary mark in replace rules.
04ce2d3 Changed include of hfst-scanner.h to hfst-scanner.hh.
fcab62f Changed include of xfst-parser.h to xfst-parser.hh
bc5b1c4 Changed PmatchUtilityTransducers variable utils into a pointer and added a function get_utils() to avoid variable initialization issues.
c67ec6c Reverted INTERNAL_EPSILON etc to internal_epsilon..
5501ab9 Pop the local state stack too when exiting rtn calls
966fd9f Changed internal_epsilon etc to INTERNAL_EPSILON also in pamtch utilities.
22d93fa Use xfst instead of my local copy of it..
90db8f4 Now using macro INTERNAL_EPSILON etc instead of string internal_epsilon to avoid variable initialization issues (noticed on cygwin).
8f8f9a0 Now fsmbook and hfst-xfst2fst tests should work correctly whether they are enabled or not.
cf28d27 Added condition WANT_XFST2FST to tools/src/Makefile.am. If it's true, directory parsers is included.
4359279 Make parser even more messy by having bodyless tagging contexts at the toplevel - for now
91a171c Added missing file.
7ba73be Now using directory fsmbook-tests instead of the old fsmbook-examples. Renamed h2hh.sh script to generate-cc-files.sh.
bdb4ca7 Roll back broken empty-bodied contexts for now
9932c0c Removed the HH_EXTENSION_USED conditional from configure and makefiles since it didn't work. Now assuming that the flex/bison-generated header files have the extension hh. If this is not the case, the script scripts/h2hh.sh should be run between configure and make.
ba5562b Added a script for cygwin compilation.
f373aed Edited ifdef conditionals based on warnings when compiling on cygwin. Also added a script for handling errors in flex/bison header file naming.
5c3fe43 HfstTransducer::cross_product should expand ?:? transitions to ?:?|?, added this but inside comments until a bug in xerox rule compilation is fixed.
9bd6dcd Fixed a couple of typos.
2ab61ee Added more tests to the xfst parser.
9b4ec9b Finally full support for quoted literals in xre parser.
8693d53 Tentatively adding quoted literals in xre parsing, this time step by step..
6d2fcd9 Trying again to make quoted literals work better in xre parser..
5f46435 Reverted recent changes to xre parser, noticed bugs in hfst-regexp2fst..
26cc935 Symbols that contain quoted literals supported in xre parser.
65f1798 Better support for quoted literals in xre parser.
afd146a Now allowing '0' in symbol names.
6c08400 Give verbose information about networks while parsing
c9200d4 Merge from devbranch
eba09c9 Added cstdio header, it isn't included in cygwin.
15d6ecb Dynamically allocate input tapes in case of overlong input - in case of insufficient memory, truncate overlong lines.
ea5ad01 Don't attempt to find regular arcs with symbols newly discovered from input.
4641571 Added option --eliminate-flags to hfst-compare. Commented debug info printing from LexcCompiler that polluted its output. Edited command line tool tests accordingly.
f4dd70e Fixed a bug in calling FomaOutputStream.
068bd6a Added variable print-weight (default NO).
11a3786 Added examples to hfst-regexp2fst help message. Now xre parser accepts also semicolon and one or more spaces as weight separator. Moved flag handling from FomaTransducer to HfstTransducer as it does not depend on foma backend.
6089640 Restore tape state after processing rtns; this should fix leftmost greediness bugs wrt rtns. Also: do some checking and print errors in case of broken tags instead of hard crashing.
db770ec Now make check passes also when some of the back-ends is disabled.
ef59b18 Output symbol tables were missing when writing transducers in native OpenFst binary format. This is now fixed.
626d023 Cosmetic changes to Xerox rules
ab840e4 Removed HH_EXTENSION_USED conditional from configure as it still fails on some platforms. Added function HfstTransducer::get_first_input_symbols and using it in hfst-summarize.
aa76b2c Ready for release 3.4.6.
38c638e Now empty comments starting with one or more exclamation marks are also parsed right.
ea40e35 Fixed errors noticed during hfst-xfst2fst tests.
adfea76 One more special symbol spelling to rectify...
3fb5812 Fixed a bug in fsmbook tests.
1935b06 Forgot to make context markers non-special too!
770efa4 Add a couple of useful combining accents
1578436 Use (faster) string disjunction instead of full disjunction for word lists
9fe2800 Make only TOP, networks with EndTag() and networks with Ins() get the special @PMATCH_ENTRY@ etc. delimiters. This should make various operations on networks that don't use these features work.
c201bf6 Added missing file.
3b4ae5e Now allowing several prolog networks in one file, separated by empty lines.
e26899d Added tests to fsmbook test set.
786baf3 Now Palindroms test works.
7c635b5 Fsmbook test Esperanto is now divided in several script files.
c165109 Now using prolog format to convert between transducer formats in fsmbook tests. Also added the option --prolog to hfst-txt2fst and fixed a small bug in prolog parsing in HfstTransitionGraph::read_in_prolog_format.
864120d Remove generated files in hfst scripts.
728b769 Correct internal endtag format in utils too
84bccf9 Endtags look like @PMATCH_ENDTAG_foo@, not @PMATCH_ENDTAGfoo@
c14098e Fixed a couple of bugs.
5976146 Correct typo that was suppressing the functioning of CATENATE_N
88455f1 Rename internal symbols not to match is_special_symbol() to let them pass through eg. composition operations
88ae522 Added tentatively directory fsmbook-tests which will replace the directory fsmbook-examples.
7f5ff0d Added the variant 'write att outfile isymfile osymfile' of 'write att' to the xfst parser.
e4f2985 A hopefully working test for HH_EXTENSION_USED - unfortunately it appears that automake lines not in a make target are included regardless
7d7a225 Eliminate else branch
ee4e69d A better value for the HH_EXTENSION_USED variable?
dc42b09 Eliminate references to .h targets when HH_EXTENSION_USED is unset, which was perhaps bothering automakes >=1.12
6a89896 A (more) correct understanding of IDENTITY.
510955c Switched the *.h - *.hh dependency around, from *.hh: *.h, to *.h: *.hh. This seems to have fixed the *.hh build issues on MacOSX. I have no idea whether this will work or break (on) other systems. Please test!
0a1a72e Some systems need to be told that .ll files depend on .hh files so things get generated in the right order
dc1d37e Added '@pl' operator to the xre parser. Testing it with hfst-xfst.
9a7a132 Fixed the h/hh issue once again, now trying to copy instead of linking. Also added assertions when converting from sfst to basic transducer format.
3b37bde Removed redundant filenames, added whitespace.
de4fcde Changing the BUILT_SOURCES to list the cc files instead of the hh files seems to fix the header file issues for twolc.
0d56127 Some support for identity transitions
9bc40e0 Implemented '@txt' and '@stxt' operators in xre parser. Also testing them through the xfst parser.
2f7e9a7 Make the alphabet know about IDENTITY
9fc2d09 Uncomprehendingly fill in the blanks for the LEXC_BUILT target
03cf074 When composing incompatible types, instead of throwing an exception, convert the latter type to the former for composition (as per hfst-compose's message).
91db5fb --force symlink to .h/.hh target
eeaf345 Add combining accents to the end of the capitalizing and lowercase-ifying utility transducers. Also more aggressive minimize utility transducers.
bd1cd1b Character constants like \xNN are two hexadecimal numbers, not decimal
df7f071 Allow escaped quotes on quoted literals
479cff7 Implemented '@re' operation in xre parser. Added test cases for it in the regexp parser tests.
b3a3c2a Fix bug with input tape initialization in certain circumstances.
a0c966a Changed rm -f to ln -s in the Makefiles when handling the h/hh rules.
327a83d ln instead of mv in .h/.hh rule
1fc8b14 Check for .h/.hh yacc convention and mv accordingly
faf3847 Made some modifications to the xfst print format, removed comments from xfst compiler.
8b608d4 Added missing files.
394603e Added substitute defined tests to xfst parser.
b38ac6e Added -f to mv header files
e441292 Fixed a bug in converting an empty foma transducer into an HfstBasicTransducer.
3cc8efa Added more substitute tests for xfst parser and fixed some issues with alphabet in substitution functions.
fd05f5f Fixes bug with Ins()erted transducers named "name"
1b4b69c Add toplevel delimiters (in addition to ones associated with EndTags). Fixes a bug when endtags aren't used but insertions are.
e91d560 ToLower, ToUpper and OptCap made (more) compatible with Karttunen's pmatch
5613090 Fixed bug in LexcCompiler.cc, one that was causing wrong compilation of noflags in cases when there were only identities in lexicon
9178deb Forgot to remove old test file.
c757950 Divided xfst substitute tests to several files. Fixed a bug in printing unknowns in xfst format.
479c64b Now allowing more variable names.
4307079 Now substituting symbol with symbols(s) removes the original symbol from the alphabet.
cf3fd28 Fixed a bug in xre substitution: disabled harmonization that caused unknown symbols to be expanded to temporary variables.
9e1c4f9 Minor changes to lexc2fst and xfst rules
9b45963 Not substituting a symbol pair with an empty set of symbols removes the transitions equivalent to the symbol pair.
c222f0f Added tests for substitute functionalities.
d9ba4ce Added 'substitute defined' to the xfst parser.
afa6747 Now configure checks the version of automake and renames the header files generated by flex, if needed.
de7b521 Now 'make clean' removes also files generated by flex/bison.
7b6831d capitalisation modes: last on wins
0fee2aa update
a8383d4 Oops, forgot to exclude xfst testing from make check..
5c2ce1b Added documentation.
8c9f157 Now most of the keywords are interpreted as such unless they are at the beginning of the line.
d488c08 Added substitute functionalities to xfst parser.
cec7f36 Added function 'remove_from_alphabet' to HfstTransitionGraph.
e4ae2b8 Added command 'assert' to xfst parser.
f163569 Now all commands are given by tab if readline is used.
961ff5d Added tests for writing and reading prolog files in hfst-xfst2fst. Also fixed a small typo in writing in prolog format.
a24c10b Removed implicit readline requirements from hfst-xfst.cc and foma.c. Also edited prolog functions in HfstTransitionGraph.h.
eab0624 Now reading and writing in prolog format works. Also addded some exceptions for prolog handling.
1765035 Added prolog printing functions to HfstTransitionGraph.
2cd5695 more header renaming (see commit for r3377)
2c12e0c change header file names -- this will break compatibility with automake < 1.12 see http://lists.gnu.org/archive/html/automake/2012-09/msg00017.html
055ff21 changes to names of generated header files --- pending testing
77f5280 Now apropos and help messages work in xfst parser.
d5fc1bd bump required versions of automake and libtool
9feac55 Ready for release 3.4.5.
c9b1cc5 Solved bug in replace rules with term complement in context, fsmbook tests are now completed.
ea426a7 Edited help messages.
d6b3866 Small fixes to the xfst parser.
0286e7f Additions to apropos and help commands.
21d49c1 Tentatively added enums and maps for xfst commands.
cdc7f85 Now using variables outstream_ etc instead of stdout etc.
1e43194 Tentatively added commands 'apropos' and 'help' to the xfst parser.
ed0fe8e Backport pretty dot files from that better branch
0fe1b1c Added more tests to xfst parser.
8ccc79d Added new functions write_in_xfst_format(ostream/FILE*) to class HfstTransitionGraph. Using it now in the xfst parser when calling 'print net' instead of write_in_att_format.
df3ab64 Added more documentation to the xfst parser.
90d0462 Added documentation to xfst parser functions. Also fixed a bug in prompt handling when calling apply up/down.
987fe0a Basic size indications in verbose mode
b66c98d Xfst parser: now apply up and down work also in pipemode.
dd5f2e1 Fixed some bugs in the xfst parser. Now readline/getline is used consistently in the command line tool and in the actual parser. Also the option quit-on-fail now works as it should.
cc65d1e ToLower() and ToUpper() were missing from the rule parser
5662445 Substituting symbol A with B on both levels now correctly removes A from the alphabet.
4316ee7 Occasionally minimize while collecting large @txt lists (this is a significant efficiency improvement for those cases)
de4c50d Fixed a couple of bugs in including back-end libraries noticed during cygwin compilation.
648dd3e Tests for version and help checking as well as empty input now check only a set of listed tools instead of hfst-*.
41f0dff Commented foma headers from hfst-file-to-mem.cc. Also fixed a bug in regex parsing in hfst-xfst2fst.
c77d4ab Let's use malloc until we figure out where xxmalloc lives (and why we want to use it in the first place)?
945c765 Error messages from xre parser are now handled through the function get_error_message instead of printing them directly to standard error stream. Also changed 'Transducer' to 'SFST::Transducer' in SfstTransducer tests to avoid ambiguousness.
9898075 Some header files stil used installed back-end libraries instead of those in directory 'back-ends'. That is now fixed.
d12c4da Now prefix xre is used when generating xre parsers. This should fix bug #185.
19efda5 Now regex comments can contain semicolons. Partially fixes bug #177. There is still some ambiguousness in weighted mode, consider e.g. the input 'cat ; 3'.
66ae905 Fixed a bug in function hfst_file_to_mem, when using stdin.
1db3b53 lex joiner's format changed to special symbols which do get harmonized
c4e5502 Moved function hfst_file_to_mem from hfst-xfst2fst.cc as a separate file.
43006eb Fixed comment parsing in xre parser, hfst-regexp2fst still needs some fixing so that semicolons can be used in comments.
3b8d538 Added facility for inserting undefined transducers as long as they are defined later
78b697d stringify() can and should take a const parameter
4c74eab If no boundary markers were used, we need to advance the input tape accordingly to where the real input starts
a0171d9 Print newlines after printing match result
b6798c7 Check for nonzero length before trying to omit final newlines
f978c01 Omit final newlines from input
be48f32 When in newline mode, no need to print final line because it's already been printed
9ac5d42 Reworking of pmatch tool, added default of blank line as separator
1d9d235 Tentatively added a function compile_first to the xre parser that allows compiling onely the first regex in a string.
459a518 Now hfst-fst2strings obeys the -S flag. Also removing generated files after make check in xfst parser.
6856a76 Added 'print defined' to xfst parser. Also temporarily commented the bison > 2.6 request in configure.ac until bison on hfst gets updated...
ee6e8d0 Added a test for bison version 3.6+ - now it will stop if older versions are found. This seems to fix at least one of the build issues reported.
18a9ff8 Added tests for defining functions in xfst parser. Also fixed a bug in function definition.
acd3a94 Add diagnostic for rulesets that don't recognise anything
a62e3f5 Avoid compiling transducers that "recognise" the empty string
e196215 New insertion arc handling
8301449 Move insertion arc checking out of flag diacritic handling after all
692840f Collect and harmonize alphabets
0e895a7 Detect inserted transducers and only pass them and TOP on from parsing
5bcd4a2 Recognise insertion arcs as flag diacritics
5904514 Skipping regexp tests that do not work. Also added new tests.
62202a7 Changed regex.tab.c and regex.tab.h filenames to regex.c and regex.h because make distclean of some version of autotools removes tab.c files. Should fix bug #184.
d829e21 Catenate functions were giving wrong results, now fixed
0cf3e8d No flags part now seems to be working
4155dd9 Don't treat automake warnings as errors.
e568b99 Regexps that contained comments were skipped, now that is fixed. Fixes bug #182.
8590e32 Reverted back to earlier version.
c93a90c Changed bison-generated h-files to hh-files. Recent versions of bison and automake are needed for compilation.
cbfe23e Now binary transducer operations don't modify their second argument. Fixes bug #180.
bcbc96f Refactor ol conversion to allow giving a harmonization hint
1219953 Fixed error messages.
0bc38c1 Fixed error reporting in xre and xfst parsers.
23eb58d Now reentrant parsers are used in all xre parsing.
7ab97de Fixed replace rules with empty languages
6f2b708 Added missing file.
2fc9432 Added reentrant versions of all xre functionalitites.
e1ac0fd Tentatively added reentrant versions of the xre functionalitites.
7d683c4 Avoid extraneous copying and sorting
707fe42 Now comment lines are correctly parsed by hfst-regexp2fst. Also added error diagnosticprints to proc test.
8cb8ab2 Maybe fixed bison anf lfex header problem. Revert, if it doesn's work.
9b22f58 Added more function handling to xre parser.
31fcb4e Fix invert logic and remove debug print
758a647 Added more function handling mechanisms to xfst and xre parsers.
fb73e65 Now xfst parser recognizes function definitions.
9d699fb Make the starting position of the input tape be before BOUNDARY_MARKER. This is to enable referring to the boundary in expressions, not just contexts. It should gracefully always fail to match and be ignored in the absence of boundaries in expressions.
368efab Refactor parsing; rip out tool-specific parts (and, for the time being, some probably unnecessary control options)
3267693 Debug printings and rearrange noflags part to first
59bb75d xre parser now recognizes function calls. xfst parser supports command 'source'.
185a9ea Add some noflags to parsers
9aa9a7c Add support for selecting flagged and composed morphotaxis.
6c6f4f0 Command 'source' implemented in xfst parser.
65fc6de Added test case for bug https://sourceforge.net/p/hfst/bugs/177/ .
57132d8 Added symbol substitution to xfst parser.
f7bec38 Remove use of strndup to fix bug #175
d63cd8c Added more functions to the xfst parser.
c9048a3 Fix a bug caused by shifts in the input tape caused by entering left-hand contexts that weren't being undone.
4945095 Added more functions to the xfst parser.
16ef95d Removed WeightedString classes as they are replaced by HfstOneLevelPath etc datatypes. Also added new functions to the xfst parser.
9962f1e Moved lookup_fd functionalities in hfst-lookup.cc to HfstTransitionGraph as member functions. Also moved HfstEpsilonHandler.cc and .h from tools/src/ to libhfst/src/. Removed is_flag_diacritic from hfst-lookup.cc and using FdOperation::is_diacritic instead.
41c6f4e Removed extra code.
5dfb71b Moved EpsilonHandler from hfst-lookup.cc to HfstEpsilonHandler.h and .cc.
24c1781 Moved is_lookup_infinitely_ambiguous(HfstBasicTransducer ...) in hfst-lookup to member function of HfstTransitionGraph: is_lookup_infinitely_ambiguous(...).
3aee716 Added flag tests for hfst-fst2strings.
944e477 Use ANOTHER_EPSILON for tokenising
b39c704 Now option combination --obey-flags --random works in hfst-fst2strings.
6597da6 Insert epsilons to all lexicons
5f8d5e2 Maybe align end flags
66184c6 Now hfst-xfst warns about transducer type conversions. The state and arc information should also be correct.
2ab6f01 Added missing test file.
ff916bf Many changes to how utilities are handled, prepare to move all parsing away from the hfst-pmatch2fst tool
6379325 Another bug in flag elimination fixed. Now flags that are not eliminated are included in the filter so that they are not blocked.
2c0c535 'load stack' didn't print state and arc info for all transducers, this is now fixed. Should fix bug #171.
37b640a Calling eliminate_flags for a transducer that has no flags to be filtered caused a segfault, this is now fixed.
ecb18bc Updated eliminate flag tests.
95c91cc Fixed a problem with default symbols and harmonization in compiling guessers.
72040dc Ready for release 3.4.4.
e0e9996 Now proc functionality test proceeds until compounds2.strings would be tested and instead returns a skip value 77.
eef4bc1 Now tests for tools that are disabled by default are only carried out if the tools are enabled...
e25f409 Fixed a couple of bugs noticed when installing HFST on Hippu.
b7b567d hfst-xfst2fst tests are now skipped if --enable-xfst2fst is not specified. lexc-compiler-functionality-sh and proc-functionality.sh also return a skip value.
dacc7d1 Now foma and sfst backends are enabled by default in configure.
aea2aa0 Now hfst-lexc uses only foma. hfst-lexc2fst tests skipped as experimental. Flag tests in hfst-fst2strings are also skipped until flags are thoroughly tested.
f0a52e9 Fixed missing parameter in fprintf in warning message in hfst-compose-intersect. It was causing a segfault on some systems.
d374ccc At least unification flags are now handled correctly in hfst-fst2strings.
0f5f715 Moved HfstLookupFlagDiacritics functionality from tools/src/ to libhfst/src/.
60e6533 Tentatively added HfstTransducer::extract_random_paths_fd. Flags still need to be processed right.
d55617c Now hfst-lookup warns about infinitely ambiguous transducers. Fixes bug #168.
2c03ec7 Added missing files to Makefiles noticed during distcheck.
c608af2 Added warnings to xfst compiler when using longest-string(-size) with flags.
1fc3922 Tentatively added commands longest-string and longest-string-size to xfst parser.
34771a9 Added function HfstTransitionGraph::longest_string_size.
ec8a45d Tentatively added function HfstTransitionGraph::top_sort().
3c946a6 Added option --version to hfst-pmatch.
117677d Make sure minimization always gets done
e93a1ca Added more test to xfst parser. Now comments are handled correctly in xfst. Edited documentation in HfstTransducer.h.
c1e3447 Merge from pmatch dev branch, with a change related to read_lexc -> read_lexc_ptr.
bd3e893 Now xre parser can parse expression '{foo}:{bar}' correctly. Also added variable tests to xfst parser.
6a48024 Added a test for xfst's shortest-string. Also fixed a bug in including the epsilon string in HfstTransducer::extract_paths.
c0d6df4 Added new xfst tests, also added function HfstTransducer::is_automaton.
21b6858 Invert lexc flags
2b3399e Renamed again some testfiles in xfst..
6c55cb5 Edited xfst tests.
fbe891e Documented flag elimination function. Fixed some bugs in using transducer definitions.
d1a4df5 Added a new test for flag elimination.
7e73c8e Added tests for xfst parser's function eliminate flags.
4f1b6ed Wrote more flag elimination functionalitites.
3ef35f3 Added x: center rules and tests for them.
9bac4fc Restored long option name --resolve.
2ddf6e2 Fixed problem where definitions in pair expression were caught too late which resulted in weird syntax warnings.
dde58d7 Maybe use flags instead of joiners in legacy lexc compilation.
855907f Started implementing flag elimination function.
0417d5c Added new functions to xfst parser.
37f8117 Added new test for xfst parser.
f4117b7 Set property executable ON: test.sh.
4f0096e Again doing flag diacritic filtering in flag diacritic harmonization, since it turned out not to be a problem after all.
d7b530c insert_freely_missing_flags(...) no longer uses insert_freely(...) becauses successive invocations of insert_freely without intervening minimizations cause exponential state growth.
9cc367a compose-call in hfst-compose now does flag filtering only if flag harmonization was done
96db17f Changes that permit new filtering of illegal flag paths hopefully wothout affecting performance.
1015c1e Added more tests for xfst.
2626251 Added more tests for xfst parser.
bb869de Now readline is disabled by default in xfst parser unless configure is run with --with-readline.
e9df9dd Changed 'HFST_THROW' into 'HFST_THROW_MESSAGE' in HfstTransducer::eliminate_flag(s). Should fix bug #163.
64baca6 Renamed some of the functions in xfst parser.
b9fb1df Now readline history of 'inspect net' is ignored when returning from 'inspect net'.
4e8a8d6 Commented code.
2921e59 Now 'inspect net' should work for all cases.
b3bda03 Tentatively added 'inspect net' to xfst parser.
28fe040 Edited XfstCompiler
281668c Added new tests to xfst parser and implemented interactive lookup mode.
4d9d801 Added more tests to xfst parser.
7461e46 Tentatively added functions HfstTransducer::eliminate_flag(s). Added function 'compose_net' to xfst parser.
f40c185 Updated to foma version 0.9.17.
c822503 Added test for HfstTransducer::remove_from_alphabet.
03593c7 Changed bool static is_safe_conversion(...) to static bool is_safe_conversion(...) because syntax error.
b708ce1 Now regex parser substitution does not add substituting symbols to the alphabet of the transducer to be substituted. It is not needed anymore since there is a parameter 'harmonize' in HfstTransducer::substitute(StringPair&, HfstTransducer&).
ee00f6c Now HfstTransducer::n_best throws an HfstFatalException if OpenFst fails to allocate enough memory. Kind of fixes bug #159.
2dd7fe5 Now hfst-regexp2fst and hfst-xfst2fst both give a warning message if the user writes a replace rule that contains special symbols.
4f1842f Added more symbol harmonization tests.
782cbb7 Commented code.
1e7be91 Now the regex parser parses [x:?] so that it includes the transition [x:x]. This also happens in xfst and foma.
2b62822 Now regex parser interprets ?:? in the same way as xfst or foma, i.e. as unknown-to-unknown OR identity-to-identity.
19fddc2 Implemented substitution with an empty set in regex parser.
5d3bcdc Fixed hfst-twolc.bug which inserted all diacritics into the first set that was declared.
acdf5e8 Now xre parser never performs harmonization in substitution or ignoring.
2a1e3d4 Fixed a bug in xre substitution.
5a033b8 Added a bool parameter to control harmonization in HfstTransducer::substitute(SymbolPair, HfstTransducer). Fixed xre parser so that it doesn't harmonize when performing substitution. Also changed call to substitute function in twolc but preserved the harmonization.
5282e25 Now binary operations never harmonize special symbols of form @_.*_@. To make xerox rules work, the marker symbols @_SOME_MARKER_@ had to be changed to @SOME_MARKER@ so that they get harmonized as they did earlier.
4312b8c Now insert_freely also takes a parameter that tells whether harmonization is carried out.
e979ac2 Now there is an option --do-not-harmonize for all binary command line tools.
72d8c60 Fixed some bugs in harmonization.
b7ad8b6 Fixed missing bool parameter problem.
5890a60 Added a flag -H to hfst-compare and hfst-regexp2fst that turns off unknown harmonization.
004dfbd Added a bool argument to binary transducer functions that controls whether harmonization is carried out. Also added harmonization tests.
12c3111 Now hfst-lexc writes all messages to stderr. Fixes bug #68.
cdf4d16 Now FomaTransducer::define_transducer(symbol1, symbol2) works also if symbol1 == symbol2 == identity. Fixes bug #110.
20dfb3f SfstTransducer.h no longer uses generic word 'Transducer'. Fixes bug #130.
4aa59f6 Now hfst-sfstpl2fst finds included files in the same directory where its input file is located. Fixes bug #145.
aea2877 Now hfst-fst2txt prints an error message instead of segfaulting when given an input stream whose transducers do not have the same type. Fixes bug #115.
1cefd4e Implemented functions number_of_states and number_of_arcs.
b081353 Now all xfst files are read in a buffer and parsed with XfstCompiler.parse_line.
0c08820 Now xfst parser can parse regexps that are on several lines.
d2f3f2e Edited xfst parser.
9c5a616 Added tests for hfst-xfst2fst.
721c603 Added main program from XfstCompiler.cc to hfst-xfst2fst.cc.
21981c9 More command line options added to hfst-xfst2fst.
f036bab Parameter handling added to hfst-xfst2fst.
fea1693 Now hfst-xfst2fst works with readline functionalities if a temporary file is used for buffering commands.
e2fa01a Replace rules compilation - boundary marker added to begining/end of all contexts which don't containt it
f674390 Added more functions to the xfst interface.
51d468d Added more functions to the xfst interface.
d806ad2 Added variable undefinition to xfst parser.
21d5809 Now definition expansion can be controlled in XreCompiler through a function set_expand_definitions. If they are always expanded, lexc syntax is not parsed correctly.
48adbe9 Added definition expansion to the xre parser.
6ee87cc Made a couple of fixes to xfst parser.
bcc413b Tentatively added tools/src/parsers as a part of the automake build.
87e7cf8 Updated morphology installation tests, now omorfi is included.
0f1d6b3 Updated version number in swig setup file.
dce9471 Now installation tests skip sfst or foma if they are not found.
9dd6577 New apertium proc space handling test from bug [#153].
5092648 Small fixes to swig and installation checks.
f28cddb Added documentation.
ad582a8 Made small fixes to swig tests.
daeca5d Made permutate-file-params test silent. Edited check-tools.sh so that it handles --prefix parameter right.
a34e969 Updated hfst tool checks-
cb402e9 Fixed Lingala and MonishGuesserAnalyzer fsmbook tests, minor changes in regexp
bec768d Updated installation checks.
1b414e4 In substitute (symbol_pair, transducer) added harmonization with the symbol_par. Also, updated substitute in regexp
da8b67f Ready for release 3.4.3.
ce48e9e Prune alphabet added to regexp, also ? -> x rules fixed
310a1e1 Added a couple of fixes to swig tests.
9ecc837 Now tests pass on MinGW, file handling still has some problems.
4b8f264 Disabled tagger tools if HFST is compiled on MinGW. They are also skipped in tool tests.
fbad2ce Added tests for hfst-xfst and hfst-twolc in installation tests.
a7a109a Added 'AUTOMAKE_OPTIONS=std-options' to foma Makefile so that foma wrapper gets installed right on Cygwin.
6fb4082 Renamed option --enable-windows as --enable-mingw.
6430247 Edited configure.ac so that setting executable mode on script files should finally work. Added missing windows conditional to foma Makefile.
3e49944 Prune alphabet changed for easier use in parser
04dbf51 Removed local versions of hfst-xfst as well as hfst-xfst tests.
4dc1af2 Setting executable mode on scripts.
fcd3abf Fixed a typo in Makefile.am.
4451854 Fixed a bug in foma wrapper. Also added missing test file to EXTRA_DIST.
6ae0cb3 Set executable mode on scripts.
57f2152 Made a local version of hfst-xfst that is used in tests.
9afcfd1 Ignore generated files.
280ff8c Added a test for hfst-xfst. Modified permutate-file-params.sh so that it doesn't use curly braces which are not expanded correctly on all environments. Also modified hfst-compose-intersect.cc so that it doesn't use dollar signs when specifying how strings are printed as they don't work on Cygwin.
ae57f52 Now swig tests use libhfst.hfst_get_exception() instead of catching HFST exception classes. This should make the tests compatible with python 3 where catching exceptions that do not inherit from BaseException is not allowed.
6849b3e Disabled fsmbook tests from 'make check' by default (enable with --enable-fsmbook-tests) because they currently depend on foma command line tool.
c08fbcc Added function HfstTransducer::prune_alphabet which is also used in the tool hfst-prune-alphabet.
c23504b Added filtering to harmonize_flag_diacritics. It will filter out superfluous flag diacritic paths, which are created during harmonization.
be0e3fd Prevented foma from harmonizing id- and unk-symbols with flag diacritics in insert_freely.
8cecacd Removed 'words.txt' from EXTRA_DIST as it is not found in svn repository and make check passes without it.
653b608 Added a function 'std::string hfst_get_exception()' that can be used to get the name of the latest exception thrown by HFST library. This is useful if the version of Python used does not support catching exceptions that do not inherit from BaseException.
d4ceb69 Making sure that flag diacritics are added to the alphabets of both arguments in harmonize and harmonize_
9e56f9f Changed id- and unk-symbols harmonization so that flag diacritics are no longer harmonized.
2d801c6 Fixed a bug in flag diacritic harmonization yet again. This time actually using the parameter bool harmonize in freely_insert.
b2139f2 Fixed problem which resulted from changing the insert_freely-function in the library.
65f17f0 Re-fixed flag diacritic harmonization problem, where identity and unknown symbols were expanded by flag diacritics. To prevent expansion in this case, I added a bool parameter harmonize to the insert_freely library function.
220b78c Now configure gives warning on missing hfst-xfst correctly.
8499a59 Added a new tool, hfst-prune-alphabet. Also added a new bool parameter to HfstTransitionGraph::prune_alphabet.
0de2ffe Changed variable name 'infinity' to 'infinity_' to avoid compilation error on Cygwin.
58531f1 Added -D switch to enable reading from file in htwolcpre2 and htwolcpre3 (for debug purposes).
aa1f445 Now tagger and twolc tools also pass version and help checks.
edc2f45 Added a missing <cstdio> header that caused an error on cygwin (FILE was not defined).
0e0d644 Now all back-end files are included in the distribution package.
626ae88 Fsm-book test FinnishProsody
40ae452 Fsm-book test Palindromes updated
157ee16 All back-end directories are now included in the distribution package.
bafa96d Added PlusOrMinus fsm-book test
5534b00 Updated README. Fixed hfst-info so that it passes version and help checks.
14b7427 Changed the order of paragraphs in README. Added a todo comment about XreCompiler.compiler conflicting with a built-in name 'compile' in python.
b4c8b7b Updated README.
52eff40 Ready for release 3.4.2.
b38adb5 Now -lncurses is used in foma instead if -ltermcap is not found.
dd13b4c Fixed a bug in foma wrapping-
eaa693c Added regexp substitution
55d2750 regex parser - added left replace epenthesis rule
145ac04 Added note to remind people to also increment the HFST_LONGVERSION. This is important to ensure consistent behaviour between different types of version checks. The different types of version info returned by hfst-info --version should be consistent, the long version should just be a representational variant of the other values.
d2ff139 Longversion.
9d02b3e Ready for release 3.4.1.
7741485 fsmbook examples change
d73849e Epenthesis rules fixed
557c7bf Updated results of morphology tests.
251f753 Now hfst_foma is called directly from where it is installed.
533a2d0 Bring the long version string in sync with the other version numbers.
95f8764 hfst-xfst now uses hfst_foma instead of installed foma.
70c60e9 Added test sentences for all morphologies.
eb0a148 Renamed a duplicate symbol.
7bcb083 A variable was shadowed by a another variable with the same name, renamed the previous.
2939cf2 Made the names of the tests scripts shorter.
4334be5 Tentatively added tests for installed HFST morphologies.
b7a55de Added lower priority union to regex parser
2eaa8ba Added foma script under back-ends/foma. It is now used by hfst-xfst. Fixes bug #147.
999b9c5 Fixed a couple of typos in swig test file.
5db5e5a Updated the installation check script.
e754a74 Tentatively added tests for installed swig bindings.
c6763af Now installation tests pass on Hippu.
1bb3ebc Updated tests.
faca499 Now installation tests should pass. Also fixed a cople of minor bugs in test/tools.
7b55aac Added new tests.
ecffb8d Tentatively added a test for all installed HFST functionalities.
7844d8a Another warning-silencer flag that had to be removed to get hfst to build on MacOSX.
8dceaad Made hfst build on MacOSX again, by removing the -Wno-unused-result GCC flag. Sorry about the additional warnings, but I really need it to build.
3825fab Suppressed warnings caused by back-end libraries.
b087ac1 Ignored generated files.
ac73af7 SWIG bindings should now work with both Python version 2 and 3. As FILE* conversions are no longer supported in Python 3, added more wrapper functions that use an HfstFile argument instead.
1a71ed7 Priority union bug fix
52a9da3 Fsmbook tests updated
b151760 Removed old command line tool hfst-strip-header from tests.
d4e491c Removed old command line tool hfst-preprocess-for optimized-lookup-format from tests.
f0fbd83 Removed old command line tool hfst-duplicate from tests.
921e328 Revert previous unintended change to unrelated files
bb4a001 The header from r2888 that accidentally didn't get re-committed with the most recent bugfix in this directory..
5bcd481 Fix typo-bug wrt tape movement
6c8fc3d Roll back previous changes pending bug fixes
7e449db Avoid reallocations in the input tape
55d96ea New scheme for input and output tapes to allow them to be arbitrarily large
86942e1 Handle the case where we want to take a default transition when it's the only transition
7a27e64 Changed all includes of type <backend-header.h> to "backend-header.h". Made small modifications to NSIS installer.
34355d9 Added more information to swig/README.
a29cb34 Ready for release 3.4.0.
049de83 Fixed some bugs noticed when making distcheck-
a66c16d Fixed a couple of bugs that occurred during installing a dist package..
809b5e7 Fixed a small bug in HfstTransducer::read_lexc, suppressed some warnings from openfst headers.
6daff3e Updated python tests.
85104fb Added all python examples given in the doxygen documentation to tests.
d41d5b3 Added more tests to the Python interface and added throw statements to functions.
1113a6c Renamed hfst-duplicate as hfst-multiply.
2ab080e Now tests remove extra files that they generate.
b7d3c8c Moved Python tests to a separate directory.
173566e Python documentation is now in a separate directory.
48a0405 Now all functions in Python interface are documented.
14a3048 Added documentation on rule functions of the Python interface.
b648563 Added new internal documentation to the Python interface.
0093e29 More documentation on the Python interface.
dcd281c More swig documentation...
2b97711 Added more documentation for class libhfst.HfstTransducer.
33c493e Updated swig documentation.
964d693 Added - as a comment - parts of my attempts at building all version info from only one definition. No success so far.
23b3cbd Added missing directory 'openfst'.
61e0e19 Added a new constructor to HfstTransitionGraph, also added it to swig interface.
75891d8 Updated swig documentation.
3624f88 Updated python documentation.
43a3301 Added more documentation of the python interface.
7145563 Tentatively added doxygen documentation for Python interface.
f888bfc Now all back-ends (SFST, OpenFst and foma) are moved under HFST and statically linked to the HFST library.
5835516 Forgot to change the directory name to opefstwin in Makefile.am.
f617fb6 Renamed directory 'windows' as 'back-ends'.
35cccd1 Added a variant of HfstBasicTransducer::add_symbols_to_alphabet that takes a StringSet as an argument.
46143e2 Added all substitute functions to swig.
1a742cf Added XreCompiler functionalities to Swig.
009986e Now enumerators can be used normally in python, e.g. libhfst.SFST_TYPE.
53db94f Added Lexc functions to Swig.
417ecba Added more tests.
0d90bbb Fixed a typo in configure file.
3b8de30 Bring the HFST_LONGVERSION in sync with the other version strings (again - is there a way to calculate some version strings from others? I.e. change once, update everywhere?).
dc6f3df More updates to tests.
4146e0f Updated swig tests.
49df758 Fixed MonishAnalysis fsmbook-examples test
2468270 Added before and after to HfstXeroxRules
99ce678 Added new test for tokenizer.
0b4a0c8 Fixed restriction in HfstXeroxRules
b375541 Updated restriction in HfstXeroxRules
4b4e102 Small updates and added documentation to tests and swig bindings.
eace5f2 Added new test cases and functions to the Swig interface. Also moved HfstFile to HfstTransitionGraph.h.
c262db7 Added restriction to HfstXeroxrules
ddf3b77 Added one more test.
79d2758 Updated tests, added some throw statements to the interface file.
7de5e18 Tentatively added a new test for swig bindings.
d7e0b48 Added HfstTokenizer to the python interface.
1676fda Added some files to _BUILT that probably should have been there all along (is this the cause of those build problems on some platforms?)
1a14d8f Tentatively added python version of libhfst test.
8328757 Added yet one missing lexc file.
77eda4a Added a missing file in lexc tests.
22ae13f Ready for release 3.3.15.
8164899 Fixed a re-introduced bug in insert_freely(SymbolPair&), now unknown and identity symbols are expanded during the operation. Also made svn ignore some generated files in swig directory.
f3e79ad Fix unnecessary shell call that breaks portability
e008efd Fixed bug in flag diacritic handling. Transducers were harmonized before insertion of missing flags, which lead to undesired behavior when identity/unknown symbols mixed with flag diacritics.
0fe8065 Updated doxygen documentation. Made svn ignore some generated files.
9944411 Removed bashisms from scripts.
012e4d9 Fixed some GNUisms ans bashisms in twolc and tagger scripts to make them work also on non-Linux platforms.
4075404 Fixed some GNUisms ans bashisms in tests to make them work also on non-Linux platforms.
7a9c6c7 Enabled hfst-tagger when compiling on Windows. Excluded the Windows versions of tagger and twolc tools from hfst-check-help.sh. Also updated Doxygen documentation.
5b310da Added versions of hfst-twolc and hfst-train-tagger that use system calls instead of shell commands.
a5e86c5 Made small modifications to hfst-optimized-lookup so that it works on Windows. Now the transducer file is opened in binary mode and carriage returns are removed from the input strings.
16e7620 Just a test commit, nothing to see here
e4f6871 Parse 0 within @ flags as if it was known multichar symbol
d389a29 Fix escaped 0 at the end
c22c68d Warn users about wrong case in keywords
c1e59ec Removed hfst-strip-header from tools, use hfst-fst2fst instead.
77c82f3 Removed hfst-preprocess-for-optimized-lookup-format tool.
5d6f85e Updated swig bindings, edited NSIS scripts.
126e4b8 Parse version vector with serial strtouls
ff4d11e Bring the HFST_LONGVERSION in sync with the other version strings (?).
1cc8a0e Changed HfstTransducer(FILE*, ...) so that epsilon symbol is an obligatory argument and added a new obligatory argument that keeps track of the line number in the AT&T file. Did the same to HfstBasicTransducer::read_in_att_format.
92e7a51 Added a directory for NSIS installer scripts.
685194f Now FomaTransducer::read_net accepts also files with CR+LF newlines.
9844e68 Added wrapper class HfstFile for FILE* that can be used in Python.
4198158 upper- and lowercase versions of convenience acceptors
6d8f659 Added missing files to test/tools, fixed a typo in libhfst.i.
6ed258d Ready for release 3.3.14.
e7f7786 Added new testing scripts.
047964b Now HfstTransitionGraph::read_in_att_format throws an EndOfFileException if the argument FILE is exhausted. Also fixed a typo in HfstOutputStream.
791ead1 Added an alias for HfstOutputStream::operator<<, HfstOutputStream::redirect. Also added a Python test test_streams.py.
6b162d9 Fixed a namespace issue in libhfst.i, now omor_query.py works.
3536086 Added HFST exceptions to the Python interface. Renamed HfstBasicTransducer::at(HfstState) to transitions(HfstState).
7918308 Added more functionalities to the swig interface.
7300635 Fixed missing symbol table in optionalization bug for tropical weight type transducers.
56211a1 Added wrappers for operators [], = and << so that they can be used in swig bindings.
8610408 Added the function write_in_att_format(char * buffer, bool) to HfstTransducer. This makes it possible to call python's print() to a transducer.
cf19853 Tentatively added HfstBasicTransducer to the swig interface.
3fb1743 Changed Replace rules interface
923eb25 Fallback utf-8 tokenization
13e36ea Tokenize string literals for pmatch parsing
f1319a3 Now hfst-lookup handles the '\r' character right when parsing input strings on Windows.
f9b9344 HfstOlTransducer streams are now opened in binary mode.
5380f65 Added dlfcn include path to Makefiles.
65e903e Remove mentions of apertium and xml stuff
89ad025 Multiline handling like regexp2fst
ccfccde Fixed a typo in Makefile.am.
0902aaf Tentative names for charclasses
56e9149 Semicolon separation is default
0166e9e Context-checking markers in pmatch2fst parsing
29b863a Make , reserved. Fixes [#134].
f0a7795 print yacc debug traces if hidden sekrit debug mode is on.
6e9dbd0 Added missing file sfst/interface.h
a359eb1 Changed ../../tools/src to  in fsmbook-examples to make tests more flexible.
55b39ed Changed the pathname ../../tools/src to . This makes testing more flexible.
bfba7cb Fixed a couple of typos.
b75728a Renamed .c files to .cc so that c++-compiler is used.
06fdbfa Fixed sfst include path.
e30d3a0 Tentatively added SFST under directory windows.
d300ea2 #include <stack> (seems to be necessary when compiling without openfst at least)
0285685 Some compilers don't believe in universal characters given from the ASCII range, so instead of \u000D, have \r
3ab2f3a Fixed format specifiers when printing long long on Windows. Also specified windows/foma as include directory in Makefiles.
6c9ef91 Added regex test case from the GT/Divvun sources that presently does not compile with hfst-regexp2fst.
a40bd43 Made fixes to foma compilation.
16309e0 Add charset-defining hardcoded transducers to pmatch utilities for now
85e6ba7 Foma's io functions are now skipped.
7593089 Fixed left replace rule
3270f5d Stateful context-checking mechanism; remove recursive functions for same
8c59f9e OpenFst library check is now skipped and flag -D_MSC_VER is used when compiling on Windows.
9bde6a2 Fixed a typo in library name.
dfeb54c Tentatively added foma under directory windows.
30a6942 Added generate_threshold parameter to prevent generating suboptimal forms.
9baaaeb Generate several forms if several forms can be generated.
67950e5 Added missing --format flag in hfst-strings2fst functionality test.
1a53070 Modified guessify to handle gradation in omofi.
898af0b ACX tool disabled by default, requires libxml/tree.h and -lxml2
94a22e0 Temporary file - does not belong in svn.
e666602 Test parallel left arrows [#132] part 2.
9053c67 Ignore expressions that _only_ contain whitespace. Fixes [#132]?
5d48615 Test not contains and multiline expressions with comments and emptylines, Bug [#132]
d92fb74 Bail on parse errors
fe31f8c No semicolons, huh
8c7725c Allow windows or mac legacy \r's to end strings
3e51123 Stop recording input data to headers until header support for longer data gets fixed
5189a69 Only test fsmbook-examples if other tools actually work...
33aedf5 Do not use AC_PACKAGE_URL or resp. parameter to support autoconf's older than 2.63 or so
4ade921 Test that dos strings compile like unix
635fa08 tests should pass 77 if tool is not available
3d32324 Test that strings2fst handles M$-strings
335c414 Fixed a bug in testing if compilation is done on Windows.
58d3876 Disabled using hfst-lexc with multiple input files on Windows.
2076847 Fixed some problems with automatic LF-to-CRLF conversion on Windows.
3843896 Now hfst-twolc test uses 'uname' to find out if Windows is used.
a0d41f4 Added excutable file extension '.exe' to tests when running them on Windows.
1da9324 Disabled tagger tools on Windows until they work there too.
3d2c49b hfst-substitute now recognizes '@0@' as epsilon, fixes bug #133.
4ff55bd Noticed missing files and tests that should be skipped during distcheck.
d7576d0 make dist revealed an extra file listed in Makefile, skipped that
348deb6 Ready for release 3.3.13.
970f355 Merged changes from old repository.
5bf1aa1 Undo conditional compilation of pmatch compiling stuff
004ee60 Limit memory consumption a bit by clearing SequenceModelComponentPair state tables between tagging sentences.
73a589c Make pmatch parsing code conditional on WANT_PMATCH2FST
805b765 Changed sizeof(input_symbol) and so on to sizeof(SymbolNumber) since sizeof can only be applied to static data members according to standard (though not all versions of gcc).
c08e4ec Changed WeightedStringVectorCollection test so that it takes into account the model penalty weight line.
e7567b3 Fixed incorrect loop condition in previous commit to this file
0691b07 Added newline between results for different words.
2205e95 Recommit fix to ol -> basic conversion bug (hopefully)
634912e missing part of %0 handling
c931c66 Option --enable-windows is passed to Makefiles.
58812ce hfst-twolc test is currently skipped on Windows. hfst-twolc is a bash script that will be replaced by a batch file on Windows.
60f72ef Fixed multiple defintions of SequenceModelComponent::START_STATE in make check
8e66c75 Comment out xre-specific tests for the time being
8c0cb26 Merge missing header file
d8f8aca Merge changes to tools pmatch2fst and pmatch
87cc83a Merge pmatch parser changes
b51a7be Merge inclusion of hfst.h in HFST_HDRS
b6d5c39 Merge pmatch library parts from devbranch
786ec44 Improved error messages and fixed help message.
8e887c8 Maybe fixed undefined ref. to SequenceModelComponent::START_STATE problem.
b7caa51 Set revision for info
ffbd4f2 Newlines are now handled correctly when testing command line tools on Windows. Version and help checks also skip the tool hfst-info.
849f3df Further porting for info
d65900a Missing square bracket in define
536ebc8 Port basic info app to trunk.
2ab0507 Fix conversion bug where when converting to BasicTransducer, we would stop looking for a particular flag diacritic when seeing symbols that aren't epsilon or this flag. This caused misses of flags when they weren't arranged in a lucky way.
fc0dca3 Uh, actually declare hfst.hpp in headers and not files to get the meta
cdddfe1 Port some versioning and meta from branch
6f2af57 Replaced size_t format specifier %zu with a macro that is expanded to %zu or %Iu (Windows).
a86675b Removed debugging prints from HfstTransitionGraph.h. Added --help information to hfst-summarize.
d5084f0 Fixed python and utf-8 related problems.
126e249 Added preprocessing flag -DWINDOWS to Makefiles if HFST is compiled on Windows.
afc1e60 Streams to files in TropicalWeightInputStream and TropicalWeightOutputStream are now opened in binary mode. The global variable _CRT_fmode is also set to binary mode in file globals-common.h when compiling on Windows. This makes sure that streams are always processed in binary mode on Windows, with no extra or missing carriage return or line feed characters.
fe9eab0 Add Mac OS X abort trap to faqs
8218c9b Move "stuff that is next to colons" to another level to support quoted literals and some other constructs maybe
375ee88 Provide more info about itnae's in lexc for mac users
cba4f88 A convenience header file including HFST headers necessary for doing practically anything
ea4342f Added OpenFst and dlfcn functionalities as a part of HFST (at least when it is compiled on Windows).
afc1c34 Added a script for Monish guesser-analyzer test.
81c654e Installation instructions for dependencies that are not in our control...
223b5fb No alphas in trunk
3c0d30d retab!
86115d2 Add port and gentoo installation instructions
1d50ca0 Added reweighting tool.
e5cc768 Added config-file.
1be6c2b Added size() function.
0d8cfdd Added dummy-tags to guesser and lexicon entries without real tags, so that the could be parsed in the same way as regular lexicon and guesser entries.
4813945 Fixed output file, so that output is printed to stdout, when no output file is given instead of file "<stdout>".
93d16de Changed guesser tags.
2826e47 Some default-transitions were missing. Added them.
999fb85 Another #include <hfst/HfstTransducer.h> -> #include "HfstTransducer.h"
68a396b Added case sensitivity to suffix counts.
b9123f8 Added 'MonishGuesserAnalyzer' test for parser
8778016 Continued adventures in unsigned short reading: cast to unsigned char, not unsigned short, before shifting and adding the bytes.
aa6d431 (Regular) containment fixed in 'Regexp2fst'
c975b5f Fixed kitwiki addresses in the help-messages of hfst-guess and hfst-guessify.
c4e6d9e Fixed printouts so that they match for guesses and generated forms. Fixed printing of guesses, so that they're not printed reversed.
2bca1a1 Fixed hfst-guess help-message and exit value.
34be6db Implement a little-endian unsigned short stream-reader on top of the char-reader. Use it to read the header size rather than fiddle around with signed shorts.
429f813 Initial commit of hfst-guess utility.
bfb86fa Initial commit of the hfst-guessify utility.
ed7b61d Complement added to the 'regexp'
167a297 When reading header size, which on the disk is a two-byte unsigned number, we need to call stream_get(short &) because it's not implemented for unsigned short. Then cast it to unsigned short.
73db847 header_size needs to be able to handle MAX_HEADER_LENGTH=65535 bytes, so make it an int instead of a short
ab3de48 Excluded source files under windows directory from distribution.
f6b281f Curly brackets added to regexp
2e53d41 Appended $(EXEEXT) to program names in Makefile.
18a5340 Temporarily disabled 'hfst_mkstemp' in windows compilation, because it uses 'mkstemp' that is not standard C++.
92e33ad Fixed a typo in Makefile.
f3eaf25 Complete purge of trap_transition in try_epsilon_transitions
0165a14 When following epsilon index, set found_index *after* returning. Added rather verbose comment to try_epsilon_transitions to clarify when transition-taking needs to be trapped and when it doesn't.
9090b55 if -> endif in Makefile
6dbeb66 Now all Makefiles should be aware of windows compilation.
838b7d9 Added include flags to Makefiles when using windows compilation.
a1e0022 Made autotools more aware of windows compilation.
25dae67 Edited Makefiles.
17599d9 Added directories to windows/openfst and a script to fetch openfst.
b870284 Added a script to retrieve dlfcn.
216f22b Tentatively added support for windows compilation.
79cb942 Implement PCKIMMO output mode for fst2txt
b898898 Ready for release 3.3.12.
519d798 Tentatively added the possibility to compile hfst in Windows to configure.ac.
a2f1970 Maded changes in tests required by previous edit in WeightedStringVectorCollection.
5c7f95a Changed a bool to weighted_string_type in add_sequence function.
9dd0f5e Changed the bool lexical_model parameter in the constructor of WeightedStringVectorCollection to an enum.
9b684a7 Fixed comments
5f89671 Fixed initial comments
01daf8d Fixed statistics.
7a6ac4c Fixed statistics.
a644eb1 Added documentation.
d49bd99 Use convenient STL set constructor with InputIterator objects for returning alphabet sets
9bd0833 Maybe find unit_test_aux.cc now...
f4da277 Support StringSet get_alphabet() for hfst_ol(w) types
e09ccf7 use am_prog_ar for automake >= 1.12
2ef3838 include unistd for systems where it does not leak in
38af24a Handle the "only default transitions in state" case
39979b4 Now perhaps building unit_test_aux.o properly...
1ba1277 Build tagger tools only if the enable-tagger flag has been used during configure.
a3441eb Changed NewLexicalModel.h so that it includes the local header before instllation.
1f098c5 Added unit_test_aux.h to the sources of the test for string_handling.cc
692544e Added DataTypes.h to the sources of appropriate testa and hfst-tag.
62b23b0 Added version of hfst-tag.cc that compiles.
accb745 Tidied up code in use_model_src.
655db72 Now we pass all use_model_src/ tests.
ca283ad #include "HfstTransducer.h" instead of <hfst/HfstTransducer.h>, since often people will be building this without having headers already installed
7c8a1d6 Remove lexc tests as long as parser is not reusable
fc6488f Added skeleton for hfst-tag.cc and got all tests in use_model_src compiling. Some tests still fail.
dda2e74 Removed printout from the unit test of FstBuilder.cc
9d1942b Fixed WeightedStringVectorCollection.cc and string_handling.cc
0dc17bc Tidied Makefile.am.
e165644 Tests are now run using make check.
5e922c6 Added invert parameter to compose_intersect in HfstTransducer
b501a64 Added hfst-open-input-file-for-tagger.cc, which was missing.
cc9b901 Use intree headers for intree build
8f32733 Added some printouts to verbose mode.
278e9b1 Fixed issue with exit value.
017d653 Updated build files so that tagger tools can be built. They are not built by default.
fc1c501 Fixed some in input handling bugs.
c9f5a01 Changed comput_data_statistics.py to hfst_tagger_data_statistics.py
fcf3028 Some changes that actually allow compiling and installing hfst-train-tagger
3897533 Added test dependencies.
ebbf67f Fixed test defines in build_model_src/.
6cd2357 Added unit tests to hfst-tagger.
0fa55e4 Added Makefile.am files to hfst-tagger.
84f7434 Added src and test subdirs to hfst-tagger.
ba485b6 Initial commit of new tagger lib architecture.
e705d55 Copy input to output for default transitions
268a4b9 Allow reserved chars between quotation marks, avoid one more segfault on parser error
bae41c7 Add quotation example that breaks the legacy parser; use all tests on both lexc front-ends
f944ea5 Emit parse error on unopenable file...
df6fb6e Pass parse errors upwards from LexcCompiler
b8e33d7 Remove temporary lexc concatenations on successful exit
c948c23 Reword -q to "print only fatal errors and requested output" as it does
30a2947 Notice @_DEFAULT_SYMBOL_@ in other Alphabet contructors than the copying one as well
7a0372e Add an internal boolean found_index to keep track of whether an index has been successfully hit in the current iteration, and try to find a default symbol otherwise
8cbdcfa Make alphabet know about default
5a95634 @_DEFAULT_SYMBOL_@ and hfst::is_default()
16d45a5 Remove warning messages and XFAIL tests that may confuse users. Fixes bug #3525179
9e9e421 Added READMEs under BrazilianPortuguese 1 and 2 to EXTRA_DIST.
cbd7431 Updated Doxygen documentation.
a9e2b8a Ready for release 3.3.11.
6cc30cf Now hfst-shuffle is disabled by default.
4f3fcee Added the command line tool hfst-shuffle and functionality tests for that.
b455911 Tentatively added function shuffle under HfstTransducer.
eb6581f A nicer single production for INSERT
c560a0f Fixed a bug in SfstTransducer::random_path that happened when indexing states of the argument transducer.
bd37642 Introduction of semifunctional pmatch2fst
7870bf5 No plus sign in front of tags in CG mode, not even the special carry-through tag.
24449d5 Added experimental support for printing the raw analysis tag as part of the Xerox analysis. It is printed as a tag with a special prefix, and with the full analysis string enclosed in quotes. The idea is that vislcg, if enhanced to read the Xerox format, should carry  this "tag" through the disambiguation (and possibly analysis), and print it out on the other end. The tag can then be used as input to morphological generation.
9646a73 Print the raw analysis string as a specially prefixed tag at the end, instead of as a sub-reading in the beginning of the analysis.
92ee0bd Now hfst-fst2txt functionality test tests whether the program dot exists before using it.
d4a39a7 Changed call to the installed hfst-regexp2fst to the one in tools/src/.
22d4fcd Changed call to the installed hfst-regexp2fst to the one in tools/src/.
ab7257c Add containments and term negation, remove nasty windows formatting markers
f321bb3 Added the option --print-separator in hfst-fst2strings. The option will print a separating line "--" after each input transducer is processed.
70fe665 FomaInputStream::write_transducer and SfstInputStream::write_transducer now call fflush after writing the transducer. Now the command line tools should work correctly when pipelined.
f6c6432 The regexp [?] was parsed as [?|?:?] by FomaTransducer::define_transducer(const &string, const &string). This is now circumvented by using FomaTransducer::define_transducer(const &string) instead. This bug will be reported and fixed soon.
e4d291d Manpages are not created unless 'make man' is executed in directory man.
e47d8c0 Added support for inverting the composition i.e. computing intersect(rules) .o. lexicon instead of lexicon .o. intersect(rules).
37684b7 Added possibility to invert the composition in compose_intersect i.e. to compute intersect(rules) .o. lexicon instead of lexicon .o. intersect(rules). Also added added support for indetity symbols in the lexicon to compose_intersect.
71a2579 Removed HfstConstantTransducer as it is not used.
a9cbc17 Removed FinnishOTProsody from tests as it does not yet work. Also added HfstXeroxRulesTest.cc to EXTRA_DIST.
7cdf97c Temporarily excluded hfst-lookup from man page creation (due to a bug in help2man?). Also updated hfst-tail and hfst-head man pages.
4d54a06 Heads and tails with plusses and minuses
6b15fcf Test negative heads and positive tails
58723c3 Added Brazilian Portuguese tests to fsmbook-examples, they now work because missing files were added.
72d1a80 adding missing files
9daa986 Added missing files
31d424e Ready for release 3.3.10.
9bc5df7 Brazilian Portuguese tests disabled in fstmbook-examples to get a working release. They probably had missing files.
afce28b Updated man pages of binary tools as their help messages were changed.
70d24a5 Added documentation to the binary tools about the restored feature that the first input can have several transducers while the second has only one.
7c2399f TropicalWeightTransducer::extract_random_paths now also calculates the weight of each path. Fixes bug ID 3514912.
67a377e All binary tools now handle the case where the first input contains several transducers and the second input exactly one transducer.
e23d373 BrazilianPortuguese2 fmsbook example added to tests
f941d02 BrazilianPortuguese1 test should pass now
6ad4d9a hfst-compare now handles the case where the first input contains several transducers and the second input exactly one transducer.
c55bc43 hfst-compose now handles the case where the first input contains several transducers and the second input exactly one transducer.
abbff8c hfst-summarize now prints at the end how many transducers it has read in total.
d765699 Added documentation to different cases that are gone through when checking binary parameters.
e32b8a2 Changed the variable name stdout to stdout_ to avoid namespace conflicts.
1bc12a0 forgotten Makefile.am
8672375 Configure file had to be changed to compile fsm book examples
84850a8 Fsmbook examples update
c1f2510 Fsmbook examples update
7eb574c In xfstRules, removed TMP marker from the result alphabet
610da46 Updated hfst-compose-intersect help message.
dad67cc Appended namespace std:: in front of vector and pair where avoiding it might cause compilation problems. Also changed calls to 'rindex' to 'strrchar'.
74fe14d Doxygen for the speller and some ol internals
c3f629e Forgot to commit ChangeLog and NEWS, now ready for release 3.3.9.
e55c21a Ready for release 3.3.9
34460a6 Fixed help messages of tools.
814a1a9 Changed hfst-lexc so that the implementation format defaults to openfst-tropical.
7b97b48 Changed expected output to reflect current cg printing
97a0786 Integrate ospell, first commit
2624bf6 updated so that tests don't fail
20ab796 updated one fsmbook example, still not working properly
9b826c1 Double greek PI met opposition in #apertium, hereby changed to double dagger instead (‡). It is still open whether the raw analysis string should end up in this position at all.
03772d6 Fixed a bug in command-line option parsing, where -e conflicted with all case-handling options.
c96e52e Fixed bug wrt option counting
a708855 Added a new option to turn on printing of raw analysis string in CG format output. Default is to not print it.
7c8830a Replaced triple ] (]]]) with double ∏ (∏∏ - capital greek PI) as the CG sub-reading separator. It turned out that inserting +]]] into the transducer was not working very well.
f06aac3 Deleting the + char in front of GT/Divvun tags for the CG output. Also, there is no problem with printing the raw analysis string, all is fine (contrary to what I said in my previous commit). This tool is now pretty close to what we need in the Divvun work :)
990581e Experimenting with the CG output mode. Using the -C option, the output will now include the original lemma+tags string, including all tags within compounds etc. It is printed before the CG-formatted tag list, and separated with ]]] (suggested CG sub-reading separator). For some reason, the tags have lost their initial + sign at this point, so the result is pretty ugly and unusable. I don't know why or where - feedback appreciated.
f3c746f Make sure the outputs are sorted before filtering and printing
4905d54 Corrected key string for getopt_long
05fe312 Corrected condition in LookupPathW::operator< (or at least made it more understandable)
58c61e5 Correct terminating condition
ec52c63 Stop before end of output iterator
ae8461f Clarified comment
074a25e Added an option to only print analyses belonging to N equal-weight classes
6844c9d Typo in error message
9792641 Adding identification and processing of GT-style tags beginning with +. When formatting for CG output, this is important, otherwise the output is more or less useless. This change should probably be enhanced with tests for the requested output format, so that it does not contaminate the Apertium analyser. Please have a look.
756d724 indentation
3e37af1 Added a quietFlag variable, which makes the -s/-q flags actually useful. Using either -s or -q the warning "Warning: Transducer contains one or more multi-character symbols made up of ASCII characters which are also available as single-character symbols." will be suppressed. There might be other output that should also be suppressed, but that is not implemented yet.
784bd21 Removed "e.g." to avoid ambiguity in the interpretation of the output.
cdb80a2 Boundary mark in contexts is beeing handled now.
e999674 Xfst rules parser should be working now
8d53567 regexp updated, precedance seems to be fine now, but parallel rules with one comma and context still don't work.
9359776 FomaTransducer::extract_paths bug fixed, now it does not produces duplicate paths.
76e4f7a Ready for release 3.3.8.
b55429b Add expected arcs per symbol statistic
410a9ec Ready for releasing version 3.3.7.
8292ce4 Updated tool man pages.
939b6a9 Updated hfst-regexp2fst help message.
3df0bcb Fixed an error in composition introduced by the newest version of OpenFst (1.3.1).
bb6e2fd Fixed identities in regexp parser, now [?] is parsed as identity pair.
8f950ad Updated regexp2fst parser - rules work fine for themselves, but there remains precedance issue.
2512805 Ready for HFST release 3.3.6.
3d290d3 Changed include directives of implementation headers (foma, sfst, openfst) so that they search for headers in default include path first. Also modified memory leak script.
69f0619 Updated memory leak test.
5ece7d7 Tentatively added a script for testing memory leaks.
cb48f49 Fixed memory leaks caused by hfst_get_name in command line tools.
679d8ca Fixed a bug in hfst-lookup man page generation, now option --no-discard-stderr is used in help2man. (However, cannot figure out why this helps..)
942a563 Removed yet some extra code.
b50e384 Removed commented code.
6e87a25 Fixed some compilation warnings and removed commented code.
bf0508d Fixed some more compilation warnings.
1ebf3aa Fixed more compilation warnings.
3fd5aee Fixed some compilation warnings.
b802621 Ready for HFST release 3.3.5.
e096676 Man pages updated.
1aa5686 Modified Makefile.am.
2de5917 Added some comments
d155bca Fixed a small bug in hfst-fst2txt functionality test.
e6560e2 Output dot/graphviz
967a5be Maybe finally the reversion is working..
5373734 Made yet some changes to revert to the older revision.'
b4318de Reverted to revision 2076.
1101e6a Forgot to add the new files for BooleanWeightTransducer.
21b3949 Tentatively added a new HfstTransducer type, BOOLEAN_WEIGHT_TYPE.
6e99a0b Fixed a small error in the options handling part of the help message for hfst-strings2fst.
8514c03 Added new man pages for hfst-name and hfst-format and links to man pages for tool aliases.
76d8934 Now the man pages also have a short description of the tool, i.e. the one given by hfst-foo --help.
4f996f5 Added a test case for the bug fixed in revision 2016 (out_of_range thrown by substitute function).
5e80597 De-spanishify hint string ("argumentes" -> "arguments")
d29274e Now HfstTransitionGraph:substitute(const HfstSymbolSubstitutions & subs) adds all symbols in subs to the global alphabet before relabeling, thus avoiding the out_of_range exceptions thrown earlier by this function.
467599e Changed hfst-compose-intersect so that it gives a warning (1) when the rule does not contain the identity symbol and the output tape of the lexicon contains a symbol not found on the input tape of the first rule, and (2) when the output tape of the lexicon contains multi character symbols not found on the input tape of the first rule.
6bbf8e1 Removed spurious consts from optimized lookup conversion functions. Fixes bug ID 3469227.
7315fe4 Bug on mac os caused by its sort order (bug ID: 3462485) fixed.
e9f37ff Added function write_in_att_format_number. Also fixed hfst-optimized-lookup so that --version and --help print to standard output instead of standard error.
2284d13 hfst-optimized-lookup softlink and some package strings for hfst-optimized-lookup.h
4821cdf Heavy-handed bundling of the hfst-optimized-lookup standalone
a7e3dd1 Rules parser - added epenthesis
4f70b1d Still needed to fix some errors caused by in-place sed.. Now everything should work in commandline tools tests.
33130ab Fixed problems in test/tools that came after updating the Makefile.
bc413c7 Removed commas from filename suffixes (e.g. .hfst.sfst -> .sfst) in test/tools. Also replaced compilation rules using %-symbol in Makefile.am with old-fashioned suffix rules. Now commandline tool tests should work with older versions of make as well.
43082be Ready for HFST release 3.3.4.
5b163cb Added some documentation for HfstSymbolSubstitutions and HfstSymbolPairSubstitutions. Also updated the doxygen documentation.
53c7b7c Now HfstTransducer supports making several substitutions at the same time. Also made a test for these functions. Adding documentation soon.
7aa7026 Tentatively added a symbol-to-symbol substitution function to HfstTransitionGraph.
e9db3ce newer version of the rules parser
31446ff weighted expressions
668878f added some xerox rules to xre_parser (hfst-regexp2fst). Also,  slightly modified xerox rules
aed6b50 Remove some debugging printout
45c7b45 Merge xre precedence fixes
81ba993 Now const_cast<char*> is used instead of strdup/free in ConvertFomaTransducer::hfst_basic_transducer_to_foma. Also modified hfst-substitute so that the alphabet is not pruned after substitution.
cff563a hfst-substitute now implements the option --in-order.
5020e2d Fixed bug, which prevented compilation of grammars with both set constructions and : or ?.
0aa972c Unit test for grammars which have both set constructions and : or ?
0183928 FomaTransducer::repeat_n now takes a copy of its argument transducer to avoid double delete caused by new foma interface. Fixes bug 3468373.
253b0fe OpenFst linking bug is fixed in 1.2.10; also gqipped some paras
db1e066 Warn end users about disabled "important" tools
93c0be8 Xerox rules added to the library
a6b1982 more alphabet hacks
2f13574 more alphabet hacks
d41ccd2 Added new substitution functions. Now it is possible to perform many string pair-to-string pair mappings at the same time on a transducer. Also tentatively added a functions that performs many string-to-string mappings at the same time.
b19e466 Now the test EinsteinsPuzzle uses local tools instead of the installed ones.
f574cae Now the test FinnishNumerals uses local tools, no the installed ones.
dfcd75d Conversions between HfstTropicalTransducer and HfstBasicTransducer also made more modular.
e64a2eb Made conversion functions more modular by moving code to subroutines. Also removed commented code from files.
ab8c339 Now conversions between transducer formats openfst-tropical, sfst and foma are carried out using numbers always when possible.
4defd16 Tentatively added functions for fast conversion between transducer formats.
90fd45f Now also conversion from foma to basic transducer uses numbers. Some commented code also removed.
45cfa97 Some conversion functions are not carreid out using numbers directly.
0335646 Renamed HfstTropicalTransitionData variable 'number2symbol_vector' to 'number2symbol_map' to avoid a strange linking error. Also removed transducer conversion functions that are not used.
d964d9c Removed some extra code.
1f9516e Tentatively added alphabet pruning to substitute functions, it still has to be removed from hfst-substitute.
6e4838f Conversion between HfstBasicTransducer and TropicalOpenfstTransducer is now more efficient as it is done using numbers directly.
fc65b31 Changed conversion from HfstTropicalTransducer to HfstTransitionGraph so that it uses transition numbers directly. This improves the performance greatly.
6fe2006 Fixed bug which made tests always pass.
7f0d49a Ready for HFST release 3.3.3. Also disabled DateParser test because it relies on hfst-calculate.
a2b7ede Fixed speces so that the space symbol is used in the transition instead of @_SPACE_@.
8bceff5 Skip copying the foma transducer name to the net, leaving an empty string. Rationale: foma has a limiting 40 char ceiling on names, and we might as well use HFST3-style names for everything.
e90d002 Removed hard tab indentation from hfst-txt2fst.cc
5d4e1ab Replaced tabs with spaces in hfst-lexc.cc
932a780 Fixed HfstTransitionGraph:swap_states so that the variable new_target is no longer used uninitialized.
b85d91d DateParser test fixed, the path os hfst-calculate was not correct.
7de913d Fixed ComposeIntersectLexicon test
97e620d Now the escape '% ' should work everywhere.
abebc48 Added test for rules with space symbols.
f7341b2 Added a conversion test and made the LIBHFST_TST actually run, causing test failure for the time being
69b7b7b Optimized HfstTropicalTransducerTransitionData::get_symbol(...)
aabea1c Call sizeof() on type names, not variable names due to OSX compilation thinking the variables are non-static uninitialized fields. Fixes #3445592.
5024ecd Fixed (serious) typo
f8f2dfd (re-)added a char* constructor for tables, made it the default to only read disk once per table
6754159 Eliminate use of non-portable strdupa(const char *). Fixes #3444950.
43cc4e9 Removed some garbage
5ffe969 Removed some of the unnecessary work done in tokenizing for ol-lookup
0c09d91 Fixed a name shadowing bug
bca34ef Added a lookup_fd(char *), reduced code duplication in interface functions
c9ff453 Added note about current OpenFst / Ubuntu problems.
0ce433e Fixed error message.
c289dcd Now terminating compilation, if an empty pair is found in a defintion, rule center or rule context.
652f927 Ready for release 3.3.2
aefa028 Use NumArcs as a member function instead of using NumArcs from the OpenFst internal namespace.
366a483 OpenFst has moved NumArcs into the namespace ::internal. This commit reflects that, fixing compilation with the newest version of OpenFst. NOTE: THIS BREAKS BACKWARDS COMPATIBILITY, you need the newest OFST to compile after this commit.
67a5402 Fixed svn properties of executables in fsmbook-tests.
128e390 Remove PKG_ macros from end user builds since they do not work nicely on all platforms.
0deefb7 Tentatively added to fsmbook-examples tests where the results from foma and xfst are compared.
fff58c2 Now all tests in fsmbook-examples use the tools in tools/src/ instead of installed tools.
955e760 Tentatively fixed parts of fsmbook-tests where intalled commandline tools are used instead of tools in tools/src/.
b6a2bd3 Ready for release 3.3.1
3a9f7c8 Fixed memory leak in HfstTransducer::get_property
48232c1 Remove escape sequence __HFST_TWOLC_ from symbols in rule names.
37800e6 Changed flag diacritics handling in hfst-compose, hfst-conjunct, hfst-concatenate and hfst-subtract. Now flags are harmonized using HfstTransducer::harmonize_flag_diacritics(...).
9313133 Added a function for querying if a transducer has flag diacritics and added a parameter in harmonize_flag_diacritics for diabling insertion of missgin flags (needed for hfst-concatenate).
e207282 Added function bool HfstTransducer::has_flag_diacritics(void) const
47f9d3e When looking for indexes with a particular symbol (which is used for converting to other transducer types), handle flag diacritics specially as per their indexing rules.
8eb758b Added HfstTransducer::hamonize_flag_diacritcs(HfstTransducer &another)
f2c0a98 1) Report the correct indexing offset for epsilon as 0 even when there aren't     any epsilons 2) Put the exception about trying to find the offset of absent symbols in     the proper scope so it actually gets thrown
cafa28f Tentatively added to fsmbook-examples tests where results from foma and xfst are compared. Also fixed a small bug in remove-complement.sh.
3431ec3 Extra files created during tests are no ignored by svn.
5f9386f Added flag diacritics to the epsilon filter.
0780530 Added missing file.
f76ac72 Added a note on configure enable/disable switches to README. Changed fsmbook-examples so that now all examples that can be compiled with the current HFST tools are included in the tests.
9c006b5 Forgot to commit configure.ac last time, now automake should work.
e437010 Made some modifications to fsmbook tests.
a651082 Same leak issue as in r1846
b2b4ba8 free() char*'s that get strdup'd in a loop (which leaks memory if not free()'d).
edcc8e6 Added missing file.
f24568b Made small changes to the tests under fsmbook-examples, now make distcheck should work also when scripts in an upper-level directory are used.
e167573 Tentatively added one subdirectory of fsmbook-examples to the tests performed in make check.
31af546 Tentatively added some testing scripts to fstbook-examples
23978b1 Remove spurious space character from error message
26fac37 Give more guidance when ejecting the user for having a non-UTF-8 charset
a4981ce Comment out fsmbook-subdir; it's broken
47a618d fsmbook-examples are now included in tests.
08e8edd Ready for HFST release 3.3.0.
e3819ad missing  added to some tests so that they pass distcheck
3b0667c lexc-functionality updated
511b33b make distcheck should now find the missing files
b4f2e58 remove hfst2 broken flags from main tree
dccfa4f Remove experimental, hard to compile and dependent programs from main compilation
1bad1e8 Added missing files to Makefiles. Also replaced HfstExceptions.h with HfstExceptionsDefs.h in include declarations.
f29786a Option -s in hfst-lexc and hfst-substitute now suppresses warnings as it should.
a0e51bd Made yet some tests less verbose.
fdaba54 The option -s in hfst-lookup now suppresses warnings as they should.
6a4461e Made tests less verbose.
0c79b59 Testing if a library exists is now done with the command 'hfst-format --test-format'.
a29cc4f Now all tests in test/tools should pass.
667fb48 Fixed a bug in hfst-format. Also fixed some tests so that they don't assume that all backend libraries are available.
9522737 Indicate whether apertium2fst is enabled
32d7528 A spurious space was apparently preventing libhfst.la from getting built (!!)
9fe6a3d Fixed some small bugs.
373ab19 Added a missing file.
87d5c05 Now all Esperanto tests work.
311297f Now EsperantoNounsAndAdjectivesWithTags passes.
5703bf4 Mention both forms of <SFST-1.4.6 error messages
1ccdbf2 Fixed some bugs.
1a85f7d EsperantoNounsAndAdjectives works now.
dbe3858 Remove hfst-lexc and feature check that requires newer autotools from default builds
c8bc7d0 Updated the tests.
3dab55c New tests.
1158256 HFST scripts for Monish tests ready.
fd3adfe Fixed a bug in the xfst file.
1f57439 Adding a new test.
3cb051d Added files to MonishAnalysis
5275de0 Adding a new test case
f306463 Added an alternative solution to Brazilian Portuguese.
65521fb Now unicode characters are transformed correctly from xfst att format to hfst att format.
6f3e3f2 Added new test
bb769eb Added a new test
7b5175a Fixed some tests so that they work when only openfst is installed.
e44b704 use const char * in bison's error handler
2a8afa1 Fixed tests in test/libhfst so that they also pass when only OpenFst is installed. Also changed HfstTransducer::convert so that exceptions are handled correctly.
4e85a00 Ultimate fix to implementation type availability checking in twolc unit-testmake check
9df0c5f Tentatively fixed unit-test in OtherSymbolTransducer.cc
ba60bbe Added checks for existence of implementation types to unit tests.
3a313b0 Added support for new backend format strings in commandline arguments.
7007722 Added check for existence of backend formats to tests.
3465429 HfstTransducer::is_implementation_type_available changed from protected to public. HfstTransducer and HfstRules tests now work also when all backend libraries are not available.
298d1f7 Only respond to nick + , or nick + :. Fixed flood-handling.
d04b936 Update instructions for easier svn source
e38782a Disable problematic backend libraries from default build
3157352 Remove all version requirements and dependencies and tools that require versions, features, or dependencies to support older tool chains and systems
bbb40b1 allow modifying only end state
65e2029 Fixed character coding in Finnish examples.
7ed6905 YaleShooting test complete, rules must be implemented in HFST commandline tools before it can be tested.
48bc808 Changed 'Palindroms' into 'Palindromes'.
ca49d0c Add tool for modifying weights
e737f9d Now the test passes also for foma implementation type when revision 47 of foma is used.
47696bd Added test files.
5952156 A bug in HfstTransitionGraph::swap_states fixed, now final states are swapped too.
f41b96e Added documentation
42e26a2 alias old proc
098f6a3 add bugs
32eb73d Missing -r option
1b3e724 document input format requirements in the beginning
fc22386 few less apertiums in directories
d01f072 s/proc/apertium-proc/g
62c5a1f Added a test case, Lingala. The test is not included until an issue with writing transducers in AT&T format with the limited licence of xfst.
cc616c6 Added a new test, FinnishOTProsody
3fd80a9 Added a new test, PlusOrMinus
a5b9ea0 Fixed a bug that occurred in binary operators when the calling transducer and the argument transducer were the same.
ef31d2f Added more fsmbook examples. The tests have to wait until we get rules in hfst-regexp2fst.
00da234 Terminate xre parsed quoted string, fixes 3400657
444dffd Declare BUILT_SOURCES
5ccdf40 We are one step closer to get the tests in test/tools/fsmbook-examples under autotools..
33db26f Added Makefiles to fsmbook-examples. The tests in fsmbook-examples are not executed until they use autotools.
97d473c Now the tests do not depend on the tool hfst-xfst. Instead, they use hfst-txt2fst and a transducer in att format.
0d6b28e A bug in FomaTransducer::write_net fixed. Now the the function fsm_count is called before writing the transducer so that the number of lines is always correct.
8d28394 A testfile added.
cd193f6 Use basic transducer and trie disjunction
91754de Works minus regexps
1b73d6e dl before openfst though
86921ca do we still need pthreads and m?
0c4b53f Yet another missing header and using
b5671cd Removed file training_data, which was accidentally added.
60e4777 Initial add of hfst-tagger. A lot of stuff is still missing and the tool needs to be integrated in the build system.
c284259 tests modified
9799eb2 some of the tests rewritten
01f60b6 Now the tests are performed for sfst, openfst-tropical and foma types.
f60e580 typedef HfstTransducerPairSet removed from HfstDataTypes.h because it is not used anywhere and would cause problems because HfstTransducer::operator< is not defined
9c47fc1 Now HfstTransducer::operator= works for the case of optimized lookup transducers and copies the name of the transducer for all cases.
efa33f5 copy props in copy constructor (excepting type); fixes bug #3405831
02e7be9 make compilation work without inclusions leaking from backend headers
16b4d3c add include for std::map
461af24 add include for std::set
2f44bbc apertium2fst tool added
ca39ff3 Avoid writing name, version or type twice
8119d69 Added necessary std::qualifiers. (With --without-sfst some using-directives apparently disappear.)
1917eee Fix UTF-8 bugs in strings: * use error_at_line instead of error when linens are known * remove bogus double try blocks * throw UTF-8 errors with messages
9e80b3e Fix formulae for filenames and data starting with non-ASCII
62ce06d Revert tab-containing commit r1576
b3f24d8 fix missing paths in tests (would've used $PATH instead of newly builts)
7e11d85 Use new metadata handling in tools * fix signedness in -grep * fix erroneous error printout in -compose-intersect * add few features to summarize
b2b3010 * common practices headers and utils for metadata in cli tools * a simple cli tool to edit metadata
9829fce metadata header name registry
ab02d03 Make use of HFST3 transducer format metadata: This is basically upwards and downwards compatible change with HFST3 automata and header parsing, but be cautious that everything may break at random
0857f80 Added Beesley and Karttunen's fsmbook examples to the tests. The fsmbook tests are not yet run when doing make check.
5fe1844 Fixed bug that caused hfst-twolc to segfault when variable rules with keyword matched had unequal value lists. Now it will give an error message and terminate compilation.
6fd297d Cross product added to the library
cc5671e Universal pair update and added identity pair
459bcaa A bug in transducer conversion fixed.
711df75 Don't do foma-SFST conversion if we don't HAVE_SFST
7317a58 Warning about broken foma handling
09419fa More accurate comment
3993996 Do foma-sfst -conversion for substitute also (not just pruning); HfstTransducer doesn't know about foma's substitute.
c3c3dbc Typo
687639e Swing foma substitutions through sfst to prevent horrible leaks
24a982d Don't leak 1 (!!) byte of empty string when constructing foma transducer
727bcd3 Fixed mismatched free/delete when freeing foma transducers.
f199f2b Don't leak "fallback" transducer for each label
a53ea7f Strip server junk from username reply
f66229d Fixed wrong exception name
3894ab1 Make composition with OL throw a Mismatch exception, which compose is looking for (it still needs to know about the second tranducer being OL)
2029017 add 'using std::set' to fix compilation problem in hfst-affix-guessify.cc
78a820a Standardized the transducer format options of commandline tools.
544cc06 MAde ComposeIntersectRulePair::get_transitions virtual.
5c6e7e4 made size_t size(void) a const method.
de981e7 Added method ComposeIntersectFst::get_symbol_number, because HfstTropicalTransducerTransitionData::get_number is not accessible from outside the library.
86d4582 Edited Makefile.am so that it installs compose intersect headers properly
5330f22 Fixed bug in tokenizing string with lonely backslashes. Backslash can now escape any symbol and it is ignored when it occurs as the last symbol in the input.
719f6db Updated Rules - left replace down karttunen + tests
16dc6a6 Whitespace, punctuation, cleaned double conjunction.
2c63a89 Updates on building on the Mac.
f5d1d51 iterators can no longer be used...
921c6ab Update SFST requirements
00217c4 Resurrect the affix guesser maker
6c5a5fb Flood protection for channel messages
3cf20f7 Improved message finding and private messaging
2518451 autoconf doesn't understand dnl without space?
72f9a36 apparently mac os x "readline" does not support rl_completion_matches...
c1c9e07 HfstTransitionGraph::iterator is now protected, the public HfstTransitionGraph::const_iterator should be used instead.
894ef7b Replaced all mentions of HfstBasicTransducer::iterator with HfstBasicTransducer::const_iterator in OtherSymbolTransducer.cc.
75f4158 Changed syntax of negative context rules.
e7bbb19 Modified OtherSymbolTransducer::harmonize_diacritics, so HfstBasicTransducer::add_transition is used instead of adding transitions directly to the transition vector.
5bf71cb Enabled sfst tests.
9712ab8 untabify recent changes
5a7c0e8 Updated HFST so that it works with the newest version of SFST, 1.4.6.
feee342 detab hfst3 tree - feel free to revert (and report) if this causes any problems
fb6ae1c add few trivial features to list
1084eab Be more verbose about type mismatch
4e99e3c check for empty strings
53ba386 Modified test script so that it runs tests for all backend formats. Disabled tests for sfst and log-openfst formats, since sfst segaults and log-openfst gives weird results. These are errors from the library.
495b307 Added test for negative rule contexts in rules with variables.
251fd1d Added negative context rules.
f6a78c2 Added tests for negative contexts in twolc rules.
22e60a9 Now HfstTransducer::operator= throws a FunctionNotImplementedException instead of a TransducerHasWrongTypeException.
6fe2113 Fallback to anything or nothing from settings; resolves bug #1061990
3323325 kind reminder about importance of the README
c7e5b9f Some logging changes
25d4d18 end readme in nl
25d328a Save misses to logfile
bd2ec42 Send notice if word is unknown
29b3b9c Don't print ugly weights
f67e60c Just dump privmsg to console and don't do anything for now..
3bec150 hfst-bot demo
ba26d48 Lookup demo
e36e700 Full api and distutils-based build & install
453163e clarify SFST libraries relation to HFST; thanks to spectie for the suggestion
7803cb8 * Save temporary files for foma in /tmp/ * import old readline UI from 2nd branch * deprecation warning if input = stdin * error if output = stdout
a03e3a2 Another readline editline libedit fix maybe
bd5472b CHECK_DECLs to support mac os X's fake cripple readline
2ce7d2d Tests of replace functions now work for transducers of foma type as well.
0b8ab2c A new function rules::replace_down_karttunen added. This function returns the actual downward replacement rule transducer instead of rules::replace_down that returns a downward rule transducer as defined in SFST-PL.
b113008 New target for root makefile: valgrind
aaf0734 >=autoconf-2.62 not 2.22
b942a23 Remove libtool 1 support tricks, more trouble than it's worth
86718c2 Mark version requirements in autogen
c5094f1 Move version requirement to beginning of the line to be sure...
95bd525 debugging tool
063de32 A bug in function random_path fixed
77c7bed help messages to message_out instead of stderr
a9faa38 Do not cut unknown word forms from first non-alphabetic character in analysis, test case added
700c106 Fixed a piece of documentation on HfstTransitionGraph and added a test case too.
e9f9a79 Now extract_random_paths works for an empty tropical transducer. A test case also added.
89e4743 print more than fstinfo only if verbose
0b38d0b Fix arc-based leak in conversion
68c0d39 Make old lexc default lexc again; fixes: 100000000 Mac OS X bugs
637ea55 allow options to be passed via env.var. HFST_OPTIONS
3cf23c8 works but leaks
77a1c0d and default to glib to test if end users are ready for that in next rel
20c88c9 Release changes
6ccfd1c delete created test files
ec44172 include new proc tests in dist
25d6530 extra dist m4 macros
ce67ad2 fix how compound words are generated
6c78061 try generating first without compound interpretations; possible fix for bug #3290585
b71cb2e do not pipe error messages from tests to /dev/null
a8b0720 fix some -a tests
267a212 add valgrind to stress test targets
4af5dca Add a blurb about Unicode support requirements
b2f03ac calculate spaces in length of print forms; fixes #3296854
0bd58aa Allow use of glib for proc unicode casing (most likely breaks all Macs because of pkg-config and all that)
0662084 Implement type mismatch error messages for all binary tools
dec54f2 add quotation and neat printing in twolc tests
6048824 incompatible format messages in composition
f43dfaf document new tests in troubleshooting with examples.
dc5c1c5 Move latin-1 string tests to XFAIL
c30f3d3 Neat error output when incompatible formats; bug #3287458
eb6e0ed Troubleshooting for Mac OS X's getopt
0325c9f Check for GNU compatible getopt, try to fallback to broken getopts more aggressively on Mac OS X
23223af Implement print-sigma and acceptor
4a279dd include new command line option in man pages
8e4177d Disable reserved characters and escapes in raw mode.
3b5d30f remove roundtrip test: mac os x sed does not support new lines
6ce1996 hfst-lexc around lexc_read
df3b87e more alphabet hacks, i should really fix this properly as opposed to just patching the bugs i find
83fe100 New analysis mode for finntreebank people: * no lowercase * no uppercases * no compounds * probably some ohter things
c46b8e8 Detail all external backend libraries in troubleshooting section also...
332fb8d here here
0a94bbc possibly hippu-compatible alias installation
917ae73 Now HfstTransducer::substitute(StringPair, HfstTransducer) does not modify the argument transducer. A test for this case also added.
795926d Provide preliminary autoconf scripts for HFST
52f0884 Replaced HfstFastTransducer with HfstBasicTransducer in conversions. This might make HFST slower, but it is needed until the bugs in alphabet handling are fixed.
2fd9cfb When performing transducer conversion through HfstFastTransducer, symbols leaked from one alphabet to another. Now HfstBasicTransducer is used instead and a test case is added that will fail if HfstFastTransducer is used instead. The original bug is to be fixed at some point.
9779c9b Added a test case for replace_up with FOMA_TYPE that fails. Currently it is commented so that make check will pass, but it will be fixed soon.
12dfacf Now binary operations HfstTransducer::disjunct etc. do not modify the argument transducer.
f933bbb HfstTransducer::insert_to_alphabet and HfstTransducer::remove_from_alphabet now pass the tests.
25fd77f Added function HfstTransducer::remove_from_alphabet. Currently throws FunctionNotImplementedException because it does not pass all tests.
5447d49 verbose test failures
1085c4f Documentation updated.
c6d0e14 Removed check for null-termination of string in the test of string_mainpultion.cc, because it fails on Mac.
49f1dff Now distcheck passes the tests but still complains about att files..
e884243 Reverted r1418 - these files are supposed to be here, and 'make clean' is over-cleaning.
e7725b4 Deleted supposedly generated files - they were removed by the make clean target in test/.
6225431 assert fopen != NULL
cf16a6f Modified compose-intersect, so that it will treat only flg diacritics not found in the alphabet of the rules as epsilons.
80c2c7a A small bug fixed in the newest version of foma in the foma svn. Now the FomaTransducer test works correctly.
ea2f619 Code reorganized. There seems to be some errors in openfst's minimization, temporarily fixed the situation by pushing weights before minimization.
5aca45b Retain root joiners while applying morphotax
afa9e11 kill some warnings by settings
16eabd9 New NEWS
ae3d193 New releasenum (3.1.1) and libvector (6:0:0)
aa7bf17 Changed the test script so that it removes temp.twolc.hfst0, when it's done
97268c2 Fixed typo where $SRCDIR wasn't getting assigned to the file we want to test
077da45 Changed [[ ... ]] in regular expression center rules to <[ ... ]> to avoid introducing syntax errors in previously working twolc-grammars.
7d9a60d Added new tests in Makefile.am.
97963cf Added support for rules which have centers that are regular expressions of pairs.
9ebc8b9 Added tests for rules with regular expression centers.
b583a0c Improved test script so it won't exit normally if hfst-twolc crashes.
e358dbe The public and protected interface of HfstTransitionGraph revised. Unnecessary comments and debugging code removed from HfstTransducer.
e1ef196 class HfstTransition moved to file 'HfstTransition.h'
147b99d Substitute functions in HfstTransitionGraph should now be easier to read.
add8e68 Code of HfstTransitionGraph made clearer, a bug in including HfstTransitionData.h fixed.
dc9da7b Code simplified.
4ad9718 Transducer conversion done through HfstTransitionGraph.
f23ba67 Included <algorithm>...
3856fdc Changed member const HfstBasicTransducer &t to HfstBasicTransducer t in ComposeIntersectFst
96e5e02 Fixed bug in ComposeIntersectFst which forced all states in a transducer to have the same final weight as the initial state.
6b74eb4 Sorting the arcs of the argument transducer in the constructor of ComposeIntersectFst, because they need not be sorted nowadays.
0a5bcef Added arc_sort member function to HfstTransitionGraph.
2f9d3a9 Some additional experimental container-hacking layer stuff
e238723 left arrow rules
f49a6e1 some more hacks to alphabet.cc
6a3b642 Added support for rule centers which have the form 'X:a' where 'X' is a set and 'a' is a symbol.
972561a Added tests for rules with centers 'X:a' where 'X' is a set and 'a' is a symbol.
475b6f9 Undid some changes that stopped working after the interface of HfstBasicTransducer apparently changed. Also changed HfstTransducer OtherSymbolTransducer::get_transducer(void) to const and added the function void OtherSymbolTransducer::get_initial_transition_pairs(SymbolPairVector &pair_container) const.
48487f2 Made max() and is_subset() static functions.
83cfb8a hfst-twolc unit tests added.
3bdda4e New NEWS
d04a96b New release number, 3.1.0 release and 5:0:0 libvector
420e273 Removed an extra debugging print
c23e919 HfstTransducer::insert_freely(HfstTransducer&) now uses HfstFastTransducer
1bc6962 check locale in some tools
f063366 HfstTransducer::insert_freely now uses HfstFastTransducer for foma type. More tests also added for insert_freely.
5490095 Conversion functions between HfstFastTransducer and backend implementations added. Now HfstTransducer::convert also converts through HfstFastTransducer. HfstFastTransducer will gradually replace HfstBasicTransducer for HfstTransducer functions that require conversion.
79722bf HfstTransitionGraph now has a vector of states instead of a set. All programs and functions updated accordingly.
3b908de Reword slow lookup warning
ccad335 An apparently functional way to expose sets by copying them in order to vectors
8708944 Fixed typo
92ef298 Fixed includes in XreCompiler.h and LexcCompiler.h
8fa2481 added some functions to ConversionFunctions that will be used in fast conversion between transducer types instead of HfstConstantTransducer
0aea7c9 Fixed test script so that it should work in distcheck
6ee3d24 Made lookdown functions consistent with lookup ones, updated docs
1c9307d Updated doxygen to reflect changes in lookup functions
5f28355 simple archive tool for duplicating repeats
a43ca61 Added a faster version of lexc compiler for a transducer of foma type.
228feee decrease stress by order of 8
b15d21d Keep information about the parent FdTable of an FdState in the class to avoid invalid reads if it gets deleted
153862d Changed lookup functions to return pointer to HfstOneLevelPaths, and take no argument for it. Calling functions are (still) responsible for freeing memory.
577e62c Add more common problems
cd64166 Added HfstTransducer::get_symbol_number, it will be used in faster conversion between transducer formats.
5588a3b A small bug in testing the EmptyStringException fixed.
bda6a25 EmptyStringException is now thrown if the empty string is used as a transition symbol.
6923dc2 Universal pair function added
92ece0d Need more inclusions and usings when they don't leak in from backend libraries
ed3be66 Pathnames of testfiles in test/libhfst are now given as getenv("pwd") + "/tesfilename"
a5e5bce Tests in test/libhfst now create the test files in EXTRA_DIST, so that distcheck works. This is a temporary solution.
7c1efd3 Remove obvious memory leaks
e8bb168 Added missing test files.
2241e2d Prepare 3.0.3
552dadf New NEWS
0bf8b35 LexC and regexp functionalities moved from tools/src/parsers under the HFST interface at libhfst/src/parsers.
f13146e Link with libhfst/src/libhfst.la (is this portable?)
5c8d213 allow debianised names for libraries and headers
348aaed Global wrapper - could it be this easy?
e8d8870 Fetch headers from srcdir instead and assume libhfst is installed in the sense of ldconf
34f8ddf Fetch headers and libraries from a (presently) hardcoded /usr/local
81cd53b Testing a simple swig module
9fb915e SWIG generates files with .cxx extensions
52949a7 Removed debugging message
a3e829a Initial framework
f08623e stress test for proc
99c1d62 Optional stress tests now
09baeff Move stress tests out
8dada87 Priority union function and unit tests
e095ef1 Starting to implement a faster conversion, added file HfstFastTransitionData.h for that purpose. Also separated HfstTropicalTransducerTransitionData into its own file.
50deff4 HfstTropicalTransducerTransitionData moved to a separate file from HfstTransitionGraph.h.
0d2bda5 The pathname of HfstDataTypes.h fixed in HfstTransitionGraph.h
13e5805 A non-existing file removed from Makefile and an include command.
b91dcd8 Including HfstTransducer.h and HfstTransitionGraph.h into header files avoided in most cases by using HfstDataTypes.h instead.
fbcb690 Made some changes to header files in order to avoid cyclic dependencies.
3b54b9d A bug in TropicalWeightTransducer::determinize fixed. The weights must not be encoded, only the label pairs.
f723fba Typo in license conflict warning
d23279b Now the subtraction of two empty transducers works also for SfstTransducer. A test case also added for all transducer implementations.
e6549a3 Now read_in_att_format and hfst-txt2fst also accept a single newline as an empty transducer, if that is the only transducer in a stream.
373523d The library now uses is_epsilon, internal_epsilon etc. instead of string literals.
ceb1942 Added pkgconfig definitions
06fbbe0 Fixed variable LIBHFST_VERSION
40fde0e HfstTransducer::get_alphabet declared public. Now the foma implementation returns also special symbols that are not included in the sigma of foma, but foma is still aware of them. Some unit tests also added.
9069af7 Added some unit test for SfstTransducer that test if the alphabet is pruned in operations.
6d7f356 Fixed possible bug in ComposeIntersectUtilities and the find member function of SpaeSavingSet.
d3170ff Put stl sort function in stl namespace (which for unknown reasons is apparently unnecessary but it's probably clearer this way).
2d5c8d5 Added more test cases for hfst-subtract that test if the identities and symbols are handled right. Now subtraction seems to work for SfstTransducer, too.
430ab83 Print --version and --help to stdout in standard style
faa9758 clean generated scripts
c5381c7 Remove all binaries and texts after tests
693e2d6 remove generated files on expected failures
ded09b1 remove generated files after testing
dd1143e Skip broken aliases
da774c3 exempt scripts and tools with broken option parsing from distcheck
014ad55 Use $srcdir in tests and include all sources
26aeef9 Missing text file reconstructed by guessing
6aedb92 Now input and output projections keep all symbols in the alphabet and have unknowns replaced with identities. Subtraction of tropical transducers handles weights correctly, all paths in the second transducer are subtracted from the first one.
41d2454 Comment out failing test prereq for now (it broke make dist).
9e69826 Unit tests reinstated for .cc files under implementations, compose_intersect forced under same regimen (whether MAIN_TEST is #defined).
6d8743b Dummy unit tests for every .cc file in libhfst/src except HfstExceptionDefs.cc
693540f Tests added for hfst-lookup with transducers that contain identitites.
0dc5bd2 Document openfst version requirements in troubleshooting also
c7a37f1 An error in harmonization fixed: now flags are excluded when expanding unknowns. A test also added that checks this case.
b5c65bd Missing lexc test cases
b9c7075 Reinstate final weights (they've apparently been absent since the missing state -relabeling code was added...)
7677a39 Added overloaded lookup functions passing untokenized strings
f354be5 print verbosely tokenization and blocked flags unless ol
faeb874 use GNU getopt if available
3248829 declare more built sources
008db77 More weight tests
e772be3 More sanity checks for escaped colons
c68e38b Give weighted example
38440f9 partial fix for 3269636
0cc3569 remove debug printout
70af6e0 filenames and line numbers in error reporting again
339413e Build obsolete lexc compiler by default: * remove handbuilt lexicon combinatorics and revert to sfcm2009 algo * use pretty name for lexicons * more verbose printing instead of xerox style again
a75d492 Now hfst-twolc rule names are unquoted.
72ca575 Add missing files with likely sources
5f924ab Now composition should work correctly
2b052f5 Changed population of symbols sets in HarmonizeUnknownAndIdentitySymbols.cc so that it relies on the get_alphabet method in HfstTransitionGraph. get_alphabet sometimes returns an empty set, so that needs to be fixed in order for unknown and identity symbols harmonization to be reliable.
492ddf3 Added HarmonizeUnknownAndIdentitySymbols.{cc,h} to Makefile.am
08d99b5 Unknown and identity symbols should now work correctly in composition. Still some errors in other tools in test/tools, trying to get them pass all.
ec358ec HarmonizeUnknownAndIdentitySymbols.{h,cc}: A model for how identity and unknown symbol harmonization can be done...
8ab38b4 New NEWS for 3.0.2 rerelease
0bcd861 Added missing ComposeIntersectUtilities.cc
d5d1a14 Added missing ComposeIntersectUtilities.h to Makefile.am
e8c4fd6 Prepare 3.0.2 release
2e151c7 s/warning: Warning/warning: /
d108e8d Moved the #ifdef HAVE_SFST a bit higher..
a9645b3 Fix ; separated regexps: * add compatibility function for mac OS X missing getdelim * use delim for all cases in regexp2fst
bdc8a29 Converting to and from SFST format is now faster because function 'check_visited' is used instead of a set of visited states.
32a3161 Conversions between tropical, sfst and foma transducers are now carried out through HfstConstantTransducer, which is faster than using HfstBasicTransducer.
893c4eb Make fallback dependent on fnie
25b8f73 Removed some unused code..
8711cbc Reduced hfst-compose-intersect memory consumption a bit.
0d9c166 Added hfst-pair-test for testing hfst-twolc rule files.
7594c1a HfstConversionTransducer renamed to HfstConstantTransducer, most functions in ConvertTransducerFormat.cc splitted to separate files
bcc537f Some optimizations done on conversion functions. Class HfstConversionTransducer added, it will be soon used in a more efficient conversion.
4e74b65 Note that we have licence exception for foma and warn if foma is missing
3fd96ed Epsilon filter compose intersect
77f9748 Only ascii-tokenize a-zA-Z to avoid breaking eg. generating transducers
3155440 3.0.1 release news
24b0883 Prepare bugfix release 3.0.1
4d7a04d Use library function to check for epsilon in to_ol
78f7e60 Pass strings to internal representation checking functions by reference
6a5608c More control over floor lift behaviour
aeee89f Clean up conflict resolution mess
47f3823 Better floor jumping + relabel non-contiguous states (could slowdown in extreme cases, possibly revisit)
eaa606b Alias hfst-optimized-lookup -> hfst-lookup for compatibility
04d7b57 use default names to trace hfst operations neatly
f6fecd5 Allow name truncation for neater names
af00906 describe input/output formats in man pages
2b9fa09 Extend example to contain all sections and stuff --Denn och nedanstående rader kommer inte med i loggmeddelandet--
da36afc New tool for preprocessing automata before optimizing
8bcd08b A state in an HfstTransitionGraph now has a vector of transitions intead of a set of transitions.
eeb7acc ...I meant HfstException
0a73c66 Catch HfstFatalException in convertion
e6a8e3d HFST_CATCH macro
456a02e hfst-lookup now handles flag diacritics correctly
35ea903 Added functions to check for equality to internal reserved strings
9f4d70b a bug in printing epsilons in hfst-fst2strings for sfst transducers fixed
ab54f89 Add custom error message and README blurb for ::malloc errors
618d064 When the floor sticks for (even) a single round, jump up to most recent successful allocation. This makes things a lot faster, at the cost of some waste of space. --quick should not be necessary for most purposes now.
ff86e49 hfst-fst2strings now implements option --random for sfst and openfst transducers
bcc68cd Misc. speed improvements, in particular removed the need to separately track which indices begin states
17007d8 Actually write the index table finality markers
f436f9a see previous commit message
9d863c2 Fixed oopsie wrt removed profiling variables
51049ce Run twolc tests in numerical order.
b9bdc86 Fixed problem with word boundries breaking the alphabet.
5234e0e Added tests for ensuring that word boundary conversions wonät break the alphabet.
b532c93 Changed word boundaries so that they perhaps emulate what xerox does. A "#" in a rule file is always compiled to both an absolute word boundary (which is an internal symbol in hfst-twolc) and a relative word boundary "#". The absolute word boundary will always correspond to zero in output strings and the relative one will correspond to whatever it corresponded to in the rule file.
f9788b0 Word bounadry tests added.
d797484 Major improvements to --quick and some speed cases
0062cd4 option --random added to hfst-fst2strings, not yet implemented
724121b Removed trailing whitespace that broke make
c45ba6b Fix lexc tests: * make script in build dir +x * put script dir in beginning of PATH in tests
6621d1c git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@1181 941e2c2b-deac-454f-805a-451daa25f33c
b4d68df Missing files added to Makefiles, make distcheck still fails.
fface6c Turned option passing back on for convert function
a31e269 test-script fixed. Previously it required installed hfst-compare, hfst-txt2fst and hfst-fst2txt
94b2b83 fix partial commit of portability c xfunctions
71b3bcd remove unnecessary read
3091271 use pretty names by default
c122b04 Fix name tool installation
45f3fda Use automata names from metadata instead of filenames
a3e6bcd Fixed a bug with %-escaped variable values, which weren't properly unescaped previously.
39651a4 Added tests for %-escaped variable values.
f59a7e9 hfst-fst2strings help message and special character printing fixed
584c57c hfst-fst2strings fixed
3c0acb5 filter flags should've defaulted true
7e4b6cd Typo in usage message
8f464a9 Added some files for a transducer testing environment.
c53f605 fix spacing in examples and describe
28680a9 update version
38108b8 "environment" variables fed to configure get saved
c35977b Added note about local libraries
8077e3e Fixed a bug in compiling <=>-rules with list centers.
97dd6fc Added test for compiling <=>-rules with list centers.
c748ca3 Changed target to first release version
35fb221 hfst-lookup should now work correctly
5d4421d Added deprecated (and undocumented) -w switch to hfst-calculate for backwards compatibility
d8842d4 Update substitute documentation
d4c7e58 hfst-lookup should now work, still some problems in printing strings
d7773da A more organized version of lookup function added to hfst-lookup. It will soon replace the original function.
f850e58 Don't expect hfst-lexc-compiler to have a proper help and version string because it's just a humble wrapper.
09adc46 Added a way for ol-conversion to be done really lazily (for when you want it done quickly - should still be faster, honestly). Also to pass a string parameter to conversion functions if they ever want a hint about something.
d4f4c94 Don't run obsolete tests that break when run after make clean
252e225 more distcheck fixes
53d54a4 small changes made to hfst-lookup
200b383 Added single quotes around and space after the filename for file-not-found errors. Makes it easier to read the filename (earlier the cause of the error was printed flush after the filename, with no space between).
63b19be be explicit explicit about foma requirement for lexc and xfst
9b1f9c2 Only claim to convert files that test -f
df545ae Small changes made to helps and arguments in tools that convert between text, string and binary formats.
65bebfd Maybe INFINITY will be defined more portably this way
a9183a2 INFINTY in hfst-lookup was not defined properly. fixed it.
b97ed62 string::npos changed to std::string::npos
105b4b3 Generate binaries needed to test hfst-proc correctly
65b4602 More utf-8 checks added. Command line tool helps also updated.
2342e28 Fixed Makefile.am so that make dist works.
59e6347 Initialise weight
3f70a6d Missing std qualifier
49eaf2b Added HfstTokenizer utf-8 validation tests.
4991081 Enlightened hfst-proc as to the true nature of HFST3 headers
de50e34 made HfstTokenizer::check_utf8_correctness a public static
1850f9f Added utf-8 validation to the tokenizer and added the exception IncorrectUtf8CodingException.
6361b29 Fix brokentests
3a4ce96 remove coding style requirements and unit tests since they aren't followed anyways
7752960 Do not print all multicharacter symbols unless in verbose mode
206bef0 Update man pages
92a9924 mark 3.0_rc1
3d3e70b Fixed a bug in compiling rules with word boundaries and added test45.
7ff8200 Don't use sed in scripts since it's broken on MacOS
cb7b1ee Fixed bug in compose_intersect which discarded the second rule systematically.
948b372 @_SPACE_@, @_TAB_@ and @_COLON_@ added to HfstStrings2FstTokenizer.cc
7ad47d7 hfst-lookup now uses HfstFst2StringsTokenizer to tokenize the input strings.
9ab6592 Don't convert same stack more than once
b9dbb3a fisx some warnings
fa5dd09 workaround mac''s missing strndup
6870c09 Allow pair substitutions...
01d746d Fix test for Mac OS
677ddde Replace tabs with random number of spaces
0990ea4 More tests limited to available formats
8ee702b Only test available automata formats
96af3b2 Test all string formats in strings2fst
834cf29 Fixed hfst-string2fst parsing of input strings and reading of multichar symbols.
e188081 hfst-strings2fst should now be able to handle all kind of string formats.
ecc96f2 # is no longer a special symbol, because it is not a special symbol in xerox twolc.
2ca9026 Fixed print in left arrow rule conflict warnings.
3d41fc2 Fixed word boundaries in compose-intersect, again...
b4270dd Special symbol handling changed in the commandline tools.
0e1bea8 Fixed word boundaries in compose_intersect. When HfstBasicTransducer substitute starts working correctly, this can be done more efficiently.
161d903 Fixed word boundaries in compose_intersect.
f507cee Added printing the conflicting context in left a row rule conflict warnings.
2851f19 Unknown and identity symbols are now correctly handled in composition. AT&T format now allows spaces in symbols if they are ascaped as '@_SPACE_@'
caaf109 Tests for ? symbols
a57fcaa Use unknowns and identities in regexp2fst
a6c9834 Documentation added. Exceptions now throw a message.
dfceeee Modified test so that the script does not have to be changed when new tests are added.
71f1022 Removed a unnecessary newline printed when not reporting left arrow conflicts.
6b311ec Added a rule for compiling a local version of hfst-twolc used in tests.
d08533d Added hfst-twolc tests.
e032ace Added hfst-twolc tsts.
98a5402 Added hfst-twolc tests.
aacbe75 Cahnged order of arguments so that 'cat fst1 | hfst-binary-tool fst2' will treat fst1 as the first and fst2 as the second argument.
262d19e Fixed compose-intersect
1bb5dec Handle the empty transducer in ->ol
34d3577 Now hfst-compare does not print anything in silent mode. More tests added to test/libhfst.
ea10912 Fixed definitions
232b892 Fixed d:0 where d is a diacritic.
58836d2 Fixed symbol offset calculation in certain flag diacritic-concerning cases
d2a766d Fixed problem with diacritic pair syntax. Now d d: and d:0 are equivalent for the diacritic d and d:x will give a warning.
a5505dc Remove debugging strings
0590acd Properly copy flag state vector on stack
25a5b9e Fixed a bug with definitions
32e01ec Add tests for UTF-8 and latin-1 encoded strings
ef669e3 follow print-epsilons in all formats
9331c51 format options like other tools have them
5b8b2b1 More documentation added.
6197f2f More careful quotation and informative error messages for foma scripts
5d6a2ff Use "HfstTransducer.hh" instead of <hfst/HfstTransducer.hh> for in tree builds
49db47a hfst-strings2fst fixed
6bf1608 Now hfst-strings2fst should handle escaping correctly.
2825f8d A bug in hfst-strings2fst fixed: now the backslash is reserved for escaping characters. Now the library also throws more informative exceptions.
54aaa9e Fixed operation retrieval
1a0485a Fixed problem wrt subtracting from an end() iterator of an empty state_placeholder index placeholder map
85d0602 Removed some more debug prints.
fe95ee7 Removed some debugging prints
28844b5 Fixed bug in defining lista using other lists and another bug which prevented variable rules unless there was a Definitions section.
613d39f Fixed diacritics handling.
86c8335 Fixed output file handling.
103d9b2 Gave up on making }, ] and ) literal if they are not preceeded by opening {, [ and [, since this is too difficult in flex when brackets get nested. Fixed commandline argument handling properly and separated { and } from [ and ] i.e. they get their own bison symbols.
52ebbb7 Fixed handling of commandline paramters so that the input filename can be given as a free argument without -i or --input.
d630caf Fixed syntax so that a lonely }, ] or ) is not interpreted as abracket, parenthesis and so on. Fixed numbers so that they need not be quoted unless preceeded by ^
ade6ed8 nore quotation and script guessing
60c8023 Added hfst-compose-intersect.cc
2b7a964 Changes needed for libhfst/src/implementations/compose_intersect
5718b7b Added fiels needed in compose_intersect
803e57e configure.ac changed because hfst-twolc was added.
51b9cd2 Removed file hfst-twolc/.txt_fst which was added by mistake.
cfa3c5c Removed geerated dir hfst-twolc/src/.deps which was added by mistake
38ec61e Added hfst-twolc.
e195be6 Miscellaneous packing tricks & fixed a typo
7815222 Correction to flag indexing
51cdc7a Corrected alphabet ordering (eps-input-flag-other)
a07a6e4 Small changes done in documentation.
f8658bb More refactoring
20e8d36 Miscellaneous improvements and refactoring
4e9475f Fixed bug wrt finding the largest used index
859a1b2 missing test file added
b4ecec5 Parametrized packing aggression. Packing is still a little slow for decent levels of aggression. Consider adding a mechanism to ask conversion to be maximally aggressive (eg. for binary distribution).
fcaea64 Redo flag handling in ol-conversion
dfb2690 A bug in hfst-lookup fixed: reading input strings in spaced format now works.
7856fc9 Fix stupid mistake in previous fix to optimized-lookup internals
2f32111 More coding style conformism
bc68f4c Coding style conformism
7d828ce Fixes to optimized-lookup internals and conversion
d53894f Exception classes refined
844d881 All exceptions changed to HFST_THROW(HfstException).
66ebc7d Renamed TWROW HFST_THROW in HfstExceptionDefs.h
661b615 Added some macro definitions, which make it easy to declare, define and throw informative exceptions.
829953a Free memory earlier and do some less work in ol-conversion
3c70901 The order of weight and string/string pair changed in HfstOneLevelPath and HfstTwoLevelPath. Now the elements of a set are sorted in ascending weight order.
44f919c HfstLookupPath(s) is now replaced with HfstOneLevelPath(s)
ae4f393 Flag diacritics should now be handled right in functions lookup and extract_strings
95eca27 HfstTransducer::lookup now takes a StringVector as argument and HfstTransducer::extract_strings stores its results in an 'HfstTwoLevelPaths'
38c5e2b Now coding-style.sh should pass.
7470a48 Overlong lines separated on several lines.
525715b More more more OFST->OPENFST renaming
bd8523a More more OFST->OPENFST renaming
1a56267 more OFST -> OPENFST name renaming
645f7af HfstTransducer::read_in_att_format is now protected
4b37099 A bug in tests fixed.
b106796 test.cc renamed to test_rules.cc and only tests for functions in namespace hfst::rules preserved
708daa6 TROPICAL_OFST_TYPE and LOG_OFST_TYPE changed to TROPICAL_OPENFST_TYPE and LOG_OPENFST_TYPE.
4a1fd2f Fixed a bug in reading binary SFST and FOMA transducers.
80e53b4 update requirements and reorganise
44a7d7a Fixed bug in reading and writing transducer binary headers.
d9ac6ec AT&T format now accepts all whitespace characters as field separators.
6fd24d8 Added a const-version of HfstTransitionGraph::operator[], so that it is possible to access the transitions of a const HfstTransitionGraph.
3cae2e6 Added appropriate type checks in a HfstTransducer constructor - probably many more needed...
af09e6f Fixed typo and clarified help message
ce45fb5 Represent epsilon as blank string when returning representations from ol-library
13f7061 More tests added. Substitute bugs fixed in TropicalWeightTransducer.
5632573 New tests added. Substitution functions cause a segfault in TropicalWeightTransducer, debugging under way.
2c7c03e More tests added.
91ab109 Fixed some problems (due to epsilon symbol) in SfstTransducer functions that take a String or StringPair as argument. FomaTransducer::is_cyclic now works correctly, function 'fsm_topsort' must be called before 'is_loop_free'.
2ea6659 More tests added.
ebfd3f9 File test_transducer_functions fixed.
47066b2 Extra debugging messages removed.
e48223d Test for tokenizer added.
8ff3b25 FomaTransducer::define_transducer(char*) now calls fsm_count(struct *fsm) on its return value. In this way the number of fsm lines is correct when the transducer is written to file.
8485a6d More tests added.
10901e1 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@925 941e2c2b-deac-454f-805a-451daa25f33c
a68b3e3 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@924 941e2c2b-deac-454f-805a-451daa25f33c
0df13e9 File test_constructors.cc added
e1e8ffe Documentation added.
7e81b7b Documentation updated. Tool hfst-name added.
ab09fac Now conversion from OpenFst and SFST to HFST handles special symbols correctly. How special symbols should be handled in the reverse case is still an open question.
aaa59d3 remove deprecated conversions and unused parameters
8cbe4a2 ol-basic -conversion
b3a201a Function HfstTransducer::insert_freely(const HfstTransducer &tr) added.
63eb030 Now the output symbol table of a tropical or log HFST transducer is assigned a copy of the input symbol table when the transducer is written. This should make conversions between OpenFst and HFST transducers easier.
473157e Fixed two typedefs in trunk/hfst3/tools/src/hfst-summarize.cc
c8c1af8 Fixed an #include in trunk/hfst3/libhfst/src/implementations/ConvertTransducerFormat.h and a typedef in trunk/hfst3/libhfst/src/implementations/convert.h
25fc8e1 Fix verbose messages about HFST3 headers
e856b97 epsilon tests
0e91640 hfst-lookup now prints the results in the FILE requested instead of stderr
758a956 hfst-lookup has two options more. It is possible to print results in pairstring format (not implemented for optimized lookup format) and defined how the epsilon is printed.
76acf98 extract_strings and WeightedPath changed so that it is possible to include a StringPairVector representation of the extracted paths
d17c308 implement flags and epsilons and xfst options print-space print-flag obey-flags etc.
290b861 Avoid testing missing backends if possible
843e8d6 Documentation added. Thrown exception types changed for some functions.
ed4c9af Documentation improved and added. Keyword 'const' added to functions and their arguments when appropriate.
532d9e9 Documentation added.
026a095 Documentation changed.
a76fccf HfstFileException changed to HfstStreamException
740219f Missing flag diacritics now yield a warning in hfst-compose, hfst-conjunct and hfst-subtract, unless freely inserted by option -F.
fbd97a1 One more portability fix... could be the last one!
200fae1 Yet another without-openfst portability fix
da6dfc8 Another without-openfst portability fix
beae820 Another without-openfst portability fix
b8f670a Added using-directives to hfst-lookup.cc - apparently they don't get included from somewhere else when not HAVE_OPENFST, so this is necessary for portability
e9a2fa5 More optimized-lookup conversion fixes... produces correct results at least for trivial cases now.
37717b2 Various bits of format correctness for optimized-lookup conversion (lookup still doesn't give correct results)
1e51ff9 Documentation added.
c929f0b HfstAlphabet, HfstBasic and HfstUtf8 cc and h files moved to tools/src. Some functions in HfstTransducer decalred as protected.
3e5198b #if HAVE_HFST_OL perhaps not working correctly - in any case HFST_OL-writing is now enabled, although the results still don't appear to be quite correct
70441e4 Oops, forgot to remove some debugging messages
c930115 Fixed hanging hfst-ol conversion, still doesn't get written
4b3123b Ol-conversion (untested as yet)
7f0965a Added parsers/xre_utils.h to noist_HEADERS in tools/src/Makefile.am
52ba7b8 Added parsers/XreCompiler.h to noinst_HEADERS in tools/src/Makefile.am
0382dde Documentation added.
3210012 Documentation added.
94820cd Intersecting composition functionalities removed, they will be implemented using HfstBasicTransducer format in the future.
90253d69 Updated configure.ac to reflect dropping of tools/src/parsers/Makefile.am
081c986 Deleted unneeded Makefile.am in tools/src/parsers
aa3fcaa Substituting unknowns and identities now works correctly in composition.
3e96e8f A bug in reading transducers in AT&T format fixed. Also an alphabet bug in converting HfstBasicTransducers into SFST transducers fixed.
7c9ed80 Corrected documentation link.
7bd61df Documentation added.
b03365d Redundant code removed from class HfstTransitionGraph.
ee7689a HfstNet renamed to HfstTransitionGraph, HfstFsm to HfstBasicTransducer and HfstArc to HfstBasicTransition
9c3b6a6 extra_dist all scripts
b930429 fix VPATH _build_
324cd91 Substitute functions of HfstNet updated.
366b783 HfstCompiler.h in noinst_HEADERS
d545f77 Added items to noinst_HEADERS in tools/src/parsers/Makefile.am, evidently needed for building
150715f Remove intermediate files and be selectively verbose to avoid confusion
e2e9e79 YAUH: HfstUtf8.h
4c75eab YAUH: HfstBasic.h
f0e1f0c Another unshipped header (HfstGrammar.h)
c9d913a oops
b258f1f HAVE_OPENFST guards
ab00fb4 A bug in transducer conversion functions fixed.
ab4c840 Missing \ in another header shipping directive in libhfst/src/implementations/Makefile.am
8bb83a4 Another missing header, TropicalWeightComposeIntersect.h included for shipping (this is probably temporary and the header shouldn't be needed anymore, but for now building fails without it on clean machines)
558f4f3 The alpha testing instructions
cc748cc skip all tests that might fail
1ac0e16 * some new autoconfiscate features (silent make!) * misc. stylistic changes
e8b5ac0 Made HfstNet::get_final_weight(...) a const member function.
eafc2e8 Added missing headers to Makefile
2427785 changed void HfstNet::add_state(HfstState s) so that it returns s and added HfstState HfstNet::add_state(void), which adds a new state.
51d9a4a HfstNet.cc added
6468e7f Conversions between transducer formats updated so that they use HfstFsm
236c20d Fix the missing includes and linkage
5a74f1f Remove hfst's lexc, xfst and xre parsers and use foma's instead of reinventing wheel and doubling update burden
f569bf3 New transducer conversion functions added.
b1b2c7c git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@841 941e2c2b-deac-454f-805a-451daa25f33c
569ec24 A bug in FomaTransducer::define_transducer fixed.
bb4aa6b A bug fixed in HfstNet::substitute(StringPair, HfstNet)
7b4516e More functions added to HfstNet
c80a341 More functions added to HfstNet
59c60c3 Use local compare for all tests
4babf62 don't fclose infile too early
f4ecf79 * Separate bug report from further info by newline * remove old and deprecated tools
7db67bb * intersect has been called conjunct since forever * move external links in SEE ALSO
c49c6c7 enforce coding style in make check :-)
a5774a9 File new-ConvertTransducerFormat.cc added, it will soon replace ConvertTransducerFormat.cc
3c75438 testing results of parameter vcombinations
1c76cc0 add hfst-lexc script
1fe743b Update xfst/foma scripts
9259bbe Added a substitute function to class HfstNet.
ea06012 HfstNet.h has now a function write_in_att_format
a0e38e2 HfstNet.h updated and tested.
692777c Added file HfstNet.h that will soon replace file HfstInternalTransducer.h.
8bd8247 Fix internal tests
a988bf9 HfstInternalTransducer_ updated.
d511822 A new HfstInternalTransducer class HfstInternalTransducer_ added.
5105bb2 class HfstTransition changed to struct HfstTransition
de7d5ce A bug in converting empty transducers between internal and OpenFst's formats fixed.
b2b1ee8 use #if's on backend specific code
75bc715 LogFstTrieFunctions .cc and .h removed and their functions moved to LogWeightTransducer .cc and .h and copied also to TropicalWeightTransducer .cc and .h
e5cacc1 Files renamed and deleted.
19e3b71 Including headers should now work correctly. Doxygen documentation also added.
871e1e0 The minimal lexc compiler functionality tests and minor fixes to compilation formula
4d00374 implementations unit tests
4233b9d Add unit tests for extensions
e7213c6 don't test default input stream since it requires stdin
135c50f EXIT_* needs cstdlib on newere gcc's/glibc's
f008438 main interface unit tests 3/6
7923d45 It's a verbose test suite
3fdf3ea hfst-strings2fst now tokenizes multicharacter symbols
4280623 A bug in hfst-strings2fst with (not pairstrings && not spaces) fixed.
dc4b60b hfst-strings2fst now uses the internal trie transducer format and converts to the format requested before writing the output. This should make hfst-strings2fst faster.
178b960 Documentation added.
39f6553 Added a simple example transducer library, MFSTL,  that can be included by command './configure --with-mfstl=true'
37fa173 An implementation type FOO_TYPE added for testing, by default it is not included in HFST.
779e27e Fix lookup_printf and test it
91cbdca Reading hfst version 2 transducers whose implementation type is OpenFst should work now.
7248131 Documentation added. A bug in hfst version 2 transducer handling detected, fixing started.
8a0f692 Full-ish HfstTransducer unit test
f65f228 A bug in hfst-strings2fst fixed, now the newline character is discarded before converting the weight string into a float. More Doxygen documentation also added.
76ebbd1 Lookup implemented for optimized-lookup API with new style headers, though hfst-lookup doesn't print anything due to commented-out printing function (it seems to break when I uncomment it..?)
606ddf6 This could be unit test
8887df6 hfst-lookup should now work correctly for transducers having epsilon input cycles. Also added some documentation.
da33a15 fix more of mac os x getline substitute
b88d9ff skip hfst-proc dir for empty input tst
b41e574 Revert man bootstrapping stuff and add stubs for rest of the tools
539ef76 add set and use zu for size_t prints
100123f hfst-lookup fixed, still allows too many cycles although limits the results to a finite number
0aeebd2 A bug in hfst-lexc-compiler fixed, still some errors in help2man and tool tests.
1e97875 File added.
b185125 Outputs of help commands of commandline tools fixed.
12ff3ef A bug in HfstTransducer(StringPairVector&, type==FOMA_TYPE) fixed.
edbb55d A version of infinitely_ambiguous (necessary for at least lookup)
d7f8ba0 Removed conditional compilation guards from non-template lookup-performing function (which way do we want it to be?)
d9efbee enable lexc, missing ) added
64c686b Some additional HFST_OL functionality
75934b3 Update man pages for release :-)
2dc49a3 Tests added.
209fd04 Some fixes to make the right names defined down the line, plus a first attempt at the lookup API
39692d3 Lookup for hfst_ol in hfst3
4887fe9 More documentation added.
07b4838 Files documented, a skeleton file added for adding a new backend implementation
2831646 Files renamed
cc61692 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@769 941e2c2b-deac-454f-805a-451daa25f33c
185380b Now libhfst/src/implementations contains files that modify backend transducers directly and libhfst/src other files
a98715a Files moved
636599b Files moved
a373e2d Set executable bit + line endings.
fcf17ac Fake hfst-xfst using foma-wrapper hacks
2158187 HfstCompiler.cc and HfstCompiler.h added to tools/src/
485406c Files moved.
b7d01a8 hfst-lookup updated, handling flag diacritics and multiple paths with the same weights not yet implemented
bccb155 hfst-lookup no works on unweighted acyclic transducers
565d990 set type after exiting error for mac compiler
4cba1aa Fix -o handling
a62adc6 hfst-lookup implementation continued, works not yet
edd084b beginning to implement hfst-lookup
cb1cd03 Recover from make-breaking stupid mistake
2b402e5 Standardization of HfstOlTransducer's iostreams
b10fc11 just testing that everything works
d20527d Clean up few minor details
d1968d9 Documentation added.
0fd33bd More documentation added, exceptions divided into subclasses.
f55820f Documentation added.
bac207d Catch stuff to provide meaningful error messages, provide better error replacement
7bec3d4 Small changes done, documentation added.
ddaf962 Lookup and related functions
94bb62d Update string_to_format and some polishing
3067f9f A bug in reading transducers with old header format fixed.
8bde7d5 harmonization function optimized
29c3abc don't use bsd err functions
19df4c8 return paths even if !HAVE_SFST
cba810b using std::map, have SFST stuff ifdef'd
218faa6 A name field added to hfst header. Doxygen documentation updated.
7a3b473 HfstInputStream changed
c7509d0 Missing restriction rule operators added.
fd80271 A bug in HfstRules fixed. FEXPORT must be added in front of foma functions 'sigma_create', 'xxmalloc' and 'sigma_max' in file fomalibconf.h until new version of foma is released.
1a306b5 A bug in HFST header handling fixed. Todo: update HFST code to work with the newest version of SFST.
42e2594 Fix lexc compiler
8739b07 Detail GNU bison version and feature requirements
801ad20 Use obsolete syntax for name-prefix to support slightly older bisons
2d9b05b More checks for backend installation sanity§
da681d1 Make compiler tools dependent on relevant features in library
a6f4b68 Also include current foma diffs
322b7f8 Function open() removed from HfstInputStream and HfstOutputStream. HFST transducer header structure changed.
5d197d0 Added MacOS X notes.
6532861 Remove bundled foma and SFST
bf5053e Mergefail→delete duplicate lines
cba481f Add preliminary xfst compiler for offline compilation of static files
ea740e3 Add autogen.sh for problematic oses and systems
2ce19e8 Document library backend requirements
d911ae3 add replacement strndup and error_at_line for mac again
dbcfbf1 commandline tool hfst-compose-intersect and test added
45290c5 more tests added to tools, SFST_TYPE transducer header now has an optional field that tells if the transducer is minimal
0db6a0b more tests added to test/tools and small bugs fixed in commandline programs
ce7f006 HfstInputStream and HfstOutputStream now work correctly with tropical and log weight transducers if their file argument is an empty string
b36fd19 Alphabet definition in HfstCompiler is now not done through internal format when using SfstTransducers. This optimization makes hfst-calculate slightly faster.
aaa2eae tropical transducer harmonization code optimized
5421e3d Moved the #endif to the actual correct place...
3a950b7 Moved #endif to correct place. Suggestion: when there's a lot of distance between #if and #endif, consider placing a comment after #endif to indicate what it's ending (emacs can't figure it out).
8ab00a7 Removed obsolete definitions
15369a7 transducer harmonization code optimized
bf6dfa1 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@692 941e2c2b-deac-454f-805a-451daa25f33c
7eb10ec missing files added
3d244cb git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@690 941e2c2b-deac-454f-805a-451daa25f33c
3bc4705 HfstCompiler's code optimized
0a590ff git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@685 941e2c2b-deac-454f-805a-451daa25f33c
dfd5143 replace_in_context's code optimized
0ba8366 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@683 941e2c2b-deac-454f-805a-451daa25f33c
2cad021 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@682 941e2c2b-deac-454f-805a-451daa25f33c
e1f264b Class 'HfstTrie' added for quick disjunction of words in hfst-calculate
41328b3 reading words from a file should now be faster in hfst-calculate
c973556 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@679 941e2c2b-deac-454f-805a-451daa25f33c
c075ba7 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@678 941e2c2b-deac-454f-805a-451daa25f33c
256cd2d a bug in hfst-calculate's 'make_mapping' fixed
f46a184 now hfst-calculate does not unnecessarily handle unknown symbols in composition
f2a5359 A bug in alphabet definition in hfst-calculate fixed. It seems that handling unknown and identity symbols in composition is very slow for some reason.
d194e80 A bug in HfstCompiler::complement_range fixed
a489950 HfstCompiler::new_transducer is now faster
e74f90f some bugs in command line programs fixed
d78c45a some bugs fixed in tests and hfst library
2a64259 4cats.txt now contains a transducer accepting 4 consecutive 'cat's, and 4_cats.txt contains 4 consecutive transducers that each accept 'cat'.
acd96ec a bug in function 'harmonize' fixed
553674c Commit missing inputNamed
d9f9551 hfst now compiles with all combinations of available backend libraries
c5e0973 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@661 941e2c2b-deac-454f-805a-451daa25f33c
68b326b Make --without-foma work
71d08c4 optimized-lookup bigfix wrt weights in flagged transducers
41a235e Function HfstTransducer::push_weights added. Foma is now dynamically linked to hfst, option -fvisibility must not be used when compiling foma, because then some of foma's functions will not be visible to hfst.
fbe2969 Tentatively implement HFST 2 format
9d4aeed move things around
917827d Add space separated strings2fst; parse - as filename in hfst_fopen
6c69309 hfst now works without making any changes in sfst files
11f5420 Configuring out broken parts
17cd8d3 Add preliminary lexc compiler
ab50a96 HfstMutableTransducer defined in namespace hfst, small bugs fixed in TropicalWeightTransducer
226ae4c unknown and identity symbols are now handled correctly in composition
c016a92 HfstMutableTransducer replaced with HfstInternalTransducer
792dfb3 Document the XRE compiler
a2d0367 functions that read and write in AT&T format now use HfstInternalFormat
6f0770d some modifications made to HfstInternalTransducer
0d5c47d freely_insert and substitute functions now do not depend on any specific library. They must still be tested.
2bf97cb some functions added to HfstInternalTransducer
f26af42 invert XRE autoconf logic
1a8a586 missing files added
b2da1a4 conversions between internal transducers and sfst, tropical and foma transducers implemented
9b896d2 Enable XRE parser
058a409 conversions between SFST and internal transducers added
66cdf07 summarize more statistics
3260a0a Wrap backends around autoconf have_
2c140fa Enable basic compose-intersect again; no flags, no #'s
ef39e84 HFST code updated so that it works with OpenFst version 1.2. New internal transducer format added.
47cdd70 fix signs when wno-sign-compare fails
00e079b Delete generated files
4590274 Now hfst-calculate does not depend on SFST.
1a56a05 hfst-calculate now does not depend on SFST, function implementations not yet written
ea974ba a bug in substitute(StringPair, HfstTransducer) fixed, agreement variables added to hfst-calculate
90d6275 Add some functionality tests for hfst-proc
a884b27 Remove obsolete source files of hfst-proc predecessors
4ebe348 Fix bug causing final weights not to be included in weight calculations
32abdec Remove obsolete flag diacritics code from hfst-proc
b5a8fe8 Add license information to hfst-proc and optimized-lookup files
eeedf41 Modify the extract_strings algorithm to avoid the non-standard std::vector::data() for older GCCs
2cad922 Perform initial migration of hfst-proc into its new home in the proper HFST tools directory, while refactoring to share as much code as possible with the optimized-lookup library backend
0fc222e Add 'exclude' option to hfst-fst2strings
af172ca git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@616 941e2c2b-deac-454f-805a-451daa25f33c
c457724 Fixed memory leaks in harmonize
fb3fab5 Added libhfst/src/implementations/TropicalWeightComposeIntersect.h and libhfst/src/implementations/TropicalWeightComposeIntersect.cc
ebef427 Composition can soon handle identity and unknown symbols correctly. Substitute can now also modify symbol pairs through a function pointer.
0207962 Refactor extract_strings to invoke a callback on paths, allowing neat features in hfst-fst2strings
25b747a Credit self in AUTHORS
fddd8b2 SFST updated to new test version, some modifications still needed to HFST library so that it will work with the new version
17125a4 error_at_line for parsing errors
5df88b5 Allow empty lines and comment lines starting with # except when followed by a tab.
b8da331 Add the basic summarize functionality and the test
5ff805e rule operators added to hfst-calculate
3618b20 More functionlaity tests
1daa938 Reimplement extract_strings in all backends with cycle-limiting functionality
8ebdabe substitute and test
4c2697a Add test for fst2stings
3c0db1e more functions added to hfst-calculate
527b7cf Ubuntu (dash?) workaround
07c8832 Make string pair implementation work
5a1646a Add lexc compiler's old skeleton
768cc07 Reverse how the upper and lower strings are stored during string extraction in the SFST backend
d822832 Implement string extraction in the optimized-lookup backend
3901e85 Fix a compiler warning about missing a return statement
e55cdab Remove debug output that should never have been committed
5238401 Fix calculation of several header properties during conversion
7130a07 more functions added to hfst-calculate
35c3e57 Start a document on formats
dbf650c test strings2fst and fail
bc5f85d Rollback bogus repeat throw
23f95bc comment out broken repeats
84d6b26 Fix bug in binary op parameter handling and add test for it
5f3b317 Invert upper and lower to follow upside-down terminology of HFST
8cb2196 Fix no parameters parsing for unary tools and project
72cb416 Test basic algebra
8ce8483 functionality test for concatenate
cbecff2 Add text based transducers for functionality tests
124df37 Fix fst2strings to use set_program_name and rest of the basic tools
893186f small changes done in functions that are needed for hfst-calculate
8e30ba1 Check handling of empty input file for all tools
11692b2 Default to -Wall -Wextra -Werror for developers
041c510 Convert txt2fst; * use const for global char-* * default stdin, stdout for global FILE*
58dff6e Add initial support for evaluating and filtering flag diacritics when extracting transducer paths
90f49d9 Add a class for dealing with flag diacritics in string extraction
7f8204c Port and enable fst2strings
44154e0 Hook up is_cyclic for the optimized-lookup backend
74dcdfc Use warn and err if warnc, errc, error are not available
a425f2b Instead of gnulib-tool, handle gnulib manually
aaa4a7f Missing file
34e3ecb Gnulib weirdness?
075e63c Convert some simpler command line tools to HFST3 (preliminary)
e535c05 Perhaps gnulib for portability problems? Temp. disabled few tools.
c49af70 Few sample tests for tools in automated tests
02a08ed Include licensing blurb in every single file since it's very important
721ee49 Update docs a bit
56b426a First steps towards unbundling backend libraries
caa7704 Remove debug output inadvertently left enabled
6296b07 Fix sign-problem causing format-guessing for OFST transducers to fail
9444698 Add support for converting to the HFST optimized-lookup format
39bc89e Add support for the HFST-optimized-lookup implementation in the tool for format conversion
134c6ea Add support for the new implementation in the interface files
0fd24ab Modify the semantics of the InputStream function is_good in the OFST implementations to match those in the SFST and foma implementations
4883d03 Update makefile for building the new backend
ec67b18 Add conversion HFST-optimized-lookup -> OFST Tropical Weight
41d7301 Initial commit of bridge classes for the HFST-optimized-lookup backend
f8b849b Initial commit of an HFST-optimized-lookup backend
3532dca Add is_cyclic to the transducer API and implementations
532c668 Add hfst-strip-header tool for removing any HFST3 headers from a file
c7a64f3 Port hfst-compare to the new library
96b6ca7 Pass the correct output type when creating the output stream
29dc489 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@471 941e2c2b-deac-454f-805a-451daa25f33c
5834ac0 Port hfst-fst2txt
9d0ade8 Fix the already-ported tools to properly handle IO using the standard streams
5cc5100 Add ability to read the type of the first transducer in an HfstInputStream without first having created a transducer from it
ab1d96c git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@465 941e2c2b-deac-454f-805a-451daa25f33c
714fc13 HfstCompiler class added
7580d1d Fix fst2fst to correctly handle inputs with multiple transducers
e34e30c Teach HfstInputStream how to deal with inputs containing multiple transducers, of which none have HFST3 headers
50d5686 Port hfst-invert to HFST3
1a86ae3 Port hfst-fst2fst to work with HFST3
de1feba Add support in HfstInputStream for loading transducers lacking the HFST3 header
a5e7ec0 Make foma transducer type guesser able to detect non-gzipped transducers as well
7f785cf Add call to open the output stream so the tool doesn't crash
dc85825 Add static methods to the InputStream implementations allowing to check whether a FILE* or istream contains a transducer of a give type
1e6b6cd Remove HFST2 include from file with helper functions for tools
0ab49ed hfst-calculate files updated to hfst version 3. At the moment, hfst-calculate only parses single transition expressions and produces a dummy transducer as a result.
fbe8acb anonymous transducer constructors added
ef4f1c5 Added HfstTransducer &operator=(const HfstTransducer &)
e352b9a hfst-compiler.yy added
544c418 more rule operator functions implemented
6886485 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@409 941e2c2b-deac-454f-805a-451daa25f33c
13ec682 'HfstRules.cc' added.
d72acf3 rule functions added
cfe73d0 stub for flag diacritic support doc
916c12d documentation added
dbd6d6e more Doxygen documentation added
9759312 Convert binary integers to decimal for old gcc support (OS X)
0067fa6 Tried to make the autoreconf text a bit clearer.
7ded1cc ImplementationType parameters removed from most functions
91ac6d0 Require fst/fstlib.h as <fst/fstlib.h> and include as such.
55b3eb5 * Remove openfst from source directory because bundling Apache licenced    software with GNU GPL softare is a bad thing * Add licence disclaimers to auxiliary files because FSF suggests so
c695e9a documentation added
4905303 Documentation and type handling added, some problems in type conversion.
7d5dbb0 add ofst include dir for compilation
f76eb45 sgi.h is visible outside libhfst
7bf66dd Work around problems with missing hash
c618bd7 Add extra doc stuff to build tree
df2f412 more doxygen documentation added
8804a38 some typedefs in SymbolDefs.h and ExtractStrings.h renamed
a05ba5d copy hfst-proc verbatim to 3
ce06b13 Swap stable old trunk to branches and proposition to trunk