Number, clock and date fst's are upside down for half the Oahpa languages (Bugzilla Bug 1538)

[albbas]

This issue was created automatically with bugzilla2github

Bugzilla Bug 1538

Date: 2012-12-12T16:13:25+01:00
From: @snomos (Originally created by @snomos)
To: @Trondtr
CC: heli1401, @leneantonsen, ryan.txanson, @Trondtr

Last updated: 2017-10-04T23:00:35+02:00

[albbas]

Comment 7541

Date: 2012-12-12 16:13:25 +0100
From: @snomos

I'm trying to add proper compilation of the numbers, clock and date fst's to the new infrastructure, but I come out short because the intended use and behaviour of the fst's are not documented.

Examples:

• The old infra refers to a filter that removes +Use/NA tagged strings, but every string except 24:00 has that tag, and the intention can't possibly be to make such a transducer.

• The old infra uses the "analyzer" and "generator" terms even for these transducers, but exactly what that means in terms of +Use/NA and +Use/NG is far from clear.

What I need is a clear specification of exactly what transducers are needed for the different usages:

• web page
• oahpa (with sub-uses, like showing facit, accepting input, etc.)

I have put a relatively high priority on this, since I need to move the new infra work forward as soon as possible, this included.

Any feedback is greatly appreciated :)

[albbas]

Comment 7542

Date: 2012-12-12 16:22:09 +0100
From: @Trondtr

This is for me and Ryan, we were the ones introducing the +Use/NA business without documenting it...

[albbas]

Comment 7552

Date: 2012-12-13 00:28:59 +0100
From: Ryan Johnson <ryan.txanson>

Me and Trond have been discussing what this is. For sma-oahpa and univ_oahpa we are using the clock and date FSTs for checking user answers and generating question and answer text. +Use/NG is still used in these not to prevent generation as a whole, but to prevent generation of substandard forms for presentation as question/answers in Oahpa. These forms are still accepted as possible answers of course, because user-input is taken in a non-normative sense-- we just want to be normative in what we show.

In the compilation of the clock FST (according to revision 59359, which is the "frozen" version we are using for smaoahpa). I think what the purpose was supposed to be was to present a specific set of forms with clock-sma.fst, and restrict analyses with +Use/NA, and then in iclock-sma.fst, restrict generation of words with +Use/NG. However, now I think I am having difficulty getting the correct/intended behavior with the FSTs in this version, which is problematic because it is affecting the output of Numra.

[ryan@gtoahpa ~]$ lookup -flags mbTT /opt/smi/sma/bin/clock-sma.fst 
njieljie
njieljie    4:00
njieljie    4.00
njieljie    4
njieljie    04:00
njieljie    04.00
njieljie    04
njieljie    16:00
njieljie    16.00
njieljie    16

nieljie
nieljie 4:00
nieljie 4.00
nieljie 4
nieljie 04:00
nieljie 04.00
nieljie 04
nieljie 16:00
nieljie 16.00
nieljie 16

[ryan@gtoahpa ~]$ lookup -flags mbTT /opt/smi/sma/bin/iclock-sma.fst 
4:00
4:00    njieljie
4:00    njielje
4:00    nieljie
4:00    nielje

4.00
4.00    njieljie
4.00    njielje
4.00    nieljie
4.00    nielje

This is currently resulting in users seeing:

15:30
bielie njieljie, bielie njielje, bielie nieljie, bielie nielje

However, the other way is fine, because the first result is the accepted form. I am trying to remember now if this is what happened, is we switched away from using +Use/NA, opting to use the order of generation to present a normative question to the user. The rest of Oahpa otherwise works by not presenting NG forms, but keeping them around to accept nonstandard answers.

[albbas]

Comment 9323

Date: 2014-04-21 18:17:06 +0200
From: @Trondtr

This bug has come back to us with the extension to several new Oahpa languages. Heli and I am working on an improved Numra documentation. Input welcome.

[albbas]

Comment 9558

Date: 2014-08-20 11:40:30 +0200
From: @Trondtr

Heli and I have fixed the behaviour in Numra, so now it works. Documentation is found here:

http://giellatekno.uit.no/ped/common/Numra.html

I suggest we close the bug.

[albbas]

Comment 9614

Date: 2014-10-07 09:46:44 +0200
From: @Trondtr

No closing yet, no:

Sjur, Heli and Trond finally got a new setup made in Helsinki. Appr 1/3 of the lexc files are now in correct shape. The Oahpa versions still have the old versions.

TODO:

• Update the transducers to digit:text
• Evaluate
• Update Oahpa

[albbas]

Comment 9652

Date: 2014-10-21 06:52:09 +0200
From: @Trondtr

All languages have been moved to the new format. As for testing, I have done functionality testing, but cannot guarantee for correct content.

Oahpa inclusion: I think we are there. Heli, what do you say?

[albbas]

Comment 10599

Date: 2015-08-13 17:25:18 +0200
From: @Trondtr

Now, we have 37 languages with correct order digit-text, and 46 with wrong order.

hmm, there are languages in both camps. But the issue needs to be looked into.

[albbas]

Comment 10691

Date: 2015-09-21 09:23:10 +0200
From: @snomos

The original issue ("Number, clock and date fst's are not properly documentet, no new infra version") is at least partly solved, there has been support for these fst's in the (not so new anymore) infra for a long time. Documentation is probably still weak.

During the course of this bug another issue has been added, that of the direction of the lexc code. That is solved on an infrastructure level, but as commented:

(In reply to Trond Trosterud from comment #7)

Now, we have 37 languages with correct order digit-text, and 46 with wrong
order.

the content might not always be correct according to the defined format.

Trond, can you have a look, or kick the bug to someone else?

[albbas]

Comment 10696

Date: 2015-09-21 10:26:11 +0200
From: @snomos

No need to have Biret Ánne on the CC list anymore.

[albbas]

Comment 10737

Date: 2015-09-21 11:33:42 +0200
From: @Trondtr

These are 3 different fst types.
It might be possible just to script this, let us have a look (I will have a look next week), evt. we will have to share work between us.

[albbas]

Comment 11865

Date: 2016-12-15 20:56:44 +0100
From: @Trondtr

"As soom as possible" (2015) seems to be a good description of last year. Oahpa is not on the agenda now, there is a deadline coming up, so P4 ("within a month") is unfortunately too high. But I will look into the languages now and then.

[albbas]

Comment 12643

Date: 2017-10-04 22:27:45 +0200
From: @snomos

Status?

[albbas]

Comment 12650

Date: 2017-10-04 23:00:35 +0200
From: @Trondtr

Clock:
grep '58:' */src/transcriptions/transcriptor-clock-digit2text.lexc |cut -d":" -f1|uniq|wc -l
grep ':58' */src/transcriptions/transcriptor-clock-digit2text.lexc |cut -d":" -f1|uniq|wc -l

I langs-katalogen: 42 korrekt, 10 feil. Dei 10 feil er:
kal,mns,mrj,olo,sjd,sje,sma,smj,smn,udm

For dei andre to er ssv situasjonen tilsvarande.