Bad corpora should be flagged

[johnml1135]

If there is an extraneous tab in a corpora, it will crash the SMT build process with an error "Object must be of type String." This error is cryptic and does not allow the user to debug what is going on. It should either be clear "this file on this line is not correctly formatted - here is the correct format" or clean the files in a deterministic way before being passed down to the engines. This may need to be specific per engine type.

[Enkidu93]

Would you prefer a 'silently clean' solution or a 'throw an error from the machine side' solution? Seems like the latter might be more straightforward and less likely to cause difficult-to-debug headaches. But if it this trailing tab or similar issue is likely to be common, maybe it is just worth cleaning.

[johnml1135]

@Enkidu93, I would rather throw an intelligent error. If we clean, I would want to define it clearly - I would hate for people's data to be changed unexpectedly.

[Enkidu93]

OK, sounds good. I tried to recreate this by adding a tab with no luck (it completed and worked properly). Where did you add the extra tab?

[johnml1135]

At least before, this type of file caused errors:
MyFileWithOneTab.txt

my long sentence\t
my other long sentence
my third long sentence

Note the tab (\t) at the end of the first sentence
I believe the trailing tab specifically causes errors - in that it makes a zero length line as per these to pieces of code:
https://github.com/sillsdev/machine/blob/940ca147dc83638cc9c3da2128b7573af511381d/src/SIL.Machine/Corpora/TextFileAlignmentCollection.cs#L52-L59
https://github.com/sillsdev/machine/blob/940ca147dc83638cc9c3da2128b7573af511381d/src/SIL.Machine/Corpora/TextFileText.cs#L43-L48

The idea was seeing that but a bit more general. We have an odd format - either text per line or a keys (separated at times by '_' and by '-') that is then separated by a tab from the rest of the text.
A tab at the beginning, a tab at the end, keys separated by '-', a tab in a sentence that is unintentional - all of these would create bad data or crashing.
To remedy this here is a proposal:

• When uploading files, if they are of type "text", run a "text" check:
  • Verify that all lines either have one tab or none
• Change the behavior of separating keys to only underscores "_". Document it in the API. Update the code in TextFileAlignmentCollection.cs to only split on underscores.

[ddaspit]

What is the issue with dashes in a key?

[ddaspit]

As @Enkidu93 said, we could make the code a bit more fault-tolerant and simply ignore any trailing whitespace, such as tabs, when reading the text file.

[johnml1135]

(1) the issue with the dashes is that it is inconsistent (used in one place but not the other) and undocumented. trimming white space sounds fine - that is unless we get a whole raft of verse references with no sentences! I would be more inclined to have the clients clean the data and then they can establish the intention better. It is really a data format that we are specifying - I wouldn't want to "clean" csv files saying "you didn't follow the rules, but I think this is what you intended". I would rather say "here are the rules, and if you don't follow them, I will error out and tell you why."

[ddaspit]

We can document that we support either dashes or underscores to separate different components of a key.

I would like to keep the Machine corpus classes as fault-tolerant as possible, since those are often used by researchers, but I don't have a problem with being more strict in Serval and validating the files when they are uploaded.