Unicode normalized comparison

[harendra-kumar]

Fix the problem described in revert commit cf18703.

[harendra-kumar]

I copied the characters from inside the .cabal file (package name field) and pasted them on the terminal to list the file using ls and it correctly listed the file. This means the filename and the package name inside the cabal file are identical.

The problem is likely to be with the comparison check in init as well as in build.

[harendra-kumar]

This turns out to be pretty interesting!

@borsboom noted the following in his update in cf18703:

show (FP.takeBaseName $ toFilePath fp,show $ gpdPackageName gpd) in this case gives me
("\12399\12441\26085\26412-4\26412","\12400\26085\26412-4\26412");
note the first two bytes in the fst are different than the first byte in the snd.

It happens that those are two equivalent unicode representations of the same visual character representation. I printed those on my terminal after converting the decimal into hex representation. Here is the output:

$ echo -e "\u306f + \u3099 = \u306f\u3099 = \u3070"
は + ゙ = ば = ば

So it seems you can combine two codepoints and it gives you the same character as a third pre-combined codepoint.

So I googled and found this (https://en.wikipedia.org/wiki/Combining_character) on wikipedia:

------quote------
Unicode also contains many precomposed characters, so that in many cases it is possible to use both combining diacritics and precomposed characters, at the user's or application's choice. This leads to a requirement to perform Unicode normalization before comparing two Unicode strings and to carefully design encoding converters to correctly map all of the valid ways to represent a character in Unicode to a legacy encoding to avoid data loss.
------unquote-----

In previous update I wrote that the filename and the package name are identical. But that's not correct, hexdump shows that they are actually different but the shell and the editor treat them as equivalent.

It seems we need to use some sort of a normalized comparison. So there are two questions to be answered:

• How do we do a normalized comparison?
• How did we end up with two different representations - filename using one representation and package name a different one? The source was the same for both of them i.e. the name given on command line and it was a combined character. Perhaps some API somewhere is splitting a codepoint into two. This is just for curiosity. If we do a normalized comparison it won't matter.

[harendra-kumar]

https://hackage.haskell.org/package/text-icu-0.7.0.1/docs/Data-Text-ICU-Normalize.html has a normalize function, we can use that for comparison.

[harendra-kumar]

A bigger question is how to find and fix all places in the code susceptible to the "normalization bug"? And make sure we do not introduce them in future?

[harendra-kumar]

@borsboom since this is a pre-existing bug and not a regression you may want to go ahead with the release irrespective of this. Though, we can quickly fix the two known places i.e. stack init and stack build still we don't know if there are other places which may need fixing.

[harendra-kumar]

Here is a FAQ about unicode normalization: http://unicode.org/faq/normalization.html. Its looking like a PITA to me. And this report - http://www.unicode.org/reports/tr15/ .

[harendra-kumar]

I think this needs to be understood better before we decide on a fix. There has to be a better way than explicitly normalizing at all places. Not sure if normalization happens implicitly somewhere? Perhaps String and Text are using different normalization forms? Need to dig more.

Any Unicode expert out there?

[luigy]

Just wanted to chime in with a test I ran:
Oh unicode! o.O

λ stack new ば日本-4本
λ ls -a
.                 ..                ば日本-4本
λ stack runghc ../stack-1810.hs .
Normalization mode: None
[".","..","\12399\12441\26085\26412-4\26412"]
Normalization mode: NFD
[".","..","\12399\12441\26085\26412-4\26412"]
Normalization mode: NFKD
[".","..","\12399\12441\26085\26412-4\26412"]
Normalization mode: NFC
[".","..","\12400\26085\26412-4\26412"]
Normalization mode: NFKC
[".","..","\12400\26085\26412-4\26412"]
Normalization mode: FCD
[".","..","\12399\12441\26085\26412-4\26412"]

-- ../stack-1810.hs
import qualified Data.Text as T
import Data.Text.ICU.Normalize
import System.Directory
import System.Environment

main :: IO ()
main = do
  [path] <- getArgs
  contents <- map T.pack <$> getDirectoryContents path
  mapM_ (logNormalize contents) modes
  where
    modes = [None,NFD,NFKD,NFC,NFKC,FCD]
    logNormalize contents mode = do
      putStrLn $ "Normalization mode: " ++ show mode
      print $ map (normalize mode) contents

[borsboom]

If at all possible, we want to avoid using text-icu in Stack as that requires installing extra system libraries.

[harendra-kumar]

I was wondering if ghc (7.10.3) handles the same thing correctly. Observations from an experiment:

• It allows precomposed form in an identifier and correctly handles the case when the filename is in decomposed form.
• produces an error when the decomposed form is used in a module name - I guess this should be a bug

cueball $ cat main.hs 
-- Using decomposed form U+306f U+3099
-- GHC produces a lexical error with this
--import Aば日本4本

-- Using precomposed form U+3070
import Aば日本4本

main = func "Hello"

cueball $ cat Aば日本4本.hs 
-- prcomposed form U+3070
module Aば日本4本 where

func = putStrLn

cueball $ runghc main.hs 
Hello

Note the filename is in decomposed form and therefore different than the module name inside the file. If ghc treated them differently it would have produced an error like this:

A.hs:1:8:
    File name does not match module name:
    Saw: ‘B’
    Expected: ‘A’

[harendra-kumar]

Now let's see if ghc compares strings correctly:

cueball $ cat string-equal.hs 
main = if "ば" == "ば" then putStrLn "Pass" else putStrLn "Fail"
cueball $ runghc string-equal.hs 
Fail

If ghc used a normalized form to store strings then the test above would have passed and even our comparison code leading to this issue would have worked correctly. I guess we should discuss this on ghc dev list.