Revise tools scripts to be python3 compatible

[lygstate]

Revise tools scripts to be python3 compatible
We introduce setup_stdio function to setup stdout/stderr properly.
For python <-> python pipe, we always use 'utf8'/'ignore' encoding for not lost
characters.
For tty <-> python, we using native encoding with xmlcharrefreplace to encode, to
preserve maximal information.
For python <-> native program, we use naive encoding with 'ignore' to not cause error

For reading/writing to file, we use 'utf8' /'ignore' encoding for not lost
characters.

JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com

[LaszloLango]

LGTM

[ossy-szeged]

Do we need it to be able to use our harness for ES5.1 version of test262? If yes, it would be great to mention it in the commit log at least.

The CI fails now, must be reviewed again after the patch is fixed.

[lygstate]

Do we need it to be able to use our harness for ES5.1 version of test262? If yes, it would be great to mention it in the commit log at least.
Doesn't need that, only python3 or windows need that

[akosthekiss]

During the review of the current version of the PR, I've found a potentially useful (and definitely interesting) approach to the problem, which may reduce the necessary code changes considerably. With just two lines of code (per tool), the unicode encode/decode errors can be disabled/hidden/ignored:

import codecs
codecs.register_error("strict", codecs.ignore_errors)

(source: https://stackoverflow.com/questions/9537356/can-i-make-decodeerrors-ignore-the-default-for-all-strings-in-a-python-2-7-p)

This would be worth a try. Then, no stdour/err would be necessary, no polluting of the code with encodes/decodes, no codecs.open instead of open, etc. (Hopefully.)

[akosthekiss]

Please, edit the commit message. It is unacceptably long. Tracebacks and diagnostics and console logs should not go into a commit message. They belong to the discussion comments of the corresponding PR.

[lygstate]

During the review of the current version of the PR, I've found a potentially useful (and definitely interesting) approach to the problem, which may reduce the necessary code changes considerably. With just two lines of code (per tool), the unicode encode/decode errors can be disabled/hidden/ignored:

import codecs
codecs.register_error("strict", codecs.ignore_errors)

(source: https://stackoverflow.com/questions/9537356/can-i-make-decodeerrors-ignore-the-default-for-all-strings-in-a-python-2-7-p)

This would be worth a try. Then, no stdour/err would be necessary, no polluting of the code with encodes/decodes, no codecs.open instead of open, etc. (Hopefully.)

Yeap, I know this method, but the following are still better.

def setup_stdio():
    (out_stream, err_stream) = (sys.stdout, sys.stderr)
    if sys.version_info.major >= 3:
        (out_stream, err_stream) = (sys.stdout.buffer, sys.stderr.buffer)
    # For tty using native encoding, otherwise (pipe) use 'utf-8'
    encoding = sys.stdout.encoding if sys.stdout.isatty() else 'utf-8'
    # Always override it to anvoid encode error
    sys.stdout = codecs.getwriter(encoding)(out_stream, 'ignore')
    sys.stderr = codecs.getwriter(encoding)(err_stream, 'ignore')

codec ignore is just hidden the error when output isatty, but when we do piping, we always using the 'utf8-8' encoding.
Otherwise the characters will lost in tests script :(
That's why setup_stdio are important, not just simple ignore it

[lygstate]

@ossy-szeged any progress of the PR? after this, I can create PR that support python3 debugger