Dogwrap fails to event with unicode in stderr, stdout when using buffer_outs option #201
Description
[invalid] Real issue from -b (buffer_opts) with stderr,stdout redirect
Environment
- Shared Hosting (cPanel)
- Python 2.7.13
~$ ./.localpython/Python-2.7.12/python --version
Python 2.7.13
- Dog/Dogwrap v0.14.0
~$ ./.local/bin/dog --version
dog 0.14.0
~$ ./.local/bin/dogwrap --version
dogwrap 0.14.0
Context
Using dogwrap via cron to run a PHP cli script:
/home/MYUSER/.local/bin/dogwrap -n "My Cron" -k APIKEY -b --submit_mode all --notify_error="Notifying @MYEMAIL1 @MYEMAIL2" --tags=cron,test "php -q cron.php" >/dev/null 2>&1
The cron runs every 15 minutes. When there are no tasks, a small summary string of (ascii) text is sent to stdout from PHP cli (dogwrap correctly captures it, and I see it in my events view):
My Automation Task
===================================
...
[OK] Completed
However, each and every time cron.php does have a task to do, dogwrap fails to send the event to datadog server. The dogwrap command doesn't seem to exit or crash, it simply finishes and prints buffered output from stdout and stderr like everything is fine, but the event never seems to send. The output in these cases, seen by removing null route >/dev/null is:
0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 0% < 1 sec/< 1 sec 16.0 MiB
1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100% 2 secs/2 secs 34.0 MiB
My Automation Task
=================================
...
[OK] Completed
Thus, the only seemingly significant difference I can see between the outputs is the use of extended character encoding via UTF-8. Specifically the use of:
(UTF-8) Light Shade (U+2591) | ASCII: 9617
https://www.utf8icons.com/character/9617/light-shade
(UTF-8) Dark Shade ( U+2593) | ASCII: 9619
https://www.utf8icons.com/character/9619/dark-shade
I believe this is likely a unicode encoding issue.
Without redirecting stdout, there is no crash, everything prints fine in my console (where sys.stdout.encoding=UTF-8), and the event is successfully transmitted.
If I test with some unicode, and redirect to a file instead of to /dev/null, I see an exceptions:
Traceback (most recent call last):
File "/home/static/.local/bin/dogwrap", line 11, in <module>
load_entry_point('datadog==0.14.0', 'console_scripts', 'dogwrap')()
File "/home/static/.local/lib/python2.7/site-packages/datadog-0.14.0-py2.7.egg/datadog/dogshell/wrap.py", line 316, in main
print >> sys.stderr, stderr.strip()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2591' in position 30: ordinal not in range(128)
So maybe there is an issue with unicode support, conversion, handling, etc?
I mostly use Python 3+ these days, but, if I understand correctly, most datadog related python is based on version 2.7.x where unicode support is handled differently? (I.e. in 3+ a str is unicode by default, but no so in 2.7.x I think?)
- + Unicode output in
stdout,stderr(with/without IO redirection). - + Better consistency when handling unicode.
- ~ Would checking for I/O redirection be useful? Any unintended side effects?
- + Tags including unicode are ignored? Ex:
--tags=test░tag, tag does not appear on event.
According to dogwrap --help:
-b, --buffer_outs displays the stderr and stdout of the command only
once it has returned (the command outputs remains
buffered in dogwrap meanwhile)
It seems like the buffer_outs option causes different path for string printing, that doesn't include / support unicode encoding. I believe the culprit here in datadogpy.dogshell.wrap:
if options.buffer_outs:
print >> sys.stderr, stderr.strip()
print >> sys.stdout, stdout.strip()Following the state of stdout, it is concatenated with a unicode string, changing its type to unicode as well. However, it is never encoded back before printing.
Pull Request for minimal fix for just this localized issue: #203