Newsfeed module cannot show description right. Wrong HTML handling?

[Jochbart]

Hi together,
somehow, the newsfeed module cannot handle the RSS feed right. I tried different sources but every time the same at my MagicMirror:

Thank you in advance.

[sdetweil]

can u post the rss url

[Jochbart]

https://www.main-echo.de/rss/stadt_kreis_aschaffenburg.xml

[sdetweil]

thanks.. the data is encoded, but the module does not attempt to decode it

<item>
<title>
<![CDATA[ &raquo;Thomaidis Gewerbepark Hanau&laquo; fast fertig ]]>
</title>
<link>https://www.main-echo.de/7421848?utm_source=rssfeed&utm_medium=rss&utm_campaign=rss-aschaffenburg</link>
<dc:creator>
<![CDATA[ Pressestelle der Stadt Hanau ]]>
</dc:creator>
<pubDate>Thu, 18 Nov 2021 16:54:09 +0000</pubDate>
<description>
<![CDATA[ &raquo;Da&shy;mit er&shy;f&auml;hrt der Ha&shy;nau&shy;er Ha&shy;fen den gr&ouml;&szlig;&shy;ten Ent&shy;wick&shy;lungs&shy;schub seit lan&shy;gem&laquo;, freut sich Ha&shy;n&shy;aus Ober&shy;b&uuml;r&shy;ger&shy;meis&shy;ter Claus Ka&shy;mins&shy;ky &uuml;ber die An&shy;sied&shy;lung des in&shy;ha&shy;ber&shy;ge&shy;f&uuml;hr&shy;ten Spe&shy;di&shy;ti&shy;ons&shy;un&shy;ter&shy;neh&shy;mens Gebr. Tho&shy;mai&shy;dis GmbH auf dem ehe&shy;ma&shy;li&shy;gen Ca&shy;bot-Ge&shy;l&auml;n&shy;de im Ha&shy;nau&shy;er Ha&shy;fen. Der Im&shy;mo&shy;bi&shy;lien&shy;in&shy;ves&shy;tor und Pro&shy;jekt&shy;ent&shy;wick&shy;ler Al&shy;pha In&shy;du&shy;s&shy;trial schafft dort bis En&shy;de 2021 den &raquo;Tho&shy;mai&shy;dis Ge&shy;wer&shy;be&shy;park Ha&shy;nau&laquo; auf dem 3,5 Hektar gro&shy;&szlig;en Areal in der Jo&shy;sef-Bautz-Stra&shy;&szlig;e, das er vom Main-Kin&shy;zig-Kreis er&shy;warb. ]]>
</description>
<content:encoded>
<![CDATA[ &raquo;Da&shy;mit er&shy;f&auml;hrt der Ha&shy;nau&shy;er Ha&shy;fen den gr&ouml;&szlig;&shy;ten Ent&shy;wick&shy;lungs&shy;schub seit lan&shy;gem&laquo;, freut sich Ha&shy;n&shy;aus Ober&shy;b&uuml;r&shy;ger&shy;meis&shy;ter Claus Ka&shy;mins&shy;ky &uuml;ber die An&shy;sied&shy;lung des in&shy;ha&shy;ber&shy;ge&shy;f&uuml;hr&shy;ten Spe&shy;di&shy;ti&shy;ons&shy;un&shy;ter&shy;neh&shy;mens Gebr. Tho&shy;mai&shy;dis GmbH auf dem ehe&shy;ma&shy;li&shy;gen Ca&shy;bot-Ge&shy;l&auml;n&shy;de im Ha&shy;nau&shy;er Ha&shy;fen. Der Im&shy;mo&shy;bi&shy;lien&shy;in&shy;ves&shy;tor und Pro&shy;jekt&shy;ent&shy;wick&shy;ler Al&shy;pha In&shy;du&shy;s&shy;trial schafft dort bis En&shy;de 2021 den &raquo;Tho&shy;mai&shy;dis Ge&shy;wer&shy;be&shy;park Ha&shy;nau&laquo; auf dem 3,5 Hektar gro&shy;&szlig;en Areal in der Jo&shy;sef-Bautz-Stra&shy;&szlig;e, das er vom Main-Kin&shy;zig-Kreis er&shy;warb. ]]>
</content:encoded>
</item>

[sdetweil]

the spec says html-entities are allowed.. these seem language specific.

­
ä
ö
«
ü
ß

but they display correctly here (5 of the 6)
­
ä
ö
«
ü
ß

none of the decoders I can find handle those.

there is an approach it seems

create a textarea html object,
post this as the html content
extract the text content
https://stackoverflow.com/questions/7394748/whats-the-right-way-to-decode-a-string-that-has-special-html-entities-in-it/7394787

function decodeHtml(html) {
    var txt = document.createElement("textarea");
    txt.innerHTML = html;
    return txt.value;
}

[rejas]

ah, good old german umlauts :-)

[rejas]

but the "­" dont get decoded neither :-(

[Jochbart]

the spec says html-entities are allowed.. these seem language specific.

­
ä
ö
«
ü
ß

but they display correctly here (5 of the 6) ­ ä ö « ü ß

none of the decoders I can find handle those.

there is an approach it seems

create a textarea html object, post this as the html content extract the text content https://stackoverflow.com/questions/7394748/whats-the-right-way-to-decode-a-string-that-has-special-html-entities-in-it/7394787

function decodeHtml(html) {
    var txt = document.createElement("textarea");
    txt.innerHTML = html;
    return txt.value;
}

Hi sdetweil,
thanks for your help. I will try to realize your mentioned solution, but I am still a big noob if it is about programming and its logic. But I cannot imagine that I am the only German user of this module and has this problem. How can this be?

[sdetweil]

I was not suggesting YOU solve this.

I confirmed the problem and the cause.

The newsreader module does not handle the encoded description.

I do not know how to resolve it. The few things I tried do not provide a fix.

[rejas]

@Jochbart you need to add

encoding: "ISO-8859-1",

to the section of that newsfeed in your config.js. Then the umlaute etc should appear correctly.

[Jochbart]

@Jochbart you need to add

encoding: "ISO-8859-1",

to the section of that newsfeed in your config.js. Then the umlaute etc should appear correctly.

I added this already and my news headline is right.

[rejas]

Happy to hear that. So this issue can be closed @Jochbart ? Or would you say that the documentation would need some clarification regarding this?

[Jochbart]

Happy to hear that. So this issue can be closed @Jochbart ? Or would you say that the documentation would need some clarification regarding this?

No, we miss understood us. I added this already before I opened this issue. It helped me to encode the news headline right, but I have this problem in the news description. You can see it in the screenshot I posted: „wächst“ is right, but the description is still wrong with html commands.

[rejas]

Ah I see, yes, the description still has the encoding issues.

Not sure if it is something the MM should fix (replacing the special html entities with "correct" chars) or if that is something the used iconv-lite library needs to adress.

What doy ou think @sdetweil ?

[sdetweil]

iconv is used to decode the entire fetch payload, so it 'should' have worked on the description too..

response.body.pipe(iconv.decodeStream(encoding)).pipe(parser);

but adding debug shows title not converted correctly

"title": " �PNV-Ausschuss:

which matches iconv doc

Untranslatable characters are set to � or ?. No transliteration is currently supported.

so I think something else would have to be done.
and iconv has already mangled the content