Port liquid-c bug compatible whitespace trimming

[dylanahsmith]

Problem

As part of the work to integrate Shopify/liquid-c#58, we need to opt-out of using Liquid::C::BlockBody in some code to debug verification failures as mentioned in that PR:

As part of our storefront rewrite we have some code to debug differences in the rendered output, which is quite coupled to the liquid parse tree node ruby objects, but doesn't seem to be performance sensitive, so I added a :disable_liquid_c_nodes parse option so this can be used without disabling liquid-c globally (which wouldn't be thread-safe).

Unfortunately, using Liquid::BlockBody for parsing can result in an incompatibility with liquid-c when relying on its bug_compatible_whitespace_trimming: true parse option. This is the same incompatibility issue we were seeing into when testing TruffleRuby with liquid and not liquid-c. So I would like to port that parse option to unblock these use cases.

Solution

The bug prevents pre-trimming from removing the first character in the preceding string token. So we can just restore that byte after stripping if previous_token.rstrip! fully strips the token (i.e. previous_token was a non-empty blank string).

The tests were copied from the above mentioned liquid-c pull request, which we can now remove them from that repo after this is merged.

[pushrax]

If it makes anything easier, we're not really using the highly-coupled Liquid diff debugging system in SFR, it's actually somewhat buggy itself. It could be removed entirely.

[dylanahsmith]

Ok, I'll delete that code if it causes more problems. However, I still think it would be useful to be able to dynamically use ruby liquid in a thread-safe way for verification in the future, in which case having them be compatible will be necessary. That could make it easier to detect incompatibilities, like this whitespace trimming bug, in production.

[dylanahsmith]

pushed a commit to port this test too, which is the corresponding test without the bug_compatible_whitespace_trimming: true option

[XrXr]

My memory on this is hazy but the change makes sense when I review it with related changes.
Related PRs:

• Shopify/liquid-c#51
• Shopify/liquid-c#54