System.Text.Json: add ability to do semantic comparisons of JSON values à la JToken.DeepEquals()

[dbc2]

API proposal

namespace System.Text.Json;

public partial struct JsonElement
{
    public static bool DeepEquals(JsonElement element1, JsonElement element2);
}

// Existing API
public partial class JsonNode
{
    public static bool DeepEquals(JsonNode? node1, JsonNode? node2);
}

Alternative Designs

The static method approach mirrors the shape used by JsonNode.DeepEquals, however in that case we are forced to use a static because null is a valid representation for the case of JsonNode (representing JSON null). This concern does not exist here so might as well just use an instance method instead.

In terms of implementation, property comparison always uses case sensitive ordinal comparison however we might consider introducing case insensitive comparison as well.

Original post

Json.NET has the ability to do a deep semantic comparison of two JSON tokens via [`JToken.DeepEquals()`](https://www.newtonsoft.com/json/help/html/M_Newtonsoft_Json_Linq_JToken_DeepEquals.htm):

Compares the values of two tokens, including the values of all descendant tokens.

It also provides JTokenEqualityComparer for hashing and comparing of JSON tokens.

There does not appear to be an equivalent functionality in System.Text.Json for comparing two JsonElement or JsonDocument objects. As this is a common requirement (e.g. in writing unit tests, or checking for differences between object versions) it would be useful to have.

Sample Stack Overflow questions:

• What is equivalent in JToken.DeepEqual in System.Text.Json
• Compare two JObjects or JArray
• How to compare two Json objects using C#
• How can I create a unique hashcode for a JObject?

Issues:

1. Formatting should be ignored.

2. Differences due to string escaping probably should be ignored. (E.g. "a" should be equivalent to "\u0061".)

3. Differences due to trailing zeroes probably should be significant. When deserializing to decimal trailing zeroes are preserved; financial, engineering and scientific apps sometimes make use of this.

   Example Stack Overflow questions:

   • JsonConvert deserializing decimal to jtoken converts it to scientific number, how do i convert it back to decimal?.
   • JObject.Parse modifies end of floating point values.
   • Parsing JSON rounds Negative Values.
   • NewtonSoft Json deserialize decimal numbers with more then 8 decimals.

   Utf8JsonReader preserves the underlying character representation when parsing a number so you have the advantage over JsonTextReader here, as the latter discards the character representation after recognizing a token as a number.

4. Differences due to ordering of unique property names should be ignored since the original JSON proposal states, An object is an unordered set of name/value pairs.

5. Differences due to ordering of duplicate property names require some thought.

   I have noticed that, surprisingly, JsonDocument fully supports duplicate property names! I.e. it's perfectly happy to parse {"Value":"a", "Value" : "b"} and will store both key/value pairs inside the document. (Thanks for that, I guess.)

   A close reading of https://tools.ietf.org/html/rfc8259#section-4 seems to indicate that such objects are allowed but not recommended, and when they arise, interpretation of identically-named properties may be order-dependent. So I'd propose that relative order of identically-named properties should be significant while relative order of distinctly named properties should not. Such a proposal could be implemented by stably sorting the properties by name, then comparing the sorted property lists in order.

6. Differences due to casing of property names should be significant by default, but possibly there should be a configuration argument to ignore property name casing. (JToken.DeepEquals() doesn't support this.)

One possible demo comparer can be found here, but better performance may be possible by using internal methods.

[layomia]

Triage: this isn't on our roadmap right now.

It would require discussion on what the behavior would be. It would be good to know what scenarios this would be used for. You provided a set of exceptions here, would you hardcode these or would you want options to control these?

[bjbr-dev]

I'd appreciate this being considered for the next iteration. We use Newtonsoft's JToken.DeepEqual extensively in unit test code, and we were hoping to see the equivalent in System.Text.Json.

This stack overflow question also highlights community desire for this feature: https://stackoverflow.com/questions/60580743/what-is-equivalent-in-jtoken-deepequal-in-system-text-json

[mattdarwin]

me too please. I want to be able to compare two JsonDocument objects for logical equality, ignoring formatting and property ordering.
Maybe we should be able to vote on these issue, so you can see how much demand there is for this fix.

[svick]

@mattdarwin

Maybe we should be able to vote on these issue, so you can see how much demand there is for this fix.

You can, you do that but giving the issue the "+1" (👍) reaction.

[gregsdennis]

I've been able to define some deep equality functionality in my Json.More package, but doing .GetHashCode() efficiently still eludes me (although I really like that first SO answer).

JsonElement doesn't override this, which means that different instances (remember it's a value type, so you can only practically get different instances unless you start using pointers, which... ew) return different hash values, even if they are otherwise exactly equal.

In trying to implement my own hash code generation, I'd like to use .GetRawText(), but it only returns the input string. This presents a problem for objects and arrays. For example, [ 1, 2, 3 ] and [1,2,3] return those values, respectively, so they don't work for getting a "JSON hash code." And there's not another way (that I'm aware of) to get a consistent output short of putting the element through the serializer.

So that leaves us with performing a serialization in the midst of a method that is supposed to be as efficient as possible.

(Marginally related to #42502.)

[sashafp10]

As a workaround, you can remove all non-valuable symbols like spaces, tabs, newlines, and so on. Then just sort all symbols in alphabetical order. There is a very small chance that for big JSON files such strings will be equal to each other.

[eiriktsarpalis]

Related to #56592, which proposes a DeepEquals method for JsonNode.

@steveharter would it make sense to also include this functionality for JsonElement? I suspect we might need to implement it since certain JsonNode instances are backed by JsonElement representations.

[bd]

I, too, could really use the ability to compare JsonDocuments for content.