Preserve comments when re-writing text Ion

[tgregg]

When a user has some text Ion that contains comments, and that user wants to change a few values in that stream but keep it in text Ion, there is currently no way to preserve the comments.

Maybe we can find a way to do this via an existing API like IonWriter.writeValues(IonReader). Or maybe we need a different API that makes it easy to make targeted changes to existing Ion streams.

[austnwil]

We could add an IonReader.nextOrComment() that, instead of skipping all whitespace and placing the cursor on the next value, halts if it finds a comment and gives you a chance to do something with it.

final String ion = """
"hello"
/* I am a comment */
// me too
"world"
""";

final IonReader reader = IonReaderBuilder.standard().build(ion);

// Skip any comments, place cursor at start of next value.
reader.next();
reader.stringValue(); // Returns "hello"

String comment;
while((comment = reader.nextOrComment()) != null) {
    // A comment was encountered while seeking to the next value.
    // Cursor is not positioned on the next value yet, need to call nextOrComment() until it
    // is null or just call next() to skip all whitespace (including remaining comments).
    System.out.printf("Found a comment: %s\n", comment);
}

reader.stringValue(); // Returns "world"

Or a IonReader.commentValue() or similar that treats comments almost like any other value and a configuration on the builder to recognize comments:

final String ion = """
"hello"
/* I am a comment */
// me too
"world"
""";

final IonReader reader = IonReaderBuilder.standard().withCommentPreservation().build(ion);

reader.next();
reader.stringValue(); // Returns "hello"
reader.next();
reader.commentValue(); // Returns "I am a comment"
reader.next();
reader.commentValue(); // Returns "me too"
reader.next();
reader.stringValue(); // Returns "world"

// Alternatively, IonTextWriter.writeValues would now write values and comments directly from the reader with comment preservation on

IonTextWriterBuilder.standard().build(System.out).writeValues(reader);

A tricky case is comments that appear between long strings that are concatenated together, since the breaks in the string are not preserved during a round-trip:

[austnwil@14147d973d49 ~ % echo "  '''I am''' /* what about me? */ ''' a string'''  " | ion cat
"I am a string"

[austnwil]

These are some cases in which I foresee comments being difficult to deal with.

Between concatenated long strings:

'''some''' /*comment*/ '''string'''

Between annotation separators:

hello /*comment*/ :: 123
hello :: /*comment*/ 123

In structs:

{ /*comment*/ struct : 123 }
{ struct /*comment*/ : 123 }
{ struct : /*comment*/  123 }
{ struct : 123 /*comment*/ }

In lists:

[ /*comment*/ 1, 2 ]
[ 1 /*comment*/ , 2 ]
[ 1, /*comment*/ 2 ]
[ 1, 2 /*comment*/ ]

For a lot of these cases, the comment appears in the middle of a single "value", so how would you expose this information from a reader? How do you differentiate between a comment on the left and right side of a colon in a struct or comma in a list?