Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 22 additions & 8 deletions lib/prism/lex_compat.rb
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# frozen_string_literal: true
# :markup: markdown

require "delegate"

module Prism
# This class is responsible for lexing the source using prism and then
# converting those tokens to be compatible with Ripper. In the vast majority
Expand Down Expand Up @@ -201,27 +199,43 @@ def deconstruct_keys(keys)
# When we produce tokens, we produce the same arrays that Ripper does.
# However, we add a couple of convenience methods onto them to make them a
# little easier to work with. We delegate all other methods to the array.
class Token < SimpleDelegator
# @dynamic initialize, each, []
class Token < BasicObject
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to not just subclass Array?
That would avoid an extra indirection and (in theory at least) be more compatible.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this but then ignores the == from the subclass. Somehow also slower on cruby

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, that's subtle, I think it's because of these semantics: https://github.com/truffleruby/truffleruby/blob/a48ecf894f19636f09235f5ac70ea6d9889be255/src/main/ruby/truffleruby/core/array.rb#L129-L132
i.e. if other is not an Array then always use other == self, but if other is an Array then just compare all elements and don't call other.==

# Create a new token object with the given ripper-compatible array.
def initialize(array)
@array = array
end

# The location of the token in the source.
def location
self[0]
@array[0]
end

# The type of the token.
def event
self[1]
@array[1]
end

# The slice of the source that this token represents.
def value
self[2]
@array[2]
end

# The state of the lexer when this token was produced.
def state
self[3]
@array[3]
end

# We want to pretend that this is just an Array.
def ==(other) # :nodoc:
@array == other
end
Copy link
Member

@eregon eregon Jan 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For speed it might be relevant to define to_ary, since that will be checked by Array#== and more methods when using this Token as an Array (e.g. https://github.com/truffleruby/truffleruby/blob/a48ecf894f19636f09235f5ac70ea6d9889be255/src/main/ruby/truffleruby/core/array.rb#L130).
Defining to_ary will avoid going through respond_to_missing? for that case.


def respond_to_missing?(name, include_private = false) # :nodoc:
@array.respond_to?(name, include_private)
end

def method_missing(name, ...) # :nodoc:
@array.send(name, ...)
end
end

Expand Down