Issues with the coordinates action space

[imenelydiaker]

Hi,

In the following step, the action mouse_click(1219, 228, button='left') is not executed properly on WorkArena L1, even if the blue cursor shows the mouse is on the right element. I'm using GenericAgent with coord action space. See screenshots below:

Before:

After:

[G-JWLee]

Hi! Have you resolved this issue?

[imenelydiaker]

Hi! Have you resolved this issue?

The problem is that the dropdown is not accessible by Playwright.

Claude's analysis: Native <select> dropdowns render their option list as a browser/OS-level popup. Synthetic mouse events (move + down + up) land outside Playwright's DOM context, so they're silently swallowed. Even the original page.mouse.click() has the same problem the dropdown opens on the first click but the option click goes nowhere.

Some possible solutions:

• Removing playwright and moving to pyautogui
• Add a new function that finds the element targeted and applies a select, then add this function to the coord action space:

# Create a custom action 

def select_option_at(x: float, y: float, option: str):
    """
    Find the select element at the given coordinates and select an option by
    value or visible label.
    Use this for native select/combobox dropdowns instead of mouse_click.
    Mouse events cannot interact with native browser select popups.

    Examples:
        select_option_at(270, 167, "blue")
        select_option_at(270, 167, "Red")
    """
    if demo_mode != "off":
        smooth_move_visual_cursor_to(page, x, y)
        highlight_by_box(page, {"x": x, "y": y, "width": 1, "height": 1})
    element_handle = page.evaluate_handle(
        "([x, y]) => document.elementFromPoint(x, y)?.closest('select')",
        [x, y],
    ).as_element()
    if element_handle is None:
        raise ValueError(f"No select element found at coordinates ({x}, {y})")
    element_handle.select_option(option, timeout=500)

# Create a new action set with the existing coord action set + custom action
@dataclass
class CoordActionSetArgs(HighLevelActionSetArgs):
    """Coord action space augmented with select_option_at for native <select> elements.

    Drop-in replacement for HighLevelActionSetArgs(subsets=("coord",), ...).
    Passes select_option_at as a custom action so the function lives in the
    gui_interactions library rather than patching browsergym's venv.
    """

    def make_action_set(self) -> HighLevelActionSet:
        return HighLevelActionSet(
            subsets=list(self.subsets) + ["custom"],
            custom_actions=[select_option_at],
            multiaction=self.multiaction,
            strict=self.strict,
            retry_with_force=self.retry_with_force,
            demo_mode=self.demo_mode,
        )

Hope this helps!