retrieving a textarea value with ChromeDriver for Chrome 91 headless vs non-headless

[timdiggins]

Full details below, but here's a summary of what happens on Chromedriver 91 headless only.

e = find("textarea#sometextareaid")
e.value # => nil
# and 
e.native['_capybara_raw_value'] # => nil
# but  
e.native.property(:value) # => "some value"

So this could be a bug in chromedriver, but capybara is doing some weird stuff with textarea values (native['_capybara_raw_value']) and I can't follow where this gets set on update, so it's possible it's a problem in capybara

Meta

Capybara Version:

capybara (3.35.3)

Driver Information (and browser if relevant):

selenium-webdriver (3.142.7) with Chrome 91.0.4472.77, chromedriver 91.0.4472.19

Additionally:

Nokogiri.respond_to?(:HTML5) && Capybara.allow_gumbo
=> false

This started happening when Chrome went from 90 -> 91 (on CI (ubuntu) as well as locally (MacOS)).

This only happens on chrome headless, not on chrome headed.

Setup

find("#sometextareaid").fill_in(with: "some value")`
expect(find("#sometextareaid").value).to eq("some value")

NB: I can validate that the current value is actually "some value" by creating a png screenshot with capybara-screenshot (either in a REPL, or in a line before the expect)

FYI also occurs with the following (just to rule out any timing issues)

find("#sometextareaid").fill_in(with: "some value")`
sleep(5)
expect(page).to have_field("sometextareaid", with: "some value")

Expected Behavior

expect to get the current value of the textarea

Actual Behavior

get nil always

Steps to reproduce

<textarea id="sometextareaid">
initial value
<textarea>

FYI I have only tested with a much more maximal version, and it's possible that there is some weird detail in the maximal version that is critical. However it's suddenly (post 91) happening for me across multiple tests, all to do with testing the values of different textareas in different contexts. If you require a realistic minimal repro, let me know.

[twalpole]

'_capybara_raw_value', Nokogiri.respond_to?(:HTML5) and Capybara.allow_gumbo are only relevant when using the rack_test driver or simple nodes, so don't have anything to do with using Chrome.

[twalpole]

Looking at this it seems like this is the same issue where chromedriver would append to fields rather than replacing the current selection- it's a chromedriver problem

[twalpole]

To work around the issue you can specify to use backspaces as the clear option as shown in https://gist.github.com/twalpole/df24f13b5416971142d7b2ffd32891cc

[zthompson-conga]

@twalpole I think I'm experiencing the same issue, but I don't think it has to do with appending vs. replacing. The original poster states he's trying to get the value of a textarea, but it always returns nil, even though he can verify via screenshot that the textarea does have a value.

For me, I'm experiencing this on an input instead of a textarea, and it's affecting both headless and non-headless. Like tim said, I can verify the input does have a value.

One other thing I noticed while prying during my tests, find() is now always returning an "Obsolete" element, even when I know it's not obsolete.

I'm on an older version of Capybara (3.5.1), but am also on Chrome 91.0.4472 with the corresponding Chromedriver.

I can repro by doing the following:

visit('https://zach.staging.octiv.com')
fill_in('user[login]', with: 'test') # verified this works as expected
find('input[name="user[login]"]').value # should return 'test', but returns nil

Still sounds like this is probably a Chromedriver issue, but wanted to provide some detail to get verification, and see if there are any potential workarounds.

[Petercopter]

Hello I'm experiencing something similar. Now here's an interesting wrinkle. I was not able to replicate the problem on a MacBook Air with an M1 processor, but I am seeing it on a different MacBook without an M1 processor. Headless only as well.

First error:

Failure/Error:
expect(page).to have_field(
  'delivery_request[instructions_pickup]',
  with: delivery_request_previous.instructions_pickup
)

expected to find visible field "delivery_request[instructions_pickup]" that is not disabled with value "Nesciunt quo laborum et." but there were no matches. Also found "Nesciunt quo laborum et.", "", which matched the selector but not all filters.

Second error:

Failure/Error:
expect(page).to have_field(
  'delivery_request[instructions]',
  with: delivery_request_previous.instructions
)

expected to find visible field "delivery_request[instructions]" that is not disabled with value "Voluptas natus libero aut." but there were no matches. Also found "Voluptas natus libero aut.", which matched the selector but not all filters. Expected value to be "Voluptas natus libero aut." but was nil

Edit:
Definitely chromedriver related, this seems to be a workaround:

Webdrivers::Chromedriver.required_version = '90.0.4430.24'

[twalpole]

@zthompson-conga Works fine for me - https://gist.github.com/twalpole/fe2d063a4660ba5428b58d1776651733 - If you can make that gist exhibit the behavior you claim to be seeing then there's something for me to investigate - otherwise ...

[twalpole]

@Petercopter Your issue isn't related to this, the error message you've posted shows that it's finding an element, with what appears to be the value you're searching for, but that element doesn't match all the required filters for the have_field matcher. Are you sure the field is visible and not disabled? Possibly different default headless browser window sizes on different laptops causing different page layouts to be used?

[twalpole]

The initial posting here is a chromedriver/chrome issue with not replacing its contents when the current contents are selected and new keystrokes are sent. The other issues mentioned are unrelated to that issue. If either of the other posters can create a reproducible example please file them as new issues. The current workaround for the OPs issue is to specify to use backspace clearing until chromedriver/chrome are fixed

[timdiggins]

@twalpole thanks very much for your helpful gist. My original problem was not with the issue you pointed out there (to do with changing a value from capybara), but with the value of a retrieved element being nil only in chrome headless.

I have discovered the underlying issue and created a minimal repro. I am documenting that here in case someone else runs into this (it doesn't require any fix in chromedriver or capybara).

The issue was the use of an antique workaround (for a problem with getting logs out of chrome) of setting w3c mode to false.

options.add_option("w3c", false)

removing this line from my config fixes it.

[timdiggins]

I spoke too soon and misread my own notes. Although it doesn't error with the w3c mode to false, it does not seem possible to access chrome logs without it.

@twalpole If I read #2447 correctly, then non-w3c mode is no longer supported/encouraged in capybara.
Is there another suggested means of accessing logs?

[twalpole]

@timdiggins Correct, non-w3c because obsolete quite a while ago. You can access logs via driver specific methods - https://github.com/teamcapybara/capybara/blob/master/spec/selenium_spec_chrome.rb#L185

[zthompson-conga]

@timdiggins Thanks a bunch! The w3c thing solved my issue as well.

[timdiggins]

thanks @twalpole.

FYI I was using the driver specific method already but erroneously thought that non-w3c mode was needed to get "ALL" logs. But actually as long as you use the newer option "goog:loggingPrefs" you can get the logs (with the older "loggingPrefs" you have to have non-w3c mode.

ie.

      capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
        # https://stackoverflow.com/a/59778547/109175
        "goog:loggingPrefs": { browser: "ALL" }
      )

See https://gist.github.com/timdiggins/65c4371a71d025ec9d9192ee4848a4e0 for full example.
Just documenting here in case anyone else finds this.

[jeremyhaile]

Does anyone know why disabling w3c mode causes finding field values to fail? We are having a lot of trouble running our build without the w3c: false option, and are having to use the find_field(name).native.property(:value) workaround, but that doesn't feel very good.