Support using an Older Chrome Release

[mars]

A workaround for recent Chrome 128 release incompatibility with Selenium,

Selenium::WebDriver::Error::WebDriverError: disconnected: not connected to DevTools

This changeset provides a new build-time config var setting to install an older release.

See the branch's new README section, Using an Older Chrome Release.

Try this branch version on your app

heroku buildpacks:remove heroku-community/chrome-for-testing
heroku buildpacks:add -i 1 https://github.com/heroku/heroku-buildpack-chrome-for-testing.git#chrome-milestone-offset

To get the previous stable version of Chrome (currently v127), configure the app & then rebuild:

heroku config:set GOOGLE_CHROME_MILESTONE_OFFSET=-4

[unRARed]

Would it be reasonable to assume this is what's been causing our heroku builds to fail on a handful of specifc tests since 8/20?

debugging with heroku ci:debug --pipeline our-pipeline uncovered this error:

I think we're still on heroku22 with the deprecated buildpack, for the record, but we also tried bumping the platform and the buildpack with the tests failures staying relatively consistent.

[mars]

@unRARed "…with the deprecated buildpack"

If you're using the old deprecated buildpack, I really don't know what's going wrong. Maybe it's the same issue, but the error reported was Selenium::WebDriver::Error::WebDriverError: disconnected: not connected to DevTools.

[mars]

@unRARed now I hear of another problem happening with that newest Chrome v128 & Selenium, hung network connections in Ruby Net::ReadTimeout. Still cannot confirm if this is your issue, but seems ever more likely.

[unRARed]

Dunno if it helps, but it sounds like one of our team members may have fixed our build by adding the chrome flag --disable-search-engine-choice-screen where we instantiate the driver.

[edmorley]

I'm curious to know more about the choice to use an offset approach rather than supporting the ability to request a specific Chrome version? I know in general we want people to stay up to date with Chrome releases, but I think being permanently on an offset may not be what people should be doing longer term anyway (and pinning more of a temporary workaround)?

[mars]

@edmorley Chrome is evergreen. The constant automatic updates to individuals' web browsers across the internet means that unless this buildpack continues updating the Chrome installation, that its testing context falls behind. Therefore, the premise of Chrome for Testing buildpack is to provide apps with the current stable, beta, or dev version of the browser.

This pull request directly addresses a new, stable Chrome release breaking tests due to other dependencies (Selenium in this case) not keeping ahead of Chrome releases to maintain compatibility.

This "milestone offset" provides a graceful escape hatch, to install an older stable major version Chrome. It's an equivalent functionality to Chrome's release channels, but supports following an older release, instead of the leading edge.

With this milestone offset strategy, the buildpack supports picking up minor releases & security patches on the older version, and will also automatically upgrade to the next version when Chrome Developers promote a milestone to stable, creating a new dev milestone.

A developer could forget about GOOGLE_CHROME_MILESTONE_OFFSET being set, and the browser would still automatically update regularly, just behind the leading edge.

Support to set (pin or lock) a specific Chrome+Chromedriver version is at odds with this design, and the very premise of Chrome's evergreen auto-updates. A locked version setting would fall further behind over time, increasing potential for test false positives, and mask that the app's own dependencies (like Selenium) are falling behind too.

[dzuelke]

I find the GOOGLE_CHROME_MILESTONE_OFFSET numbering a bit unintuitive. I'd like to suggest the following alternative procedure:

1. fetch https://googlechromelabs.github.io/chrome-for-testing/LATEST_RELEASE_${GOOGLE_CHROME_CHANNEL} (that will return e.g. "128.0.6613.84" for GOOGLE_CHROME_CHANNEL=STABLE);
2. to the first digit, add GOOGLE_CHROME_MILESTONE_OFFSET (assuming you want to keep the offset value negative; otherwise, subtraction is maybe easier to grasp for people, or use the absolute value and subtract), so for let's say an offset value of "-2", we get a $RESULT of "126";
3. fetch https://googlechromelabs.github.io/chrome-for-testing/LATEST_RELEASE_${RESULT}, which for RESULT=126 will return "126.0.6478.182".

[edmorley]

Yeah an offset approach makes sense given that. Though I like David found the current numbering of the offset counter-intuitive (ie: I thought zero might be stable). Perhaps an alternative to numbering might be named offsets, so beta, stable, previous-stable etc (or similar)? I'm happy to defer to you/David on this though :-)

[schneems]

I see that it's been 3 weeks and no progress since these comments. They are suggestions on the implementation and interface though acting as blockers. I would like to move this forward.

@mars do you have strong feelings on either of the suggestions? If there was a PR to your branch with something either David or Ed would accept would you be able to review and merge it?

FWIW internal support ticket number 1424804 is using this branch.

[mars]

I would be happy to see the suggested approaches implemented @schneems, but I would also be happy to merge/release as is. @dzuelke said he would implement his suggestion, but I assume he has not been able to prioritize the work.

[mars]

Also, hopefully the lag in the Selenium release that was causing this issue has passed. Two release have been made since this issue popped-up. (Not that it won't come up again.)