Update scripts and README

[navarone-feekery]

The script files were a bit bloated as we haven't looked at them since porting over base crawler. There were also limitations (e.g. forced env managers) that I don't think would be healthy for an open-code repo.

• Remove script funcs and script files that are not used
• Change echo coloring for progress statements from yellow to blue
  • Yellow should be for warnings only
• Remove hard requirement for rbenv and jenv when running make install
  • Users can opt-in to this by passing in env var CRAWLER_MANAGE_ENV=true
• Add currently enabled Java and Ruby versions to bash output when running make install
• Remove detailed instructions for env management from README.md
  • Instead we just have a recommendation to use an env manager, and links to official docs

[artem-shelkovnikov]

I tried to just make install and got caught in the hell of jenv/renv version incompatibility that I have installed from ent-search times.

What's the idea now about those? Does user install their own versions and make install sets up other stuff?

[navarone-feekery]

@artem-shelkovnikov my first thought for this was that the users should manage their env's themselves. For connectors we don't enforce pyenv for example. It doesn't seem like we even mention it in the repo. It's expected the user will manage their versions.

We could make this into its own command (like make env or something) and document that it sets the rbenv and jenv versions for the user. But I'm still hesitant to do that because we would need to support it and fix it if there's bugs, and I think version management should be a user's responsibility.

But this is a more complicated project to set up with having two languages to worry about. I can see value in providing a simple command, not to install languages but just make sure the correct one is being used. WDYT?

[navarone-feekery]

@artem-shelkovnikov @seanstory I re-implemented the rbenv and jenv management checks. They're now optional, and can be run by setting the env var CRAWLER_MANAGE_ENV=true.

$ CRAWLER_MANAGE_ENV=true make install

I can also make this an in-line argument if that feels cleaner. I couldn't decide between the two.

[artem-shelkovnikov]

I think something is missing - have you added the script to setup rubies?

artemshelkovnikov-m2@Work-M2 crawler % CRAWLER_MANAGE_ENV=true make install
script/environment
Homebrew 4.3.2
Checking version constraints...
Required Ruby version: jruby-9.4.7.0
Required Java version: 21

Checking if rbenv is installed...
/opt/homebrew/bin/rbenv
rbenv: OK

Enabling rbenv support...
Done!

Checking if Ruby jruby-9.4.7.0 is installed...
Ruby version jruby-9.4.7.0 is not installed. Running: 'script/setup-rubies'
script/functions.sh: line 147: script/setup-rubies: No such file or directory
make: *** [install] Error 1

[navarone-feekery]

I think something is missing - have you added the script to setup rubies?

Bleh sorry, the call to setup-rubies was meant to be removed.
However, as a user would you prefer the missing ruby version be automatically installed or not? If so I can re-add it.

[navarone-feekery]

I guess at this point of having the env managed almost entirely by Crawler, installing the ruby version automatically is probably fine.

[navarone-feekery]

@artem-shelkovnikov I've re-implemented the ruby installation, and also the bundler installation. I also fixed the coloring so yellow log lines can be reserved for warnings. I think it reads much easier now.
I also added a check against RBENV_VERSION variable as ent-search sets this automatically, and it causes clashes when switching between projects.

[artem-shelkovnikov]

Did some testing and everything looks good after I installed everything!