[Improvement]: Decouple in-built bal tools from the distribution

[azinneera]

Description

Currently, the most commonly used bal tools (bal openapi, bal graphql, bal grpc, bal persist) are packaged in the distribution. Due to this, we cannot issue bug fixes and improvements to the tools without releasing a ballerina distribution. Decoupling the tools from the distribution will make the delivery of features/fixes fast.

Describe your problem(s)

Ballerina's support for the above features includes CLI tools, package-provided compiler-plugins, and also VSCode features. The Java dependencies required for the CLI tool and VSCode features are shipped in the <distribution>/bre/lib directory and some of these are also used in compiler plugins. Therefore, a distribution release is required when there is a requirement to update a tool or a compiler plugin.

Describe your solution(s)

A plugin architecture must be designed to remove the IDL-specific Java dependencies from the distribution. CLI tools, compiler plugins and VSCode extensions must be decoupled from the distribution. The owners must be able to independently release and maintain these individually.

Related area

-> Compilation

[TharmiganK]

This should also resolve the issue in resolving compiler-plugin dependencies for a new package which has dependency to one of the tool's dependency(which is packed with the distribution). For example consider the following scenario:

The package ballerinax/ai.agent:1.0.0 has a dependency to ballerina-to-openapi package. The openapi-tool also has the same dependency. We cannot publish ballerinax/ai.agent by depending on a new version of ballerina-to-openapi because the dependencies will first resolve to the libraries packed with the distribution(in this case the openapi tool). The only solution for now is to release it with the new distribution by changing all the dependent packages and tools.