Add modules_to_not_convert in quantized model

[jiqing-feng]

Fix modules_to_not_convert to skip the unquantized linear. As some models have unquantized modules, we should skip these modules in quantization.

This PR could enable qwen2_vl-awq model.
Without this change, the error will be:

Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 10, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.11/site-packages/typer/main.py", line 323, in __call__
    return get_command(self)(*args, **kwargs)
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.11/site-packages/typer/core.py", line 743, in main
    return _main(
  File "/opt/conda/lib/python3.11/site-packages/typer/core.py", line 198, in _main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.11/site-packages/typer/main.py", line 698, in wrapper
    return callback(**use_params)
  File "/usr/src/server/text_generation_server/cli.py", line 119, in serve
    server.serve(
  File "/usr/src/server/text_generation_server/server.py", line 316, in serve
    asyncio.run(
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete
    self.run_forever()
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
> File "/usr/src/server/text_generation_server/server.py", line 268, in serve_inner
    model = get_model_with_lora_adapters(
  File "/usr/src/server/text_generation_server/models/__init__.py", line 1592, in get_model_with_lora_adapters
    model = get_model(
  File "/usr/src/server/text_generation_server/models/__init__.py", line 1388, in get_model
    return VlmCausalLM(
  File "/usr/src/server/text_generation_server/models/vlm_causal_lm.py", line 354, in __init__
    super().__init__(
  File "/usr/src/server/text_generation_server/models/flash_causal_lm.py", line 1289, in __init__
    weights_loader = get_loader(quantize, model_id, revision)
  File "/usr/src/server/text_generation_server/utils/quantization.py", line 159, in get_loader
    return GPTQWeightsLoader(
TypeError: GPTQWeightsLoader.__init__() got an unexpected keyword argument 'modules_to_not_convert'
2025-02-25T13:01:44.894419Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:

[jiqing-feng]

Hi @Narsil , could you please help to trigger the tests and review this PR? Thanks!

[danieldk]

Thanks for adding support for this. I added some small comments.

[jiqing-feng]

Hi @danieldk , I have fixed your comments, please review the new changes. Thanks!