Add compression support to Pulsar sink

[zamazan4ik]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We want to compress messages before sending them to Apache Pulsar.

Attempted Solutions

No response

Proposal

Add compression support to Pulsar sink. Pulsar supports several algorithms.

I guess compression could be implemented in similar way as it's done for Kafka sink.

References

No response

Version

No response

[zamazan4ik]

Compression could be enabled via https://docs.rs/pulsar/latest/pulsar/producer/struct.ProducerOptions.html#structfield.compression option. Compression routine is already implemented inside pulsar-rs library. We just need to enable it properly.

[zamazan4ik]

pulsar-rs does not support tweaking compression levels (yet), so I'ev created the issue in upstream: streamnative/pulsar-rs#239

[zamazan4ik]

@bruceg as you mentioned in #14689 Vector decided to go with another approach to handle different compression levels.

Am I right that it also affects the compression level implementation in the Pulsar sink? In this case, I will enable compression with a default level in the Pulsar sink.