[Questions] Publish a historical messages at stream #14080
-
Community Support Policy
RabbitMQ version used4.1.0 Erlang version used27.3.x Operating system (distribution) usedLinux - Fedora 39 How is RabbitMQ deployed?RPM package rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics
Status of node rabbit@basestation ...
[]
Runtime
OS PID: 93472 Plugins Enabled plugin file: /etc/rabbitmq/enabled_plugins
Data directory Node data directory: /var/lib/rabbitmq/mnesia/rabbit@basestation Config files
Log file(s)
Alarms (none) Tags (none) Memory Total memory used: 0.144 gb reserved_unallocated: 0.0453 gb (31.45 %) File Descriptors Total: 0, limit: 32671 Free Disk Space Low free disk space watermark: 5.0 gb Totals Connection count: 1 Listeners Interface: [::], port: 15672, protocol: http, purpose: HTTP API Logs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs rabbitmq.confSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location
disk_free_limit.absolute = 5GB
message_interceptors.incoming.set_header_timestamp.overwrite = false
message_interceptors.incoming.set_header_routing_node.overwrite = false
Steps to deploy RabbitMQ clusterVery easy way, I perform a RPM install and systemctl start ;-) Steps to reproduce the behavior in questionI have a json file representing a financial tick historical data. (3 months - ≃ 800 thousand records). advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location Application code# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKSWhat problem are you trying to solve?After creating a stream I need to load a set of historical messages keeping the creation date (Message Context Timestamp field) to perform replay using a offset-timestamp. Is there any way to create messages keeping the creation date in the past at a stream? Of course the order of these dates is chronologically correct. I searched all the documentation, performed several tests but was unsuccessful. Example: using Redis stream it is possible to send an unixtimestamp-0 field where 0 is the sequence of message at the same millisecond. Thank you so much. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
The system sets the timestamp that is used for attaching at a given time and also for retention. An application cannot set it. Message content is opaque to the broker, so it does not know about e.g. the creation time property. A workaround consists in always reading from the beginning of the stream and filtering on the client side. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for reply. Is there any other way to load old messages into a stream keeping metadata or documentation about the file structure in the file system? I'm thinking in convert the messages directly to files (00000000000000000000.index 00000000000000000000.segment). Thank you again! |
Beta Was this translation helpful? Give feedback.
-
|
We most definitely do not expect RabbitMQ users to generate Osiris (stream) segment and index files. You can re-publish your messages if you have a different place to get them from. |
Beta Was this translation helpful? Give feedback.
The system sets the timestamp that is used for attaching at a given time and also for retention. An application cannot set it. Message content is opaque to the broker, so it does not know about e.g. the creation time property.
A workaround consists in always reading from the beginning of the stream and filtering on the client side.