1

Considering a docker container which receives logs on UDP and forwards to a central logging server using TLS, I was wondering if I could be satisfied with one queue or if I needed several.

Indeed, knowing that I won't be doing anything else than sending the logs to the collector, I don't really understand the need for several queues. Surely they make sense if other actions exist such as writing logs on files or duplicate outputs for failover, but what if forwarding is the only task? Wouldn't it be enough to have the main queue and the "send to collector" action in direct queue mode (i.e. no queue)? If the central server goes down (or during network outage) then logs will simply be re-enqueued to the main message queue ?

For example in this scenario below the Action Q makes total sense, but if we discard logging on /var/log/messages wouldn't Action Q be useless ? It's not only useless but also slows down the forwarding, right?

Scenario diagram: https://i.stack.imgur.com/8kFQP.png (can't post picture)

1
  • Moved post from SO to here, thanks for the hint @eDonkey
    – Ashlanfox
    Jul 28, 2022 at 13:52

2 Answers 2

0

To start, a quick introduction into rsyslog and queues.

Each input to rsyslog is through one or more threads, which gather the log messages and add them to the main queue. Worker threads then pull messages off the main queue and deliver them to their destinations and/or add the message to an action queue. (This is basically what the image you posted is doing)

If a worker is unable to deliver messages to a destination, all progress of that queue will block until that delivery is able to succeed (or it hits the retry limit and permanently fails). If you don’t want this to block all log processing, you should make an action queue for that destination (or group of destinations).

Now in your case, the last bit probably doesn't interest you, as you only want to relay the logs.

So to come to the point, each action has a dedicated queue. This queue can be in-memory, it can be on disk or it can be a combination of both. There is also direct mode, which means by design there is a queue but the actual driver does forward messages to the action without action queueing.

Therefore direct queues are probably the queue type you are looking for. In case the output action fails, the action processor notifies the action queue, which then takes an unprocessed element back and after some time interval, the action is attempted again. So direct queues also have the benefits of a normal queue.

Now to the last question: It's not only useless but also slows down the forwarding, right?

I am not really sure, if direct queues are (that much) faster than action queues in linkedList or fixedArray mode, as I have never tested it (or even used direct queues) before.

Edit: For more in-depth knowledge see the answer below.

2
  • Thank you for your time, you have indeed put your finger right on my situation. If not having a queue isn't better, it's at least equivalent, right?
    – Ashlanfox
    Jul 28, 2022 at 14:49
  • I'm glad I could help. And yes, I'd say your statement is correct, I don't see why direct queues would exist otherwise.
    – eDonkey
    Jul 28, 2022 at 15:13
1

you want to use queues where you need to decouple processing of messages. In your simple (and common) example, you are writing to a local file and sending messages to a remote machine via TCP.

Sending logs via TCP is a process that can block. If there is a network problem or a server problem, the sender pauses delivery of logs. If there is not a queue for that action, then this will mean that you will also not write logs to the local files (including such useful information as logs saying that the remote system is down ;-) )

so in this case, you do want an action queue, so that the TCP connection can block without blocking writes to /var/log/messages.

A couple other things to think about here.

  1. queues can be placed on actions or rulesets, if you are sending to multiple places (or want to failover from one to another), you want to group them in a ruleset and put the queue on the ruleset.

  2. you REALLY want to use the new action() syntax if you are putting a queue on an action, the old syntax ($foo lines before the action) is FAR too easy to misunderstand what's happening

  3. you can still lose messages in transit with TCP, when rsyslog gives the message to the OS on the sending machine, if that OS accepts it (i.e. doesn't say "the queue is full, wait"), rsyslog on the sender is forced to assume that it's going to be delivered. But if there is an outage (network or receiving server), rsyslog will never know that the message has been lost. The RELP protocol is designed to handle this case and resend messages if there are network/receiver issues.

as far as queues go, a 'direct queue' is 'no queue', it's the normal thing that happens when you have multiple actions

so a traditional

mail.* /var/log/mail kern.* /var/log/kern

is an example of 'direct queues' They are the fastest way to process messages.

disk queues save every message to disk before processing and are EXTREMELY slow (something on the order of 1000x slowdown), but will survive a system crash

memory queues store the logs in ram (linkedlist of fixedarray differ in how they allocate ram and a bit of speed)

Disk Assisted queues are memory queues that will overflow to disk files rather than blocking further processing (draining the queues does not happen in order, the disk portion is drained in parallel with processing logs from the memory queue). This is a good choice to handle long outages, just be aware that after the outage, you will start getting current logs quickly, but older logs will arrive at a slower rate.

It is possible to overdo it with queues. When you have a memory queue, thread A needs to lock the queue to add the messages to the queue, then thread B needs to lock the queue to retrieve the messages from the queue to process them. If you have a lightweight output (i.e. writing to a file) you spend far more CPU locking and unlocking the queue than you would just writing the messages out.

1
  • I've been waiting for this day (when you finally appear on any SE site :)). Regarding 3. - Do I understand correctly, that no queue can "save" you from message loss (if there is any kind of outage) after the OS accepted the message, if you don't handle it with e.g. RELP?
    – eDonkey
    Aug 26, 2022 at 9:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .