Events not flushing until next event occurs because of multine code

[shanmukha511]

Problem

I have added multi line code with regex because of that the most recent event not getting flushed until any new event occurs.The problem is that the event is buffered and isn't delivered until another event has started.If i remove my multi line code everything seems to be fine.The most recent event will logged in splunk.

For example, if event-a is delivered, then it buffers somewhere in fluentd. Let's say 10 mins pass, and then event-b comes into the log, at that point event-a is sent to Splunk .

...

Steps to replicate

Provide example config and message

<filter  tail.containers.var.log.containers.*test-*.log>
        @type concat
        key log
        timeout_label @SPLUNK
        stream_identity_key stream
        multiline_start_regexp /^\d{4}-\d{2}-\d{2}\s\d{2}/
        multiline_end_regexp /\\n$/
        flush_interval 5s
        separator ""
        use_first_timestamp true
</filter>

Expected Behavior

Use the multiline code the most recent event should reflect in splunk.

...

Your environment

• OS version
• paste result of fluentd --version or td-agent --version
  fluentd --version ---> fluentd 1.7.3
• plugin version
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

activemodel (3.2.22.5)
activesupport (3.2.22.5)
addressable (2.7.0)
aes_key_wrap (1.0.1)
async (1.21.0)
async-http (0.46.3)
async-io (1.25.0)
attr_required (1.0.1)
bigdecimal (default: 1.4.1)
bindata (2.4.4)
builder (3.0.4)
bundler (default: 1.17.2)
cmath (default: 1.0.0)
concurrent-ruby (1.1.5)
connection_pool (2.2.2)
console (1.4.0)
cool.io (1.5.4)
csv (default: 3.0.9)
date (default: 2.0.0)
dbm (default: 1.0.0)
did_you_mean (1.3.0)
dig_rb (1.0.1)
domain_name (0.5.20190701)
e2mmap (default: 0.1.0)
etc (default: 1.0.1)
fcntl (default: 1.0.0)
ffi (1.11.1)
fiddle (default: 1.0.0)
fileutils (default: 1.1.0)
fluent-plugin-concat (2.2.2)
fluent-plugin-jq (0.5.1)
fluent-plugin-kubernetes_metadata_filter (2.1.2)
fluent-plugin-prometheus (1.3.0)
fluent-plugin-splunk-hec (1.2.0)
fluent-plugin-systemd (1.0.2)
fluentd (1.7.3)
forwardable (default: 1.2.0)
gdbm (default: 2.0.0)
http (0.9.8)
http-accept (1.7.0)
http-cookie (1.0.3)
http-form_data (1.0.3)
http_parser.rb (0.6.0)
httpclient (2.8.3)
i18n (0.9.5)
io-console (default: 0.4.7)
ipaddr (default: 1.2.2)
irb (default: 1.0.0)
json (2.2.0, default: 2.1.0)
json-jwt (1.9.4)
kubeclient (1.1.4)
logger (default: 1.3.0)
lru_redux (1.1.0)
mail (2.7.1)
matrix (default: 0.1.0)
mime-types (3.3)
mime-types-data (3.2019.1009)
mini_mime (1.0.2)
minitest (5.11.3)
msgpack (1.3.1)
multi_json (1.14.1)
mutex_m (default: 0.1.0)
net-http-persistent (3.1.0)
net-telnet (0.2.0)
netrc (0.11.0)
nio4r (2.5.2)
oj (3.5.1, 3.3.10)
openid_connect (1.1.8)
openssl (default: 2.1.2)
ostruct (default: 0.1.0)
power_assert (1.1.3)
prime (default: 0.1.0)
prometheus-client (0.10.0, 0.9.0)
protocol-hpack (1.4.1)
protocol-http (0.8.1)
protocol-http1 (0.8.3)
protocol-http2 (0.9.5)
psych (default: 3.1.0)
public_suffix (4.0.1)
quantile (0.2.1)
rack (2.0.7)
rack-oauth2 (1.10.0)
rake (12.3.2)
rdoc (default: 6.1.2)
recursive-open-struct (1.0.0)
rest-client (2.1.0)
rexml (default: 3.1.9)
rss (default: 0.2.7)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
serverengine (2.1.1)
shell (default: 0.7)
sigdump (0.2.4)
stringio (default: 0.0.2)
strptime (0.2.3)
strscan (default: 1.0.0)
swd (1.1.2)
sync (default: 0.5.0)
systemd-journal (1.3.3)
test-unit (3.2.9)
thwait (default: 0.1.0)
timers (4.3.0)
tracer (default: 0.1.0)
tzinfo (2.0.0)
tzinfo-data (1.2019.3)
unf (0.1.4)
unf_ext (0.0.7.6)
validate_email (0.1.6)
validate_url (1.0.8)
webfinger (1.1.0)
webrick (default: 1.4.2)
xmlrpc (0.3.0)
yajl-ruby (1.4.1)
zlib (default: 1.0.0)

[shanmukha511]

Anybody can help on these issue?

[okkez]

You can check the behavior using fluent-cat command like the below.
(I used the tag dummy in my local configuration)

$ echo '{"log":"A line matched with multiline_start_regexp"}' | fluent-cat dummy

You can see the following logs as fluentd log:

2020-05-30 09:36:16 +0900 [info]: #0 Timeout flush: dummy: