2

Say I have the following events in Splunk (ordered chronologically):

  1. operator=zeroPointModule id=123 action=start
  2. operator=foobar id=123 action=start
  3. operator=zeroPointModule id=123 action=stop
  4. operator=zeroPointModule id=123 action=stop

(Yes it's stopping twice, or can be restarted. Please don't assume anything about the order of the events.)

I'd like to get all events when (1) then (3) happened. Something like:

operator=zeroPointModule id=123 (action=start THEN action=stop)

I don't care too much about the output (both events, just the first or just the last).

2

That's a classic example of the transaction command. It can group events based on shared fields and a start and/or stop text. For example:

index=foo
| transaction startswith="start" endswith="stop" operator

The transaction command is not very performant so, depending on how much data you need to search, you may have better results with stats.

index=foo
| stats range(_time) as duration values(*) as * by operator
3
  • 1
    there is no "duration()" stats function in Splunk – warren Jun 16 '20 at 20:30
  • 2
    Correct. My fingers got ahead of me. I fixed my answer. – RichG Jun 16 '20 at 20:35
  • Thanks! I'll try it out tomorrow – pyb Jun 16 '20 at 22:46
1

Is it ever possible for a "stop" to happen before a "start"? I'd suspect not - unless you have some pretty bad timestamp extraction going on.

Which means that you'll always have 1 (or 2) events per operator (a "start" and an eventual "stop").

And I also suspect you won't have more than one "start" for a given operator and id.

Presuming that is true:

index=ndx sourcetype=srctp operator=* id=* action IN("start","stop")
| stats values(_time) as times values(action) as action by operator id
| where mvcount(action)>1
| eval elapsed_time_in_seconds=mvindex(times,-1)-mvindex(times,0)
| table operator id elapsed_time_in_seconds

EDIT due to updated question

If you can have multiple "start" and "stop" events, yet you want them grouped, you can do something similar to this (I've had to on several occasions to get goofy data out of Splunk):

index=ndx sourcetype=srctp operator=* id=* action IN("start","stop")
| eval comb=_time+"|"+action
| stats values(comb) as combs by operator id
| where mvcount(combs)>1
| eval cfirst=mvindex(combs,0), clast=mvindex(combs,-1)
| search cfirst="*start" clast="*stop"
| rex field=cfirst "(?<start_time>\d+)"
| rex field=clast "(?<stop_time>\d+)"
| where stop_time>start_time
| table operator id start_time stop_time
| eval start_time=strftime(start_time,"%c"), stop_time=strftime(stop_time,"%c")

PS - nice Stargate reference

3
  • 1
    Actually yes, I think that's possible and those are the ones I'd like to track. For the sake of simplicity, I've worded my question to search for stop after start, but I'm looking for anomalies, so the opposite. Both can happen but I'm only interested in anomalies. – pyb Jun 16 '20 at 22:48
  • 1
    Also, (I'll add it to the question), the ZPM sometimes needs to be restarted so you can have start, stop, start, stop. Or anything else since it's finicky. Well done for recognizing the reference :) – pyb Jun 16 '20 at 22:50
  • @pyb - see updated suggestion – warren Jun 17 '20 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.