How to extract numeric values from messages and graph them

gsl_ops_practice · Post by **gsl_ops_practice** » Thu Apr 16, 2015 2:59 pm

Hello folks,

Admittedly I am new user to the Nagios Log server, but I am not able to figure out how we can pull out one numerical value from a log entry and graph those values over time (and generate alerts if it is out of spec)

Example log entry is below - this is an extract from the Message field.

2015-04-16 17:54 - TotalRequests=17,TotalTime=120,AverageTime=7 NameRequest:Requests=6,TotalTime=40,AverageTime=6 docRequest:Requests=5,TotalTime=53,AverageTime=10 loginRequest:Requests=6,TotalTime=27,AverageTime=4

Any help or a document on how to do this would be appreciated.

Thanks,
Alex

jolson · Post by **jolson** » Thu Apr 16, 2015 3:53 pm

I just double-checked your post, and it looks like you've already provided what I'm asking for (the message field):

Code: Select all

2015-04-16 17:54 - TotalRequests=17,TotalTime=120,AverageTime=7 NameRequest:Requests=6,TotalTime=40,AverageTime=6 docRequest:Requests=5,TotalTime=53,AverageTime=10 loginRequest:Requests=6,TotalTime=27,AverageTime=4

We need to develop a grok filter that will parse out the information that you need. Grok uses regular expression to parse your logs out into appropriate fields. You will need some regular expression knowledge to configure the filter properly.

Please use this tool to help develop your filter: http://grokdebug.herokuapp.com/

Input the contents of 'message' into the 'Input' field of the debugger. Now we can start matching patterns using either regular expression or pre-defined grok patterns, which are available here: http://grokdebug.herokuapp.com/patterns#

Keep in mind that the following filter is not optimal, but it is meant to show you what you can do:

Code: Select all

.*TotalRequests=%{NUMBER:TotalRequests},TotalTime=%{NUMBER:TotalTime},AverageTime=%{NUMBER:AverageTime} NameRequest:Requests=%{NUMBER:NameRequest},TotalTime=%{NUMBER:TotalTime2},AverageTime=%{NUMBER:AverageTime2} docRequest:Requests=%{NUMBER:docRequest},TotalTime=%{NUMBER:TotalTime3},AverageTime=%{NUMBER:AverageTime3} loginRequest:Requests=%{NUMBER:loginRequest},TotalTime=%{NUMBER:TotalTime4},AverageTime=%{NUMBER:AverageTime4}

I skipped the date and matched all of your numbers to fields appropriate to their names. Please note that with this filter, there is no tolerance for unexpected characters - the logs need to match the format you've provided me with exactly.

The grok debug site will look like this if you've defined your filter properly:

2015-04-16 15_56_22-Grok Debugger.png

Now, the above will take the 'numbers' and split them from messages into their own fields. From there we want to graph them, so we need to specify that these numbers are integers, and not strings. We do that by adding ':int' to the end of a field, like so:

Code: Select all

%{NUMBER:AverageTime2:int}

You can change all of them like that.

This way, Nagios Log Server can graph the information properly. I hope this helps you get started - I recommend learning regex for this project, it's immensely helpful.

Once the filter is done, it's a matter of adding it to Nagios Log Server. Let me know when you've reached this point, and I can answer any questions you might have. Thanks!

gsl_ops_practice · Post by **gsl_ops_practice** » Fri Apr 17, 2015 10:14 am

Hello,

I followed your example and added :int to the end of all the fields, but now Grok debugger is showing me only this:

Code: Select all

.*TotalRequests=%{NUMBER:TotalRequests:int},TotalTime=%{NUMBER:TotalTime:int},AverageTime=%{NUMBER:AverageTime:int} NameRequest:Requests=%{NUMBER:NameRequest:int},TotalTime=%{NUMBER:TotalTime2:int},AverageTime=%{NUMBER:AverageTime2:int} docRequest:Requests=%{NUMBER:docRequest:int},TotalTime=%{NUMBER:TotalTime3:int},AverageTime=%{NUMBER:AverageTime3:int} loginRequest:Requests=%{NUMBER:loginRequest:int},TotalTime=%{NUMBER:TotalTime4:int},AverageTime=%{NUMBER:AverageTime4:int}

What would be the next steps?

jolson · Post by **jolson** » Fri Apr 17, 2015 10:20 am

My guess is that the debugger doesn't support the :int tag for some reason.

Below is some documentation stating that the :int tag will work when applied to logstash - not sure why it's not built-in to the debugger:
http://logstash.net/docs/1.4.2/filters/grok

Optionally you can add a data type conversion to your grok pattern. By default all semantics are saved as strings. If you wish to convert a semantic’s data type, for example change a string to an integer then suffix it with the target data type. For example %{NUMBER:num:int} which converts the ‘num’ semantic from a string to an integer. Currently the only supported conversions are int and float.

Could you please post the input that you are using for these logs? You can find your inputs at:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

gsl_ops_practice · Post by **gsl_ops_practice** » Fri Apr 17, 2015 10:31 am

Hello,

I am trying to reverse-engineer the apache log filter to see if I can at least get the total requests working and then add the other ones one a time.

As you requested:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 02 Apr 2015 01:49:33 -0400
#

#
# Global inputs
#

input {
    syslog {
        type => 'syslog'
        port => 5544
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
}

#
# Local inputs
#

gsl_ops_practice · Post by **gsl_ops_practice** » Fri Apr 17, 2015 10:35 am

I will first get more coffee because this is not really intuitive - Is there a guide that has a step-by-step how to actually pull this data out of the provided apache filter? I can then use the same to build one for our log.

jolson · Post by **jolson** » Fri Apr 17, 2015 11:10 am

Thanks for providing your inputs. I assume that these logs are being sent to port 5544?

There are three parts to using logstash to import data properly:

1. Inputs - these are simple listeners setup by logstash that listen for incoming logs. The most common inputs are tcp, udp, and syslog (which listens on both tcp and udp). After the data is received, it's passed to step 2.

2. Filters - this is how data is structured, and this is the area we will add your grok filter to. Filters take in cruddy unstructured log data and add 'fields' to it. For instance, %{NUMBER:TotalTime:int} would find a NUMBER and add the 'TotalTime' field to it. Also, :int would convert the number from a string to an integer. Post-filter, data is sent to step 3.

3. Outputs - simply takes the structured data from step 2 and puts it into the elasticsearch database - allowing you to query it and build graphs as you wish.

Using the above information, I would define your inputs/filters as follows as a test:

Input:

Code: Select all

tcp {
        type => 'extractnumbers'        #type is used to 'tag' the data entering through this input so that you can simply match a filter to it.
        port => 9001
}

filter {

Code: Select all

if [type] == 'extractnumbers' {          #this matches the type as described above
     grok {
         match => [ 'message', '.*TotalRequests=%{NUMBER:TotalRequests},TotalTime=%{NUMBER:TotalTime},AverageTime=%{NUMBER:AverageTime} NameRequest:Requests=%{NUMBER:NameRequest},TotalTime=%{NUMBER:TotalTime2},AverageTime=%{NUMBER:AverageTime2} docRequest:Requests=%{NUMBER:docRequest},TotalTime=%{NUMBER:TotalTime3},AverageTime=%{NUMBER:AverageTime3} loginRequest:Requests=%{NUMBER:loginRequest},TotalTime=%{NUMBER:TotalTime4},AverageTime=%{NUMBER:AverageTime4}' ]     #this matches the 'message' field (as you described in your first post) and runs your custom filter against it. As long as the 'message' field contains nothing other than what you posted, this should work properly.
  }
}

A couple of notes:
The syslog input is not extremely desirable because it automatically applies a 'syslog' filter to any data that enters through it. While this can be useful sometimes, it can also be confusing - which is why I recommend a simple tcp/udp port to start with.

You can either use your existing syslog input, or use a new tcp/udp input. You can use the setup-script provided with Nagios Log Server to get your servers to log to the correct area.

Please let me know if you need any clarification - I understand that this can be confusing initially.

gsl_ops_practice · Post by **gsl_ops_practice** » Fri Apr 17, 2015 11:19 am

Ok, thank you, now it is starting to make more sense.

Just so I understand correctly - below filter means I should have an entry in the Fields section called "TotalRequests"?

Code: Select all

if [program] == 'APP_1_METRICS' {
    grok {
        match => [ 'TotalRequests', 'TotalRequests=%{NUMBER:TotalRequests:int}']
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
}

gsl_ops_practice · Post by **gsl_ops_practice** » Fri Apr 17, 2015 11:29 am

Ok, I got it to display the TotalRequests field and it is picking up values

thank you.

Can you please help me understand how I can write a filter to show only those values? Querystring Must, Query TotalRequests shows me below, even when it is 0

Code: Select all

if [program] == 'APP_1_METRICS' {
    grok {
        match => [ 'message', 'TotalRequests=%{NUMBER:TotalRequests:int}']
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
}

jolson · Post by **jolson** » Fri Apr 17, 2015 11:58 am

You can easily filter by TotalRequests by clicking the magnifying glass:

2015-04-17 11_56_07-Dashboard • Nagios Log Server.png

After adding it as a filter, you can modify it:

2015-04-17 11_57_15-Dashboard • Nagios Log Server.png

You can also specify a range:

2015-04-17 11_59_46-Dashboard • Nagios Log Server.png

Hope this helps!

Nagios Support Forum

How to extract numeric values from messages and graph them

How to extract numeric values from messages and graph them

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th

Re: How to extract numeric values from messages and graph th