Multiple filters

Post by **WillemDH** » Wed Apr 01, 2015 9:10 am

What you say definitely makes sense, but it sill doesn't work.

This is a raw json of one of the last events, after I applied and set the filter matching on info:

Code: Select all

{
  "_index": "logstash-2015.04.01",
  "_type": "syslog-f5",
  "_id": "yCzvmzT4QCS2r35FqRt-LQ",
  "_score": null,
  "_source": {
    "message": "<155>Apr  1 15:43:51 slot1/ca_f5_1_vir_pr err dcc[13273]: 01310033:3: [SECEV] Request blocked, violations: Attcak signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 2913722834563104156, source ip: 212.123.0.8, xff ip: 212.123.0.8, source port: 44601, destination ip: 20.20.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_LAMP_Policy, scheme HTTP, geographic location: <BE>, request: <GET / HTTP/1.1\\r\\nTE: deflate,gzip;q=0.3\\r\\nConnection: TE, close\\r\\nHost: www.digipolis.be\\r\\nUser-Agent: libwww-perl/6.04\\r\\n>, username: <N/A>, session_id: <1b5b12e54080e419>\n",
    "@version": "1",
    "@timestamp": "2015-04-01T13:43:51.838Z",
    "type": "syslog-f5",
    "host": "20.40.24.159",
    "tags": [
      "grokked"
    ],
    "priority": 13,
    "severity": 5,
    "fcaility": 1,
    "fcaility_label": "user-level",
    "severity_label": "Notice",
    "syslog5424_pri": "155",
    "logsource": "ca_f5_1_vir_pr",
    "loglevel": "err",
    "program": "dcc",
    "pid": "13273",
    "info": "01310033:3: [SECEV] Request blocked, violations: Attcak signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 2913722834563104156, source ip: 212.123.0.8, xff ip: 212.123.0.8, source port: 44601, destination ip: 20.20.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_LAMP_Policy, scheme HTTP, geographic location: <BE>, request: <GET / HTTP/1.1\\r\\nTE: deflate,gzip;q=0.3\\r\\nConnection: TE, close\\r\\nHost: www.digipolis.be\\r\\nUser-Agent: libwww-perl/6.04\\r\\n>, username: <N/A>, session_id: <1b5b12e54080e419>"
  },
  "highlight": {
    "message": [
      "<155>Apr  1 15:43:51 slot1/ca_f5_1_vir_pr @start-highlight@err@end-highlight@ @start-highlight@dcc@end-highlight@[13273]: 01310033:3: [SECEV] Request blocked, violations: Attcak signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 2913722834563104156, source ip: 212.123.0.8, xff ip: 212.123.0.8, source port: 44601, destination ip: 20.20.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_LAMP_Policy, scheme HTTP, geographic location: <BE>, request: <GET / HTTP/1.1\\r\\nTE: deflate,gzip;q=0.3\\r\\nConnection: TE, close\\r\\nHost: www.digipolis.be\\r\\nUser-Agent: libwww-perl/6.04\\r\\n>, username: <N/A>, session_id: <1b5b12e54080e419>\n"
    ],
    "loglevel": [
      "@start-highlight@err@end-highlight@"
    ],
    "loglevel.raw": [
      "@start-highlight@err@end-highlight@"
    ],
    "program.raw": [
      "@start-highlight@dcc@end-highlight@"
    ],
    "program": [
      "@start-highlight@dcc@end-highlight@"
    ],
    "type": [
      "@start-highlight@syslog@end-highlight@-@start-highlight@f5@end-highlight@"
    ]
  },
  "sort": [
    1427895831838,
    1427895831838
  ]
}

So with this filter file:

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 01 Apr 2015 15:49:30 +0200
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [type] == "syslog-brocade" {
        grok {
          match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }     
      }
    
    if [type] == "syslog-f5" {
        grok {     
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        break_on_match => "false"
        }   
    }
    
 if [program] == "dcc" {
    grok {     
      match => [ "info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}",
                 "info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
}

#
# Local filters
#

As you can see I did had to split the dcc filter, as you only cut the first half of the filter. As the some of the dcc events have a user-agent part and others didn't I had to make to matches. i'm not sure however I'm doing it the right way, as it's the first time I'm trying this and I never saw a working complex example..

So i'm still stuck. Some basic question I have in order to continue testing:

Code: Select all

break_on_match => "false" or break_on_match => false

Is this a filter that should work? (multiline)
if [program] == "dcc" {
grok {
match => [ "info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}",
"info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
}

When I use http://grokconstructor.appspot.com/do/match#result to test the filter, i have a match.. (with of course the extra grok pattern

Code: Select all

HOSTNAME2 \b(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62})(?:\.(?:[_0-9A-Za-z][_0-9A-Za-z-]{0,62}))*(\.?|\b)

So I have the feeling the break on match might not be working as expected. I never saw it goe one time in the dcc filter..

Tx and grtz

jolson · Post by **jolson** » Wed Apr 01, 2015 9:53 am

Willem,

The correct syntax is as follows:

Code: Select all

break_on_match => false

Is it possible to send these different logs to a different port and tag the traffic accordingly? For instance:

Code: Select all

    syslog {
        type => 'dcc'
        port => 5550
        tags => dcclogs
    }

Please note the added tag. Your filter definition could be something like:

Code: Select all

if "dcclogs" in [tags] {
    grok {     
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}", "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
}

This way you could have this filter up top to match via tag. Any chance you can send logs to different IPs as described? Figured I would let you know this is possible just in case.

Post by **WillemDH** » Thu Apr 02, 2015 2:20 am

Jesse, sorry but the f5 syslog can only send to one specified port. All F5 'programs' all come into the input

Code: Select all

syslog {
    type => 'syslog-f5'
    port => 5548
}

I had no other choice, as when I'm not setting this input, all the syslogs get the _grokparsefailure tags.. I found this bug today, which is solved on Logstash 1.5. Do you think I might be hitting this bug? https://github.com/elastic/logstash/issues/1547

I just changed the filters a bit. Moved the "break_on_match => false" to the top of each filter and also changed the " add_tag => "grokked_syslog_brocade"" so each filter gets a specific tag. The strange thing is that all the F5 syslog events still receive the 'grokked' tag, while this is no longer in my filters. I have the feeling something else is wrong and my filters are no applied properly.... It's like they match on an old filter config...

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 02 Apr 2015 10:24:18 +0200
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [type] == "syslog-brocade" {
        grok {
          match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked_syslog_brocade"
        }     
      }
    
    if [type] == "syslog-f5" {
        grok {     
          break_on_match => false
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
          remove_tag => "_grokparsefailure"
          add_tag => "grokked_syslog_f5"      
        }   
    }
    
    if [program] == "dcc" {
        grok {   
          break_on_match => false  
          match => [ "info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
          add_tag => "grokked_syslog_f5_dcc"
    }
}

#
# Local filters
#

Grtz

Willem

jolson · Post by **jolson** » Thu Apr 02, 2015 9:23 am

Let's take a look at your configurations on the GUI to make sure they're implemented properly:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

If not, check the permissions:

Code: Select all

ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/

If the owner/group is apache:apache, you could change the permissions to Nagios to get them working.

Code: Select all

chown -R nagios:nagios /usr/local/nagioslogserver/logstash/etc/conf.d/*

Apply Configuration to push the configs down.

Let me know if that works for you - seeing the 'grokked' tag is a red flag.

Post by **WillemDH** » Thu Apr 02, 2015 2:56 pm

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 26 Mar 2015 13:01:21 +0100
#

#
# Global inputs
#

input {
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    syslog {
        type => 'syslog'
        port => 5544
    }
    syslog {
        type => 'syslog-eternus'
        port => 1516
    }
    syslog {
        type => 'syslog-esx'
        port => 514
    }
    syslog {
        type => 'syslog-infoblox'
        port => 5545
    }
    syslog {
        type => 'syslog-linux'
        port => 5546
    }
    syslog {
        type => 'syslog-brocade'
        port => 5547
    }
    syslog {
        type => 'syslog-f5'
        port => 5548
    }
    syslog {
        type => 'naf'
        port => 5549
    }
}

#
# Local inputs
#


#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 26 Mar 2015 13:01:21 +0100
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }

    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [type] == "syslog-brocade" {
        grok {
          match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }
      }

    if [type] == "syslog-f5" {
        grok {
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }
      }

}

#
# Local filters
#


#

Hmmm this is really problematic... It seems like the f5 dcc filter is not even there... Damned all these tests I did I'll have to redo. What could be the cause of this?

Code: Select all

ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/
total 12
-rwxrwxr-x 1 nagios nagios 1164 Mar 26 13:01 000_inputs.conf
-rwxrwxr-x 1 nagios nagios 1730 Mar 26 13:01 500_filters.conf
-rwxrwxr-x 1 nagios nagios  534 Mar 26 13:01 999_outputs.conf

SO Nagios is still the owner... Please advice how to continue.

jolson · Post by **jolson** » Thu Apr 02, 2015 3:24 pm

Willem, this is likely being caused by a very small syntax error - unfortunately 'Verify' does not catch everything.

The one that I dug out is that in your filter, there were too few closing brackets. This could be why the filter is not pushing down. Please try using the following filter (keeping in mind that it's one of the originals you posted):

Code: Select all

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [type] == "syslog-brocade" {
        grok {
          match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }     
      }
   
    if [type] == "syslog-f5" {
        grok {     
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }   
    }
   
    if [program] == "dcc" {
        grok {     
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}User-Agent: %{GREEDYDATA:useragent}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}", "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME2:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
        }
    }
}

Note that I don't think this will fix your original problem - but it may fix your configuration push issue.

Post by **WillemDH** » Sat Apr 04, 2015 3:16 am

Jesse,

Managed to get the filters applied again. The dcc filter was indeed missing a } .

Code: Select all

if [type] == "syslog-f5" {
    grok {     
      break_on_match => false
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
      remove_tag => "_grokparsefailure"
      add_tag => "grokked_syslog_f5"      
    }   
}

Code: Select all

if [program] == "dcc" {
    grok {   
      break_on_match => false  
      match => [ "info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
      add_tag => "grokked_syslog_f5_dcc"
      }
}

New dcc events do get the "grokked_syslog_f5,_grokparsefailure". As non dcc events get only grokked_syslog_f5, the dcc filter will probabaly egt started but fails to parse. I'll test with a debugger if there should be a match. TO be continued...

Grtz

Willem

Post by **WillemDH** » Sat Apr 04, 2015 3:30 am

I'm not able to get into the dcc filter, whatever I'm trying.

Code: Select all

    if [type] == "syslog-f5" {
        grok {
          break_on_match => false
          match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} %{SYSLOGPROG}: %{GREEDYDATA:info}" ]
          remove_tag => "_grokparsefailure"
          add_tag => "grokked_syslog_f5"
        }
    }

    if [program] == "dcc" {
        grok {
          match => [ "info", "%{GREEDYDATA:info}, source ip: %{IP:sourceip}, xff ip: %{IP:xffip}, source port: %{NUMBER:sourceport}, destination ip: %{IP:destinationip}, destination port: %{NUMBER:destinationport}, route_domain: %{NUMBER:routedomain}, HTTP classifier: %{GREEDYDATA:httpclassifier}, geographic location: , request: %{GREEDYDATA:request}, username: %{GREEDYDATA:username}, session_id: %{GREEDYDATA:sessionid}" ]
          add_tag => "grokked_syslog_f5_dcc_test"
          }
    }
}

Checked with "cat /usr/local/nagioslogserver/logstash/etc/conf.d/*" and is applied pproperly. When I put some dcc events (without a useragent, as there are two kinds of dcc logs, but I want to get those without useragent working first) in http://grokconstructor.appspot.com/do/match#result with the above dcc filter and only the "info" field, I get a match.

Example two dcc logs that match in the debugger:

Code: Select all

01310033:3: [SECEV] Request blocked, violations: Attack signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 2705742410508804324, source ip: 188.226.186.150, xff ip: 188.226.186.150, source port: 48903, destination ip: 20.25.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_LAMP_Policy, scheme HTTP, geographic location: , request: , username: , session_id: 
01310033:3: [SECEV] Request blocked, violations: Attack signature detected. HTTP protocol compliance sub violations: N/A. Evasion techniques sub violations: N/A. Web services security sub violations: N/A. Virus name: N/A. Support id: 2705742410508789699, source ip: 188.227.186.150, xff ip: 188.227.186.150, source port: 48902, destination ip: 20.25.1.138, destination port: 80, route_domain: 0, HTTP classifier: /Common/F5_External_1_LAMP_Policy, scheme HTTP, geographic location: , request: , username: , session_id: <32bce266bac599d2>

So what else can I do to make the dcc filter work?

Willem

jolson · Post by **jolson** » Mon Apr 06, 2015 1:15 pm

Willem,

I know this is unrelated - but I just thought of some information that could get rid of those pesky 'grokparsefailure' tags. Below are some of your example inputs:

Code: Select all

syslog {
     type => 'syslog-brocade'
     port => 5547
 }
syslog {
     type => 'syslog-f5'
     port => 5548
}

Before the inputs forward the log data to your custom-built grok filters, they will attempt to parse the log data using the generic 'syslog' filter. This filter is hard-coded and designed to parse syslog-compliant logs. Please read the below quote from http://logstash.net/docs/1.4.2/inputs/syslog:

Of course, ‘syslog’ is a very muddy term. This input only supports RFC3164 syslog with some small modifications. The date format is allowed to be RFC3164 style or ISO8601. Otherwise the rest of RFC3164 must be obeyed. If you do not use RFC3164, do not use this input.

To get rid of some of the parsefailures that you've been receiving, I recommend changing the input type to tcp or udp.

Also, I want to mention that any custom field containing an integer must contain the ':int' tag if you intend to use that variable in charts or graphs.

For example:
NUMBER:routedomain:int would register the result of 'routedomain' as an integer instead of a string - this will enable graphing functionality for this field. I was thinking that this is something you would like to know.

Optionally you can add a data type conversion to your grok pattern. By default all semantics are saved as strings. If you wish to convert a semantic’s data type, for example change a string to an integer then suffix it with the target data type. For example %{NUMBER:num:int} which converts the ‘num’ semantic from a string to an integer. Currently the only supported conversions are int and float.

Regarding the 'dcc' filter - I verified that the syntax is proper, and as far as I can tell it looks like it should be working.

Let's take a look at the following filter:

Code: Select all

if [type] == "syslog-brocade" {
        grok {
          match => [ "message", "<[\d]+>[a-z]+ [\d]+ [\d\:]+ %{IPV4:logsource}%{GREEDYDATA:program}: %{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day}-%{TIME:time}%{GREEDYDATA:data1}WWN %{IPV6:wwn}%{GREEDYDATA:data2}%{LOGLEVEL:loglevel}\, %{HOSTNAME:hostname}\, %{GREEDYDATA:info}" ]
        remove_tag => "_grokparsefailure"
        add_tag => "grokked"
        }     
      }

It looks like the above filter is tagging 'program' - this is where 'dcc' will be tagged. Can you be sure that the above filter has break_on_match set to false as well? Otherwise the 'dcc' logs will stop processing there.

Post by **WillemDH** » Tue Apr 07, 2015 6:18 am

Jesse,

Thanks for the input. I'll try you suggestion to use tcp as input.

The syslog-brocade filter however has nothing to do with the f5 filter. As it starts with "if [type] == "syslog-brocade" " and the tyoe is set on the input with

Code: Select all

syslog {
    type => 'syslog-brocade'
    port => 5547
}

Just like the type for f5 logs is set with:

Code: Select all

syslog {
    type => 'syslog-f5'
    port => 5548
}

So the program dcc is NOT set in the syslog-brocade filter, but in the syslog-f5 filter. So I do not have to set break_on_match there? The program is set with:

Code: Select all

if [type] == "syslog-f5" {
    grok {     
      break_on_match => false
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} slot1\/%{HOSTNAME:logsource} %{LOGLEVEL:loglevel} [b]%{SYSLOGPROG}[/b]: %{GREEDYDATA:info}" ]
      remove_tag => "_grokparsefailure"
      add_tag => "grokked_syslog_f5"      
    }   
}

Imho the dcc filter above should just work...

About the tcp input, I tried changing the f5 input to

Code: Select all

tcp {
    type => 'syslog-f5'
    port => 5548
}

And the f5 logs immediately stopped coming in. So I reverted back to syslog. Is there something I'm missing with this tcp input?

Grtz

Willem

Nagios Support Forum

Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters

Re: Multiple filters