Importing from file - assistance please

tomslmonitor · Post by **tomslmonitor** » Wed Feb 24, 2016 8:37 pm

Thanks jolson,

I've appeared to get it sort of working with some redundant fields but i'll fix that up soon enough.

if [host] ==xxx.xxx.xx.xxx' {
    grok {
        match => ['message', '%{TIMESTAMP_ISO8601:im_timestamp}']
    }
}

date {
        locale => "en"
        match => [ "im_timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
        timezone => "Australia/Sydney"
        target => "logTimestamp"
    }

The histogram is now able to graph the data, but I still get occurences of the line below.

Code: Select all

FacetPhaseExecutionException[Facet [0]: (key) field [im_timestamp] not found]

I'm assuming that because I have some data that doesn't have this field and it can't graph just those specific lines, it is shooting out this message.

Thoughts?

jolson · Post by **jolson** » Thu Feb 25, 2016 10:49 am

I'm assuming that because I have some data that doesn't have this field and it can't graph just those specific lines, it is shooting out this message.

That's what the error says to me - that some of the data you're parsing through does not include the 'im_timstamp' field - though if data is displaying properly there is no cause for concern. It's more informational.

I'm happy to hear this is working for you. May I close this thread or do you have some further questions?

tomslmonitor · Post by **tomslmonitor** » Mon Feb 29, 2016 12:16 am

I do have another question related to this topic.

This may be a simple fix, but as you previously know my coding is on the basic side.

With the python script shipper.py, I've noticed that some of the messages from my log file are not making it through and also some additional junk in the message field (not as important as data being removed from lines).

I'm guessing the script is picking up some lines that may be python script and not bringing it through to the message field, examples below:

Original text:

Code: Select all

2016-02-19 23:58:27,789 DEBUG [UserManager.logAccess:720  ] <regularProcess> has logged out.

message field after being shipped to NLS using shipper.py

Code: Select all

{"message": "2016-02-19 23:58:27,789 DEBUG [UserManager.logAccess:720  ]  has logged out.\r\n"}

as you can see the <regularProcess> has been filtered out and there is junk at the start and end of the line.

My initial thoughts were, I am happy for the lines of my logs to come through as is (non-json). Something like a read line by line and echo each line and push it through to the log server.

Code: Select all

while read line
do
	echo **read line** | nc 123.456.78.912 2056
done < server.log

Sorry I don't know the proper syntax, but something along those lines would it work and would it be okay to do?

jolson · Post by **jolson** » Mon Feb 29, 2016 10:53 am

2016-02-19 23:58:27,789 DEBUG [UserManager.logAccess:720 ] <regularProcess> has logged out.

The reason that you're not seeing <regularProcess> is due to the fact that your browser is interpreting the HTML tag and stripping it from your view. I bet that if you clicked the 'raw' section of any log, you would be able to see the entire thing.

Example:

2016-02-29 09_51_44-Dashboard • Nagios Log Server - Firefox Developer Edition.png

Nagios Support Forum

Importing from file - assistance please

Re: Importing from file - assistance please

Re: Importing from file - assistance please

Re: Importing from file - assistance please

Re: Importing from file - assistance please