the Logstash service on our NLS nodes is crashing continuously. I saw these errors passing:
Code: Select all
Oct 29, 2015 10:48:00 AM org.elasticsearch.transport.netty.NettyInternalESLogger warn
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1063)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:118)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Error: Your application used more memory than the safety cap of 500M.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace
Code: Select all
Specify -J-Xmx####m to increase it (#### = cap size in MB).Code: Select all
grep -i 'out of memory' /var/log/messagesJVM settings:
Code: Select all
curl -XGET localhost:9200/_nodes/jvm?pretty
{
"cluster_name" : "ee9e60a0-f4cb-41ec-a97f-8f17434b748e",
"nodes" : {
"JHyDfTFIT82MUiBkxpmGmw" : {
"name" : "95f9ab14-da22-4144-bb0b-6bbc5662115c",
"transport_address" : "inet[/10.54.24.141:9300]",
"host" : "nls02",
"ip" : "127.0.0.1",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[localhost/127.0.0.1:9200]",
"attributes" : {
"max_local_storage_nodes" : "1"
},
"jvm" : {
"pid" : 32338,
"version" : "1.7.0_85",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.85-b03",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1446106732260,
"mem" : {
"heap_init_in_bytes" : 4294967296,
"heap_max_in_bytes" : 4260102144,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 4260102144
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
},
"MpQpx1GcTOmWEoILA4sn7w" : {
"name" : "c4d16075-9bc2-4095-9f00-e7de7f96930c",
"transport_address" : "inet[/10.54.24.140:9300]",
"host" : "nls01",
"ip" : "127.0.0.1",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[localhost/127.0.0.1:9200]",
"attributes" : {
"max_local_storage_nodes" : "1"
},
"jvm" : {
"pid" : 1587,
"version" : "1.7.0_85",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.85-b03",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1446112726670,
"mem" : {
"heap_init_in_bytes" : 4294967296,
"heap_max_in_bytes" : 4242669568,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 4242669568
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
}
}
}
Code: Select all
free -m
total used free shared buffers cached
Mem: 10014 6530 3483 0 41 1597
-/+ buffers/cache: 4892 5122
Swap: 1999 0 1999
Code: Select all
cat /etc/sysconfig/logstash
###############################
# Default settings for logstash
###############################
# Override Java location
#JAVACMD=/usr/bin/java
# Set a home directory
APP_DIR=/usr/local/nagioslogserver
LS_HOME="$APP_DIR/logstash"
# set ES_CLUSTER
ES_CLUSTER=$(cat $APP_DIR/var/cluster_uuid)
# Arguments to pass to java
#LS_HEAP_SIZE="256m"
LS_JAVA_OPTS="-Djava.io.tmpdir=$APP_DIR/tmp"
# Logstash filter worker threads
#LS_WORKER_THREADS=1
# pidfiles aren't used for upstart; this is for sysv users.
#LS_PIDFILE=/var/run/logstash.pid
# user id to be invoked as; for upstart: edit /etc/init/logstash.conf
LS_USER=root
LS_GROUP=nagios
# logstash logging
#LS_LOG_FILE=/var/log/logstash/logstash.log
#LS_USE_GC_LOGGING="true"
# logstash configuration directory
LS_CONF_DIR="$LS_HOME/etc/conf.d"
# Open file limit; cannot be overridden in upstart
#LS_OPEN_FILES=2048
# Nice level
#LS_NICE=0
# Increase Filter workers to 4 threads
LS_OPTS=" -w 4"
if [ "x$1" == "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" ];then
GET_LOGSTASH_CONFIG_MESSAGE=$( php /usr/local/nagioslogserver/scripts/get_logstash_config.php )
GET_LOGSTASH_CONFIG_RETURN=$?
if [ "$GET_LOGSTASH_CONFIG_RETURN" != "0" ]; then
echo $GET_LOGSTASH_CONFIG_MESSAGE
exit 1
fi
fi
Code: Select all
cat /etc/sysconfig/elasticsearch
# Directory where the Elasticsearch binary distribution resides
APP_DIR="/usr/local/nagioslogserver"
ES_HOME="$APP_DIR/elasticsearch"
# Heap Size (defaults to 256m min, 1g max)
ES_HEAP_SIZE=4g
# Heap new generation
#ES_HEAP_NEWSIZE=
# max direct memory
#ES_DIRECT_SIZE=
# Additional Java OPTS
#ES_JAVA_OPTS=
# Maximum number of open files
MAX_OPEN_FILES=65535
# Maximum amount of locked memory
MAX_LOCKED_MEMORY=unlimited
# Maximum number of VMA (Virtual Memory Areas) a process can own
MAX_MAP_COUNT=262144
# Elasticsearch log directory
LOG_DIR=/var/log/elasticsearch
# Elasticsearch data directory
DATA_DIR="$ES_HOME/data,/mnt/data"
# Elasticsearch work directory
WORK_DIR="$APP_DIR/tmp/elasticsearch"
# Elasticsearch conf directory
CONF_DIR="$ES_HOME/config"
# Elasticsearch configuration file (elasticsearch.yml)
CONF_FILE="$ES_HOME/config/elasticsearch.yml"
# User to run as, change this to a specific elasticsearch user if possible
# Also make sure, this user can write into the log directories in case you change them
# This setting only works for the init script, but has to be configured separately for systemd startup
ES_USER=nagios
ES_GROUP=nagios
# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true
if [ "x$1" == "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" -o "x$1" == "xforce-reload" ];then
GET_ES_CONFIG_MESSAGE="$( php $APP_DIR/scripts/get_es_config.php )"
GET_ES_CONFIG_RETURN=$?
if [ "$GET_ES_CONFIG_RETURN" != "0" ]; then
echo $GET_ES_CONFIG_MESSAGE
exit 1
else
ES_JAVA_OPTS="$GET_ES_CONFIG_MESSAGE"
fi
fi
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Sat, 17 Oct 2015 17:16:13 +0200
#
#
# Required output for Nagios Log Server
#
output {
elasticsearch {
cluster => 'ee9e60a0-f4cb-41ec-a97f-8f17434b748e'
host => 'localhost'
document_type => '%{type}'
node_name => 'c4d16075-9bc2-4095-9f00-e7de7f96930c'
protocol => 'transport'
workers => 4
}
}
#
# Global outputs
#
#
# Local outputs
#
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Sat, 17 Oct 2015 17:16:13 +0200
#
#
# Required output for Nagios Log Server
#
output {
elasticsearch {
cluster => 'ee9e60a0-f4cb-41ec-a97f-8f17434b748e'
host => 'localhost'
document_type => '%{type}'
node_name => 'c4d16075-9bc2-4095-9f00-e7de7f96930c'
protocol => 'transport'
workers => 4
}
}
#
# Global outputs
#
#
# Local outputs
#
elasticsearch.yml
[root@srvnaglog01 ~]# cat /usr/local/nagioslogserver/elasticsearch/config/elasticsearch.yml
##################### Elasticsearch Configuration Example #####################
# This file contains an overview of various configuration settings,
# targeted at operations staff. Application developers should
# consult the guide at <http://elasticsearch.org/guide>.
#
# The installation procedure is covered at
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html>.
#
# Elasticsearch comes with reasonable defaults for most settings,
# so you can try it out without bothering with configuration.
#
# Most of the time, these defaults are just fine for running a production
# cluster. If you're fine-tuning your cluster, or wondering about the
# effect of certain configuration option, please _do ask_ on the
# mailing list or IRC channel [http://elasticsearch.org/community].
# Any element in the configuration can be replaced with environment variables
# by placing them in ${...} notation. For example:
#
# node.rack: ${RACK_ENV_VAR}
# For information on supported formats and syntax for the config file, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html>
################################### Cluster ###################################
# Cluster name identifies your cluster for auto-discovery. If you're running
# multiple clusters on the same network, make sure you're using unique names.
#
cluster.name: nagios_elasticsearch
#################################### Node #####################################
# Node names are generated dynamically on startup, so you're relieved
# from configuring them manually. You can tie this node to a specific name:
#
# node.name: "Franz Kafka"
# Every node can be configured to allow or deny being eligible as the master,
# and to allow or deny to store the data.
#
# Allow this node to be eligible as a master node (enabled by default):
#
# node.master: true
#
# Allow this node to store data (enabled by default):
#
# node.data: true
# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
# This will be the "workhorse" of your cluster.
#
# node.master: false
# node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
# to have free resources. This will be the "coordinator" of your cluster.
#
# node.master: true
# node.data: false
#
# 3. You want this node to be neither master nor data node, but
# to act as a "search load balancer" (fetching data from nodes,
# aggregating results, etc.)
#
# node.master: false
# node.data: false
# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_nodes] or GUI tools
# such as <http://www.elasticsearch.org/overview/marvel/>,
# <http://github.com/karmi/elasticsearch-paramedic>,
# <http://github.com/lukas-vlcek/bigdesk> and
# <http://mobz.github.com/elasticsearch-head> to inspect the cluster state.
# A node can have generic attributes associated with it, which can later be used
# for customized shard allocation filtering, or allocation awareness. An attribute
# is a simple key value pair, similar to node.key: value, here is an example:
#
# node.rack: rack314
# By default, multiple nodes are allowed to start from the same installation location
# to disable it, set the following:
node.max_local_storage_nodes: 1
#################################### Index ####################################
# You can set a number of options (such as shard/replica options, mapping
# or analyzer definitions, translog settings, ...) for indices globally,
# in this file.
#
# Note, that it makes more sense to configure index settings specifically for
# a certain index, either when creating it or by using the index templates API.
#
# See <http://elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html> and
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/indices-create-index.html>
# for more information.
# Set the number of shards (splits) of an index (5 by default):
#
# index.number_of_shards: 5
# Set the number of replicas (additional copies) of an index (1 by default):
#
# index.number_of_replicas: 1
# Note, that for development on a local machine, with small indices, it usually
# makes sense to "disable" the distributed features:
#
# index.number_of_shards: 1
# index.number_of_replicas: 0
# These settings directly affect the performance of index and search operations
# in your cluster. Assuming you have enough machines to hold shards and
# replicas, the rule of thumb is:
#
# 1. Having more *shards* enhances the _indexing_ performance and allows to
# _distribute_ a big index across machines.
# 2. Having more *replicas* enhances the _search_ performance and improves the
# cluster _availability_.
#
# The "number_of_shards" is a one-time setting for an index.
#
# The "number_of_replicas" can be increased or decreased anytime,
# by using the Index Update Settings API.
#
# Elasticsearch takes care about load balancing, relocating, gathering the
# results from nodes, etc. Experiment with different settings to fine-tune
# your setup.
# Use the Index Status API (<http://localhost:9200/A/_status>) to inspect
# the index status.
#################################### Paths ####################################
# Path to directory containing configuration (this file and logging.yml):
#
# path.conf: /path/to/conf
# Path to directory where to store index data allocated for this node.
#
# path.data: /path/to/data
#
# Can optionally include more than one location, causing data to be striped across
# the locations (a la RAID 0) on a file level, favouring locations with most free
# space on creation. For example:
#
# path.data: /path/to/data1,/path/to/data2
# Path to temporary files:
#
# path.work: /path/to/work
# Path to log files:
#
# path.logs: /path/to/logs
# Path to where plugins are installed:
#
# path.plugins: /path/to/plugins
#################################### Plugin ###################################
# If a plugin listed here is not installed for current node, the node will not start.
#
# plugin.mandatory: mapper-attachments,lang-groovy
################################### Memory ####################################
# Elasticsearch performs poorly when JVM starts swapping: you should ensure that
# it _never_ swaps.
#
# Set this property to true to lock the memory:
#
bootstrap.mlockall: true
# Make sure that the ES_MIN_MEM and ES_MAX_MEM environment variables are set
# to the same value, and that the machine has enough memory to allocate
# for Elasticsearch, leaving enough memory for the operating system itself.
#
# You should also make sure that the Elasticsearch process is allowed to lock
# the memory, eg. by using `ulimit -l unlimited`.
############################## Network And HTTP ###############################
# Elasticsearch, by default, binds itself to the 0.0.0.0 address, and listens
# on port [9200-9300] for HTTP traffic and on port [9300-9400] for node-to-node
# communication. (the range means that if the port is busy, it will automatically
# try the next port).
# Set the bind address specifically (IPv4 or IPv6):
#
# network.bind_host: 192.168.0.1
# Set the address other nodes will use to communicate with this node. If not
# set, it is automatically derived. It must point to an actual IP address.
#
# network.publish_host: 192.168.0.1
# Set both 'bind_host' and 'publish_host':
#
# network.host: 192.168.0.1
# Set a custom port for the node to node communication (9300 by default):
#
# transport.tcp.port: 9300
# Enable compression for all communication between nodes (disabled by default):
#
transport.tcp.compress: true
# Set a custom port to listen for HTTP traffic:
#
# http.port: 9200
# Set a custom allowed content length:
#
# http.max_content_length: 100mb
# Disable HTTP completely:
#
# http.enabled: false
# Set the HTTP host to listen to
#
http.host: "localhost"
################################### Gateway ###################################
# The gateway allows for persisting the cluster state between full cluster
# restarts. Every change to the state (such as adding an index) will be stored
# in the gateway, and when the cluster starts up for the first time,
# it will read its state from the gateway.
# There are several types of gateway implementations. For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html>.
# The default gateway type is the "local" gateway (recommended):
#
# gateway.type: local
# Settings below control how and when to start the initial recovery process on
# a full cluster restart (to reuse as much local data as possible when using shared
# gateway).
# Allow recovery process after N nodes in a cluster are up:
#
# gateway.recover_after_nodes: 1
# Set the timeout to initiate the recovery process, once the N nodes
# from previous setting are up (accepts time value):
#
# gateway.recover_after_time: 5m
# Set how many nodes are expected in this cluster. Once these N nodes
# are up (and recover_after_nodes is met), begin recovery process immediately
# (without waiting for recover_after_time to expire):
#
# gateway.expected_nodes: 2
############################# Recovery Throttling #############################
# These settings allow to control the process of shards allocation between
# nodes during initial recovery, replica allocation, rebalancing,
# or when adding and removing nodes.
# Set the number of concurrent recoveries happening on a node:
#
# 1. During the initial recovery
#
# cluster.routing.allocation.node_initial_primaries_recoveries: 4
#
# 2. During adding/removing nodes, rebalancing, etc
#
# cluster.routing.allocation.node_concurrent_recoveries: 2
# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
#
# indices.recovery.max_bytes_per_sec: 20mb
# Set to limit the number of open concurrent streams when
# recovering a shard from a peer:
#
# indices.recovery.concurrent_streams: 5
################################## Discovery ##################################
# Discovery infrastructure ensures nodes can be found within a cluster
# and master node is elected. Multicast discovery is the default.
# Set to ensure a node sees N other master eligible nodes to be considered
# operational within the cluster. Its recommended to set it to a higher value
# than 1 when running more than 2 nodes in the cluster.
#
# discovery.zen.minimum_master_nodes: 1
# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
# discovery.zen.ping.timeout: 3s
# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html>
# Unicast discovery allows to explicitly control which nodes will be used
# to discover the cluster. It can be used when multicast is not present,
# or to restrict the cluster communication-wise.
#
# 1. Disable multicast discovery (enabled by default):
#
discovery.zen.ping.multicast.enabled: false
#
# 2. Configure an initial list of master nodes in the cluster
# to perform discovery when new nodes (master or data) are started:
#
discovery.zen.ping.unicast.hosts: ["localhost"]
# EC2 discovery allows to use AWS EC2 API in order to perform discovery.
#
# You have to install the cloud-aws plugin for enabling the EC2 discovery.
#
# For more information, see
# <http://elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html>
#
# See <http://elasticsearch.org/tutorials/elasticsearch-on-ec2/>
# for a step-by-step tutorial.
# GCE discovery allows to use Google Compute Engine API in order to perform discovery.
#
# You have to install the cloud-gce plugin for enabling the GCE discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-gce>.
# Azure discovery allows to use Azure API in order to perform discovery.
#
# You have to install the cloud-azure plugin for enabling the Azure discovery.
#
# For more information, see <https://github.com/elasticsearch/elasticsearch-cloud-azure>.
################################## Slow Log ##################################
# Shard level query and fetch threshold logging.
#index.search.slowlog.threshold.query.warn: 10s
#index.search.slowlog.threshold.query.info: 5s
#index.search.slowlog.threshold.query.debug: 2s
#index.search.slowlog.threshold.query.trace: 500ms
#index.search.slowlog.threshold.fetch.warn: 1s
#index.search.slowlog.threshold.fetch.info: 800ms
#index.search.slowlog.threshold.fetch.debug: 500ms
#index.search.slowlog.threshold.fetch.trace: 200ms
#index.indexing.slowlog.threshold.index.warn: 10s
#index.indexing.slowlog.threshold.index.info: 5s
#index.indexing.slowlog.threshold.index.debug: 2s
#index.indexing.slowlog.threshold.index.trace: 500ms
################################## GC Logging ################################
#monitor.jvm.gc.young.warn: 1000ms
#monitor.jvm.gc.young.info: 700ms
#monitor.jvm.gc.young.debug: 400ms
#monitor.jvm.gc.old.warn: 10s
#monitor.jvm.gc.old.info: 5s
#monitor.jvm.gc.old.debug: 2s
EDIT1: it seems 10 seconds after I start Logstash, I get
Code: Select all
service logstash status
Logstash Daemon (pid 8892) is running...
[root@srvnaglog01 ~]# Error: Your application used more memory than the safety cap of 500M.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace
service logstash status
Logstash Daemon dead but pid file exists
[root@srvnaglog01 ~]# service logstash start
Starting Logstash Daemon: WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
[ OK ]
[root@srvnaglog01 ~]# Oct 29, 2015 11:31:03 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [c4d16075-9bc2-4095-9f00-e7de7f96930c] loaded [], sites []
Oct 29, 2015 11:31:04 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [c4d16075-9bc2-4095-9f00-e7de7f96930c] loaded [], sites []
Oct 29, 2015 11:31:04 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [c4d16075-9bc2-4095-9f00-e7de7f96930c] loaded [], sites []
Oct 29, 2015 11:31:04 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [c4d16075-9bc2-4095-9f00-e7de7f96930c] loaded [], sites []
Oct 29, 2015 11:31:04 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [c4d16075-9bc2-4095-9f00-e7de7f96930c] loaded [], sites []
service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon (pid 9767) is running...
[root@srvnaglog01 ~]# service logstash statusOct 29, 2015 11:32:10 AM org.elasticsearch.transport.netty.NettyInternalESLogger warn
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
logstash start^Cstart
[root@srvnaglog01 ~]# service logstash status
Logstash Daemon dead but pid file exists
I edited the /etc/sysconfig/logstash
uncommented LS_HEAP_SIZE and set it to 1024m
Code: Select all
# Arguments to pass to java
LS_HEAP_SIZE="1024m"
LS_JAVA_OPTS="-Djava.io.tmpdir=$APP_DIR/tmp"
The Logstash service seem more stable since I changed the LS_HEAP_SIZE. I also changed the elasticsearch HEAP_SIZE in /etc/sysconfig/elasticsearch from 4g to 6g, https://support.nagios.com/forum/viewto ... h&start=10:
Code: Select all
ES_HEAP_SIZE=6gEDIT4: Seems I'm still having issues. The webgui is superslow or unresponsive. Logs are not coming in.
EDIT5: Logstash is no longer stopping it seems after a few restarts. I guess Logstash needed catching up..
Grtz