You might even know that MarkLogic also sends important events to the operating system. On linux this means syslog, and important events are those at NOTICE and higher by default.

But are you monitoring these events?

How can you set up your MarkLogic deployment so that it will automatically alert you to errors, warnings, or other important events?

Most linux deployments now use rsyslog as their system logging facility. The full documentation is available, but this brief tutorial will show you how to set up email alerts for MarkLogic using rsyslog version 4.2.6.

All configuration happens in /etc/rsyslog.conf. Here is a sample of what we need for email alerts. First, at the top of the file you should see several ModLoad declarations. Check for ommail and add it if needed.

$ModLoad ommail.so  # email support

Next, add a stanza for MarkLogic somewhere after the ModLoad declaration.

# MarkLogic
$template MarkLogicSubject,"Problem with MarkLogic on %hostname%"
$template MarkLogicBody,"rsyslog message from MarkLogic:\r\n[%timestamp%] %app-name% %pri-text%:%msg%"
$ActionMailSMTPServer 127.0.0.1
$ActionMailFrom your-address@your-domain
$ActionMailTo your-address@your-domain
$ActionMailSubject MarkLogicSubject
#$ActionExecOnlyOnceEveryInterval 3600
daemon.notice   :ommail:;MarkLogicBody

Be sure to replace both instances of your-address@your-domain with an appropriate value. The ActionMailSMTPServer must be smart enough to deliver email to that address. I used a default sendmail configuration on the local host, but you might choose to connect to a different host.

Note that I have commented out the ActionExecOnlyOnceEveryInterval option. The author of rsyslog, Rainer Gerhards, recommends setting this value to a reasonably high number of seconds so that your email inbox is not flooded with messages. However, the rsyslog documentation states that excess messages are discarded, and I did not want to lose any important messages. What I would really like to do is buffer messages for N seconds at a time, and merge them together in one email. But while rsyslog has many features, and does offer buffering, it does not seem to know how to combine consecutive messages into a single email.

Getting back to what rsyslog can do, you can customize the subject and body of the mail message. With the configuration above, a restart of the server might send you an email like this one:

Subject: Problem with MarkLogic on myhostname.mydomain

rsyslog message from MarkLogic:
[May 17 23:58:36] MarkLogic daemon.notice<29>: Starting MarkLogic Server 5.0-3 i686 in /opt/MarkLogic with data in /var/opt/MarkLogic

When making any rsyslog changes, be sure to restart the service:

sudo service rsyslog restart

At the same time, check your system log for any errors or typos. This is usually /var/log/messages or /var/log/syslog. The full documentation for template substitution properties is online. You can also read about a wealth of other options available in rsyslog.

Visit to ipv6-test.net for more tests.

The larger impact may be to application environments under heavy memory pressure. The OS can swap everything else out, but hugepage allocations are pinned. This is a double-edged sword. Preventing swapping may benefit some environments (cf vm.swappiness). But imagine a situation where you have 8-GB RAM and designate 4-GB for huge pages, but only use 2-GB. Now the OS has only 4-GB to manage, and the free 2-GB in huge pages are effectively wasted. If the system comes under memory pressure, that could lead to swapping or activate the OOM killer.