Where am I?

You probably know that MarkLogic Server logs important events to the ErrorLog.txt file. By default it logs events at INFO or higher, but many development and staging environments change the file-log-level to DEBUG. These log levels are also available to the xdmp:log function, and some of your XQuery code might use that for printf-style debugging.

You might even know that MarkLogic also sends important events to the operating system. On linux this means syslog, and important events are those at NOTICE and higher by default.

But are you monitoring these events?

How can you set up your MarkLogic deployment so that it will automatically alert you to errors, warnings, or other important events?

Most linux deployments now use rsyslog as their system logging facility. The full documentation is available, but this brief tutorial will show you how to set up email alerts for MarkLogic using rsyslog version 4.2.6.

All configuration happens in /etc/rsyslog.conf. Here is a sample of what we need for email alerts. First, at the top of the file you should see several ModLoad declarations. Check for ommail and add it if needed.

$ModLoad ommail.so  # email support

Next, add a stanza for MarkLogic somewhere after the ModLoad declaration.

# MarkLogic
$template MarkLogicSubject,"Problem with MarkLogic on %hostname%"
$template MarkLogicBody,"rsyslog message from MarkLogic:\r\n[%timestamp%] %app-name% %pri-text%:%msg%"
$ActionMailSMTPServer 127.0.0.1
$ActionMailFrom your-address@your-domain
$ActionMailTo your-address@your-domain
$ActionMailSubject MarkLogicSubject
#$ActionExecOnlyOnceEveryInterval 3600
daemon.notice   :ommail:;MarkLogicBody

Be sure to replace both instances of your-address@your-domain with an appropriate value. The ActionMailSMTPServer must be smart enough to deliver email to that address. I used a default sendmail configuration on the local host, but you might choose to connect to a different host.

Note that I have commented out the ActionExecOnlyOnceEveryInterval option. The author of rsyslog, Rainer Gerhards, recommends setting this value to a reasonably high number of seconds so that your email inbox is not flooded with messages. However, the rsyslog documentation states that excess messages are discarded, and I did not want to lose any important messages. What I would really like to do is buffer messages for N seconds at a time, and merge them together in one email. But while rsyslog has many features, and does offer buffering, it does not seem to know how to combine consecutive messages into a single email.

Getting back to what rsyslog can do, you can customize the subject and body of the mail message. With the configuration above, a restart of the server might send you an email like this one:

Subject: Problem with MarkLogic on myhostname.mydomain

rsyslog message from MarkLogic:
[May 17 23:58:36] MarkLogic daemon.notice<29>: Starting MarkLogic Server 5.0-3 i686 in /opt/MarkLogic with data in /var/opt/MarkLogic

When making any rsyslog changes, be sure to restart the service:

sudo service rsyslog restart

At the same time, check your system log for any errors or typos. This is usually /var/log/messages or /var/log/syslog. The full documentation for template substitution properties is online. You can also read about a wealth of other options available in rsyslog.

When AWS EC2 tells you that "requested instance type (cc1.4xlarge) is not supported in your requested Availability Zone (us-east-1b)", what they really mean is that you are using the wrong AMI. Switch to an HVM AMI. This enables support within Xen for Hardware Virtual Machine (HVM), using AMD SVM or Intel VT-x instructions.

Tomorrow's event prodded me into setting up IPv6 at home, where I use openwrt. The tutorial I found was helpful: I just had to change the interface names. On my system eth0.1 was eth1, and 6rdtun was called 6to4. Comcast's test page says I'm up and working. I can see the unicorn too.

Visit to ipv6-test.net for more tests.

Part 5 in a series by Mel Gorman describes how to measure the potential benefit from hugepages. The results match up reasonably well with CPU-intensive synthetic benchmarks on linux, which tend to show 10-15% improvement over ordinary pages.

The larger impact may be to application environments under heavy memory pressure. The OS can swap everything else out, but hugepage allocations are pinned. This is a double-edged sword. Preventing swapping may benefit some environments (cf vm.swappiness). But imagine a situation where you have 8-GB RAM and designate 4-GB for huge pages, but only use 2-GB. Now the OS has only 4-GB to manage, and the free 2-GB in huge pages are effectively wasted. If the system comes under memory pressure, that could lead to swapping or activate the OOM killer.

It wasn't rocket surgery, but blakeley.com now has an IPv6 stack and an AAAA record. If http://ipv6-test.com/validate.php can be believed, it even works. I had more trouble finding a server-side test tool than I had with the configuration.