How to install ElasticSearch and Logstash (and Kibana for ELK) in a TrueNAS Core Jail

Before we begin, lets see what a ELK stack is made of. ELK stacks are comprised of ElasticSearch, Logstash, and Kabina with each of these services preforming different jobs. This diagram from Logz.io explains its the best

Beats are installed to each host to collect the various metrics used by these services.

Logstash is responsible for the processing of the incoming logs and Aggregation into something in a common format.

ElasticSearch is the storage and index for the processed logs and metrics.

Kibana is the analysis and visualization for the data.

My setup is as follows:

I have a multi server setup, with 1 hypervisor, ESXi and one SAN, TrueNAS. I could just have an ELK stack running in a vm and target NFS or iscsi on the storage server, or even just use a datastore where other vms are stored. The reason is not done that was is I want the logs to be revived by the storage directly so ZFS can handle the compression and long term storage of the logs. There also is am advantage of having better uptime since the SAN reboots much less often than my hypervisor host. I have a full ELK stack deployed, but kibana is running on an ESXi VM, not on TrueNAS. This is so the brunt of the analyzing will be done on the Hypervisor CPU not the storage CPU.

Beware the RAM and CPU requirements of my setup. My TrueNAS host in this setup is somewhat powerful (10 core, 20 thread CPU, 48GB RAM, with LOG and L2ARC SSD’s). Logstash and Elasticsearch can consume a lot of RAM. I have 8GB free for just this jail (just in case the usage gets high). This can be lowered by editing Java settings for each applications. The other thing to worry about is the CPU cycles. If you are using heavy compression like gzip or zstd this can heavily effect other process. I used this report , also this report, and a few others to base my decision on the compression levels. The applications themselfs can also use a lot of CPU depending on the log size and the filters applied. I would strongly recommend setting up grafana, telegraf, and influxDB or some similar long term monitoring setup to make sure your jail is not making your TrueNAS overloaded.

[Optional] Dataset Creation

For my setup, I am having Logstash and ElasticSearch store their files on separate datasets on my TrueNAS box. This is so I can easily migrate things in the future if it breaks, and so I can set custom compression and record sizes for the datasets to decrease storage consumed and speed of everything. I set my datasets up with the following settings:

Record size: 32KiB
Compression: ZSTD-6
Encryption: Inherit

Jail creation

To start with, Fetch the latest FreeBSD release for your system in the command line(in my case 12.2 because that is what TrueNas Core 12.0-U4.1 is biased on)

# iocage fetch

Once that is done, go and create a advanced jail. Name it what you want, Mine is Log1

In jail properties, allow allow_set_hostname, allow_mlock, and allow_mount. In the dropdown for allow mount, Give it

allow_mount_devs
allow_mount_nullfs
allow_mount_procfs
allow_mount_fusefs
allow_mount_tmpfs

Next, on networking, uncheck NAT, check VNET, checkBerkeley Packet Filterand give the jail an IP address, netmask, and gateway.

Finally the jail is created! Now we need to drop to command line and run the following

# iocage set mount_fdescfs=1 your-jail
# iocage set mount_procfs=1 your-jail
# iocage set enforce_statfs=1 your-jail

This will allow java8 to run because fdescfs and proc are needed by java to run correctly. The old way involved modifying enforce statfs for the system and setting it to 1. This is not recommended! The issue is that it changes how all jails operate and lead to some system instability. With this method, it changes just this jails statfs settings.

If you have datasets to mount, this is where you would do it. Mine are setup like this:

Now we can finally start the jail. If the jail is not starting, go to command line and run the following to debug. Berkley Packet Filter sometimes likes to enable. Disable it in the networking config for the jail.

# iocage start your-jail

Entering the jail

To get into the jail, run

# iocage console your-jail

All commands from this point on are in the jail, not TrueNAS

[Optional] Changing pkg to use the latest packages

If you want to run the latest of all of the software and not wait on quarterly updates, you can change the freebsd repo to use the latest. I use quarterly for stability above all else. If you want to run the latest, change the following:

# vim /etc/pkg/FreeBSD.conf

Change

FreeBSD: {
url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",

to:

FreeBSD: {
url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",

 

Installation

Install elasticsearch

# pkg install -y elasticsearch7

 

Install logstash

# pkg install -y logstash7

If you wanted the full ELK stack, Run # pkg install -y kibana7

Configuration

All of the configurations are store in /usr/local/etc/ under the respective application name. I am usingvim to edit my config files (install with # pkg install -y vim) If you want a simpler editor, use nano (install with # pkg install -y nano) Just replace use nano instead of vimin the commands below.

 

For elasticsearch, you need to find and configure the following lines in the config file to what you want it to be. Here is what I changed on mine.

# vim /usr/local/etc/elasticsearch/elasticsearch.yml
cluster.name: Logs
node.name: Log1
path.data: /mnt/elasticsearch # only change if you are mounting different datasets. Else set it to /var/db/elasticsearch
path.logs: /var/log/ElasticSearch
http.port: 9200

Once done with that, start and make the service start at boot

# sysrc elasticsearch_enable=YES
# service elasticsearch start

Any errors encountered should dump a log to the console or ssh session.

 

For Logstash, you dont have to change anything, but I changed some of the values. Here is what I changed on mine.

# vim /usr/local/etc/logstash/logstash.yml

node.name: Logs  
path.data: /mnt/Logstash 
pipeline.ordered: auto 
path.config: /usr/local/etc/logstash/logstash.conf 
http.host: 192.168.42.84 
http.port: 9600-9700

Finally, start the and enable the services

# sysrc logstash_enable=YES
# service logstash start

Any errors encountered should dump a log to the console or ssh session.

If you want to configure kibana, follow this guide

Conclusion

Now you should have a ready to go ElasticSearch and Logstash Jail! If you have any issue please post about them here. Im not an expert by any means, but I may be able to help.



About: Ryan Parker

I'm a former captain of the Cyber Defense team form Cal State San Bernardino. I also have a side job helping small to medium business with anything technology, including but not limited to servers, networking, and end user devices. One of my hobbies is building out infrastructures for myself, friends, and clients. I current maintain a homelab with about 400GB of RAM, 100+ TB of storage, and tons of CPU.


Leave a Reply

Your email address will not be published. Required fields are marked *