Netdata-TSRelay
===============

What's this?
------------

This program is designed to accept JSON streams from
[Netdata](https://my-netdata.io/) clients, and write metrics to a
PostgreSQL table - specifically, [Timescale](http://timescale.com)
backed tables (though that's not technically a requirement.)


Installation
------------

You'll need a working [Nim](http://nim-lang.org) build environment and
PostgreSQL development headers to compile the binary.

Simply run `make` to build it.  Put it wherever you please.


Configuration
-------------

There are a few assumptions that should be satisfied before running
this successfully.

### Database setup

You'll need to create the destination table.

```sql
CREATE TABLE netdata (
	time timestamptz default now() not null,
	host text not null,
	metrics jsonb default '{}'::jsonb not null
);
```

Index it based on how you intend to query the data, including JSON
functional indexing, etc.  See PostgreSQL documentation for details.

Strongly encouraged:  Promote this table to a Timescale "hypertable".
See [Timescale](http://timescale.com) docs for that, but a quick example
to partition automatically at weekly boundaries would look something
like this, if you're running v0.9.0 or better:

```sql
SELECT create_hypertable( 'netdata', 'time', migrate_data => true, chunk_time_interval => '1 week'::interval );
```

Timescale also has some great examples and advice for efficient [JSON
indexing](https://docs.timescale.com/timescaledb/latest/how-to-guides/schema-management/json/#json)
and queries.


### Netdata

You'll likely want to pare down what netdata is sending.  Here's an
example configuration for `exporting.conf` -- season this to taste (what
charts to send and frequency.)

Note: This example uses the "exporting" module introduced in
Netdata v1.23.  If your netdata is older than that, you'll be using
the deprecated "backend" instead in the main `netdata.conf` file.

```
[exporting:global]
	enabled  = yes
	hostname = your-hostname

[json:timescale]
	enabled              = yes
	data source          = average
	destination          = localhost:14866
	prefix               = netdata
	update every         = 10
	buffer on failures   = 10
	send charts matching = !cpu.cpu* !ipv6* !users.* nfs.rpc net.* net_drops.* net_packets.* !system.interrupts* system.* disk.* disk_space.* disk_ops.* mem.*
```


Running the Relay
-----------------

### Options

  * [-q|--quiet]:    Quiet mode.  No output at all. Ignored if -d is supplied.
  * [-d|--debug]:    Debug mode.  Show incoming data.
  * [-D|--dropconn]: Drop the TCP connection to netdata between samples.
                     This may be more efficient depending on your environment and
                     number of clients.  Defaults to false.
  * [-o|--dbopts]:   PostgreSQL connection information.  (See below for more details.)
  * [-h|--help]:     Display quick help text.
  * [-a|--listen-addr]: A specific IP address to listen on.  Defaults to **INADDR_ANY**.
  * [-p|--listen-port]: The port to listen for netdata JSON streams.
                     Default is **14866**.
  * [-P|--persistent]: Don't disconnect from the database between samples. This may be
                     more efficient with a small number of clients, when not using a
                     pooler, or with a very high sample size/rate.  Defaults to false.
  * [-T|--dbtable]:  Change the table name to insert to.  Defaults to **netdata**.
  * [-t|--timeout]:  Maximum time in milliseconds to wait for data.  Slow
                     connections may need to increase this from the default **500** ms.
  * [-v|--version]:  Show version.



**Notes**

Nim option parsing might be slightly different than what you're used to.
Flags that require arguments must include an '=' or ':' character.

  * --timeout=1000  *valid*
  * --timeout:1000  *valid*
  * -t:1000  *valid*
  * --timeout 1000  *invalid*
  * -t 1000  *invalid*

All database connection options are passed as a key/val string to the
*dbopts* flag.  The default is:

	"host=localhost dbname=netdata application_name=netdata-tsrelay"

... which uses the default PostgreSQL port, and connects as the running
user.

Reference the [PostgreSQL Documentation](https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS)
for all available options (including how to store passwords in a
separate file, enable SSL mode, etc.)


### Daemonizing

Use a tool of your choice to run this at system
startup in the background.  My personal preference is
[daemontools](https://cr.yp.to/daemontools.html), but I won't judge you
if you use something else.

Here's an example using the simple
[daemon](https://www.freebsd.org/cgi/man.cgi?query=daemon&apropos=0&sektion=8&manpath=FreeBSD+11.0-RELEASE+and+Ports&arch=default&format=html) wrapper tool:

	# daemon \
		-o /var/log/netdata_tsrelay.log \
		-p /var/run/netdata_tsrelay.pid \
		-u nobody -cr \
		/usr/local/bin/netdata_tsrelay \
			--dbopts="dbname=metrics user=metrics host=db-master port=6432 application_name=netdata-tsrelay"

### Scaling

Though performant by default, if you're going to be storing a LOT of
data (or have a lot of netdata clients), here are some suggestions for
getting the most bang for your buck:

  * Use the [pgbouncer](https://pgbouncer.github.io/) connection
    pooler.
  * DNS round robin the hostname where **netdata_tsrelay** lives across
    *N* hosts -- you can horizontally scale without any gotchas.
  * Edit your **netdata.conf** file to only send the metrics you are
    interested in.
  * Decrease the frequency at which netdata sends its data. (When in
    "average" mode, it averages over that time automatically.)
  * Use [Timescale](http://timescale.com) hypertables.
  * Add database indexes specific to how you intend to consume the data.
  * Use the PostgreSQL
    [JSON Operators](https://www.postgresql.org/docs/current/static/functions-json.html#FUNCTIONS-JSONB-OP-TABLE),
	which take advantage of GIN indexing.
  * Put convenience SQL VIEWs around the data you're fetching later, for
    easier graph building with [Grafana](https://grafana.com/) (or whatever.)