| author | Mahlon E. Smith <mahlon@laika.com> |
| Mon, 19 Feb 2018 18:22:25 -0800 | |
| changeset 11 | 475c9942eb15 |
| parent 10 | 252cdb26f76b |
| child 12 | 450df27eaeec |
| permissions | -rw-r--r-- |
| 1 | 1 |
|
2 |
Netdata-TSRelay |
|
3 |
=============== |
|
4 |
||
5 |
What's this? |
|
6 |
------------ |
|
7 |
||
8 |
This program is designed to accept JSON streams from |
|
|
10
252cdb26f76b
Fix the netdata link to point the the right spot.
Mahlon E. Smith <mahlon@laika.com>
parents:
5
diff
changeset
|
9 |
[Netdata](https://my-netdata.io/) clients, and write metrics to a |
|
252cdb26f76b
Fix the netdata link to point the the right spot.
Mahlon E. Smith <mahlon@laika.com>
parents:
5
diff
changeset
|
10 |
PostgreSQL table - specifically, [Timescale](http://timescale.com) |
|
252cdb26f76b
Fix the netdata link to point the the right spot.
Mahlon E. Smith <mahlon@laika.com>
parents:
5
diff
changeset
|
11 |
backed tables (though that's not technically a requirement.) |
| 1 | 12 |
|
13 |
||
14 |
Installation |
|
15 |
------------ |
|
16 |
||
| 11 | 17 |
You'll need a working [Nim](http://nim-lang.org) build environment and |
18 |
PostgreSQL development headers to compile the binary. |
|
| 1 | 19 |
|
| 11 | 20 |
Simply run `make` to build it. Put it wherever you please. |
| 1 | 21 |
|
22 |
||
23 |
Configuration |
|
24 |
------------- |
|
25 |
||
26 |
There are a few assumptions that should be satisfied before running |
|
| 11 | 27 |
this successfully. |
| 1 | 28 |
|
29 |
### Database setup |
|
30 |
||
31 |
You'll need to create the destination table. |
|
32 |
||
33 |
```sql |
|
34 |
CREATE TABLE netdata ( |
|
35 |
time timestamptz default now() not null, |
|
36 |
host text not null, |
|
37 |
metrics jsonb default '{}'::jsonb not null
|
|
38 |
); |
|
39 |
``` |
|
40 |
||
| 11 | 41 |
Index it based on how you intend to query the data, including JSON |
42 |
functional indexing, etc. See PostgreSQL documentation for details. |
|
| 1 | 43 |
|
44 |
Strongly encouraged: Promote this table to a Timescale "hypertable". |
|
| 11 | 45 |
See [Timescale](http://timescale.com) docs for that, but a quick example |
46 |
to partition automatically at weekly boundaries would look something |
|
47 |
like: |
|
|
3
f34855556f4d
Add timescale creation example.
Mahlon E. Smith <mahlon@laika.com>
parents:
1
diff
changeset
|
48 |
|
|
f34855556f4d
Add timescale creation example.
Mahlon E. Smith <mahlon@laika.com>
parents:
1
diff
changeset
|
49 |
```sql |
|
f34855556f4d
Add timescale creation example.
Mahlon E. Smith <mahlon@laika.com>
parents:
1
diff
changeset
|
50 |
SELECT create_hypertable( 'netdata', 'time', chunk_time_interval => 604800000000 ); |
|
f34855556f4d
Add timescale creation example.
Mahlon E. Smith <mahlon@laika.com>
parents:
1
diff
changeset
|
51 |
``` |
| 1 | 52 |
|
| 11 | 53 |
Timescale also has some great examples and advice for efficient [JSON |
54 |
indexing](http://docs.timescale.com/v0.8/using-timescaledb/schema-management#json) |
|
55 |
and queries. |
|
| 1 | 56 |
|
57 |
||
58 |
### Netdata |
|
59 |
||
60 |
You'll likely want to pare down what netdata is sending. Here's an |
|
| 11 | 61 |
example configuration for `netdata.conf` -- season this to taste (what |
62 |
charts to send and frequency.) |
|
| 1 | 63 |
|
64 |
``` |
|
65 |
[backend] |
|
66 |
hostname = your-hostname |
|
67 |
enabled = yes |
|
68 |
type = json |
|
69 |
data source = average |
|
70 |
destination = machine-where-netdata-tsrelay-lives:14866 |
|
71 |
prefix = n |
|
| 11 | 72 |
update every = 60 |
73 |
buffer on failures = 5 |
|
| 5 | 74 |
send charts matching = !cpu.cpu* !ipv6* !users* nfs.rpc net.* net_drops.* net_packets.* !system.interrupts* system.* disk.* disk_space.* disk_ops.* mem.* |
| 1 | 75 |
``` |
76 |
||
| 11 | 77 |
|
78 |
Running the Relay |
|
79 |
----------------- |
|
80 |
||
81 |
### Options |
|
82 |
||
83 |
* [-q|--quiet]: Quiet mode. No output at all. Ignored if -d is supplied. |
|
84 |
* [-d|--debug]: Debug mode. Show incoming data. |
|
85 |
* [--dbopts]: PostgreSQL connection information. (See below for more details.) |
|
86 |
* [-h|--help]: Display quick help text. |
|
87 |
* [--listen-addr]: A specific IP address to listen on. Defaults to INADDR_ANY. |
|
88 |
* [--listen-port]: The port to listen for netdata JSON streams. |
|
89 |
Default is 14866. |
|
90 |
* [-T|--dbtable]: Change the table name to insert to. Defaults to **netdata**. |
|
91 |
* [-t|--timeout]: Maximum time in milliseconds to wait for data. Slow |
|
92 |
connections may need to increase this from the default 500 ms. |
|
93 |
* [-v|--version]: Show version. |
|
94 |
||
95 |
||
96 |
**Notes** |
|
97 |
||
98 |
Nim option parsing might be slightly different than what you're used to. |
|
99 |
Flags that require arguments must include an '=' or ':' character. |
|
100 |
||
101 |
* --timeout=1000 *valid* |
|
102 |
* --timeout:1000 *valid* |
|
103 |
* --t:1000 *valid* |
|
104 |
* --timeout 1000 *invalid* |
|
105 |
* -t 1000 *invalid* |
|
106 |
||
107 |
All database connection options are passed as a key/val string to the |
|
108 |
*dbopts* flag. The default is: |
|
109 |
||
110 |
"host=localhost port=5432 dbname=netdata user=netdata application_name=netdata-tsrelay" |
|
111 |
||
112 |
Reference |
|
113 |
https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ- |
|
114 |
PARAMKEYWORDS for all available options (including how to store |
|
115 |
passwords in a seperate file, enable SSL mode, etc.) |
|
116 |
||
117 |
||
118 |
### Daemonizing |
|
119 |
||
120 |
Use a tool of your choice to run this at system |
|
121 |
startup in the background. My personal preference is |
|
122 |
[daemontools](https://cr.yp.to/daemontools.html), but I won't judge you |
|
123 |
if you use something else. |
|
124 |
||
125 |
Here's an example using the simple |
|
126 |
[daemon](https://www.freebsd.org/cgi/man.cgi?query=daemon&apropos=0&sektion=8&manpath=FreeBSD+11.0-RELEASE+and+Ports&arch=default&format=html) wrapper tool: |
|
127 |
||
128 |
# daemon \ |
|
129 |
-o /var/log/netdata_tsrelay.log \ |
|
130 |
-p /var/run/netdata_tsrelay.pid \ |
|
131 |
-u nobody -cr \ |
|
132 |
/usr/local/bin/netdata_tsrelay \ |
|
133 |
--dbopts="dbname=metrics user=metrics host=db-master port=6432 application_name=netdata-tsrelay" |
|
134 |