netdata_tsrelay.nim
author Mahlon E. Smith <mahlon@martini.nu>
Wed, 20 Oct 2021 11:41:30 -0700
changeset 24 af18a10710be
parent 22 4250addd706f
permissions -rw-r--r--
Update quitProc for nim 1.6 deprecation warning.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
     1
# vim: set et nosta sw=4 ts=4 :
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
     2
#
24
af18a10710be Update quitProc for nim 1.6 deprecation warning.
Mahlon E. Smith <mahlon@martini.nu>
parents: 22
diff changeset
     3
# Copyright (c) 2018-2021, Mahlon E. Smith <mahlon@martini.nu>
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
     4
# All rights reserved.
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
     5
# Redistribution and use in source and binary forms, with or without
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
     6
# modification, are permitted provided that the following conditions are met:
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
     7
#
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
     8
#     * Redistributions of source code must retain the above copyright
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
     9
#       notice, this list of conditions and the following disclaimer.
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    10
#
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    11
#     * Redistributions in binary form must reproduce the above copyright
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    12
#       notice, this list of conditions and the following disclaimer in the
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    13
#       documentation and/or other materials provided with the distribution.
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    14
#
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    15
#     * Neither the name of Mahlon E. Smith nor the names of his
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    16
#       contributors may be used to endorse or promote products derived
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    17
#       from this software without specific prior written permission.
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    18
#
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    19
# THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    20
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    21
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    22
# DISCLAIMED. IN NO EVENT SHALL THE REGENTS AND CONTRIBUTORS BE LIABLE FOR ANY
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    23
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    24
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    25
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    26
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    27
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    28
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    29
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    30
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    31
import
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    32
    db_postgres,
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    33
    json,
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    34
    nativesockets,
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    35
    net,
16
fce5b4150c09 Updates for nim 0.19.
Mahlon E. Smith <mahlon@martini.nu>
parents: 15
diff changeset
    36
    parseopt,
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
    37
    posix,
24
af18a10710be Update quitProc for nim 1.6 deprecation warning.
Mahlon E. Smith <mahlon@martini.nu>
parents: 22
diff changeset
    38
    std/exitprocs,
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    39
    strutils,
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    40
    strformat,
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    41
    tables,
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
    42
    terminal,
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
    43
    times
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    44
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    45
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    46
const
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    47
    VERSION = "v0.3.0"
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    48
    USAGE = """
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    49
./netdata_tsrelay [-adDhopqtTv]
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    50
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    51
  -a --listen-addr:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    52
    The outbound IP address to listen for netdata streams.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    53
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    54
  -d --debug:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    55
    Debug: Show incoming and parsed data.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    56
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    57
  -D --dropconn:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    58
    Drop the persistent socket to netdata between samples to conserve
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    59
    local resources.  This may be helpful with a large number of clients.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    60
    Defaults to false.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    61
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    62
  -h --help:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    63
    Help.  You're lookin' at it.
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
    64
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    65
  -o --dbopts:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    66
    The PostgreSQL connection string parameters.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    67
    The default connection string is:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    68
      "host=localhost dbname=netdata application_name=netdata-tsrelay"
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    69
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    70
  -p --listen-port:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    71
    Change the listening port from the default (14866).
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    72
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    73
  -P --persistent:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    74
    Don't disconnect from the database between samples.  This may be
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    75
    more efficient with a small number of clients, when not using a
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    76
    pooler, or with a very high sample size/rate.  Defaults to false.
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    77
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    78
  -q --quiet:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    79
    Quiet mode.  No output at all.  Ignored if -d is supplied.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    80
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    81
  -T --dbtable:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    82
    Change the destination table name from the default (netdata).
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    83
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    84
  -t --timeout:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    85
    Alter the maximum time (in ms) an open socket waits for data
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    86
    before processing the sample.  Default: 500ms.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    87
21
a2fe9ec4cdf2 Fix docs: exporting.conf example and help verbose/version typo.
Mahlon E. Smith <mahlon@laika.com>
parents: 19
diff changeset
    88
  -v --version:
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    89
    Display version number.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
    90
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    91
    """
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    92
    INSERT_SQL = """
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
    93
    INSERT INTO $1
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    94
        ( time, host, metrics )
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    95
    VALUES
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    96
        ( 'epoch'::timestamptz + ? * '1 second'::interval, ?, ? )
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    97
    """
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    98
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
    99
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   100
type
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   101
    Config = object of RootObj
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   102
        dbopts:      string  # The postgresql connection parameters.  (See https://www.postgresql.org/docs/current/static/libpq-connect.html)
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   103
        dbtable:     string  # The name of the table to write to.
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   104
        dropconn:    bool    # Close the TCP connection between samples.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   105
        persistent:  bool    # Don't close the database handle between samples.
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   106
        listen_port: int     # The port to listen for incoming connections.
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   107
        listen_addr: string  # The IP address listen for incoming connections.  Defaults to inaddr_any.
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   108
        verbose:     bool    # Be informative
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   109
        debug:       bool    # Spew out raw data
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   110
        insertsql:   string  # The SQL insert string after interpolating the table name.
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   111
        timeout:     int     # How long to block, waiting on connection data.
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   112
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   113
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   114
type
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   115
    NetdataClient = ref object
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   116
        sock: Socket    # The raw socket fd
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   117
        address: string # The remote IP address
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   118
        db: DbConn      # An optionally persistent database handle
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   119
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   120
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   121
# Global configuration
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   122
var conf: Config
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   123
9
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   124
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   125
proc hl( msg: string, fg: ForegroundColor, bright=false ): string =
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   126
    ## Quick wrapper for color formatting a string, since the 'terminal'
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   127
    ## module only deals with stdout directly.
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   128
    if not isatty(stdout): return msg
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   129
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   130
    var color: BiggestInt = ord( fg )
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   131
    if bright: inc( color, 60 )
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   132
    result = "\e[" & $color & 'm' & msg & "\e[0m"
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   133
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   134
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   135
proc fetch_data( client: NetdataClient ): string =
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   136
    ## Netdata JSON backend doesn't send a length nor a separator
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   137
    ## between samples, so we read line by line and wait for stream
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   138
    ## timeout to determine what constitutes a sample.
17
96b8799a565a Fix command line usage docs, replace deprecated recvLine() with readLine().
Mahlon E. Smith <mahlon@martini.nu>
parents: 16
diff changeset
   139
    var buf = ""
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   140
    while true:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   141
        try:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   142
            client.sock.readline( buf, timeout=conf.timeout )
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   143
            if buf == "":
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   144
                if conf.debug: echo "Client {client.address} closed socket.".fmt.hl( fgRed, bright=true )
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   145
                quit( 1 )
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   146
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   147
            result = result & buf & "\n"
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   148
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   149
        except OSError:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   150
            quit( 1 )
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   151
        except TimeoutError:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   152
            if result == "": continue 
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   153
            return
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   154
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   155
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   156
proc parse_data( data: string ): seq[ JsonNode ] =
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   157
    ## Given a raw +data+ string, parse JSON and return a sequence
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   158
    ## of JSON samples. Netdata can buffer multiple samples in one batch.
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   159
    result = @[]
16
fce5b4150c09 Updates for nim 0.19.
Mahlon E. Smith <mahlon@martini.nu>
parents: 15
diff changeset
   160
    if data == "": return
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   161
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   162
    # Hash of sample timeperiods to pivoted json data
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   163
    var pivoted_data = init_table[ BiggestInt, JsonNode ]()
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   164
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   165
    for sample in split_lines( data ):
16
fce5b4150c09 Updates for nim 0.19.
Mahlon E. Smith <mahlon@martini.nu>
parents: 15
diff changeset
   166
        if sample == "": continue
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   167
        if conf.debug: echo sample.hl( fgBlack, bright=true )
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   168
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   169
        var parsed: JsonNode
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   170
        try:
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   171
            parsed = sample.parse_json
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   172
        except JsonParsingError:
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   173
            if conf.debug: echo hl( "Unable to parse sample line: " & sample.hl(fgRed, bright=true), fgRed )
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   174
            continue
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   175
        if parsed.kind != JObject: return
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   176
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   177
        # Create or use existing Json object for modded data.
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   178
        #
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   179
        var pivot: JsonNode
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   180
        try:
16
fce5b4150c09 Updates for nim 0.19.
Mahlon E. Smith <mahlon@martini.nu>
parents: 15
diff changeset
   181
            let key = parsed[ "timestamp" ].get_int
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   182
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   183
            if pivoted_data.has_key( key ):
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   184
                pivot = pivoted_data[ key ]
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   185
            else:
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   186
                pivot = newJObject()
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   187
                pivoted_data[ key ] = pivot
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   188
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   189
            var name = parsed[ "chart_id" ].get_str & "." & parsed[ "id" ].get_str
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   190
            pivot[ "hostname" ] = parsed[ "hostname" ]
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   191
            pivot[ "timestamp" ] = parsed[ "timestamp" ]
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   192
            pivot[ name ] = parsed[ "value" ]
22
4250addd706f Support storing Netdata host labels, if they are passed to this json backend.
Mahlon E. Smith <mahlon@martini.nu>
parents: 21
diff changeset
   193
4250addd706f Support storing Netdata host labels, if they are passed to this json backend.
Mahlon E. Smith <mahlon@martini.nu>
parents: 21
diff changeset
   194
            if parsed.has_key( "labels" ):
4250addd706f Support storing Netdata host labels, if they are passed to this json backend.
Mahlon E. Smith <mahlon@martini.nu>
parents: 21
diff changeset
   195
                pivot[ "labels" ] = parsed[ "labels" ]
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   196
        except:
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   197
            continue
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   198
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   199
    for timestamp, sample in pivoted_data:
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   200
        result.add( sample )
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   201
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   202
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   203
proc write_to_database( client: NetdataClient, samples: seq[ JsonNode ] ): void =
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   204
    ## Given a sequence of json samples, write them to database.
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   205
    if samples.len == 0: return
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   206
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   207
    if client.db.isNil:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   208
        client.db = open( "", "", "", conf.dbopts )
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   209
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   210
    try:
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   211
        client.db.exec sql( "BEGIN" )
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   212
        for sample in samples:
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   213
            var
16
fce5b4150c09 Updates for nim 0.19.
Mahlon E. Smith <mahlon@martini.nu>
parents: 15
diff changeset
   214
                timestamp = sample[ "timestamp" ].get_int
15
ed87882bb7f0 Lowercase all hostnames before sending to the database.
Mahlon E. Smith <mahlon@martini.nu>
parents: 13
diff changeset
   215
                host = sample[ "hostname" ].get_str.to_lowerascii
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   216
            sample.delete( "timestamp" )
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   217
            sample.delete( "hostname" )
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   218
            client.db.exec sql( conf.insertsql ), timestamp, host, sample
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   219
        client.db.exec sql( "COMMIT" )
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   220
    except:
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   221
        let
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   222
            e = getCurrentException()
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   223
            msg = getCurrentExceptionMsg()
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   224
        echo "Got exception ", repr(e), " while writing to DB: ", msg
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   225
        discard
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   226
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   227
    if not conf.persistent:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   228
        client.db.close
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   229
        client.db = nil
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   230
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   231
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   232
proc process( client: NetdataClient ): void =
9
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   233
    ## Do the work for a connected client within child process.
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   234
    let t0 = cpu_time()
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   235
    var raw_data = client.fetch_data
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   236
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   237
    # Done with the socket, netdata will automatically
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   238
    # reconnect.  Save local resources/file descriptors
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   239
    # by closing after the send is considered complete.
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   240
    #
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   241
    if conf.dropconn:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   242
        try:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   243
            client.sock.close
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   244
        except OSError:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   245
            return
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   246
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   247
    # Pivot the parsed data to a single JSON blob per sample time.
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   248
    var samples = parse_data( raw_data )
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   249
    client.write_to_database( samples )
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   250
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   251
    if conf.verbose:
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   252
        let cputime = cpu_time() - t0
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   253
        echo(
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   254
            hl( $(epochTime().to_int), fgMagenta, bright=true ),
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   255
            " ",
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   256
            hl( $(samples.len), fgWhite, bright=true ),
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   257
            " sample(s) parsed from ",
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   258
            client.address.hl( fgYellow, bright=true ),
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   259
            " in ", hl( "{cputime:<2.3f}".fmt, fgWhite, bright=true), " seconds."
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   260
        )
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   261
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   262
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   263
proc serverloop( conf: Config ): void =
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   264
    ## Open a database connection, bind to the listening socket,
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   265
    ## and start serving incoming netdata streams.
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   266
    let db = open( "", "", "", conf.dbopts )
9
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   267
    db.close
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   268
    if conf.verbose: echo( "Successfully tested connection to the backend database.".hl( fgGreen ) )
9
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   269
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   270
    # Ensure children are properly reaped.
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   271
    #
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   272
    var sa: Sigaction
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   273
    sa.sa_handler = SIG_IGN
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   274
    discard sigaction( SIGCHLD, sa )
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   275
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   276
    # Setup listening socket.
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   277
    #
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   278
    var server = newSocket()
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   279
    server.set_sock_opt( OptReuseAddr, true )
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   280
    server.bind_addr( Port(conf.listen_port), conf.listen_addr )
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   281
    server.listen()
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   282
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   283
    if conf.verbose:
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   284
        echo(
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   285
            "Listening for incoming connections on ".hl( fgGreen, bright=true ),
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   286
            hl( (if conf.listen_addr == "0.0.0.0": "*" else: conf.listen_addr) , fgBlue, bright=true ),
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   287
            ":",
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   288
            hl( $conf.listen_port, fgBlue, bright=true ),
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   289
        )
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   290
        echo ""
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   291
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   292
    # Wait for incoming connections, fork for each client.
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   293
    #
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   294
    while true:
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   295
        let client = NetdataClient.new
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   296
        client.sock = Socket.new
6
1f366fc61592 Each incoming connection requires its own client socket.
Mahlon E. Smith <mahlon@laika.com>
parents: 5
diff changeset
   297
9
aa9d537f7067 Properly reap child processes.
Mahlon E. Smith <mahlon@laika.com>
parents: 8
diff changeset
   298
        # Block, waiting for new connections.
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   299
        server.acceptAddr( client.sock, client.address )
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   300
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   301
        if fork() == 0:
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   302
            server.close
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   303
            if conf.dropconn:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   304
                # "one shot" mode.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   305
                client.process
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   306
                quit( 0 )
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   307
            else:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   308
                # Keep the connection to netdata open.
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   309
                while true: client.process
7
c0bcf3bea772 Force a GC pass after 25 cycles. Don't bother with sync() at exit, unnecessary.
Mahlon E. Smith <mahlon@laika.com>
parents: 6
diff changeset
   310
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   311
        client.sock.close
7
c0bcf3bea772 Force a GC pass after 25 cycles. Don't bother with sync() at exit, unnecessary.
Mahlon E. Smith <mahlon@laika.com>
parents: 6
diff changeset
   312
        when defined( testing ): dumpNumberOfInstances()
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   313
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   314
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   315
proc parse_cmdline: Config =
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   316
    ## Populate the config object with the user's preferences.
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   317
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   318
    # Config object defaults.
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   319
    #
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   320
    result = Config(
13
e1777929ba15 Remove port and user from the default dbopts, so they instead use the postgresql behavioral default.
Mahlon E. Smith <mahlon@laika.com>
parents: 11
diff changeset
   321
        dbopts: "host=localhost dbname=netdata application_name=netdata-tsrelay",
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   322
        dbtable: "netdata",
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   323
        dropconn: false,
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   324
        listen_port: 14866,
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   325
        listen_addr: "0.0.0.0",
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   326
        verbose: true,
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   327
        debug: false,
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   328
        timeout: 500,
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   329
        persistent: false,
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   330
        insertsql: INSERT_SQL % [ "netdata" ]
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   331
    )
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   332
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   333
    # always set debug mode if development build.
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   334
    result.debug = defined( testing )
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   335
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   336
    for kind, key, val in getopt():
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   337
        case kind
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   338
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   339
        of cmdArgument:
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   340
            discard
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   341
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   342
        of cmdLongOption, cmdShortOption:
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   343
            case key
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   344
                of "debug", "d":
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   345
                    result.debug = true
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   346
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   347
                of "dropconn", "D":
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   348
                    if result.persistent:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   349
                        echo "Dropping TCP sockets are incompatible with persistent database connections."
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   350
                        quit( 1 )
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   351
                    result.dropconn = true
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   352
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   353
                of "help", "h":
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   354
                    echo USAGE
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   355
                    quit( 0 )
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   356
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   357
                of "quiet", "q":
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   358
                    result.verbose = false
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   359
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   360
                of "version", "v":
5
a1276c3d39eb Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 4
diff changeset
   361
                    echo hl( "netdata_tsrelay " & VERSION, fgWhite, bright=true )
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   362
                    quit( 0 )
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   363
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   364
                of "timeout", "t": result.timeout = val.parse_int
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   365
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   366
                of "dbtable", "T":
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   367
                    result.insertsql = INSERT_SQL % [ val ]
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   368
                of "dbopts", "o": result.dbopts = val
11
475c9942eb15 Multiple changes:
Mahlon E. Smith <mahlon@laika.com>
parents: 9
diff changeset
   369
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   370
                of "listen-addr", "a": result.listen_addr = val
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   371
                of "listen-port", "p": result.listen_port = val.parse_int
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   372
19
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   373
                of "persistent", "P":
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   374
                    if result.dropconn:
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   375
                        echo "Persistent database connections are incompatible with dropping TCP sockets."
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   376
                        quit( 1 )
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   377
                    result.persistent = true
1f09cfb560e0 Multiple changes.
Mahlon E. Smith <mahlon@martini.nu>
parents: 17
diff changeset
   378
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   379
                else: discard
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   380
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   381
        of cmdEnd: assert( false ) # shouldn't reach here ever
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   382
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   383
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   384
when isMainModule:
24
af18a10710be Update quitProc for nim 1.6 deprecation warning.
Mahlon E. Smith <mahlon@martini.nu>
parents: 22
diff changeset
   385
    addExitProc( resetAttributes )
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   386
    conf = parse_cmdline()
7
c0bcf3bea772 Force a GC pass after 25 cycles. Don't bother with sync() at exit, unnecessary.
Mahlon E. Smith <mahlon@laika.com>
parents: 6
diff changeset
   387
    if conf.debug: echo hl( $conf, fgYellow )
8
1ef3f2d6d10e Multiple changes.
Mahlon E. Smith <mahlon@laika.com>
parents: 7
diff changeset
   388
    serverloop( conf )
0
72c9c6f0b713 Initial commit.
Mahlon E. Smith <mahlon@laika.com>
parents:
diff changeset
   389