mahlon/nim-tnetstring
1
0
Fork 0
Code Issues Pull requests Activity

10 years later... updates for modern Nim.

Browse source 
FossilOrigin-Name: 7361ae25595c1916e9701a25033b684e3cd089d8abb71ead3803a59e36189f63
This commit is contained in:
Mahlon E. Smith 2025-04-03 18:27:46 +00:00
parent ae7b77df4c
commit 9aa2912286
23 changed files with 640 additions and 371 deletions
Download patch file Download diff file Expand all files Collapse all files

4
.hgignore
View file

@ -1,4 +0,0 @@
syntax: glob
.cache
tnetstring
*.html

62
README.md
View file

@ -1,6 +1,6 @@
# README #
# Nim TNetstring
### What's this? ###
## Description
This module implements a simple TNetstring parser and serializer.
TNetString stands for "tagged netstring" and is a modification of Dan
@ -10,21 +10,30 @@ overflows and backward compatible with original netstrings. They make
no assumptions about string contents, allowing for easy transmission of
ascii and binary data mixed with strongly typed values.
See http://cr.yp.to/proto/netstrings.txt and http://tnetstrings.org/ for
See http://cr.yp.to/proto/netstrings.txt and http://tnetstrings.info/ for
additional information.
You can also read the specification [here](Specification.md).
### Installation ###
## Prerequisites
None. This is a pure-nim library.
## Installation
The easiest way to install this module is via the nimble package manager,
by simply running 'nimble install tnetstring'.
by simply running:
Alternatively, you can fetch the 'tnetstring.nim' file yourself, and put it in a place of your choosing.
> % nimble install tnetstring
### Usage ###
Alternatively, you can fetch the 'tnetstring.nim' file yourself, and put it in a
place of your choosing.
```
#!nimrod
## Usage
```nim
import tnetstring
let
@ -43,8 +52,7 @@ import tnetstring
Results in:
```
#!nimrod
```nim
1.3
true
1
@ -55,26 +63,44 @@ Results in:
This module can also be used to reasonably create a serialized
TNetstring, suitable for network transmission:
```
#!nimrod
```nim
let
number = 1000
list = @[ "thing1", "thing2" ]
tnettop = newTNetstringArray() # top-level array
tnetsub = newTNetstringArray() # sub array
tnettop.add( newTNetstringInt(number) )
for item in list:
tnetsub.add( newTNetstringString(item) )
tnettop.add( tnetsub )
# Equivalent to: @[1000, @[thing1, thing2]]
echo dump_tnetstring( tnettop )
```
Results in:
```
#!nimrod
```nim
29:4:1000#18:6:thing1,6:thing2,]]
```
```
## Contributing
You can check out the current development source with Fossil via its [home
repo](https://code.martini.nu/fossil/nim-tnetstring), or with Git/Jujutsu at its
[project mirror](https://github.com/mahlonsmith/nim-tnetstring).
After checking out the source, running:
$ nimble setup
... will install dependencies, and do any other necessary setup for
development.
## Authors
- Mahlon E. Smith <mahlon@martini.nu>

240
Specification.md Normal file
View file

@ -0,0 +1,240 @@
---
title: "TNetstrings Specification"
source: "https://tnetstrings.info/"
description: "Spec for typed netstrings."
---
## About Tagged Netstrings
TNetStrings stand for a "tagged netstrings" and are a modification of Dan Bernstein's [netstrings](http://cr.yp.to/proto/netstrings.txt) specification to allow for the same data structures as [JSON](http://www.json.org/) but in a format that meets these requirements:
1. Trivial to parse in every language without making errors.
2. Resistant to buffer overflows and other problems.
3. Fast and low resource intensive.
4. Makes no assumptions about string contents and can store binary data without **escaping** or **encoding** them.
5. Backward compatible with original netstrings.
6. Transport agnostic, so it works with streams, messages, files, anything that's 8-bit clean.
## Grammar
The grammar for the protocol is simply:
```
SIZE = [0-9]{1,9}
COLON = ':'
DATA = (.*)
TYPE = ('#' | '}' | ']' | ',' | '!' | '~' | '^')
payload = (SIZE COLON DATA TYPE)+
```
Each of these elements is defined as:
`SIZE`
A ascii encoded integer that is no longer than 9 digits long.
`COLON`
A colon character.
`DATA`
A sequence of bytes that is `SIZE` in length. The bytes **can** include any of the `TYPE` characters since the `SIZE` is used to determine the end, not a terminal `TYPE` char.
`TYPE`
A character indicating what type the `DATA` is.
Each `TYPE` is used to determine the contents and maps to:
`,`
string (byte array)
`#`
integer
`^`
float
`!`
boolean of 'true' or 'false'
`~`
null always encoded as 0:~
`}`
Dictionary which you recurse into to fill with key=value pairs inside the payload contents.
`]`
List which you recurse into to fill with values of any type.
## Failure Mode
TNetstrings are all or nothing. Either they parse cleanly and a value is returned, or it aborts and cleans up any in-process data returning nothing. As in the reference implementation below, it's normal to return the remainder of a given buffer for further processing, meaning all of a given buffer does not need to be parsed for a single parsing call to be successful.
Since the `SIZE` can be read before consuming any other data, anyone receiving a message can abort immediately if the data exceeds a limit on the number of bytes.
## Implementation Restrictions
You are not allowed to implement any of the following features:
UTF-8 Strings
String encoding is an application level, political, and display specification. Transport protocols should not have to decode random character encodings accurately to function properly.
Arbitrary Dict Keys
Keys must be **strings** only.
Floats Undefined
Floats are encoded with X.Y format, with no precision, accuracy, or other assurances.
These restrictions exist to make the protocol reliable for anyone who uses it and to act as a constraint on the design to keep it simple.
## Reference Implemenation
You should be able to work with this simple reference implementation written in Python 2.5 or greater (but not 3.x):
```python
# Note this implementation is more strict than necessary to demonstrate
# minimum restrictions on types allowed in dictionaries.
def dump(data):
if type(data) is long or type(data) is int:
out = str(data)
return '%d:%s#' % (len(out), out)
elif type(data) is float:
out = '%f' % data
return '%d:%s^' % (len(out), out)
elif type(data) is str:
return '%d:' % len(data) + data + ','
elif type(data) is dict:
return dump_dict(data)
elif type(data) is list:
return dump_list(data)
elif data == None:
return '0:~'
elif type(data) is bool:
out = repr(data).lower()
return '%d:%s!' % (len(out), out)
else:
assert False, "Can't serialize stuff that's %s." % type(data)
def parse(data):
payload, payload_type, remain = parse_payload(data)
if payload_type == '#':
value = int(payload)
elif payload_type == '}':
value = parse_dict(payload)
elif payload_type == ']':
value = parse_list(payload)
elif payload_type == '!':
value = payload == 'true'
elif payload_type == '^':
value = float(payload)
elif payload_type == '~':
assert len(payload) == 0, "Payload must be 0 length for null."
value = None
elif payload_type == ',':
value = payload
else:
assert False, "Invalid payload type: %r" % payload_type
return value, remain
def parse_payload(data):
assert data, "Invalid data to parse, it's empty."
length, extra = data.split(':', 1)
length = int(length)
payload, extra = extra[:length], extra[length:]
assert extra, "No payload type: %r, %r" % (payload, extra)
payload_type, remain = extra[0], extra[1:]
assert len(payload) == length, "Data is wrong length %d vs %d" % (length, len(payload))
return payload, payload_type, remain
def parse_list(data):
if len(data) == 0: return []
result = []
value, extra = parse(data)
result.append(value)
while extra:
value, extra = parse(extra)
result.append(value)
return result
def parse_pair(data):
key, extra = parse(data)
assert extra, "Unbalanced dictionary store."
value, extra = parse(extra)
return key, value, extra
def parse_dict(data):
if len(data) == 0: return {}
key, value, extra = parse_pair(data)
assert type(key) is str, "Keys can only be strings."
result = {key: value}
while extra:
key, value, extra = parse_pair(extra)
result[key] = value
return result
def dump_dict(data):
result = []
for k,v in data.items():
result.append(dump(str(k)))
result.append(dump(v))
payload = ''.join(result)
return '%d:' % len(payload) + payload + '}'
def dump_list(data):
result = []
for i in data:
result.append(dump(i))
payload = ''.join(result)
return '%d:' % len(payload) + payload + ']'
```
## Conformance
If your implementation does not work with the above Python implementation then it is wrong and is not tnetstrings. It's that simple.
## Streaming
Tnetstrings put the length at the beginning and the type at the end so that you have to read all of the data element and cannot "stream" it. This makes it much easier to handle, since nested data structures need to be loaded into RAM anyway to handle them. It's also unnecessary to allow for streaming, since sockets/files/etc are already streamable. If you need to send 1000 DVDs, don't try to encode them in 1 tnetstring payload, instead send them as a sequence of tnetstrings as payload chunks with checks and headers like most other protocols. In other words: If you think you need to dive into a tnetstring data type to "stream", then you need to remove one layer and flatten it instead.
Here's an example to make this concrete. Many protocols have a simple `HEADER+BODY` design where the `HEADER` is usually some kind of dict, and the body is a raw binary blob of data. Your first idea might be to create one tnetstring of `[{HEADER}, "BODY"]`, but that'd only work if you expect to limit the request sizes. If the requests can be any size then you **actually** should do one of two things:
1. Design the protocol so that it's **always** `HEADER` followed by `BODY`, with no tnetstring wrapping them. Tnetstrings APIs are designed so that you can read, parse, then take the remainder and keep reading, so this is easy. It does limit asynchronous operations.
2. Design the protocol so that messages are a limited size `[{HEADER}, "BODY"]` design, of say 64k, and then let senders use the header to indicate a UUID and a `"SENDMORE"` flag. Usually the first header would indicate the full message size, a checksum, and a first body chunk. Then it sends each new piece with a small header and the original UUID. Finally when it's done it closes it off with a final message. This has the disadvantage of taking up a few more bytes on each message, but has the advantage that you can send multiple streams at once over the same pipe.
Finally, the general rule is senders should have to completely specify the size of what they send and receivers should be ready to reject it. If you allow arbitrary streaming then your servers will suffer attacks that eat your resources.
The contents of this page were copied and maintained by [Michael Granger](ged@deveiate.org), from an old archived copy of Zed A. Shaw's tnetstrings.org site (which has since disappeared). Most of the content contained herein was written by Zed.

427
src/tnetstring.nim
View file

@ -1,105 +1,13 @@
#
# Copyright (c) 2015, Mahlon E. Smith <mahlon@martini.nu>
# All rights reserved.
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# * Neither the name of Mahlon E. Smith nor the names of his
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE REGENTS AND CONTRIBUTORS BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
## This module implements a simple TNetstring parser and serializer.
## TNetString stands for "tagged netstring" and is a modification of Dan
## Bernstein's netstrings specification. TNetstrings allow for the same data
## structures as JSON but in a format that is resistant to buffer overflows
## and backward compatible with original netstrings. They make no assumptions
## about string contents, allowing for easy transmission of binary data mixed
## with strongly typed values.
## See http://cr.yp.to/proto/netstrings.txt and http://tnetstrings.org/ for additional information.
##
## This module borrows heavily (in both usage and code) from the nim JSON stdlib
## (json.nim) -- (c) Copyright 2015 Andreas Rumpf, Dominik Picheta.
##
## Usage example:
##
## .. code-block:: nim
##
## let
## tnetstr = "52:4:test,3:1.3^4:key2,4:true!6:things,12:1:1#1:2#1:3#]}"
## tnetobj = parse_tnetstring( tnetstr )
##
## # tnetobj is now equivalent to the structure:
## # @[(key: test, val: 1.3), (key: key2, val: true), (key: things, val: @[1, 2, 3])]
##
## assert( tnetobj.kind == TNetstringObject )
## echo tnetobj[ "test" ]
## echo tnetobj[ "key2" ]
## for item in tnetobj[ "things" ]:
## echo item
##
## Results in:
##
## .. code-block:: nim
##
## 1.3
## true
## 1
## 2
## 3
##
## This module can also be used to reasonably create a serialized
## TNetstring, suitable for network transmission:
##
## .. code-block:: nim
##
## let
## number = 1000
## list = @[ "thing1", "thing2" ]
## tnettop = newTNetstringArray() # top-level array
## tnetsub = newTNetstringArray() # sub array
##
## tnettop.add( newTNetstringInt(number) )
## for item in list:
## tnetsub.add( newTNetstringString(item) )
## tnettop.add( tnetsub )
##
## # Equivalent to: @[1000, @[thing1, thing2]]
## echo dump_tnetstring( tnettop )
##
## Results in:
##
## .. code-block:: nim
##
## 29:4:1000#18:6:thing1,6:thing2,]]
##
# vim: set et sta sw=4 ts=4 :
import
hashes,
parseutils,
strutils
std/hashes,
std/parseutils,
std/strutils
const version = "0.1.1"
const TNETSTRING_VERSION* = "0.2.0"
type
type
TNetstringKind* = enum ## enumeration of all valid types
TNetstringString, ## a string literal
TNetstringInt, ## an integer literal
@ -131,88 +39,76 @@ type
TNetstringParseError* = object of ValueError ## Raised for a TNetstring error
proc raiseParseErr*( t: TNetstringNode, msg: string ) {.noinline, noreturn.} =
## Raises a `TNetstringParseError` exception.
raise newException( TNetstringParseError, msg )
proc newTNetstringString*( s: string ): TNetstringNode =
func newTNetstringString*( s: string ): TNetstringNode =
## Create a new String typed TNetstringNode.
new( result )
result.kind = TNetstringString
result = TNetstringNode( kind: TNetstringString )
result.str = s
proc newTNetstringInt*( i: BiggestInt ): TNetstringNode =
func newTNetstringInt*( i: BiggestInt ): TNetstringNode =
## Create a new Integer typed TNetstringNode.
new( result )
result.kind = TNetstringInt
result = TNetstringNode( kind: TNetstringInt )
result.num = i
proc newTNetstringFloat*( f: float ): TNetstringNode =
func newTNetstringFloat*( f: float ): TNetstringNode =
## Create a new Float typed TNetstringNode.
new( result )
result.kind = TNetstringFloat
result = TNetstringNode( kind: TNetstringFloat )
result.fnum = f
proc newTNetstringBool*( b: bool ): TNetstringNode =
func newTNetstringBool*( b: bool ): TNetstringNode =
## Create a new Boolean typed TNetstringNode.
new( result )
result.kind = TNetstringBool
result = TNetstringNode( kind: TNetstringBool )
result.bval = b
proc newTNetstringNull*(): TNetstringNode =
func newTNetstringNull*(): TNetstringNode =
## Create a new nil typed TNetstringNode.
new( result )
result.kind = TNetstringNull
result = TNetstringNode( kind: TNetstringNull )
proc newTNetstringObject*(): TNetstringNode =
func newTNetstringObject*(): TNetstringNode =
## Create a new Object typed TNetstringNode.
new( result )
result.kind = TNetstringObject
result = TNetstringNode( kind: TNetstringObject )
result.fields = @[]
proc newTNetstringArray*(): TNetstringNode =
func newTNetstringArray*(): TNetstringNode =
## Create a new Array typed TNetstringNode.
new( result )
result.kind = TNetstringArray
result = TNetstringNode( kind: TNetstringArray )
result.elems = @[]
proc getStr*( node: TNetstringNode, default: string = "" ): string =
func getStr*( node: TNetstringNode, default: string = "" ): string =
## Retrieves the string value of a `TNetstringString TNetstringNodee`.
## Returns ``default`` if ``node`` is not a ``TNetstringString``.
if node.kind != TNetstringString: return default
return node.str
proc getInt*( node: TNetstringNode, default: BiggestInt = 0 ): BiggestInt =
func getInt*( node: TNetstringNode, default: BiggestInt = 0 ): BiggestInt =
## Retrieves the int value of a `TNetstringInt TNetstringNode`.
## Returns ``default`` if ``node`` is not a ``TNetstringInt``.
if node.kind != TNetstringInt: return default
return node.num
proc getFloat*( node: TNetstringNode, default: float = 0.0 ): float =
func getFloat*( node: TNetstringNode, default: float = 0.0 ): float =
## Retrieves the float value of a `TNetstringFloat TNetstringNode`.
## Returns ``default`` if ``node`` is not a ``TNetstringFloat``.
if node.kind != TNetstringFloat: return default
return node.fnum
proc getBool*( node: TNetstringNode, default: bool = false ): bool =
func getBool*( node: TNetstringNode, default: bool = false ): bool =
## Retrieves the bool value of a `TNetstringBool TNetstringNode`.
## Returns ``default`` if ``node`` is not a ``TNetstringBool``.
if node.kind != TNetstringBool: return default
return node.bval
proc getFields*( node: TNetstringNode,
func getFields*( node: TNetstringNode,
default: seq[tuple[key: string, val: TNetstringNode]] = @[] ):
seq[tuple[key: string, val: TNetstringNode]] =
## Retrieves the key, value pairs of a `TNetstringObject TNetstringNode`.
@ -221,14 +117,14 @@ proc getFields*( node: TNetstringNode,
return node.fields
proc getElems*( node: TNetstringNode, default: seq[TNetstringNode] = @[] ): seq[TNetstringNode] =
func getElems*( node: TNetstringNode, default: seq[TNetstringNode] = @[] ): seq[TNetstringNode] =
## Retrieves the values of a `TNetstringArray TNetstringNode`.
## Returns ``default`` if ``node`` is not a ``TNetstringArray``.
if node.kind != TNetstringArray: return default
return node.elems
proc parse_tnetstring*( data: string ): TNetstringNode =
proc parseTNetstring*( data: string ): TNetstringNode =
## Given an encoded tnetstring, parse and return a TNetstringNode.
var
length: int
@ -237,17 +133,22 @@ proc parse_tnetstring*( data: string ): TNetstringNode =
extra: string
let sep_pos = data.skipUntil( ':' )
if sep_pos == data.len: raiseParseErr( result, "Invalid data: No separator token found." )
if sep_pos == data.len:
raise newException( TNetstringParseError, "Invalid data: No separator token found." )
try:
length = data[ 0 .. sep_pos - 1 ].parseInt
kind = data[ sep_pos + length + 1 ]
payload = data[ sep_pos + 1 .. sep_pos + length ]
extra = data[ sep_pos + length + 2 .. ^1 ]
length = data[ 0 .. sep_pos - 1 ].parseInt
except ValueError, IndexError:
var msg = getCurrentExceptionMsg()
raiseParseErr( result, msg )
if ($length).len > 9:
raise newException( TNetstringParseError, "Invalid data: Size more than 9 digits." )
kind = data[ sep_pos + length + 1 ]
payload = data[ sep_pos + 1 .. sep_pos + length ]
extra = data[ sep_pos + length + 2 .. ^1 ]
except ValueError, IndexDefect:
let msg = getCurrentExceptionMsg()
raise newException( TNetstringParseError, msg )
case kind:
of ',':
@ -258,22 +159,26 @@ proc parse_tnetstring*( data: string ): TNetstringNode =
result = newTNetstringInt( payload.parseBiggestInt )
except ValueError:
var msg = getCurrentExceptionMsg()
raiseParseErr( result, msg )
raise newException( TNetstringParseError, msg )
of '^':
try:
result = newTNetstringFloat( payload.parseFloat )
except ValueError:
var msg = getCurrentExceptionMsg()
raiseParseErr( result, msg )
raise newException( TNetstringParseError, msg )
of '!':
result = newTNetstringBool( payload == "true" )
of '~':
if length != 0: raiseParseErr( result, "Invalid data: Payload must be 0 length for null." )
if length != 0:
raise newException(
TNetstringParseError,
"Invalid data: Payload must be 0 length for null."
)
result = newTNetstringNull()
of ']':
result = newTNetstringArray()
@ -288,22 +193,27 @@ proc parse_tnetstring*( data: string ): TNetstringNode =
result = newTNetstringObject()
var key = parse_tnetstring( payload )
if ( key.extra == "" ): raiseParseErr( result, "Invalid data: Unbalanced tuple." )
if ( key.kind != TNetstringString ): raiseParseErr( result, "Invalid data: Object keys must be strings." )
if ( key.extra == "" ):
raise newException( TNetstringParseError, "Invalid data: Unbalanced tuple." )
if ( key.kind != TNetstringString ):
raise newException( TNetstringParseError, "Invalid data: Object keys must be strings." )
var value = parse_tnetstring( key.extra )
result.fields.add( (key: key.str, val: value) )
while value.extra != "":
var subkey = parse_tnetstring( value.extra )
if ( subkey.extra == "" ): raiseParseErr( result, "Invalid data: Unbalanced tuple." )
if ( subkey.kind != TNetstringString ): raiseParseErr( result, "Invalid data: Object keys must be strings." )
if ( subkey.extra == "" ):
raise newException( TNetstringParseError, "Invalid data: Unbalanced tuple." )
if ( subkey.kind != TNetstringString ):
raise newException( TNetstringParseError, "Invalid data: Object keys must be strings." )
value = parse_tnetstring( subkey.extra )
result.fields.add( (key: subkey.str, val: value) )
else:
raiseParseErr( result, "Invalid data: Unknown tnetstring type '$1'." % $kind )
let msg = "Invalid data: Unknown tnetstring type '$1'." % $kind
raise newException( TNetstringParseError, msg )
result.extra = extra
@ -338,7 +248,7 @@ iterator mpairs*( node: var TNetstringNode ): var tuple[ key: string, val: TNets
yield keyVal
proc `$`*( node: TNetstringNode ): string =
func `$`*( node: TNetstringNode ): string =
## Delegate stringification of `TNetstringNode` to its underlying object.
return case node.kind:
of TNetstringString:
@ -357,35 +267,14 @@ proc `$`*( node: TNetstringNode ): string =
$node.fields
proc `==`* ( a, b: TNetstringNode ): bool =
func `==`*( a, b: TNetstringNode ): bool =
## Check two TNetstring nodes for equality.
if a.isNil:
if b.isNil: return true
return false
elif b.isNil or a.kind != b.kind:
return false
else:
return case a.kind
of TNetstringString:
a.str == b.str
of TNetstringInt:
a.num == b.num
of TNetstringFloat:
a.fnum == b.fnum
of TNetstringBool:
a.bval == b.bval
of TNetstringNull:
true
of TNetstringArray:
a.elems == b.elems
of TNetstringObject:
a.fields == b.fields
return a.kind == b.kind and $a == $b
proc copy*( node: TNetstringNode ): TNetstringNode =
func copy*( node: TNetstringNode ): TNetstringNode =
## Perform a deep copy of TNetstringNode.
new( result )
result.kind = node.kind
result = TNetstringNode( kind: node.kind )
result.extra = node.extra
case node.kind
@ -409,17 +298,17 @@ proc copy*( node: TNetstringNode ): TNetstringNode =
result.fields.add( (key, copy(value)) )
proc delete*( node: TNetstringNode, key: string ) =
func delete*( node: TNetstringNode, key: string ) =
## Deletes ``node[key]`` preserving the order of the other (key, value)-pairs.
assert( node.kind == TNetstringObject )
for i in 0..node.fields.len - 1:
if node.fields[i].key == key:
node.fields.delete( i )
return
raise newException( IndexError, "key not in object" )
raise newException( IndexDefect, "key not in object" )
proc hash*( node: TNetstringNode ): Hash =
func hash*( node: TNetstringNode ): Hash =
## Compute the hash for a TNetstringString node
return case node.kind
of TNetstringString:
@ -438,7 +327,7 @@ proc hash*( node: TNetstringNode ): Hash =
hash( node.fields )
proc len*( node: TNetstringNode ): int =
func len*( node: TNetstringNode ): int =
## If `node` is a `TNetstringArray`, it returns the number of elements.
## If `node` is a `TNetstringObject`, it returns the number of pairs.
## If `node` is a `TNetstringString`, it returns strlen.
@ -454,7 +343,7 @@ proc len*( node: TNetstringNode ): int =
0
proc `[]`*( node: TNetstringNode, name: string ): TNetstringNode =
func `[]`*( node: TNetstringNode, name: string ): TNetstringNode =
## Gets a field from a `TNetstringNode`, which must not be nil.
## If the value at `name` does not exist, returns nil
assert( not isNil(node) )
@ -465,7 +354,7 @@ proc `[]`*( node: TNetstringNode, name: string ): TNetstringNode =
return nil
proc `[]`*( node: TNetstringNode, index: int ): TNetstringNode =
func `[]`*( node: TNetstringNode, index: int ): TNetstringNode =
## Gets the node at `index` in an Array. Result is undefined if `index`
## is out of bounds.
assert( not isNil(node) )
@ -473,20 +362,20 @@ proc `[]`*( node: TNetstringNode, index: int ): TNetstringNode =
return node.elems[ index ]
proc hasKey*( node: TNetstringNode, key: string ): bool =
func hasKey*( node: TNetstringNode, key: string ): bool =
## Checks if `key` exists in `node`.
assert( node.kind == TNetstringObject )
for k, item in items( node.fields ):
if k == key: return true
proc add*( parent, child: TNetstringNode ) =
func add*( parent, child: TNetstringNode ) =
## Appends `child` to a TNetstringArray node `parent`.
assert( parent.kind == TNetstringArray )
parent.elems.add( child )
proc add*( node: TNetstringNode, key: string, val: TNetstringNode ) =
func add*( node: TNetstringNode, key: string, val: TNetstringNode ) =
## Adds ``(key, val)`` pair to the TNetstringObject `node`.
## For speed reasons no check for duplicate keys is performed.
## (Note, ``[]=`` performs the check.)
@ -494,13 +383,13 @@ proc add*( node: TNetstringNode, key: string, val: TNetstringNode ) =
node.fields.add( (key, val) )
proc `[]=`*( node: TNetstringNode, index: int, val: TNetstringNode ) =
func `[]=`*( node: TNetstringNode, index: int, val: TNetstringNode ) =
## Sets an index for a `TNetstringArray`.
assert( node.kind == TNetstringArray )
node.elems[ index ] = val
proc `[]=`*( node: TNetstringNode, key: string, val: TNetstringNode ) =
func `[]=`*( node: TNetstringNode, key: string, val: TNetstringNode ) =
## Sets a field from a `TNetstringObject`. Performs a check for duplicate keys.
assert( node.kind == TNetstringObject )
for i in 0 .. node.fields.len - 1:
@ -510,7 +399,7 @@ proc `[]=`*( node: TNetstringNode, key: string, val: TNetstringNode ) =
node.fields.add( (key, val) )
proc dump_tnetstring*( node: TNetstringNode ): string =
func dump_tnetstring*( node: TNetstringNode ): string =
## Renders a TNetstring `node` as a regular string.
case node.kind
of TNetstringString:
@ -538,170 +427,10 @@ proc dump_tnetstring*( node: TNetstringNode ): string =
result = $( result.len ) & ':' & result & '}'
# Quickie round-tripper.
#
# Tests!
#
when isMainModule:
# Expected exceptions
#
try:
discard parse_tnetstring( "totally invalid" )
except TNetstringParseError:
doAssert( true, "invalid tnetstring" )
try:
discard parse_tnetstring( "what:ever" )
except TNetstringParseError:
doAssert( true, "bad length" )
try:
discard parse_tnetstring( "3:yep~" )
except TNetstringParseError:
doAssert( true, "null w/ > 0 length" )
try:
discard parse_tnetstring( "8:1:1#1:1#}" )
except TNetstringParseError:
doAssert( true, "hash with non-string key" )
try:
discard parse_tnetstring( "7:4:test,}" )
except TNetstringParseError:
doAssert( true, "hash with odd number of elements" )
try:
discard parse_tnetstring( "2:25*" )
except TNetstringParseError:
doAssert( true, "unknown netstring tag" )
# Equality
#
let tnet_int = parse_tnetstring( "1:1#" )
doAssert( tnet_int == tnet_int )
doAssert( tnet_int == parse_tnetstring( "1:1#" ) )
doAssert( parse_tnetstring( "0:~" ) == parse_tnetstring( "0:~" ) )
# Type detection
#
doAssert( tnet_int.kind == TNetstringInt )
doAssert( parse_tnetstring( "1:a," ).kind == TNetstringString )
doAssert( parse_tnetstring( "3:1.0^" ).kind == TNetstringFloat )
doAssert( parse_tnetstring( "5:false!" ).kind == TNetstringBool )
doAssert( parse_tnetstring( "0:~" ).kind == TNetstringNull )
doAssert( parse_tnetstring( "9:2:hi,1:1#}" ).kind == TNetstringObject )
doAssert( parse_tnetstring( "8:1:1#1:2#]" ).kind == TNetstringArray )
# Iteration (both array and tuple)
#
var
keys: array[ 2, string ]
vals: array[ 4, string ]
k_idx = 0
idx = 0
for key, val in parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" ):
keys[ idx ] = key
idx = idx + 1
for item in val:
vals[ k_idx ] = item.str
k_idx = k_idx + 1
doAssert( keys == ["hi","there"] )
doassert( vals == ["a","b","c","d"] )
# Deep copies
#
var original = parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" )
var copied = original.copy
doAssert( original == copied )
doAssert( original.repr != copied.repr )
doAssert( original.fields.pop.val.elems.pop.repr != copied.fields.pop.val.elems.pop.repr )
# Key deletion
#
var tnet_obj = parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" )
tnet_obj.delete( "hi" )
doAssert( tnet_obj.fields.len == 1 )
# Hashing
#
doAssert( tnet_int.hash == 1.hash )
doAssert( parse_tnetstring( "4:true!" ).hash == hash( true.int ) )
# Length checks.
#
tnet_obj = parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" )
doAssert( parse_tnetstring( "0:~" ).len == 0 )
doAssert( tnet_obj.len == 2 )
doAssert( parse_tnetstring( "8:1:1#1:2#]" ).len == 2 )
doAssert( parse_tnetstring( "5:hallo," ).len == 5 )
# Index accessors
#
tnet_obj = parse_tnetstring( "20:1:1#1:2#1:3#1:4#1:5#]" )
doAssert( tnet_obj[ 2 ].num == 3 )
# Key accessors
#
tnet_obj = parse_tnetstring( "11:2:hi,3:yep,}" )
doAssert( $tnet_obj["hi"] == "yep" )
doAssert( tnet_obj.has_key( "hi" ) == true )
doAssert( tnet_obj.has_key( "nope-not-here" ) == false )
# Adding elements to an existing TNetstring array
#
var tnet_array = newTNetstringArray()
for i in 1 .. 10:
tnet_obj = newTNetstringInt( i )
tnet_array.add( tnet_obj )
tnet_array[ 6 ] = newTNetstringString( "yep" )
doAssert( tnet_array.len == 10 )
doAssert( tnet_array[ 4 ].num == 5 )
doAssert( tnet_array[ 6 ].str == "yep" )
# Adding pairs to an existing TNetstring aobject.
#
tnet_obj = newTNetstringObject()
tnet_obj.add( "yo", newTNetstringInt(1) )
tnet_obj.add( "yep", newTNetstringInt(2) )
doAssert( tnet_obj["yo"].num == 1 )
doAssert( tnet_obj["yep"].num == 2 )
doAssert( tnet_obj.len == 2 )
tnet_obj[ "more" ] = newTNetstringInt(1)
tnet_obj[ "yo" ] = newTNetstringInt(1) # dup check
doAssert( tnet_obj.len == 3 )
# Serialization.
#
var tstr = "308:9:givenName,6:Mahlon,16:departmentNumber,22:Information Technology," &
"5:title,19:Senior Technologist,13:accountConfig,48:7:vmemail,4:true!7:allpage," &
"5:false!7:galhide,0:~}13:homeDirectory,14:/home/m/mahlon,3:uid,6:mahlon,9:yubi" &
"KeyId,12:vvidhghkhehj,5:gecos,12:Mahlon Smith,2:sn,5:Smith,14:employeeNumber,5:12921#}"
tnet_obj = parse_tnetstring( tstr )
doAssert( tstr == tnet_obj.dump_tnetstring )
# Value fetching methods
#
var tnet_null = newTNetstringNull()
tnet_obj = newTNetstringString( "Hello." )
doAssert( tnet_obj.getStr == "Hello." )
doAssert( tnet_null.getStr("nope") == "nope" )
doAssert( tnet_null.getStr == "" )
tnet_obj = newTNetstringInt( 42 )
doAssert( tnet_obj.getInt == 42 )
doAssert( tnet_null.getInt == 0 )
doAssert( tnet_null.getInt(1) == 1 )
tnet_obj = newTNetstringFloat( 1.0 )
doAssert( tnet_obj.getFloat == 1.0 )
doAssert( tnet_null.getFloat == 0 )
doAssert( tnet_null.getFloat(0.1) == 0.1 )
tnet_obj = newTNetstringObject()
tnet_obj[ "yay" ] = newTNetstringInt( 1 )
doAssert( tnet_obj.getFields[0].val == newTNetstringInt(1) )
doAssert( tnet_null.getFields.len == 0 )
tnet_obj = newTNetstringArray()
tnet_obj.add( newTNetstringInt(1) )
doAssert( tnet_obj.getElems[0] == newTNetstringInt(1) )
doAssert( tnet_null.getElems.len == 0 )
echo "* Tests passed!"
while true and defined( testing ):
when isMainModule and defined( testing ):
while true:
for line in readline( stdin ).split_lines:
let input = line.strip
try:

12
tests/parseErrors/t_hash_with_nonstring_key.nim Normal file
View file

@ -0,0 +1,12 @@
# vim: set et sta sw=4 ts=4 :
import
std/re
import tnetstring
try:
discard parse_tnetstring( "8:1:1#1:1#}" )
except TNetstringParseError as err:
assert err.msg.contains( re"""Invalid data: Object keys must be strings.""" )

12
tests/parseErrors/t_hash_with_unbalanced_pairs.nim Normal file
View file

@ -0,0 +1,12 @@
# vim: set et sta sw=4 ts=4 :
import
std/re
import tnetstring
try:
discard parse_tnetstring( "8:1:1#1:1#}" )
except TNetstringParseError as err:
assert err.msg.contains( re"""Invalid data: Object keys must be strings.""" )

12
tests/parseErrors/t_invalid_data.nim Normal file
View file

@ -0,0 +1,12 @@
# vim: set et sta sw=4 ts=4 :
import
std/re
import tnetstring
try:
discard parse_tnetstring( "totally invalid" )
except TNetstringParseError as err:
assert err.msg.contains( re"""Invalid data: No separator token found""" )

12
tests/parseErrors/t_invalid_length.nim Normal file
View file

@ -0,0 +1,12 @@
# vim: set et sta sw=4 ts=4 :
import
std/re
import tnetstring
try:
discard parse_tnetstring( "what:ever" )
except TNetstringParseError as err:
assert err.msg.contains( re"""invalid integer: what""" )

12
tests/parseErrors/t_length_too_long.nim Normal file
View file

@ -0,0 +1,12 @@
# vim: set et sta sw=4 ts=4 :
import
std/re
import tnetstring
try:
discard parse_tnetstring( "1000000000:1" )
except TNetstringParseError as err:
assert err.msg.contains( re"""Invalid data: Size more than 9 digits.""" )

12
tests/parseErrors/t_null_with_nonzero_length.nim Normal file
View file

@ -0,0 +1,12 @@
# vim: set et sta sw=4 ts=4 :
import
std/re
import tnetstring
try:
discard parse_tnetstring( "3:yep~" )
except TNetstringParseError as err:
assert err.msg.contains( re"""Invalid data: Payload.*0 length for null""" )

12
tests/parseErrors/t_unknown_symbol.nim Normal file
View file

@ -0,0 +1,12 @@
# vim: set et sta sw=4 ts=4 :
import
std/re
import tnetstring
try:
discard parse_tnetstring( "2:25*" )
except TNetstringParseError as err:
assert err.msg.contains( re"""Invalid data: Unknown tnetstring type '\*'.""" )

23
tests/parser/t_equality_checks.nim Normal file
View file

@ -0,0 +1,23 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_int = parse_tnetstring( "1:1#" )
# equal to itself
assert tnet_int == tnet_int
# equal to another object
assert tnet_int == parse_tnetstring( "1:1#" )
# type equalities
assert parse_tnetstring( "0:~" ) == parse_tnetstring( "0:~" )
assert parse_tnetstring( "3:hi!," ) == parse_tnetstring( "3:hi!," )
assert parse_tnetstring( "3:100#" ) == parse_tnetstring( "3:100#" )
assert parse_tnetstring( "3:1.1^" ) == parse_tnetstring( "3:1.1^" )
assert parse_tnetstring( "4:true!" ) == parse_tnetstring( "4:true!" )
assert parse_tnetstring( "4:true!" ) != parse_tnetstring( "5:false!" )
assert parse_tnetstring( "8:1:1#1:2#]" ) == parse_tnetstring( "8:1:1#1:2#]" )
assert parse_tnetstring( "8:1:1#1:2#]" ) != parse_tnetstring( "8:1:1#1:1#]" )
assert parse_tnetstring( "21:2:hi,1:1#5:there,1:2#}" ) == parse_tnetstring( "21:2:hi,1:1#5:there,1:2#}" )

14
tests/parser/t_type_detection.nim Normal file
View file

@ -0,0 +1,14 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_int = parse_tnetstring( "1:1#" )
assert tnet_int.kind == TNetstringInt
assert parse_tnetstring( "1:a," ).kind == TNetstringString
assert parse_tnetstring( "3:1.0^" ).kind == TNetstringFloat
assert parse_tnetstring( "5:false!" ).kind == TNetstringBool
assert parse_tnetstring( "0:~" ).kind == TNetstringNull
assert parse_tnetstring( "9:2:hi,1:1#}" ).kind == TNetstringObject
assert parse_tnetstring( "8:1:1#1:2#]" ).kind == TNetstringArray

15
tests/tnetobjects/t_can_add_array_elements.nim Normal file
View file

@ -0,0 +1,15 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_array = newTNetstringArray()
for i in 1 .. 10:
let tnet_obj = newTNetstringInt( i )
tnet_array.add( tnet_obj )
tnet_array[ 6 ] = newTNetstringString( "yep" )
assert tnet_array.len == 10
assert tnet_array[ 4 ].num == 5
assert tnet_array[ 6 ].str == "yep"

13
tests/tnetobjects/t_can_be_hashed.nim Normal file
View file

@ -0,0 +1,13 @@
# vim: set et sta sw=4 ts=4 :
import
std/hashes
import tnetstring
# Hashes to the underlying object type.
let tnet_int = parse_tnetstring( "1:1#" )
assert tnet_int.hash == 1.hash
assert parse_tnetstring( "4:true!" ).hash == hash( true.int )

43
tests/tnetobjects/t_can_be_serialized.nim Normal file
View file

@ -0,0 +1,43 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tstr = "308:9:givenName,6:Mahlon,16:departmentNumber,22:Information Technology," &
"5:title,19:Senior Technologist,13:accountConfig,48:7:vmemail,4:true!7:allpage," &
"5:false!7:galhide,0:~}13:homeDirectory,14:/home/m/mahlon,3:uid,6:mahlon,9:yubi" &
"KeyId,12:vvidhghkhehj,5:gecos,12:Mahlon Smith,2:sn,5:Smith,14:employeeNumber,5:12921#}"
let tnet_null = newTNetstringNull()
var tnet_obj = parse_tnetstring( tstr )
# full round trip
assert tstr == tnet_obj.dump_tnetstring
# objects and their defaults
tnet_obj = newTNetstringString( "Hello." )
assert tnet_obj.getStr == "Hello."
assert tnet_null.getStr("nope") == "nope"
assert tnet_null.getStr == ""
tnet_obj = newTNetstringInt( 42 )
assert tnet_obj.getInt == 42
assert tnet_null.getInt == 0
assert tnet_null.getInt(1) == 1
tnet_obj = newTNetstringFloat( 1.0 )
assert tnet_obj.getFloat == 1.0
assert tnet_null.getFloat == 0
assert tnet_null.getFloat(0.1) == 0.1
tnet_obj = newTNetstringObject()
tnet_obj[ "yay" ] = newTNetstringInt( 1 )
assert tnet_obj.getFields[0].val == newTNetstringInt(1)
assert tnet_null.getFields.len == 0
tnet_obj = newTNetstringArray()
tnet_obj.add( newTNetstringInt(1) )
assert tnet_obj.getElems[0] == newTNetstringInt(1)
assert tnet_null.getElems.len == 0

17
tests/tnetobjects/t_can_paired_elements.nim Normal file
View file

@ -0,0 +1,17 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_obj = newTNetstringObject()
tnet_obj.add( "yo", newTNetstringInt(1) )
tnet_obj.add( "yep", newTNetstringInt(2) )
assert tnet_obj["yo"].num == 1
assert tnet_obj["yep"].num == 2
assert tnet_obj.len == 2
tnet_obj[ "more" ] = newTNetstringInt(1)
tnet_obj[ "yo" ] = newTNetstringInt(1) # dup check
assert tnet_obj.len == 3

13
tests/tnetobjects/t_deep_copies.nim Normal file
View file

@ -0,0 +1,13 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let original = parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" )
let copied = original.copy
# Same values
assert copied == original
# Different instances
assert cast[pointer](original) != cast[pointer](copied)

7
tests/tnetobjects/t_index_slicing.nim Normal file
View file

@ -0,0 +1,7 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_obj = parse_tnetstring( "20:1:1#1:2#1:3#1:4#1:5#]" )
assert tnet_obj[ 2 ].num == 3

18
tests/tnetobjects/t_iterators.nim Normal file
View file

@ -0,0 +1,18 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
var
keys: seq[ string ]
vals: seq[ string ]
for key, val in parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" ):
keys.add( key )
for item in val:
vals.add( item.str )
assert keys == @["hi","there"]
assert vals == @["a","b","c","d"]

10
tests/tnetobjects/t_key_accessors.nim Normal file
View file

@ -0,0 +1,10 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_obj = parse_tnetstring( "11:2:hi,3:yep,}" )
assert $tnet_obj["hi"] == "yep"
assert tnet_obj.has_key( "hi" ) == true
assert tnet_obj.has_key( "nope-not-here" ) == false

10
tests/tnetobjects/t_key_deletion.nim Normal file
View file

@ -0,0 +1,10 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_obj = parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" )
assert tnet_obj.fields.len == 2
tnet_obj.delete( "hi" )
assert tnet_obj.fields.len == 1

11
tests/tnetobjects/t_length_checks.nim Normal file
View file

@ -0,0 +1,11 @@
# vim: set et sta sw=4 ts=4 :
import tnetstring
let tnet_obj = parse_tnetstring( "35:2:hi,8:1:a,1:b,]5:there,8:1:c,1:d,]}" )
assert parse_tnetstring( "0:~" ).len == 0
assert tnet_obj.len == 2
assert parse_tnetstring( "8:1:1#1:2#]" ).len == 2
assert parse_tnetstring( "5:hallo," ).len == 5