Next steps

add SPDX license identifier to each ubx module

http://www.spdx.org/spdx-license-list

[#A] overhaul ubx_reflection layer

switch to handle ubx_data.len via cdata
exhaustive unit tests

improve type safety of gen_[read/write] generated functions

/* assert(strcmp(#typename, port->out_type_name)==0); */

This doesn’t work because of type aliasing. Solution is to compare ubx_type_t ptrs. This ultimately requries access to node_info. Could be via: port->block-node_info.

blockdiagram: figure out how to add behavior aka apps

tools: add ubx_info <module>

pretty print contents of a module

create rotating zero copy ptr example

consider kdbus’ memfd’s

add support for enum types

add function ffi_basic_type_is_loaded(t)

such that reloading a char[50] will go ok too.

untyped samples in fifos can create havoc

luablock: wrote resposne to exec_str even though nothing had been sent (bug!) next read produced the int that was executed (and failed obviously!).

add proper logging

either:

add new ERR3 that also prints node and block name
add log function that writes messages via log ports

component identification (from string to uid?)

replace string as primary key by numeric ubx_block_id_t ? Advantage: would be easier to use for configuring trigger blocks, etc. -> easier for communicating block_id’s via ports.

mqueue: add real test.

mqueue safety: if type name given, append md5sum to mqueue name

[#C] find workaround for gcc issue 10676

[#C] webif: ubx_load_types is called upon every request.

Find a way to only call if a new type was added.

[#C] think about “fast data”

where ubx_data_t->data points to it’s own addr+1. This means the whole ubx_data_t is continuous in memory and could be sent out without copying.

[#A] blockdiagram: support for hierarchical composition

node_graph: add trigger links

usability: add Lua OO methods to node, block, port

just make it more convenient

logger: allow connection to interaction

introduce typeid’s: (like git commitish)

full name: package/struct foo or package/foo_t or
partial hash of name: 0a407a4f51ff2bb1f92a6ae611cb63fb

add md5 hash support

embeddeding microblx: prepare C only example

cleanup modules utils, ubx_utils and ubx

[#A] ubx_gen_block: add support for ports, configs, types

logger: add ports to block IF to improve viz

store seqid in node specific structure

Otherwise, loading the same type in multiple nodes will foobar maintaining the sequence. -> registered types are global!!!

test to confirm that attrs=IN_PORT etc, can be omitted

-> instead we only rely on [in|out]_type_name

file_rep: output comma separated list

rename reporter->logging

data_tolua: start table converted from arrays with index 1

CANCELED locking

coordinate such that no locking is required.
Options
- rw-locks (or uRCU?)
Required locking
- access to node_info
- access to blocks
node_info locking rules
- writer lock
  - ubx_node_cleanup
  - ubx_block_[un]register
  - ubx_type_[un]register
- reader lock
  - ubx_num_blocks, ubx_num_types
block locking rules
- writer lock
  1. ubx_node_[init|cleanup|start|stop|
  2. set ubx_config_data (TODO!)
  3. ubx_connect[_one] (writer lock on (each) involved cblock)
- reader lock
  1. stepping a block ubx_node_step (this has to use writer

lock, or would two simultaneous steps be allowed?)

reading configuration ubx_config_get[_data]

interactions must handle their own locking.

[#A] test communicating struct types

[#A] add methods [add|rm}_[port|config]

add len field for configs and ports to simplify searching.
unify cloning and dynamically adding ports
- ubx_port_add(b, name, meta, in_type_name, out_type_name, attrs)
  - resize target buffer

[#A] make an example of how to use C++

[#A] develop a generic luajit based block

test compilation with g++ and clang++ (sigh)

#ifdef __cplusplus

#else

#endif

#ifdef __cplusplus

#else

#endif

above wasn’t necessary, but only clang++ works for now because gcc doesn’t support non-trivial designated initializers.

rename ubx -> microblx

Test a minimal example with arrays of basic types

ubx.data_tolua: deal with ubx_data_t multiplicity (len)!

and detect and pretty print strings

implement real cdata reflection on top of reflect.lua

cdata_to_tab / cdata_from_tab
implement logging component (first generic luajit block)

remove BLOCK_TYPE_TRIGGER (same as COMPUTATION)

webif: add step_once button (will do start()->step()->stop() cycle)

How to deal with variable sized configuration and port input

e.g zero to many block names to be triggered by ptrig.
e.g. trig_conf: resize in resize in data_set?
maybe have two version: one that resizes and one that doesn’t.

unit tests

load a configuration

introduce ubx.unload

one three lists for block prototypes and one for instances

rationale: users shall choose sane names for their application blocks.

latest problem

type: charctstrchar* type: random/struct random_configctstrstruct random___random* /usr/bin/luajit: ./lua/ubx.lua:267: undeclared or implicit tag ‘random___random’ stack traceback: [C]: in function ‘type_to_ctype’ ./lua/ubx.lua:267: in function ‘data_to_cdata’ ./lua/ubx.lua:276: in function ‘set_config’ ./rnd_to_hexdump.lua:34: in main chunk [C]: at 0x00404ca0

Problem is that struct name parsing stops at ‘_’ !! Add Unit tests!

implement a buffered interaction

and test by writing data from the lua shell

implement a nice high level lua library.

extend the webserver with luajit support.

Test a minimal example with basic types

Fix leak upon failure: e.g. in alloc

if realloc fails the original block is untouched and NULL is returned. Then we need to “unroll”.

add functions to change life-cycle state and check that the FSM is respected.

Implement ubx_type_register/unregister

resolve types

-> in ubx_resolve_types: need to check whether port has namein or outport is

test hexdump interaction with variable types

add namespace to struct type’s string spec and load into ffi

Usefull stuff:

check exported symbols:

$ nm -C -D file.so

valgrind

supressing false positive in luajit luajit ML gmane
valgrind --leak-check=full --track-origins=yes luajit rnd_to_hexdump.lua 2>&1 | less

P99 - Preprocessor macros and functions for C99

uthash

libmowgli-2

Lock-free and interprocess libs

liblfds the lock-free data structure library

http://concurrencykit.org/

jsmn ANSI C json parser with permissive mode

Lua jit Application Programming Helper Libraries (github)

gcc plugin for luajit-ffi http://colberg.org/gcc-lua-cdecl/

javascript graph drawing

https://github.com/cpettitt/dagre
https://github.com/cpettitt/dagre-d3
http://d3js.org/
http://sigmajs.org/
http://www.graphdracula.net/
https://github.com/anvaka/VivaGraphJS
http://js-graph-it.sourceforge.net/index.html (nice!)
http://jsplumbtoolkit.com/doc/home (allows editing, flowcharts, FSM, but not layout :-( )
http://jointjs.com/demos/fsa
http://labs.unwieldy.net/moowheel/
http://cytoscapeweb.cytoscape.org/

MD5 and SHA hashes

https://github.com/andresy/lua—md5

http://stackoverflow.com/questions/11167713/pure-lua-hashing-ripemd160-or-sha2/15417980#15417980

HDF5 stuff

https://github.com/jzrake/lua-hdf5

http://lttng.org/urcu

Focus

only in-out ports (maybe instead of multi-valued ports it’s better to solve this at the type level, e.g. define a composite type instead. -> I really think so!)
dealing with C-struct types (later: automatic conversion to hdf5 and rosmsg)
separate definition and instance.

Milestones

[ ] Launch the random component stdalone and test it from the lua cmdline: configure seed, write, step, read.
[ ] Connect two components with an interaction and exchange data
[ ] Build a more complex topology

Important Links

http://gcc.gnu.org/onlinedocs/cpp/Macros.html
http://luajit.org/ext_ffi.html
http://www.zeromq.org/intro:start
https://live.gnome.org/GObjectIntrospection/
http://www.isotton.com/devel/docs/C++-dlopen-mini-HOWTO/C++-dlopen-mini-HOWTO.html
Using C++ components must be possible. Should be no problem if interface functions are defined using extern “C” {}.
ffi reflection
- http://www.corsix.org/lua/reflect/api.html
- http://www.corsix.org/lua/reflect/reflect.lua

Requirements

Block model: in, in-event/out ports
a block must have life-cycle.
Meta-data: used to define constraints on blocks, periodicity, etc. JSON? or pure lua
Ports: in/outs (correspond to in-args and out-args + retval)
Composition of blocks. different methods possible:
- using functional programming
- specifying all connections. this connections-spec can then be compiled into one single new function block or just instantiated.
Pure C and Lua. Light, embeddable.
Dynamic creation of interfaces: ie. dynamic creation of youbot arms.
- dynamically adding ports vs. dynamically instatiation subcomponents. For the youbot subcomponents would work nicely. But if you want to handle an unkown amount of identical devices (minor#) the dynamical version is better. Thread safety, no statics!

Example use cases that must be nicely satisfied

youbot driver: autodetection of arms
local function calls: i.e. how to plug services
adding support for nasty C++ types.

Interaction model: defines what happens on read-write to a port, i.e. buffering, rendevouz, sending via network. See also Ptolomy.

Elements

Should we separate between types and instances: ComponentDef vs. ComponentInst? Probably yes!

Components:

define:

set of typed in and out ports
configuration
activity

issues:

thread safety: instances must not share mutable data!

interface representation

declarative yaml vs. procedural C interface. -> both necessary, even if the former should be preferred normally.
Should modify data in-place. The system will make the copy by default. That makes it easy to switch to zero copy. But the flow of data must be represented in the meta-data (two options: inport->outport tag or bidirectional port.)

Ports

Bidirectional ports are useful for properties that can be read or written. Possible to “disable”, e.g. writing/reading will cause an error. Or should this be in the interaction? -> no, whether a parameter can be changed at runtime or not depends on the block

Port states: PORT_DISBALED | PORT_ENABLED
No OldData! Old is a too fuzzy concept, and causes a lot of problems, such as ancient data lingering and causing extreme motions etc.
The OldData can be realized by an interaction which returns a piece of data on read while it can be considered new.

Triggering

Distinguish between triggered and stepped? I.e. a component must be triggered by the availability of data before it can be stepped.

Trigger specification language?

trigger{(p1:new or p2:new) and p3:data}

Components could define is_triggered C function: If not available assumes that is always triggered.

new: new data available data: old or new, but not none dontcare: whatever

Maybe triggering should be an additional debugging layer.

Open issue Passive vs. active components:

should comm comps always be passive?
How to realize “pull” semantics, i.e. have a read trigger the generation of data.
a) via a pull communication comp: use the computational components read to trigger a producer to generate data that can be returned to the read callee.

Interactions

this is a special component that implements read and write and that can define ports itself to represent different information. e.g. statistics, errors, etc.

communication like interactions:
- dataport: just store one sample, no locks.
- buffer: store multiple.
- multiplexer: one in- multiple
control oriented interactions:
- may block the writer/reader, ie. CSP alike rendevouz:

Can all locking be contained in interactions? E.g. multiplexer:
Danger: calling read/write on a port not connected to an interaction will call a segfault. Solutions: Always attach a dummy interaction, or use a port_write(port, data) function that checks instead of doing port->write yourself.

Use cases:
- Connect one-to-one
- Connect one-to-many
- Connect many-to-one

(Where are locks needed?)

For connecting and disconnecting ports with interactions. Possibly this function pointer setting can be done using atomic ops.

Buffering and zero copy semantics

One-to-one:

c1.a ->[i]-> c2.b

write(): interaction provided write is called and data stored in interaction buffer.
read(): interaction provided read is called and returns the data.
in this case the interaction requires no activity itself! But for a remote interaction (ZMQ) there might be a thread allocated for sending out data.
Copy semantics:
1. With copying: c1 has it’s own copy of the data. When it writes

to port ‘a’, the interaction [i] makes a copy. c2.b again has it’s own copy => two copies

The c2 attaches its buffer to the read-port. When c1 writes,

the interaction directly stores the data into the c2’s read buffer.

Zero copying:

Rule: Writing means releasing data. Could check this with reference count (ie. it is an error if refcnt is != 0 on write). Thus, buffer interactions only store data-objects (pointers to data).

Collect when refcnt goes 0.

How to support both?
1. DIY version of RTT
2. v2 if possible

One-to-many:

c1.a -> [i] -> c2.b -> c3.c

write as above. read must either a) lock b)

Function calls on Function Blocks

fb { pin i1, i2; pout out1; pout out2; pout out3; }

call{name=”foo”, in={i1,i2}, out={out1&out2&out3}}

foo(1,2} -> <out1>, <out2>, <out3>

Use cases for this

pluggable functions: i.e. itasc solver
causing side-effects, ie. print_this
Making this explicit adds structure, but its not a fundamental requirement. All you need is the ability to drop in a custom fb into an existing composition, i.e. a parametrizable composite.

A C representation of a call is necessary! Plugin modules!

Type (only fixed size)

universally unique and human readable ID (or better hash struct def?)
variable sized data: e.g. a json message.
ffi spec. should this be optional or not?
attributes: fixed size/variable size
serialization
- serialize/deserialize functions
- type: boost serial, GooglePB, …
- autoserialize using ffi spec info?!

Value representation

type
attributes: VARIABLE_SZ
serialization type: STRUCT | CUSTOM |
void data*

Compilation

It must be possible to compile two or more blocks, their connections and a schedule into a new block, that exposes a specified subset of the interface.

Big questions

How to permit with dynamic block interface creating

Requirement: the block interface needs to be created (extended) dynamically at block initialization time. To do this, access to configuration may be required.

Options:

Multi-pass configuration In [preinit] set configuration, in init hook create IF. If additional configuration is added, then their values can only be check in start.
Add an extra state
PREINIT -[init]-> INITIALIZED -[configure]-> INACTIVE -[start]-> ACTIVE -[stop]-> INACTIVE -[deconfigure] -> INITIALIZED -[cleanup]->preinit
- “configured” or maybe better “create_if”

Types

local tm = ffi.cast('TimeMsg*', tm_rtt:tolud())

Types safety must be guaranteed. Hash types in some way. I.e. sha256 the struct def?
To which extent can we avoid boxing and explicit serialization. I think the latter is mandatory for non-trivial structs. We must also be able to support protocol buffers, boost serialization etc.
Options:
Constrain to structs? C++ Objects can be mapped to structs (potentially automatically) but that may be non-intuitve. Ok for first go.
Support full type serialization. Necessary eventually. But serialization should only take place when necessary, e.g. upon leaving a process boundary.
Requirements
- types must be uniquely identified throughout a (distributed system). That can be the name or some hash calculated from the struct definition, etc.
- types must be registered with Lua such it knows how to interpret these. Probably there will be several classes:
  1. plain structs (easy using ffi)
  2. protocol buffers
  3. ROS types
  4. luabind
  5. …