olsrd-extension to spread database trough the net

Hi,

a while ago I created a extension for olsrd, which now seems kinda
stable. If you find it suitable and useful, you can add it to the main
olsrd repository.

About the extension (actually these are two related extensions, one
which can only send data and one which is also able to receive data):

The extension spreads some database-values through the net, taken from a
sqlite-database or a textfile, which contains json-formatted tables (one
per line).

What do you think about it?

Nice regards, Rene

I wonder if this really should be part of the routing agent...

what is the primary reason for not making it a daemon of its own?

Henning RoggeOn Tue, Oct 30, 2012 at 6:36 PM, rene  wrote:
> Hi,
>
> a while ago I created a extension for olsrd, which now seems kinda
> stable. If you find it suitable and useful, you can add it to the main
> olsrd repository.
>
> About the extension (actually these are two related extensions, one
> which can only send data and one which is also able to receive data):
>
> The extension spreads some database-values through the net, taken from a
> sqlite-database or a textfile, which contains json-formatted tables (one
> per line).
>
> What do you think about it?
>
> Nice regards, Rene
>
> --
> Olsr-dev mailing list
> 
> https://lists.olsr.org/mailman/listinfo/olsr-...-- 
Steven Hawkings about cosmic inflation: "An increase of billions of
billions of percent in a tiny fraction of a second. Of course, that
was before the present government."

On 30-10-12 20:27, Henning Rogge wrote:
> I wonder if this really should be part of the routing agent...
>
> what is the primary reason for not making it a daemon of its own?exactly ;-)>
> Henning Rogge
>
> On Tue, Oct 30, 2012 at 6:36 PM, rene  wrote:
>> Hi,
>>
>> a while ago I created a extension for olsrd, which now seems kinda
>> stable. If you find it suitable and useful, you can add it to the main
>> olsrd repository.
>>
>> About the extension (actually these are two related extensions, one
>> which can only send data and one which is also able to receive data):
>>
>> The extension spreads some database-values through the net, taken from a
>> sqlite-database or a textfile, which contains json-formatted tables (one
>> per line).
>>
>> What do you think about it?
>>
>> Nice regards, Rene
>>
>> --
>> Olsr-dev mailing list
>> 
>> https://lists.olsr.org/mailman/listinfo/olsr-...
>
>
>-- 
Ferry Huberts

> I wonder if this really should be part of the routing agent...
> 
> what is the primary reason for not making it a daemon of its own?example for our usecase:
we have a user database (e.g. login-credentials or special properties
for some mac-addresses). ofcourse you can centralize that and let
the routers talk to each other if needed (e.g. via https), but much faster
is a local query, which is possible, if you there is a way to have a recent
copy of your central database.

how can this be solved with a daemon? multicast maybe
next step can be something like a "HNA4" ("message") announcement,
which every router can update without restarting the daemon 8-)

bye, bastian

On Tue, Oct 30, 2012 at 9:05 PM, Bastian Bittorf  wrote:
> example for our usecase:I can easily see an usecase... I just want to talk about if this
should be within the routing protocol or not.> we have a user database (e.g. login-credentials or special properties
> for some mac-addresses). ofcourse you can centralize that and let
> the routers talk to each other if needed (e.g. via https), but much faster
> is a local query, which is possible, if you there is a way to have a recent
> copy of your central database.

> how can this be solved with a daemon? multicast maybe
> next step can be something like a "HNA4" ("message") announcement,
> which every router can update without restarting the daemon 8-)OLSR does only multicast between the neighbors too.

Henning Rogge-- 
Steven Hawkings about cosmic inflation: "An increase of billions of
billions of percent in a tiny fraction of a second. Of course, that
was before the present government."

Op 30 okt. 2012, om 21:12 heeft Henning Rogge het volgende geschreven:> On Tue, Oct 30, 2012 at 9:05 PM, Bastian Bittorf  wrote:
>> example for our usecase:
> 
> I can easily see an usecase... I just want to talk about if this
> should be within the routing protocol or not.
> 
>> we have a user database (e.g. login-credentials or special properties
>> for some mac-addresses). ofcourse you can centralize that and let
>> the routers talk to each other if needed (e.g. via https), but much faster
>> is a local query, which is possible, if you there is a way to have a recent
>> copy of your central database.
> 
>> how can this be solved with a daemon? multicast maybe
>> next step can be something like a "HNA4" ("message") announcement,
>> which every router can update without restarting the daemon 8-)
> 
> OLSR does only multicast between the neighbors too.No. OLSR provides a transport of messages, that should be flooded in an efficient way. Many routing protocols do support such. I say: all grown up routing protocols do (link state, BGP is another example).

Maybe an API for message distribution could be the way forward. This is more efficient than multicast (like BMF).
mdns / p2pd are examples. pud is another.
Has the packetbb iplementation such an API, so messages from another daemon are merged with hello / tc messages in same IP packet?

BTW, we need to fix mpr flooding some day (in v2 is fine).

Teco> 
> Henning Rogge
> 
> -- 
> Steven Hawkings about cosmic inflation: "An increase of billions of
> billions of percent in a tiny fraction of a second. Of course, that
> was before the present government."
> 
> -- 
> Olsr-dev mailing list
> 
> https://lists.olsr.org/mailman/listinfo/olsr-...

On 10/30/2012 09:54 PM, Teco Boot wrote:
> Maybe an API for message distribution could be the way forward. This
> is more efficient than multicast (like BMF). mdns / p2pd are
> examples. pud is another.Why is this more efficient than multicast?> Has the packetbb iplementation such an API, so messages from another
> daemon are merged with hello / tc messages in same IP packet?
>
> BTW, we need to fix mpr flooding some day (in v2 is fine).Yes. We will!

Henning Rogge

Op 31 okt. 2012, om 07:49 heeft Henning Rogge het volgende geschreven:> On 10/30/2012 09:54 PM, Teco Boot wrote:
>> Maybe an API for message distribution could be the way forward. This
>> is more efficient than multicast (like BMF). mdns / p2pd are
>> examples. pud is another.
> 
> Why is this more efficient than multicast?- message aggregation into single packet, so less headers
 - jitter, reduction of collisions

Currently we don't have selective multicast forwarding. BMF is broadcasting.
Pseudo-broadcast to few neighbors with high rate unicast would have better performance and reliability than flooding.
Message flooding is more efficient than multicast in the mentioned examples. Maybe not always.

I would like integration of mdns, p2pd and bmf in a single plugin, where a packet filter engine can direct packet processing, e.g. BMF style *or* distribution with messages. Could / should be a separate deamon, as discussed some time ago. Something for after v2.

Teco

On 10/31/2012 08:20 AM, Teco Boot wrote:
>
> Op 31 okt. 2012, om 07:49 heeft Henning Rogge het volgende
> geschreven:
>
>> On 10/30/2012 09:54 PM, Teco Boot wrote:
>>> Maybe an API for message distribution could be the way forward.
>>> This is more efficient than multicast (like BMF). mdns / p2pd
>>> are examples. pud is another.
>>
>> Why is this more efficient than multicast?
>
> - message aggregation into single packet, so less headers - jitter,
> reduction of collisions
>
> Currently we don't have selective multicast forwarding. BMF is
> broadcasting. Pseudo-broadcast to few neighbors with high rate
> unicast would have better performance and reliability than flooding.
> Message flooding is more efficient than multicast in the mentioned
> examples. Maybe not always.
>
> I would like integration of mdns, p2pd and bmf in a single plugin,
> where a packet filter engine can direct packet processing, e.g. BMF
> style *or* distribution with messages. Could / should be a separate
> deamon, as discussed some time ago. Something for after v2.So the short term solution would be plugins for OLSRd, but the long term
solution would be to have a multicast support that allows configurable
aggregation/jitter per multicast group ?

I wonder if we could use the "openvswitch" kernel module for a more 
efficient multicast implementation.

Henning Rogge

Op 31 okt. 2012, om 08:54 heeft Henning Rogge het volgende geschreven:> On 10/31/2012 08:20 AM, Teco Boot wrote:
>> 
>> Op 31 okt. 2012, om 07:49 heeft Henning Rogge het volgende
>> geschreven:
>> 
>>> On 10/30/2012 09:54 PM, Teco Boot wrote:
>>>> Maybe an API for message distribution could be the way forward.
>>>> This is more efficient than multicast (like BMF). mdns / p2pd
>>>> are examples. pud is another.
>>> 
>>> Why is this more efficient than multicast?
>> 
>> - message aggregation into single packet, so less headers - jitter,
>> reduction of collisions
>> 
>> Currently we don't have selective multicast forwarding. BMF is
>> broadcasting. Pseudo-broadcast to few neighbors with high rate
>> unicast would have better performance and reliability than flooding.
>> Message flooding is more efficient than multicast in the mentioned
>> examples. Maybe not always.
>> 
>> I would like integration of mdns, p2pd and bmf in a single plugin,
>> where a packet filter engine can direct packet processing, e.g. BMF
>> style *or* distribution with messages. Could / should be a separate
>> deamon, as discussed some time ago. Something for after v2.
> 
> So the short term solution would be plugins for OLSRd, but the long term
> solution would be to have a multicast support that allows configurable
> aggregation/jitter per multicast group ?Sounds good.
We have to take a look to SMF also. And maybe to separate distribution trees, if standards for such become available.> 
> I wonder if we could use the "openvswitch" kernel module for a more efficient multicast implementation.Me too :-) 
Or whatever what is available. The mcastd or mcast kernel module would be compatible with other protocols.

Teco> 
> Henning Rogge
> 
> 
> -- 
> Diplom-Informatiker Henning Rogge , Fraunhofer-Institut für
> Kommunikation, Informationsverarbeitung und Ergonomie FKIE
> Kommunikationssysteme (KOM)
> Fraunhofer Straße 20, 53343 Wachtberg, Germany
> Telefon +49 228 9435-961,   Fax +49 228 9435 685
>  http://www.fkie.fraunhofer.de
>

On 10/31/2012 11:05 AM, Teco Boot wrote:
> Op 31 okt. 2012, om 08:54 heeft Henning Rogge het volgende
> geschreven:
>> So the short term solution would be plugins for OLSRd, but the long
>> term solution would be to have a multicast support that allows
>> configurable aggregation/jitter per multicast group ?

> Sounds good. We have to take a look to SMF also. And maybe to
> separate distribution trees, if standards for such become available.I think BMF is very similar to SMF.>> I wonder if we could use the "openvswitch" kernel module for a more
>> efficient multicast implementation.

> Me too :-) Or whatever what is available. The mcastd or mcast kernel
> module would be compatible with other protocols.Most kernel-multicast solutions are not useful for Manet, because we do 
not have this clear "incoming/outgoing" interface thing. On a MANET its 
perfectly fine to forward multicast out of the same interface your got 
it from.

The OpenVSwitch kernel module (without the whole daemon) looks like a 
way to add custom multicast routes with MAC filters into the kernel (and 
maybe even MPLS support later! *G*).

Henning Rogge

I wrote a little plugin for relaying messages from other daemons.
https://github.com/servalproject/olsr

Perhaps we should include something like that, with sufficient
documentation, to enable other 3rd parties to build something similar.On Wed, Oct 31, 2012 at 6:42 AM, Henning Rogge  wrote:
> On Tue, Oct 30, 2012 at 9:05 PM, Bastian Bittorf  wrote:
>> example for our usecase:
>
> I can easily see an usecase... I just want to talk about if this
> should be within the routing protocol or not.
>
>> we have a user database (e.g. login-credentials or special properties
>> for some mac-addresses). ofcourse you can centralize that and let
>> the routers talk to each other if needed (e.g. via https), but much faster
>> is a local query, which is possible, if you there is a way to have a recent
>> copy of your central database.
>
>> how can this be solved with a daemon? multicast maybe
>> next step can be something like a "HNA4" ("message") announcement,
>> which every router can update without restarting the daemon 8-)
>
> OLSR does only multicast between the neighbors too.
>
> Henning Rogge
>
> --
> Steven Hawkings about cosmic inflation: "An increase of billions of
> billions of percent in a tiny fraction of a second. Of course, that
> was before the present government."
>
> --
> Olsr-dev mailing list
> 
> https://lists.olsr.org/mailman/listinfo/olsr-...

Hi,On 10/30/2012 09:12 PM, Henning Rogge wrote:
> On Tue, Oct 30, 2012 at 9:05 PM, Bastian Bittorf  wrote:
>> example for our usecase:
> 
> I can easily see an usecase... I just want to talk about if this
> should be within the routing protocol or not.Right, a good question. By intention of the routing protocol, it
shouldn't. But olsrd does the job of efficiently spreading some
information that well, why it shouldn't spread more than only tc messages?

The nameservice-plugin was the initial motivation, which does the same,
just uses a separate message to spread information about available
services. The service-part has nothing to do with routing, but works great.

The dataservice-plugins are doing the same as the nameservice-plugin,
just with some more generalized interface to generate or receive the
data. If it should be part of a routing protocol, maybe not. But it was
an easy and straight way of using the plugin infrastructure of olsrd
instead of creating some own flooding daemon, and others might benefit
from the existing code - therefore yes :)>> we have a user database (e.g. login-credentials or special properties
>> for some mac-addresses). ofcourse you can centralize that and let
>> the routers talk to each other if needed (e.g. via https), but much faster
>> is a local query, which is possible, if you there is a way to have a recent
>> copy of your central database.right. We use it for up-to-date information about the network nodes.
This information is now easily available everywhere in the net - you
only have to run the service and you will get a database with all the
information. It's way easier that a centralised attempt and can be
queried locally for map-creation etc.

nice regards,
Rene

On 10/30/2012 10:58 PM, rene wrote:
>> I can easily see an usecase... I just want to talk about if this
>> should be within the routing protocol or not.
>
> Right, a good question. By intention of the routing protocol, it
> shouldn't. But olsrd does the job of efficiently spreading some
> information that well, why it shouldn't spread more than only tc messages?That is why I have been looking to multicast. If the routing protocol 
supplies the local agents with a "mesh-wide" multicast, this should be 
enough for data-distribution among services.> The nameservice-plugin was the initial motivation, which does the same,
> just uses a separate message to spread information about available
> services. The service-part has nothing to do with routing, but works great.Yes, I know... I would like to push this functionality to an external 
service too in the long run. Otherwise this has to be re-implemented for 
every routing protocol used in Community Mesh networks.> The dataservice-plugins are doing the same as the nameservice-plugin,
> just with some more generalized interface to generate or receive the
> data. If it should be part of a routing protocol, maybe not. But it was
> an easy and straight way of using the plugin infrastructure of olsrd
> instead of creating some own flooding daemon, and others might benefit
> from the existing code - therefore yes :)So you agree, if we can make multicast work well, your daemon should be 
able to use it?>>> we have a user database (e.g. login-credentials or special properties
>>> for some mac-addresses). ofcourse you can centralize that and let
>>> the routers talk to each other if needed (e.g. via https), but much faster
>>> is a local query, which is possible, if you there is a way to have a recent
>>> copy of your central database.
> right. We use it for up-to-date information about the network nodes.
> This information is now easily available everywhere in the net - you
> only have to run the service and you will get a database with all the
> information. It's way easier that a centralised attempt and can be
> queried locally for map-creation etc.Henning Rogge

Hi,On 10/31/2012 07:57 AM, Henning Rogge wrote:
> So you agree, if we can make multicast work well, your daemon should be
> able to use it?not beeing some expert with this, but I won't see any issues. Currently
can't see how the multicast attempt can be more efficient than the
current flooding - but I will stay tuned on this channel and see how far
I can follow the arguments :)

Nice regards, Rene

Because multicast is supposed to flood across the network, but ignore
areas of the network that don't have anyone listening to the group.On Wed, Oct 31, 2012 at 9:21 PM, rene  wrote:
> Hi,
>
> On 10/31/2012 07:57 AM, Henning Rogge wrote:
>> So you agree, if we can make multicast work well, your daemon should be
>> able to use it?
> not beeing some expert with this, but I won't see any issues. Currently
> can't see how the multicast attempt can be more efficient than the
> current flooding - but I will stay tuned on this channel and see how far
> I can follow the arguments :)
>
> Nice regards, Rene
>
> --
> Olsr-dev mailing list
> 
> https://lists.olsr.org/mailman/listinfo/olsr-...

> The extension spreads some database-values through the net, taken
> from a
> sqlite-database or a textfile, which contains json-formatted tables
> (one
> per line).wow, that sounds really....useful!
just push it to public repo for testing.

bye, bastian

Hi,On 10/30/2012 08:56 PM, Bastian Bittorf wrote:
> just push it to public repo for testing.after unintentionally pushing to the public repo I was told to first
check what the feeling of this list is about the plugin... lets see :)

Just meanwhile use the supplied patch for testing, it works against
olsrd-0.6.3

nice regards,
Rene

On 10/30/2012 11:05 PM, rene wrote:
> Hi,
>
> On 10/30/2012 08:56 PM, Bastian Bittorf wrote:
>> just push it to public repo for testing.
> after unintentionally pushing to the public repo I was told to first
> check what the feeling of this list is about the plugin... lets see :)
>
> Just meanwhile use the supplied patch for testing, it works against
> olsrd-0.6.3I would say push the nl80211 metric code to a "nl80211" branch in the 
olsr.org repository, I would like to test if it compiles well on OpenWRT 
and Android. If it does and no one wants to veto it, we can quickly 
merge it into the master branch.

Henning Rogge