Enabling mesh (ad-hoc) network on multiple Raspberry Pi’s

So I’m just going to dump my experience here hoping it helps somebody.

This is Step 1 – making your Pi’s be able to ping each other and communicate via UDP/TCP, ad-hoc, peer to peer, without any central router. This doesn’t include making them route packets with 2 or more hops between origin and destination.

Long story short I’ve tried like a hundred different things and as usual a simple thing made it work.

So here it started as a weekend / hobby project trying to learn more about mesh networking and mesh communications. I’ve purchased 2 x Raspberry Pi 2 Model B (below) along with all standard stuff including usb Wi-Fi dongles and set sail with a goal of simply making them ping each other and eventually transfer a file in a mesh / ad-hoc network mode, basically so that none of them is an access point and both aren’t connected to any 3rd access point.

two raspberry pis mesh

 

 

 

 

 

 

 

 

 

 

 

1. Study of the literature

So I started with doing some research finding some existent posts and tutorials:

This one seemed like a blessing but turned out it was for Archlinux and the instructions don’t quite work for my Raspbian. Anyway in this and other tutorials authors kind of assumed RPis are able to ping each other in ad-hoc mode and skipped to discussing next steps, while in my case I couldn’t get that basic thing work at all.
  • Smesh
  • BATMAN ADV
  • Babel
  • Project Byzantium (This project looked most promising — especially the fact that you can connect to the mesh network and connect to the same node using a single wifi card. However, the list of software they use and their configurations are not readily available.)
  • CoovaChilli (DNS Redirect to capture traffic and drive it to an internal website.)
gossip-based distributed database
Linux distributed + ham radio
useful information and background on networking and radio used for mesh
The Wireless Battle of the Mesh is an event that aims at bringing together people from across the world to test the performance of different routing protocols for ad-hoc networks, like Babel, B.A.T.M.A.N., BMX6, OLSR, 802.11s and Static Routing.

 

So I basically read stuff online, tried setting up as per few tutorials and stackoverflow posts and it didn’t work.

I’ve even installed babeld as using it was suggested in 1st tutorial. Babeld got me the “id mismatch in babel-state” error, I resolved it eventually with executing “modprobe ipv6” but babeld didn’t seem to work or help at all.

2. Choose right usb Wi-Fi adaptors

So one thing I discovered digging further is it’s actually useful to have Wi-Fi adaptors:

a) of the same make

b) that are confirmed to support ad-hoc / mesh mode

And funny thing I’ve ordered my RPis separately and they had 2 different wi-fi dongles. I’ve decided to order 2 new ones that were recommended as the ones that definitely support mesh net mode (in this thread for example http://stackoverflow.com/questions/15423325/raspberry-pi-ad-hoc-networking) so went ahead and ordered 2 x “Edimax EW-7811Un 150 Mbps Wireless 11n Nano Size USB Adapter” from Amazon.

 

3. Get correct network interface setting

Executing from command line:
ifconfig wlan0 down
    iwconfig wlan0 mode ad-hoc
    iwconfig wlan0 key aaaaa11111#WEP key
    iwconfig wlan0 essid RPi     #SSID
    ifconfig wlan0 10.0.0.1 netmask 255.255.255.0 up
Obviously changing “10.0.0.1” to “10.0.0.2” on the 2nd Pi.
This didn’t work.
I’ve put same settings into /etc/network/interfaces but still no luck.

4. Get them onto same wireless cell

Found out from some tutorials that it’s not only necessary to specify same ESSID (network name) and have same keys or an absence of those, it is also important apparently to get both devices’ wlan0 interfaces to be on the same CELL.
You can find current cell your wlan0 interface is on by executing
iwconfig
You can also check other interfaces around and what cells they are on by executing
iwlist wlan0 scan
So I’ve added explicitly stating the access point cell into /etc/network/interfaces wlan0 configuration:
wireless-ap 11:5F:02:38:5C:45
I’ve noticed that didn’t really help, for example on one of the Pi’s:
CELL as put in /interfaces: 11:5F:02:38:5C:45
CELL as reported by iwconfig:  02:11:87:BF:8F:FF
As I side note I’ve also noticed that specifying wireless channel in /etc/network/interfaces doesn’t help and according to iwlist it would end up on different Cell numeric ID and long ID, but the channel would always be 1.
So anyway I’ve done some rebooting and jumping around and finally was able to get both RPis onto same Cell (so the long ID like this one above 02:11:87:BF:8F:FF was identical on both RPis).
Still no ping.

5. (optional) Install Erlang on both Raspberry Pi’s

5.1. Why do I need Erlang on my Raspberry Pi

“WTF?” says reader.
Well I just had no freaking idea of what to do but as I’m a bit familiar with Erlang  and it’s good with setting up nodes and sending messages between them, I thought why don’t I try this. I was also kinda hoping that Erlang takes care of the mesh connectivity as it’s built in there. Well, at least according to tutorials, such as this amazing book: http://learnyousomeerlang.com/distribunomicon, if Alice can message Bob and Bob can message Charlie then Alice can also message Charlie even when she hasn’t met Charlie yet.
So I installed Erlang Mini onto both Pis by executing the following:
wget http://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb sudo dpkg -i erlang-solutions_1.0_all.deb
sudo apt-get update
sudo apt-get install erlang-mini
And also connected my laptop with standard Erlang setup to the same adhoc network.
Launch a node on each device with a global name:
erl -name laptop@10.10.10.5
erl -name Pi1@10.10.10.19
erl -name Pi2@10.10.10.2
So – still no ping or Erlang messaging between two Pi’s.
BUT –  was successfully able to send a broadcast message from laptop to both RPis:
send (laptop):
Eshell V6.3  (abort with ^G)
{ok, S} = gen_udp:open(9998, [binary, {broadcast, true}]).
gen_udp:send( S, {192,168,1,255}, 9998, <<“UDP broadcast test message.”>> ).
receive (both RPis):
gen_udp:open(9998, []).
flush().
Shell got {udp,#Port<0.702>,{192,168,1,5},9998,<<“UDP broadcast test message.”>>}
Same didn’t work from either RPi. So they were able to receive laptop but not send back.

5.2. Make sure you set the same cookie

 Now as I wasn’t able to send an Erlang node message from one RPi to the other or even to the laptop even though being able to receive a message from it, I tried to explicitly connect to the laptop node:
net_kernel:connect_node(‘laptop@10.10.10.5’).
Which rendered:
=ERROR REPORT==== 27-Jun-2015::11:47:08 ===
** Connection attempt from disallowed node ‘pi1@10.10.10.19’ **
Back to reading manuals and I stumbled upon the concept of cookies and basically realised I wasn’t setting them which is the same as allowing nodes to choose their cookies randomly, this way restricting them from connecting.
Indeed,
erlang:get_cookie().
on my laptop rendered:
QHHMBZOKAQGVKRKUVYGC
and sure enough it was a different value on each RPi.
So I’ve restarted each node in a following manner:
erl -name laptop@10.10.10.5 -setcookie FOOBAR
And it helped – I was now able to send and receive Erlang messages from laptop to any of the two Pis and back from Pi to laptop.

5.3. Alice Bob and Charlie ‘paradox’

So back to our Alice Bob and Charlie thing, we have 3 nodes working in a mesh network and one of them can message two others. Each can message back. So Alice <-> Bob and Bob <-> Charlie works. But Alice <-> Charlie doesn’t work. In our case Bob is my macbook laptop and Alice = RPi1, Charlie = RPi2.
Why is that?
Let’s actually check whether each node knows about the other ones:
on laptop:
(laptop@10.10.10.5)7> nodes().
[‘pi2@10.10.10.2′,’pi1@10.10.10.19’]
Good, laptop knows about both RPis.
on RPi 2:
(pi2@10.10.1.2)15> nodes().
[‘pi2@10.10.10.5’]
Which is very odd – this is how it should NOT be, this contradicts the Distribunomicon and other Erlang tutorials – we shall be seeing other nodes that are connected to the one we’re connected to.
So, simply put, if we can see laptop@10.10.10.5 then we shall also be seeing everybody else laptop@ is able see in its nodes().
And then I’m getting this error:
global: ‘pi2@10.10.10.2’ failed to connect to ‘pi1@10.10.10.19′
Same error displayed on RPi1.
So this is very interesting. Of course this probably means I’m stupidly overseeing something simple and fundamental, but at this point it looks like this:
1) I can Erlang message between Laptop and RPi1/2 and can also broadcast via UDP (yay!)
2) I can ping from Laptop to RPi1 and RPi2, can also ping back from either of them to Laptop
3) I still cannot ping or erlang message between RPi1 and RPi2
4) Erlang nodes() on both RPi1 and RPi2 doesn’t admit it knows anything about the other RPi which contradicts one of the core principles of Erlang as laptop@ node shall be passing its knowledge of RPi2 to RPi1 and other way round. It doesn’t work this way at the same time it tries to do something and we get that error above. Very interesting – I’ve broken Erlang with my simple set up! 🙂
End of part one – to be continued.

15 thoughts on “Enabling mesh (ad-hoc) network on multiple Raspberry Pi’s”

    1. Hi Shachindra,

      still not sure why happening but resolved this with a temporary hack – this requires each node having other ones in its ARP list – see instructions below.

      Pls let me know if you find any way to make this work w/o manually adding to ARP list.

      ==========

      To add a static ARP entry to local ARP table:

      $ sudo arp -s 10.0.0.2 00:0c:29:c0:94:bf

      This commands tells local ARP table that the host with IP address 10.0.0.2 has MAC address 00:0c:29:c0:94:bf. Once you have configured a static ARP entry, you can verify that.

      $ arp -a -n

      ==========

      Reply

      1. Hi,

        I also resolved the issue by using static ARP entries at each node. But these entries disappear when Pi is rebooted and again we have to add these entries at start. Is there any way to solve this issue.

        Best Regards,
        Nasim Abbas

        Reply

        1. Maybe you could add it to start-up scripts?

          I’m not working on this at the moment but will let you know if I find some better way once I get back to this.

          Reply

  2. Dear Sir,

    I am not able to broadcast between multiple raspberry pi’s. but they can ping each other. can you please tell me how to broadcast in ad-hoc mode.

    Yours Sincerely,
    Nasim Abbas

    Reply

    1. Hi Nasim

      1. Have you tried adding static entries into ARP table as discussed in comments above?

      2. When you connect PC or laptop into same network, does broadcast work when initiated from PC/laptop?

      Reply

      1. Dear Sir,
        thanks for your reply. Yes i have added static entries into ARP table. Raspberry Pi’s ping each other but can not broadcast. When I connect to PC or Laptop broadcast still can not work. I hope you will give me positive suggestion in solving this issue.
        Waiting for your kind response

        Yours Sincerely,
        Nasim Abbas

        Reply

  3. Dear Sir,
    thanks for your reply. Yes i have added static entries into ARP table. Raspberry Pi’s ping each other but can not broadcast. When I connect to PC or Laptop broadcast still can not work. I hope you will give me positive suggestion in solving this issue.
    Waiting for your kind response

    Yours Sincerely,
    Nasim Abbas

    Reply

  4. I have the same problem with ARP. This this has to be a bug. The whole point of ARP is to find hardware addresses and map them to IP’s automatically.

    Reply

  5. As a hack around I was able to resolve this by installing arping and setting up a continuous ping.

    sudo apt-get install arping

    In my case I am using avahi-autoipd (169.254.0.0/16)

    on each host put in the broadcast address for your network and run this after your interface comes up.
    arping -q -A 169.254.255.255 &

    Reply
  6. Pingback: Raspberry Pi – Warnaka.com

  7. I already try all of tutorial above including in every link. But no one succeed. I use raspberry 3 that i thought more easy because reduce the possibility for error because the wifi dongle.
    May i know which tutorial that work? just erlang or all of the tutorial in those website (which is typically same) + erlang? thank you

    Reply

    1. hi!

      for me if you do the manual ARP setting then it works for general connectivity and you can test just via shell ping, you don’t have to install erlang for that

      have you tried the manual ARP setting or ARPing as suggested by Josh?

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *