Telling ZeroTier not to use Peering LANs for traffic

So, I use ZeroTier as my Layer 2 Mesh network between VMs and Physical hosts, and this morning I got a message telling me I was sending non allowed traffic over the DE-CIX VLAN… This was news to me, but after some digging, and help from Marek over at Vserver.Site, i managed to find out it was ZeroTier traffic causing the problem. ZeroTier in its nature will try and use all NICs on a machine to find an internet connection… but if you are on a peering VLAN, you do not want this to happen…

So, after more digging, I found the “Local Configuration File” in ZeroTier. This includes some features I did not know about, including how to place Subnets and Interfaces into the “Black List” so they wont be used by ZeroTier.

I wont reproduce the config file here, but i will give some tips, and explain what i have done. First, on Debian and Ubuntu, the file lives in /var/lib/zerotier-one/local.conf. If the file does not exist, you can create it. Its a standard JSON file. What i ended up adding was an IP range for each IX the box was connected to and marked it as black listed. Under settings, i also set the interfacePrefixBlacklist with a list of the interfaces connected to the IXes. an example is below:

This is the example for my machine connected to DE-CIX. ens19, 20 and 21 are connected to the peering VLANs on Munich, Dusseldorf and Hamburg. Given I have stopped sending bad IPv6 traffic, my guess then is the interfacePrefixBlacklist is only required, but I added the IPs also, just in case… I should probably include V6 here too…

When you make those changes, a restart of the ZeroTier service is required. And that is that. After Merak did some checks, and he was happy it was fixed, I was allowed back on the IX… Marek When you make those changes, a restart of the ZeroTier service is required. And that is that. After Merak did some checks, and he was happy it was fixed, I was allowed back on the IX…

Quick Overview of the network

AS204994 spans 5 countries (Ireland, UK, Germany, Amsterdam and USA), with 5 servers from 4 providers (3 from Vultr, 1 from and 1 from Packet.Net) and some on-prem machines (mix of VMs and Physical routers and machines). this may change over time, so check the homepage

Upstream providers are listed on the homepage also, but in house I use Virgin Media Business broadband and from there tunnel back to the upstream servers using ZeroTier. I also connect to EVIX over ZeroTier and to both LocIX and KleyReX with the VM in

Once the connection is made, I use Bird BGP to announce and receive routes from each of the providers. This is done as follows:

  • I have a /24 block of IPs announced though Vultr, Packet, VServer.Site and also though each of the IXs I’m connected to. Currently I am only using V4 to try and get this sorted first, then will add my /48 V6 space to the mix.
  • the /24 is split into a /28 for “internal peering” (ZeroTier network), a /28 for Any cast stuff (websites hosted across multiple sites, etc), a /27 for the house and multiple /28s, one for each of the VMs.
  • Any upstream VM that gets a full (or even partial) route feed announces all those routes (bar some filtered routes) back to Dublin. It also announces the /28 it has back to all other (internal) VMs. Only the /24 is announced to the upstream.
  • If a provider does not give any routes back, we leave it as is. It will still announce its internal /28 back to the rest of the network.
  • All servers also run a copy of NGinx which then hosts any of the anycast websites ( currently being the main one). That site is “hidden” behind CloudFlare for their security and bandwidth/caching features.
  • In house, all routes are managed by a single Ubuntu VM running on Hyper-V. It has a main internet connection with Virgin Media, and then connects internally to the rest of the network. Its set to allow internal routing, and the /27 for the house is connected to an internal VLAN.
  • I have a Ubiquiti EdgeRouter POE connected to that VLAN and it gets multiple of those IPs. Workstations, phones, etc, the connect to it and use it as a connection to the internet.

So far, this works about 90% of the time. This is not currently the production network: Netflix, Amazon Prime Video, etc, wont work, so all media devices go direct to the internet over a different VLAN (Luckily Virgin Media Business gave me a /29 to use). There is some work that will need to be done before it can go full production. Ideally, if I could announce my V4 and V6 though Virgin Media, that would make life easier, but we will see if they can sort it out.

So, Any questions? You can mail me at Tiernan [at] as204994 [dot] net.