Networking on Such geek. Wow.

FreeNAS finally supports ECC crypto!

Sun, 02 Feb 2020 00:00:00 +1300

Woo! FreeNAS finally supports ECC crypto for it’s SSL certificates in 11.3!

I reported this as a bug almost exactly two years ago, and kept having to do nasty workarounds to make it work every time I installed an update. Very happy that it’s finally sorted and I can use my proper shiny elliptic curve SSL setup! :)

New things I didn't know about WireGuard

Thu, 27 Dec 2018 00:00:00 +1300

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, and I’ve been using it pretty seamlessly for over a year now. I’ve learned a couple things that weren’t immediately obvious though, so I’m documenting them here.

Easy Provisioning

Samuel Holland mentioned an interesting trinket, in his post at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003703.html:

"[…] WireGuard will ignore a peer whose public key matches the interface’s private key. So you can distribute a single list of peers everywhere."

You can combine this with wg addconf like this:

Each peer has its own /etc/WireGuard/wg0.conf file, which only contains it’s [Interface] section
Each peer also has a shared /etc/WireGuard/peers.conf file, which contains all the peers
The wg0.conf file also has a PostUp hook, calling wg addconf /etc/WireGuard/peers.conf

It’s up to you to decide how you want to share the peers.conf, be it via a proper orchestration platform, something much more pedestrian like Dropbox, or something kinda wild like Ceph. I dunno, but it’s pretty great that you can just wildly fling a peer section around, without worrying whether it’s the same as the interface.

Setting Private Key from a file

Another piece of learning, courtesy of Samuel Holland, at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003702.html.

You can read in a file as the Private Key by doing something like:

PostUp = wg set %i private-key /etc/WireGuard/wg0.key

WireGuard - Part Three (Troubleshooting)

Mon, 12 Jun 2017 00:00:00 +1200

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, but there were a couple slightly curly bits that I needed to get my head around. This troubleshooting guide is a rough dump of the issues I had, and how I fixed them.

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Think through all the bits:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

Troubleshooting

There are, actually, a myriad of ways this can fail:

Broken routing on the client PC
Broken NAT on the remote router
Broken routing on the remote PC
Broken wg0 configuration on one side
Lack of TCP forwarding on the remote computer
Lack of Proxy ARP on the remote computer

If you can’t ping the remote server yet, don’t panic. Run tcpdump to find out what you’re missing.

On the remote server: tcpdump -i wg0
On your local machine: ping -c1 10.20.40.1

That will tell you whether your packets are reaching the remote server, or if they’re not getting through the tunnel.

If they’re not making it through the tunnel at all, you’ll probably be seeing error messages in the ping. For example:

PING 10.20.10.1 (10.20.10.1) 56(84) bytes of data.
From 10.20.40.2 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available

That’s the error message I got when I set AllowedIPs too strictly. Because I was trying to ping something that was routable, but wasn’t within the AllowedIPs range, there was no applicable key for the packet.

Thanks

Huge gratitude to Jason Donenfeld (aka zx2c4) for spending his time not only reading this post, but also for sending me some fantastic feedback! I’d made some bungles in my original post on this topic, and he vastly helped my understanding.

Also, huge gratitude to another Jason (@rendition) who has helped me develop from a junior network admin into a … ‘moderate’ network admin. I’ve learned more in the last year than I ever thought possible. He’s taught me nearly everything I know about managed networking, reviewed this post for me, and is actually the guy who introduced me to WireGuard originally!

WireGuard - Part Two (VPN routing)

Sun, 11 Jun 2017 00:00:00 +1200

This is a continuation of my brief series on the new WireGuard VPN. Part One was about the simple building-blocks to get WireGuard working between two endpoints. Now that we’ve got a couple machines able to ping each other by IP address, we can carry on a bit deeper into the inter-LAN routing stuff.

Extending on from the IP addresses in Part One, instead of JUST connecting to the remote machine, I want to actually have access to everything on the whole 10.20.0.0/16 network; even the non-WireGuard devices. I want it to be like I’m there on-site. The idea here is roughly:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2 (with eth0 on 192.168.88.207)
Machines on remote LAN: 10.20.0.0/16
Remote LAN Watchguard range: 10.20.40.0/16 – note this is within the remote 10.20.0.0/16 range

From my local machine, with minimal interaction, I want to be able to ping something like 10.20.10.30, and get a response.

Happily, now that the two endpoints are talking with each other, there’s really not much that has to happen to get things working the way I want.

Remote Config

Many Linux distros disable IPv4 packet forwarding by default. But in this case, we very much want this enabled. Edit your /etc/sysctl.conf file, and look for the line: net.ipv4.ip_forward, and set the value to 1. You may need to add this line manually, or uncomment it.
You’ll also need to enable ProxyARP. This is another kernel setting that’s usually disabled, so edit /etc/sysctl.conf, and add the following line: net.ipv4.conf.all.proxy_arp = 1.
Reload your kernel settings. A reboot will do the trick, or you could just run sudo sysctl -p /etc/sysctl.conf.

Starting WireGuard on Boot

There are a variety options to do this, and if you’re a seasoned sysadmin you probably already have a favourite way.

One option is to add post-up wg-quick up wg0 to the tail of your eth0 block (or appropriate interface) in /etc/network/interfaces.

A second option (for systemd users) is to simply run systemctl enable wg-quick@wg0. This will tell systemd to bring the wg0 interface up once a network connection is established.

I only do this on the remote machine, because I don’t want my local machine to be forever spinning up it’s WireGuard connection; however I always want the remote machine listening for my packets. That said, there’s probably no reason to avoid this; I just haven’t done it yet.

Local Config

Here’s the really cool bit. Since your remote machine is now set up to perform IPv4 forwaring and ProxyARP, the only thing you need to change on the local side is a single number. Or rather, a couple numbers.

In your /etc/WireGuard/wg0.conf file, just expand the network range of your interface to include the entire remote LAN:
```
 [Interface]
 ...
 Address = 10.20.40.2/16

 [Peer]
 ...
 AllowedIPs = 10.20.0.0/16
```
Now reload your WireGuard config, either by rebooting, or running sudo wg-quick down wg0 && sudo wg-quick up wg0.

That’s all! When you make the above changes, wg-quick will modify your routing table so that any IP within 10.20.x.y will be sent over the wg0 interface. The remote host will dutifully forward the packet out into the rest of the network, and ProxyARP will take care of the rest:

$ ping -c1 10.20.10.31
PING 10.20.10.31 (10.20.10.31) 56(84) bytes of data.
64 bytes from 10.20.10.31: icmp_seq=1 ttl=63 time=91.8 ms

Troubleshooting

Okay so by now there’s a really good chance that you’ve bumped into trouble. I’ve started on a basic WireGuard troubleshooting guide, but it only covers the issues that I bumped into. If you’re still stuck after reading through that, ask a question on the WireGuard Mailing List, or reach out via IRC (#WireGuard on Freenode) - this is all on the WireGuard website.

Wrap-up

That should be all! When you reboot the remote server (I specify reboot, because you want to test that your solution can survive an outage at the remote site), you should be able to ping other things in the remote LAN without any additional interaction. I still manually run wg-quick up wg0 on my local laptop, because I don’t want to be connected remotely all the time.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

WireGuard - Part One (Installation)

Sat, 03 Jun 2017 00:00:00 +1200

WireGuard is the most excellent VPN stack around. It’s really fast, the concept of Cryptokey Routing is awesome, and I love the speed and simplicity benefits that come from opionionated cryptography. The protocol is so simple - expressed in a mere 4k lines of code - that it’s auditable by anyone.

But.

With my initial naive approach, I found myself using HTTPS, over ports forwarded over SSH tunnels, connected over WireGuard. Although it was straightforward to get WireGuard working between two endpoints, I ended up in nested-crypto hell.

So, this brief series is about making WireGuard work as a VPN onto a different site. We’ll start by making it work between two endpoints, and we’ll go forward from there. My end goal is to have access to all the resources on a remote site, just by running wg-quick up wg0.

The endpoints I’m going to set up are at:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2

NAT Setup

Chances are, your remote endpoint is behind a firewall of some sort. Pick a high port, and configure your firewall to forward UDP packets on that port through to your remote WireGuard endpoint. You don’t need to do this on your local side, because reply traffic from the remote side will generally be handled by the stateful session part of your firewall.

Many routers and firewalls offer port address translation (also known as PAT) as part of port forwarding or NAT. This is when a packet hits the firewall on (say) port 57432, and the firewall puts it on the LAN to port 22, for example. You don’t want this. I think it’s possible with WireGuard, but it adds complexity without benefit.

Config - Remote Site

Install WireGuard

Generate your keys. The following will create a public key and a stub config in /etc/WireGuard/:

 cd /etc/WireGuard
 umask 077
 printf "[Interface]\nPrivateKey = " > wg0.conf
 wg genkey | tee -a wg0.conf | wg pubkey > publickey

Edit your config to match:

 [Interface]
 PrivateKey = WhAt3v3R= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port you've forwarded from your firewall)
 Address = 10.20.40.1/24 (this will be the IP given to the wg0 interface)

 [Peer]
 PublicKey = (leave this blank for now; you'll paste in your local public key here soon)
 AllowedIPs = 10.20.40.0/24 (this is the range of WireGuard IP addresses that this Peer's key can be used from)

That should be all you need to configure on the remote side for now. Save your wg0.conf file, and bring the interface up:
```
 wg-quick up wg0
 ping -c1 10.20.40.1
```
You should see a bunch of actions performed by wg-quick, and a reply packet from your ping. Now onto the local side.

Config - Local Machine

Install WireGuard and generate your keys, as per the first two steps above.

Edit your configuration again:

 [Interface]
 PrivateKey = WhAt3v3R+PaRt-tw0= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port again; I don't think they have to be the same)
 Address = 10.20.40.2/24 (note - different IP address, but in the same range)

 [Peer]
 PublicKey = ?????????? (copy the public key from the REMOTE server here)
 Endpoint = 163.172.161.0:12345 (enter the PUBLIC IP address of the remote site, plus the forwarded port)
 AllowedIPs = 10.20.40.0/24 (specifying that packets using this key must come from within 10.20.40.x)

Now that you’ve got a local public key, take a second to paste it into the Peer public key section on the remote server.
Once you’re finished you should be able to bring the interface up and ping it:
```
 wg-quick up wg0
 ping -c1 10.20.40.2
```
… and you should even be able to ping the remote server as well:
```
 ping -c1 10.20.40.1
```

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Here’s a quick summary of everything we’ve done:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

If you’ve nailed each of those and you’re still having trouble, you can have a quick look at the brief Troubleshooting guide I’ve put together as Part Three.

Onto Part Two

That should be all you need to get WireGuard working between two machines on two different sites. So far we haven’t done anything either interesting or uncommon - this is all the basic stuff you’ll find on the WireGuard Quick Start page, although expressed slightly differently. The next article will be a bit more about intra-site routing.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

HP Procurve's warn-and-disable

Sun, 30 Apr 2017 00:00:00 +1200

Since mid-2016, I’ve been working as a Network and Security Administrator. While I’d done a fair amount of networking previously, most of my experience had been with either unmanaged switches, or in a pre-built Cisco environment. Stepping into the world of managed networking was new for me, as was stepping into the world of HP Switches.

We were having recurring issues with a certain business unit looping ports on a switch. We had loop-protect running, but it was only set to disable the port after 5 seconds, and only for 300 seconds. This wasn’t long enough (we’ve since set it to something more resilient).

During our attempts to minimise the impact of a network loop, we enabled this setting on our HP Procurve switches:

fault-finder broadcast-storm action warn-and-disable

This seems simple enough. The fault-finder module will look for broadcast storms, and when found, will throw a syslog, an SNMP trap, and will disable the offending switch port.

Don’t be fooled. This will absolutely wreck your breakfast.

The fault-finder module is a system-wide setting, which means it applies to uplink ports as well. When a broadcast storm happens, all your distribution switches will see the broadcasts coming from… their uplink port. So of course, the switches disable their uplink to the rest of the network, neatly segregating themselves away from any services or central management.

Suffice to say, the next time we had a network loop there was a lot of walking and console cable madness, while we visited each of the distribution switches and manually re-enabled the disabled ports.

Lesson learned, loop-protect it is. (Until we get STP up campus-wide, of course)